The 4DN-DCIC metadata database can be accessed using a Hypertext-Transfer-Protocol-(HTTP)-based, Representational-state-transfer (RESTful) application programming interface (API) - aka the REST API. In fact, this API is used by the import_data script used to submit metadata entered into excel spreadsheets as described on this page. This API was developed by the ENCODE project so if you have experience retrieving data from or submitting data to ENCODE, use of the 4DN-DCIC API should be familiar to you. The REST API can be used both for data submission and data retrieval, typically using scripts written in your language of choice. Data objects exchanged with the server conform to the standard JavaScript Object Notation (JSON) format. Libraries written for use with your chosen language are typically used for the network connection, data transfer, and parsing of data (for example, requests and json, respectively for Python). For a good introduction to scripting data retrieval (using GET requests) you can refer to this page on the ENCODE web site that also has a good introduction to viewing and understanding JSON formatted data.
Your script will need to use an access key and secret that you can obtain by following these instructions to connect to the server. Exactly how you format and pass the connection information to the server depends on your scripting language and the libraries that you use with it.
Base URL for submitting and fetching data are: https://data.4dnucleome.org/
You can refer to the FDN_key and FDN_connection classes in the get_field_info.py library in Submit4DN for an example of how to generate the necessary information that will be passed to the server with each request.
Your script will need to add a uniquely identifying token to the Base URL in order to GET, POST or PATCH metadata for that item. IDs that can be used include: uuids, accessions, certain type specific identifiers and aliases. See the sections on 'Using Aliases' and 'Referencing existing items' for the types of identifiers that can be used in your requests.
Because in many cases fields in one Item may refer to another Item, e.g., the biosample field in the experiment_hi_c schema, it is necessary to POST the referenced item prior to POSTing the item that refers to it. A sensible POSTing order is specified in the sheet_order array located around line 228 in the get_field_info.py library.
The most important component of your submission is the proper formatting of the data into json so it will map correctly onto the 4DN metadata schemas. The details of the schemas for all object types in the database can be viewed at https://data.4dnucleome.org/profiles/. Individual schemas can be viewed and/or retrieved via GET by appending the schema file name to the above URL (e.g., for the Hi-C experiment schema https://data.4dnucleome.org/profiles/experiment_hi_c.json). For a listing of all schema files and associated resource names see this table, which should be up to date with the current schemas in the database.
Depending on the Item type that you are submitting there may be fields that require values (e.g., experiment_type for experiments), fields for which values should never be submitted (e.g., \u2018date_created\u2019 as this is an automatically generated value) and fields with specific formatting and fields that accept values of specific types. In many cases, the values must be selected from a list of constrained choices. The documentation on field values described in detail here and the annotated json document below can be used as a guide on formatting your json. In addition, the unordered and unfiltered excel workbooks produced by get_field_info can be a useful reference for determining exactly what fields are associated with what object types. The processed workbook that is actually used for data submission does not reflect the exact schema structure should not be used as a direct reference for API submission.
JSON for an ExperimentHiC POST request
\n{\n \"lab\": \"4dn-dcic-lab\",\n \"award\": \"/awards/OD008540-01/\",\n \"aliases\": [\"dcic:myalias\"]\n \"experiment_type\": \"in situ Hi-C\",\n \"biosample\":\"4DNBS1NUPXMH\",\n \"biosample_quantity\": 2000000,\n \"biosample_quantity_units\": \"cells\",\n \"dbxrefs\": [\"SRA:SRX764985\", \"GEO:GSM1551599\"],\n \"description\": \"A useful description of this experiment\",\n \"documents\": [\"dcf15d5e-40aa-43bc-b81c-32c70c9afb01\"],\n \"experiment_relation\":[{\"relationship_type\":\"controlled by\",\"experiment\":\"4DNEX067APU1\"}],\n \"experiment_sets\":[\"431106bc-8535-4448-903e-854af460b260\"],\n \"files\": [\"4DNFI067APU1\", \"4DNFI067APU2\", \"4DNFI067APA1\"],\n \"filesets\": [\"4d6ecff9-7c67-4096-bca8-515246767feb\"],\n \"follows_sop\": \"Yes\",\n \"crosslinking_method\": \"1% Formaldehyde\",\n \"crosslinking_time\": 30,\n \"crosslinking_temperature\": 37,\n \"digestion_enzyme\": \"MboI\",\n \"enzyme_lot_number\": \"123456\",\n \"digestion_temperature\": 37,\n \"digestion_time\": 30,\n \"ligation_time\": 30,\n \"ligation_temperature\": 37,\n \"ligation_volume\": 1,\n \"tagging_method\": \"Biotinylated ATP\",\n \"fragmentation_method\": \"chemical\",\n \"average_fragment_size\": 100,\n \"notes\": \"sample dcic notes\",\n \"protocol\": \"131106bc-8535-4448-903e-854af460b244\"\n}\nField specification - how to find out what fields need what format, when to provide values and other useful tips
The three fields below are common to every Item - you don't always need to include aliases but they will make some things a lot easier.:
\n\"lab\": \"4dn-dcic-lab\", # required for POST with value of an existing lab identifier - this is an alias\n\"award\": \"/awards/OD008540-01/\", # required for POST with value of an existing award identifier - this a specific item type identifier\n\"aliases\": [\"dcic:myalias\", \"dcic:myotheralias\"], # POST or PATCH an array with values that can be used as identifiers\n
Accession and uuid are automatically assigned during initial posting so can only be used as identifiers to PATCH existing Items - not for POSTing. Note that all item types have a uuid but not all items have accessions.:
\n\"accession\":\"4DNEX067APU1\", # PATCH ONLY can be used as an identifier - automatically assigned on POST\n\"uuid\": \"75041e2f-3e43-4388-8bbb-e861f209c444\", # PATCH ONLY can be used as an identifier - automatically assigned on POST\n
While we encourage you to submit as many fields as possible there are some fields that are absolutely required to post an item. These required fields are identified in the \"required\" field of the schema.:
\n\"required\": [\n \"experiment_type\",\n \"award\",\n \"lab\",\n \"biosample\"\n],\n
If they are left out of a POST request will cause an error. You don't need to include required fields if you are PATCHing an existing Item.:
\n\"experiment_type\": \"in situ Hi-C\", # required for POST string field\n\"biosample\":\"231111bc-8535-4448-903e-854af460b254\", # required for POST with value of an alias used in Biosample sheet or previously existing biosample identifier\n
There are number fields and string fields - the field type can be found by referring to the schema directly or the workbook templates produced by get_field_info.
\n\"biosample_quantity_units\": {\n \"title\": \"Quantity Units\",\n \"description\": \"The units that go along with the biological sample quantity\",\n \"enum\": [\n \"g\",\n \"mg\",\n \"cells\",\n \"\u00b5l\",\n \"ml\",\n \"ng/\u00b5l\",\n \"nM\"\n ],\n \"type\": \"string\"\n },\nAs you can see the specific field can be found by searching through the schema. If the field is an enum type (sometimes the choices are referred to as controlled vocabularies) your value must match one of the possible entries exactly.:
\n\"crosslinking_method\": {\n \"title\": \"Crosslinking Method\",\n \"description\": \"The method by which the sample was crosslinked\",\n \"enum\": [\n \"3% Formaldehyde\",\n \"1% Formaldehyde\",\n \"Fresh Frozen\"\n ],\n \"type\": \"string\"\n },\nDates should be submitted in a standard way. Here's an example of an automatically generated date
\n\"date_created\": \"2017-06-27T16:12:02.601238+00:00\"\n
A full list of the Item types and their fields is also presented in this table. Note that the list below is not complete, but is focused on the most commonly used fields.