In order to make your data accessible, searchable and assessable you should submit as much metadata as possible to the 4DN system along with the raw files you have generated in your experiments.
These pages are designed to
- show you how to find out what kind of metadata we collect for your particular type of experiment
- introduce the mechanisms by which you can submit your metadata and data to the 4DN data portal.
For an overview of the metadata structure and relationships between different items please see the slides available on the metadata introductory page.
We have three primary ways that you can submit data to the 4DN data portal.
Notes for prospective submitters
If you would like submit data to the portal:
- You will need to create a user account.
- Please skim through the metadata structure.
- Check out the other pages in the Help menu for detailed information on the submission process.
- Of note are the required metadata for the biological samples used in experiments, which is specified on this page.
- We like to know about submissions beforehand, and we will need to grant your account submitter privileges. If you contact us at email@example.com we can set up a Zoom call to discuss the details of the submission process and the most convenient approach for your existing system.
- IMPORTANT: If you are planning to submit experiments that include genomic data from human patient samples please let us know as soon as possible. This data likely requires controlled access and dbGaP registration. If you are not sure if the data you are generating should be considered controlled access please contact the relevant offices at your institute, your NIH program officers or Ian Fingerman, who coordinates controlled data issues for 4DN, with questions. Any personal health information (PHI) should not be submitted with your experimental metadata. Generally any genomic data generated from human tissue or cell lines must be explicitly consented for broad sharing of genomic information and be considered controlled access data. For more info consult the NIH Genomic Data Sharing Policy
The online web submission forms are best used
- To submit one or a few experiments.
- To edit one or a few fields of an already submitted but not yet released item.
- As a hands on way to gain familiarity with the 4DN data model.
Documentation on how to get started with this interface is here.
Data Submission via Spreadsheet
The excel metadata workbooks
- Are useful for submitting metadata and data for several experiments or biosamples
- Can be used to make bulk edits of submitted but not yet released metadata
- Contain multiple sheets where each sheet corresponds to an object type and each column a field of metadata
- Can be generated using the Submit4DN software
- Are used as input to the Submit4DN software which validates submissions and pushes the content of the forms to our database.
Documentation of the data submission process using these forms can be found here.
For both meta/data submission and retrival, you can also access our database directly via the REST-API.
- Our implementation is analagous to the one developed by the ENCODE DCC.
If you would like to directly interact with the REST API for data submission see the documentation here.
Notes on Experiments and Replicate Sets
- The 4DN Consortium strongly encourages that experiments be performed using at least two different preparations of the same source biomaterial - i.e. bioreplicates.
- When submitting metadata you should submit two Experiments that use the same Biosource, but have different Biosamples.
- In many cases the only difference between Biosamples may be the dates at which the cell culture or tissue was harvested.
- The experimental techniques and parameters will be shared by all experiments of the same bioreplicate set.
- Multiple sequencing runs performed at different times using a library prepared from the same Biosample and the same methods up until the sample is sent to the sequencer - i.e. technical replicates.
Submitting replicate information
The replicate information is stored and represented as a set of experiments that includes labels indicating the replicate type and replicate number of each experiment in the set.
The mechanism that you use to submit your metadata will dictate the type of item that you will associate replicate information with
In excel workbooks bioreplicate and technical replicate numbers are entered in the Experiment sheet.
Using the API you directly associate the replicate information (i.e. replicate number and the experiment identifier) with the ExperimentSetReplicate objects.
Using the web submission interface the replicate numbers and linked experiments are added from the ExperimentSetReplicate page
In the database the information will always end up directly associated with ExperimentSetReplicate objects.
Specific details on formatting information regarding replicates is given in the Spreadsheet Submission page.
When submitting using the REST API you should format your json according to the specifications in the schema as described in the REST API page.