\n\n* The ``-SP`` option is used to ensure the results are equivalent to that obtained by running ``bwa mem`` on each mate separately, while retaining the right formatting for paired-end reads. This option skips a step in ``bwa mem`` that forces alignment of a poorly aligned read given an alignment of its mate with the assumption that the two mates are part of a single genomic segment.\n* The ``-5`` option is used to report the 5' portion of chimeric alignments as the primary alignment. In Hi-C experiments, when a mate has chimeric alignments, typically, the 5' portion is the position of interest, while the 3' portion represents the same fragment as the mate. For chimeric alignments, ``bwa mem`` reports two alignments: one of them is annotated as primary and soft-clipped, retaining the full-length of the original sequence. The other end is annotated as hard-clipped and marked as either 'supplementary' or 'secondary'. The ``-5`` option forces the 5'end to be always annotated as primary.\n* The ``-M`` option is used to annotate the secondary/supplementary clipped reads as **secondary** rather than **supplementary**, for compatibility with some public software tools such as ``picard MarkDuplicates``.\n* The ``-t`` option is used for multi-threading and should not affect the result.", "display_title": "Alignment", "uuid": "10000000-0000-0000-0000-ffff00001000", "@type": ["StaticSection", "UserContent", "Item"], "award": {"status": "current", "@type": ["Award", "Item"], "@id": "/awards/1U01CA200059-01/", "uuid": "b0b9c607-f8b4-4f02-93f4-9895b461334b", "display_title": "4D NUCLEOME NETWORK DATA COORDINATION AND INTEGRATION CENTER - PHASE I", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin"]}}, "status": "released", "lab": {"status": "current", "@id": "/labs/4dn-dcic-lab/", "@type": ["Lab", "Item"], "display_title": "4DN DCIC, HMS", "uuid": "828cd4fe-ebb0-4b36-a94a-d2e3a36cc989", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin", "role.lab_submitter", "submits_for.828cd4fe-ebb0-4b36-a94a-d2e3a36cc989"]}}, "@id": "/static-sections/10000000-0000-0000-0000-ffff00001000/", "content_as_html": "Hi-C reads are mapped to the GRCh38 (human) or mm10 (mouse) reference genome using bwa version 0.7.17. Specifically, we run:
\nbwa mem -SP5M -t<nthreads> <genome_index> <fastq1> <fastq2>\n
- The
-SP option is used to ensure the results are equivalent to that obtained by running bwa mem on each mate separately, while retaining the right formatting for paired-end reads. This option skips a step in bwa mem that forces alignment of a poorly aligned read given an alignment of its mate with the assumption that the two mates are part of a single genomic segment. - The
-5 option is used to report the 5' portion of chimeric alignments as the primary alignment. In Hi-C experiments, when a mate has chimeric alignments, typically, the 5' portion is the position of interest, while the 3' portion represents the same fragment as the mate. For chimeric alignments, bwa mem reports two alignments: one of them is annotated as primary and soft-clipped, retaining the full-length of the original sequence. The other end is annotated as hard-clipped and marked as either 'supplementary' or 'secondary'. The -5 option forces the 5'end to be always annotated as primary. - The
-M option is used to annotate the secondary/supplementary clipped reads as secondary rather than supplementary, for compatibility with some public software tools such as picard MarkDuplicates. - The
-t option is used for multi-threading and should not affect the result.
For filtering valid Hi-C alignments, we use pairtools (previously called pairsamtools). Specifically, we use version 0.2.2. The filtering workflow outputs a pairs file containing a list of valid contacts.
\n
This filtering workflow applies the following criteria:
\n
\n- Reads marked as duplicates are removed.
\n- Full-length alignments that are unique are kept.
\n- An unmapped portion shorter than 20bp is ignored; and the rest of the alignment is still considered as full-length.
\n- In addition, clipped (chimeric) alignments are kept, if they are valid Hi-C contacts. If one mate is clipped and the other is full-length and the 3'end of the clipped alignment is mapped within 2kb of the full-length alignment in the orientation that the two 3'ends are pointing toward each other they are considered valid contacts.
\n- As with full-length alignments, any unmapped portion shorter than 20bp is ignored.
\n
\n
One of the design choices we have made is to include a lossless bam file as an output of the data processing. This output file, containing all the sequences in the original fastq files, the alignment results, and pairtools-provided flags for read filtering, is provided as a resource. To be able to produce this output file, the contents of the bam file is carried forward in the filtering workflow in intermediate pairsam files. Users who are only interested in the valid contact lists may run the same analysis with more light-weight intermediate files.
\n
Specifically, the filtering workflow consists of the following steps:
\n
\npairtools parse- A bam file is read in, and a pairsam file is written out.
\n- The pairsam file is a pairs file, listing one read pair per line, with additional columns to track the sam-file lines, and a pairtools read classification.
\n- These classifications include information on whether the read aligned to 0, 1, or multiple places in the genome and whether it aligned end-to-end or if it was clipped.
\n- This tool also upper-triangularizes the reads, i.e. if the coordinate of second read is higher than the first, the reads are flipped.
\n- \n
For more details, see pairsamtools doc
\n \n- \n
pairtools sort
\n \n- A
pairsam (or generically pairs) file is read in, and a pairsam file is written out. \n- The rows are sorted in chr1-chr2-pos1-pos2 order.
\n- \n
Note that the flipping order and sort order of chromosomes is not identical. See the docs for more details.
\n \n- \n
pairtools merge
\n \n- One or more
pairsam (or generically pairs) files are read in, and a pairsam file is written out. \n- \n
The files are merged, preserving the sorted order.
\n \n- \n
pairtools dedup --mark-dups
\n \n- (equivalent to
pairtools markasdup) \n- Duplicate alignments that share the same pair of 5'end coordinates +/- 3 bps are marked as identified.
\n- \n
An arbitrary one is retained with the original classification, while others get a duplicate classification.
\n \n- \n
pairtools select
\n \n- Only the reads with pairtools classification
UU and UC are retained and output to a pairs file. \n
Replicates are merged in two steps before producing a matrix.
\n
\n- Sequencing replicates (replicates within an experiment)
\n- Sequencing replicates are merged after the alignment but before duplicate removal step (
pairsam dedup), since reads resulting from a single PCR duplication event may exist in both replicates. \n- Merging is performed on (intermediate)
pairsam files using pairsam merge. \n- DCIC provides a merged output as a
pairs file. \n- Biological replicates (replicates within an experiment set)
\n- Biological replicates are merged after duplicate removal step, since PCR duplication event happened independently in these replicates.
\n- Merging is performed on
pairs files using run-merge-pairs.sh. \n- DCIC provides a merged output as a merged
pairs file. \n
\n\n**Resolutions**: Both `mcool` and `hic` files contain the \nfollowing resolutions.\n\n* 1kb, 2kb, 5kb, 10kb, 25kb, 50kb, 100kb, 250kb, 500kb, 1Mb, \n2.5Mb, 5Mb, 10Mb\n\n", "display_title": "Matrix Aggregation and Normalization", "uuid": "7e577277-525a-4a04-add1-dc7ffa9c462c", "@type": ["StaticSection", "UserContent", "Item"], "award": {"status": "current", "@type": ["Award", "Item"], "@id": "/awards/1U01CA200059-01/", "uuid": "b0b9c607-f8b4-4f02-93f4-9895b461334b", "display_title": "4D NUCLEOME NETWORK DATA COORDINATION AND INTEGRATION CENTER - PHASE I", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin"]}}, "status": "released", "lab": {"status": "current", "@id": "/labs/4dn-dcic-lab/", "@type": ["Lab", "Item"], "display_title": "4DN DCIC, HMS", "uuid": "828cd4fe-ebb0-4b36-a94a-d2e3a36cc989", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin", "role.lab_submitter", "submits_for.828cd4fe-ebb0-4b36-a94a-d2e3a36cc989"]}}, "@id": "/static-sections/7e577277-525a-4a04-add1-dc7ffa9c462c/", "content_as_html": "For DNase Hi-C and Micro-C, the Hi-C pipeline is run without \nrestriction enzyme-based intra-fragment contact filtering (as \nthe digestion step does not use restriction enzymes), \nwhile matrix balancing is still performed with the ICE \nalgorithm.
\n
4DN DCIC provides a Hi-C matrix in two different formats: \n.mcool format and .hic format. The two files are generated\n from the same pairs file as input filtered contact list. \nBoth files contain multiple resolutions.
\n
** .hic format**
\n
\n- A
.hic file is produced by Juicertools (version 1.8.9-cuda8) \nand can be visualized using Juicebox \n- The matrix is normalized using the VC, VC_SQRT, KR methods.
\n
\n
** .mcool format**
\n
\n- An
.mcool file is produced by Cooler (version 0.7.6) and can be visualized using HiGlass. \n- The matrix is normalized using the ICE (iterative \ncorrection and eigenvalue decomposition) matrix balancing \nalgorithm, after the matrix-level removal of the diagonal and \nthe rows/columns with a low value.
\n- The
.mcool file also contains the normalization vectors \ngenerated by Juicertools (same as in a .hic file generated\n from the same pairs file) \n
\n
\n
Resolutions: Both mcool and hic files contain the \nfollowing resolutions.
\n
\n- 1kb, 2kb, 5kb, 10kb, 25kb, 50kb, 100kb, 250kb, 500kb, 1Mb, \n2.5Mb, 5Mb, 10Mb
\n
\n\n\n\n\n \n \n | File Type | \n File Format | \n Description | \n
\n \n \n | Alignment | \n .bam | \n Filtered alignments of raw reads to the appropriate reference are available in .bam format. These are generated separately for each replicate experiment in an experiment set. | \n
\n \n | Contact list | \n .pairs | \n Contain a list of pairwise contacts and are generated after alignments are filtered. Pairs are generated separately for each replicate, and an additional pairs file is generated after replicates are merged. Quality metrics generated by PairsQC are also provided for each pairs file (a sample report can be found here). | \n
\n \n | Contact matrix | \n .mcool | \n Multi-resolution contact matrices generated in the final step of the processing pipeline are available in .mcool format, generated by Cooler. These can be visualized on the portal with HiGlass. Mcool contact matrices are normalized with the ICE matrix balancing algorithm (iterative correction and eigenvalue decomposition). | \n
\n \n | Contact matrix | \n .hic | \n Contact matrices generated by Juicertools are also available in .hic format, and these can be visualized externally in Juicebox or in the WashU Epigenome Browser. For in situ Hi-C and dilution Hi-C, the .hic matrix is normalized using the VC, VC_SQRT, KR methods. | \n
\n
", "display_title": "dnasehic-processed-files", "uuid": "98dbd1ca-b518-4984-b2eb-4a0bf9f17334", "@type": ["StaticSection", "UserContent", "Item"], "award": {"status": "current", "@type": ["Award", "Item"], "@id": "/awards/1U01CA200059-01/", "uuid": "b0b9c607-f8b4-4f02-93f4-9895b461334b", "display_title": "4D NUCLEOME NETWORK DATA COORDINATION AND INTEGRATION CENTER - PHASE I", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin"]}}, "status": "released", "lab": {"status": "current", "@id": "/labs/4dn-dcic-lab/", "@type": ["Lab", "Item"], "display_title": "4DN DCIC, HMS", "uuid": "828cd4fe-ebb0-4b36-a94a-d2e3a36cc989", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin", "role.lab_submitter", "submits_for.828cd4fe-ebb0-4b36-a94a-d2e3a36cc989"]}}, "@id": "/static-sections/98dbd1ca-b518-4984-b2eb-4a0bf9f17334/", "content_as_html": "Below is a description of the file formats available on the portal as outputs of the data processing pipeline. \n\n
\n\n
\n\n\n| File Type | \nFile Format | \nDescription | \n
\n\n\n| Alignment | \n.bam | \nFiltered alignments of raw reads to the appropriate reference are available in .bam format. These are generated separately for each replicate experiment in an experiment set. | \n
\n\n| Contact list | \n.pairs | \nContain a list of pairwise contacts and are generated after alignments are filtered. Pairs are generated separately for each replicate, and an additional pairs file is generated after replicates are merged. Quality metrics generated by PairsQC are also provided for each pairs file (a sample report can be found here). | \n
\n\n| Contact matrix | \n.mcool | \nMulti-resolution contact matrices generated in the final step of the processing pipeline are available in .mcool format, generated by Cooler. These can be visualized on the portal with HiGlass. Mcool contact matrices are normalized with the ICE matrix balancing algorithm (iterative correction and eigenvalue decomposition). | \n
\n\n| Contact matrix | \n.hic | \nContact matrices generated by Juicertools are also available in .hic format, and these can be visualized externally in Juicebox or in the WashU Epigenome Browser. For in situ Hi-C and dilution Hi-C, the .hic matrix is normalized using the VC, VC_SQRT, KR methods. | \n
\n
The pipeline components are pre-installed in a publicly available Docker image (duplexa/4dn-hic:v43) on Docker Hub. The source code for the Docker image and pipeline description in Common Workflow Language (CWL) can be found on GitHub.
\n
\n
Content-wise, 0.2.5, 0.2.6, 0.2.7 can be considered (nearly) identical.
\n
\n0.2.7\n- CWL : https://github.com/4dn-dcic/docker-4dn-hic/tree/v43/cwl
\n- Docker : https://github.com/4dn-dcic/docker-4dn-hic/tree/v43
\n
\n0.2.5/0.2.6\n- CWL : https://github.com/4dn-dcic/docker-4dn-hic/tree/v42.2/cwl
\n- Docker : https://github.com/4dn-dcic/docker-4dn-hic/tree/v42.2
\n
\n- Old runs
\n0.2.0\n- CWL : https://github.com/4dn-dcic/pipelines-cwl/tree/0.2.0/cwl_awsem/
\n- Docker : https://github.com/4dn-dcic/docker-4dn-hic/tree/v40
\n
\n
\n
Example set of commands that were actually run as part of the pipeline can be found at https://github.com/4dn-dcic/docker-4dn-hic/blob/v43/HiCPipeline.md
![](\"https://s3.amazonaws.com/4dn-dcic-public/static-pages/InfoBoxes/DNAse_HIC.png\")
\n ![](\"https://s3.amazonaws.com/4dn-dcic-public/static-pages/hicpipeline.png\")
\n\n