\n\n* The ``-SP`` option is used to ensure the results are equivalent to that obtained by running ``bwa mem`` on each mate separately, while retaining the right formatting for paired-end reads. This option skips a step in ``bwa mem`` that forces alignment of a poorly aligned read given an alignment of its mate with the assumption that the two mates are part of a single genomic segment.\n* The ``-5`` option is used to report the 5' portion of chimeric alignments as the primary alignment. In Hi-C experiments, when a mate has chimeric alignments, typically, the 5' portion is the position of interest, while the 3' portion represents the same fragment as the mate. For chimeric alignments, ``bwa mem`` reports two alignments: one of them is annotated as primary and soft-clipped, retaining the full-length of the original sequence. The other end is annotated as hard-clipped and marked as either 'supplementary' or 'secondary'. The ``-5`` option forces the 5'end to be always annotated as primary.\n* The ``-M`` option is used to annotate the secondary/supplementary clipped reads as **secondary** rather than **supplementary**, for compatibility with some public software tools such as ``picard MarkDuplicates``.\n* The ``-t`` option is used for multi-threading and should not affect the result.", "display_title": "Alignment", "uuid": "10000000-0000-0000-0000-ffff00001000", "@type": ["StaticSection", "UserContent", "Item"], "award": {"status": "current", "@type": ["Award", "Item"], "@id": "/awards/1U01CA200059-01/", "uuid": "b0b9c607-f8b4-4f02-93f4-9895b461334b", "display_title": "4D NUCLEOME NETWORK DATA COORDINATION AND INTEGRATION CENTER - PHASE I", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin"]}}, "status": "released", "lab": {"status": "current", "@id": "/labs/4dn-dcic-lab/", "@type": ["Lab", "Item"], "display_title": "4DN DCIC, HMS", "uuid": "828cd4fe-ebb0-4b36-a94a-d2e3a36cc989", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin", "role.lab_submitter", "submits_for.828cd4fe-ebb0-4b36-a94a-d2e3a36cc989"]}}, "@id": "/static-sections/10000000-0000-0000-0000-ffff00001000/", "content_as_html": "Hi-C reads are mapped to the GRCh38 (human) or mm10 (mouse) reference genome using bwa version 0.7.17. Specifically, we run:
\nbwa mem -SP5M -t<nthreads> <genome_index> <fastq1> <fastq2>\n
- The
-SP option is used to ensure the results are equivalent to that obtained by running bwa mem on each mate separately, while retaining the right formatting for paired-end reads. This option skips a step in bwa mem that forces alignment of a poorly aligned read given an alignment of its mate with the assumption that the two mates are part of a single genomic segment. - The
-5 option is used to report the 5' portion of chimeric alignments as the primary alignment. In Hi-C experiments, when a mate has chimeric alignments, typically, the 5' portion is the position of interest, while the 3' portion represents the same fragment as the mate. For chimeric alignments, bwa mem reports two alignments: one of them is annotated as primary and soft-clipped, retaining the full-length of the original sequence. The other end is annotated as hard-clipped and marked as either 'supplementary' or 'secondary'. The -5 option forces the 5'end to be always annotated as primary. - The
-M option is used to annotate the secondary/supplementary clipped reads as secondary rather than supplementary, for compatibility with some public software tools such as picard MarkDuplicates. - The
-t option is used for multi-threading and should not affect the result.
For filtering valid Hi-C alignments, we use pairtools (previously called pairsamtools). Specifically, we use version 0.2.2. The filtering workflow outputs a pairs file containing a list of valid contacts.
\n
This filtering workflow applies the following criteria:
\n
\n- Reads marked as duplicates are removed.
\n- Full-length alignments that are unique are kept.
\n- An unmapped portion shorter than 20bp is ignored; and the rest of the alignment is still considered as full-length.
\n- In addition, clipped (chimeric) alignments are kept, if they are valid Hi-C contacts. If one mate is clipped and the other is full-length and the 3'end of the clipped alignment is mapped within 2kb of the full-length alignment in the orientation that the two 3'ends are pointing toward each other they are considered valid contacts.
\n- As with full-length alignments, any unmapped portion shorter than 20bp is ignored.
\n
\n
One of the design choices we have made is to include a lossless bam file as an output of the data processing. This output file, containing all the sequences in the original fastq files, the alignment results, and pairtools-provided flags for read filtering, is provided as a resource. To be able to produce this output file, the contents of the bam file is carried forward in the filtering workflow in intermediate pairsam files. Users who are only interested in the valid contact lists may run the same analysis with more light-weight intermediate files.
\n
Specifically, the filtering workflow consists of the following steps:
\n
\npairtools parse- A bam file is read in, and a pairsam file is written out.
\n- The pairsam file is a pairs file, listing one read pair per line, with additional columns to track the sam-file lines, and a pairtools read classification.
\n- These classifications include information on whether the read aligned to 0, 1, or multiple places in the genome and whether it aligned end-to-end or if it was clipped.
\n- This tool also upper-triangularizes the reads, i.e. if the coordinate of second read is higher than the first, the reads are flipped.
\n- \n
For more details, see pairsamtools doc
\n \n- \n
pairtools sort
\n \n- A
pairsam (or generically pairs) file is read in, and a pairsam file is written out. \n- The rows are sorted in chr1-chr2-pos1-pos2 order.
\n- \n
Note that the flipping order and sort order of chromosomes is not identical. See the docs for more details.
\n \n- \n
pairtools merge
\n \n- One or more
pairsam (or generically pairs) files are read in, and a pairsam file is written out. \n- \n
The files are merged, preserving the sorted order.
\n \n- \n
pairtools dedup --mark-dups
\n \n- (equivalent to
pairtools markasdup) \n- Duplicate alignments that share the same pair of 5'end coordinates +/- 3 bps are marked as identified.
\n- \n
An arbitrary one is retained with the original classification, while others get a duplicate classification.
\n \n- \n
pairtools select
\n \n- Only the reads with pairtools classification
UU and UC are retained and output to a pairs file. \n
\n\n**Resolutions**: Both `mcool` and `hic` files contain the \nfollowing resolutions.\n\n* 1kb, 2kb, 5kb, 10kb, 25kb, 50kb, 100kb, 250kb, 500kb, 1Mb, \n2.5Mb, 5Mb, 10Mb\n\n", "display_title": "Matrix Aggregation and Normalization", "uuid": "4b8d6e04-0a69-4a08-aafc-a4a99b45b941", "@type": ["StaticSection", "UserContent", "Item"], "award": {"status": "current", "@type": ["Award", "Item"], "@id": "/awards/1U01CA200059-01/", "uuid": "b0b9c607-f8b4-4f02-93f4-9895b461334b", "display_title": "4D NUCLEOME NETWORK DATA COORDINATION AND INTEGRATION CENTER - PHASE I", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin"]}}, "status": "released", "lab": {"status": "current", "@id": "/labs/4dn-dcic-lab/", "@type": ["Lab", "Item"], "display_title": "4DN DCIC, HMS", "uuid": "828cd4fe-ebb0-4b36-a94a-d2e3a36cc989", "principals_allowed": {"view": ["system.Everyone"], "edit": ["group.admin", "role.lab_submitter", "submits_for.828cd4fe-ebb0-4b36-a94a-d2e3a36cc989"]}}, "@id": "/static-sections/4b8d6e04-0a69-4a08-aafc-a4a99b45b941/", "content_as_html": "For Capture Hi-C, the Hi-C pipeline is run with the default \nrestriction enzyme-based intra-fragment contact filtering, \nbut matrix balancing is not performed.
\n
4DN DCIC provides a Hi-C matrix in two different formats: \n.mcool format and .hic format. The two files are generated\n from the same pairs file as input filtered contact list. \nBoth files contain multiple resolutions.
\n
** .hic format**
\n
\n- A
.hic file is produced by Juicertools (version 1.8.9-cuda8) \nand can be visualized using Juicebox \n- The matrix is normalized using the VC, VC_SQRT, KR methods.
\n
\n
** .mcool format**
\n
\n- An
.mcool file is produced by Cooler (version 0.7.6) and can be visualized using HiGlass. \n- The diagonal and the rows/columns with a low value are \nremoved from the matrix.
\n- The
.mcool file also contains the normalization vectors \ngenerated by Juicertools (same as in a .hic file generated\n from the same pairs file) \n
\n
\n
Resolutions: Both mcool and hic files contain the \nfollowing resolutions.
\n
\n- 1kb, 2kb, 5kb, 10kb, 25kb, 50kb, 100kb, 250kb, 500kb, 1Mb, \n2.5Mb, 5Mb, 10Mb
\n
The pipeline components are pre-installed in a publicly available Docker image (duplexa/4dn-hic:v43) on Docker Hub. The source code for the Docker image and pipeline description in Common Workflow Language (CWL) can be found on GitHub.
\n
\n
Content-wise, 0.2.5, 0.2.6, 0.2.7 can be considered (nearly) identical.
\n
\n0.2.7\n- CWL : https://github.com/4dn-dcic/docker-4dn-hic/tree/v43/cwl
\n- Docker : https://github.com/4dn-dcic/docker-4dn-hic/tree/v43
\n
\n0.2.5/0.2.6\n- CWL : https://github.com/4dn-dcic/docker-4dn-hic/tree/v42.2/cwl
\n- Docker : https://github.com/4dn-dcic/docker-4dn-hic/tree/v42.2
\n
\n- Old runs
\n0.2.0\n- CWL : https://github.com/4dn-dcic/pipelines-cwl/tree/0.2.0/cwl_awsem/
\n- Docker : https://github.com/4dn-dcic/docker-4dn-hic/tree/v40
\n
\n
\n
Example set of commands that were actually run as part of the pipeline can be found at https://github.com/4dn-dcic/docker-4dn-hic/blob/v43/HiCPipeline.md
![](\"https://s3.amazonaws.com/4dn-dcic-public/static-pages/hicpipeline.png\")
\n\n