Standard Omics Data Processing Guide (nf-core)¶

Overview¶

Having cleanly processed, highly standardized input features is critical for downstream statistical modeling, machine learning, and epidemiology. To handle heavy data wrangling efficiently and reproducibly, the Systems Medicine and Bioinformatics Lab utilizes nf-core: a community-driven framework of peer-reviewed Nextflow pipelines.

A complete directory of available pipelines (currently over 150+) can be explored here: https://nf-co.re/pipelines/

Key Pipelines of Interest¶

Metabolomics & Epidemiology: Refer to pipelines such as nf-core/metaboigniter or various GWAS tools.
Bulk RNA-seq (nf-core/rnaseq): This is the lab's core workflow for bulk transcriptomics. It processes raw FastQ files into complete gene expression matrices.
Single-Cell RNA-seq (nf-core/scrnaseq): A robust, standardized option for analyzing cellular heterogeneity.

In-house Expertise

While nf-core is highly standardized and the basic setup logic is nearly identical across workflows, our primary in-house expertise currently centers around the bulk RNA-seq pipeline.

Shared HPC Resources¶

To prevent redundant downloading of heavy software containers, a centralized Nextflow environment is maintained on the lab's HPC partition at:

/labs/bioinformatics/Q/nfcore

Important Note on the Cache Directory

The shared folder at /labs/bioinformatics/Q/nfcore/singularity_cache currently contains the pre-cached software containers for bulk RNA-seq.

If your project requires a different pipeline (e.g. metabolomics), the specific containers must be cached to the Q drive first to prevent job timeouts. Please reach out to coordinate adding new pipelines to the shared cache, or feel free to pull them directly to the Q drive yourself.

Pipeline Execution Workflow¶

To run a pipeline on the cluster, complete the following steps:

1. Prepare your Samplesheet (CSV)¶

Every nf-core pipeline requires a CSV file mapping your raw data. You can prepare this via command line or in Excel (Save As -> CSV).

Crucial Step

The required column headers change depending on the pipeline. Always check the "Usage" tab on the specific nf-core pipeline documentation.

For the core rnaseq pipeline, the headers must exactly match this format:

samplesheet.csv

sample,fastq_1,fastq_2,strandedness
16_3663,/labs/bioinformatics/Q/projects/P138_RNAseq/fastq/16_S16_R1.fastq.gz,,forward
17_3664,/labs/bioinformatics/Q/projects/P138_RNAseq/fastq/17_S17_R1.fastq.gz,,forward

2. Point to the Shared Cache¶

Create a text file named cache.config in your specific project directory so the pipeline points to the central Q drive resources:

cache.config

singularity.cacheDir = '/labs/bioinformatics/Q/nfcore/singularity_cache'
singularity.pullTimeout = '2 hours'

3. Submit the Job via SLURM¶

Do not run pipelines interactively. Use a standard SLURM submission script to allocate cluster resources. Adapt the following template for your specific run:

submit_nfcore.sh

#!/bin/bash -l
#SBATCH --job-name=nfcore_pipeline_run
#SBATCH --output=%A.%x.out
#SBATCH --error=%A.%x.err
#SBATCH --time=1-23:59:00
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH --cpus-per-task=16
#SBATCH --mem=60G

# 1. Define Executable and Paths
NEXTFLOW="/labs/bioinformatics/Q/nfcore/bin/nextflow"
SAMPLESHEET="/path/to/your/samplesheet.csv"
OUTDIR="/path/to/your/output_folder"
CONFIG="/path/to/your/cache.config"

# 2. Run the pipeline
$NEXTFLOW run nf-core/<pipeline_name> \
  -profile singularity \
  -resume \
  -c $CONFIG \
  --input $SAMPLESHEET \
  --outdir $OUTDIR \
  --max_cpus 16 \
  --max_memory '60.GB'

4. Utilize Pre-built Genomes & Indices¶

For sequencing-based pipelines or custom models, you do not need to pull reference genomes from scratch. Pre-built, highly standardized human and mouse reference genomes (GRCh38, GRCm38, GRCm39) and their corresponding aligner indices (STAR, HISAT2, BWA) are available here:

/labs/bioinformatics/Q/index

Further details

Further details regarding the contents of these genomic files can be found in the associated README: /labs/bioinformatics/Q/index/index.md