Custom Pipelines (nf-template & nf-modules)

When standard community pipelines do not cover your analysis requirements (e.g., custom GWAS, quality control, or specific cohort merging), the lab builds custom Nextflow pipelines.

To maintain consistency and reuse code across the lab, all custom pipelines are built using nf-template as the boilerplate and nf-modules as a shared library of tool processes.

---

The Starter Boilerplate: nf-template

The starter template is hosted at SysMedBio/nf-template. It implements several design patterns that are standard in our lab:

Core Design Patterns

1. Meta-tuple Pattern
   Every process receives a [meta, files] tuple, where meta is a Groovy map containing at least an id key (e.g., cohort or sample name). This id is used to prefix all output filenames, ensuring that downstream results are traceable back to their origin.
2. Per-Cohort Configurations
   All cohort-specific paths and identifiers live in conf/cohorts/*.config files. The main pipeline files (main.nf) remain generic and cohort-agnostic.
3. Two Entrypoint Modes
4. Single-cohort (main.nf): Designed to run a single cohort configuration. Driven by adding a -c conf/cohorts/<COHORT>.config flag.
5. Batch worksheet (main_worksheet.nf): Reads a CSV file detailing paths for multiple cohorts, executing them in parallel. Driven by passing --worksheet worksheets/my_cohorts.csv.
6. Predefined Resource Labels
   Processes are annotated with resource labels rather than hardcoded CPU/memory values. The underlying scheduling is defined in conf/compute.config:

Label	CPUs	Memory	Time Limit	SLURM Queue
process_single	1	4 GB	1 h	standard
process_low	2	12 GB	4 h	standard
process_medium	6	36 GB	8 h	standard
process_high	12	72 GB	16 h	standard
process_long	—	—	7 d	long
process_high_memory	—	200 GB	—	standard
process_high_cpu	64	—	—	standard

---

Bootstrapping a New Pipeline

Step 1: Create the Repository

Always create your pipeline under the SysMedBio GitHub organization and use the -nf suffix.

• Option A (GitHub Interface - Recommended):
• Navigate to SysMedBio/nf-template.
• Click Use this template -> Create a new repository.

• Select the SysMedBio owner, set your repository name (e.g. gwas-qc-nf), and clone it onto the cluster:

  git clone git@github.com:SysMedBio/your-pipeline-name-nf.git
  cd your-pipeline-name-nf
  

• Option B (Manual Detached Clone): If you are working strictly from the terminal on the cluster, you can clone and detach:

  git clone --depth 1 git@github.com:SysMedBio/nf-template.git your-pipeline-name-nf
  cd your-pipeline-name-nf
  rm -rf .git
  git init
  git add .
  git commit -m "Initial commit from nf-template"
  # Push to your newly created repo
  git remote add origin git@github.com:SysMedBio/your-pipeline-name-nf.git
  git push -u origin main
  

Step 2: Configure the Pipeline

1. Edit the header blocks in main.nf and main_worksheet.nf to fill in the author, creation date, and purpose.
2. Add pipeline-specific parameters (e.g., thresholds, filtering options) to conf/pipeline.config.
3. Add any cohorts you want to analyze by creating config files in conf/cohorts/.

---

Reusing Processes: nf-modules

Instead of writing custom process blocks for standard tools (like PLINK2 or BCFtools) from scratch, you should pull existing modules from the SysMedBio/nf-modules repository.

Installing a Module

Ensure you have the wrapper scripts in your PATH (see the Container Wrappers guide), then run:

nf-modules <tool>/<operation>

Examples:

# Fetch PLINK2 PCA module
nf-modules plink2/pca

# Fetch BCFtools Merge module
nf-modules bcftools/merge

This utility automatically checks out the module from the nf-modules repository and places it under modules/<tool>/<operation>/ in your pipeline directory, ready for import.

Importing a Module in main.nf

At the top of your main.nf, include the process using Nextflow DSL2 syntax:

include { PLINK2_PCA } from './modules/plink2/pca/main.nf'

workflow {
    // Channel initialization ...
    pca_results_ch = PLINK2_PCA(samples_ch, 10) // passes meta-tuple and num components
}

---

Module Development Style Guide

If you write a new module, or update an existing one, you must adhere to the style guide in nf-modules to ensure it is clean and reusable across the lab.

1. Process Names & Directories

• File path: modules/<tool>/<operation>/main.nf.
• Process name: <TOOL>_<OPERATION> in SCREAMING_SNAKE_CASE (e.g. process PLINK2_PCA).

2. Tags, Labels, and Container Setup

Configure the process directives as follows:

process TOOL_OPERATION {
    tag   "TOOL_OPERATION_${meta.id}"
    label 'process_low'
    container 'docker://coreygiles/gwas-suite:main'

    // ... inputs, outputs, script
}

• tag: Always include a unique tag starting with the process name and suffixing the meta ID: "TOOL_OPERATION_${meta.id}".
• label: Assign a resource label (process_low, process_medium, etc.) rather than hardcoding compute requirements.
• container: Reference a shared lab container (e.g., docker://coreygiles/gwas-suite:main or another official image from the cache) so the process does not depend on local host tools.

3. Staging Inputs & Genotype Formats

Declare input channels using the meta-tuple pattern:

input:
    tuple val(meta), path(input_files, stageAs: "input/*", arity: 1..3)
    val(args) // extra arguments

• Use stageAs: "input/*" to stage files into a subfolder. This avoids file collisions when processes consume files with identical names from different directories.
• Use arity: 1..3 to support single files (VCFs) and multi-file sets (PLINK binary/pgen sets).
• Pass tool parameters as input values (val(args)) rather than referencing global parameters.

4. The Genotype Helper

If the module works on generic genotype formats (VCF, BED/BIM/FAM, or PGEN/PVAR/PSAM), import and run the Genotype helper class:

import Genotype

// ... in script block:
def genotype_input = Genotype.detect(input_files)

The helper exposes properties to build command arguments dynamically:

• genotype_input.cmd: Formatted input command flag (e.g. --pfile input/sample or --bfile input/sample or --vcf input/sample.vcf).
• genotype_input.make: Format-preserving output flag (e.g. --make-pgen or --make-bed).

5. Outputs & Logs

• Every output must use named emits (e.g. emit: geno_files).
• Log files must be emitted separately:

  output:
      tuple val(meta), path("${meta.id}.{pgen,pvar,psam}"), emit: geno_files
      path("*.log"),                                         emit: logs
  

6. Command Conventions

Pass resource allocations directly from Nextflow variables to ensure the process respects cluster scheduler boundaries:

plink2 \
    --threads ${task.cpus} \
    --memory ${task.memory.toMega()} \
    ${genotype_input.cmd} \
    ${args} \
    --out "${meta.id}"

Use \ to split long command lines, and include comments to document specific flags.