Skip to content

Nextflow Overview

Nextflow is the primary workflow manager used in the Systems Medicine and Bioinformatics (SysMedBio) lab. It ensures that our genomic and bioinformatics pipelines are fully reproducible, parallelized, and easily deployable across local machines and the HPC cluster.

Shared Directory Conventions

To keep all projects organized and prevent duplication of heavy data or software, the lab adheres to standard path structures on the HPC:

Resource Path on HPC (Linux/Mac) Path on Windows Purpose
Pipelines Directory /labs/SysMedBio/Q/Pipelines Q:\SysMedBio\Pipelines Standard directory where our active Nextflow pipelines reside.
Containers Directory /labs/SysMedBio/Q/Containers Q:\SysMedBio\Containers Location where software container configurations and builds are managed.
Container Cache /labs/SysMedBio/Q/Containers/cache Q:\SysMedBio\Containers\cache Centralized cache for Singularity/Apptainer images (.sif).
Command Wrappers /labs/SysMedBio/Q/Containers/wrappers Q:\SysMedBio\Containers\wrappers Local wrappers for running containerized programs directly from the shell.

Multi-platform Access

Depending on your mounting setup, macOS users may access the cluster's network share at /Volumes/SysMedBio/.... Make sure your scripts use relative paths or environment variables when sharing across systems.

GitHub Repositories & Naming Conventions

All repositories related to the lab are hosted under the SysMedBio GitHub Organization.

To make it clear what each repository contains:

  • Nextflow pipelines must end with the -nf suffix (e.g. gwas-qc-nf, scrna-seq-processing-nf).
  • Custom Nextflow pipelines are built on top of our shared templates and modules.

Two Paths for Pipeline Development

Our lab divides pipeline execution and development into two categories:

For standard processing steps (like Bulk RNA-seq or Single-Cell RNA-seq), we utilize pre-existing pipelines from the nf-core community.

  • Use case: Standard omics data processing where the workflow is already peer-reviewed.
  • Guide: See Community Pipelines (nf-core) for how to use them with our shared caches.

For custom analysis, GWAS, cohort-specific workflows, or quality control steps, we build pipelines from scratch using our in-house starter template and modular processes.

  • Use case: Tailored analyses, custom data combinations, or steps not covered by nf-core.
  • Guide: See Custom Pipelines to learn how to bootstrap and build custom pipelines.