Nextflow: How to process multiple samples - bioinformatics

Question

I have few fq.gz files for few samples. I am trying to process all samples at once using nextflow. But somehow, I am unable to process all the samples at once . But I can process a single sample at once. Here is the data structure and my code for processing single sample.
My nextflow code
params.sampleName="sample1"
params.fastq_path = "data/${params.sampleName}/*{1,2}.fq.gz"
fastq_files = Channel.fromFilePairs(params.fastq_path)
params.ref = "ab.fa"
ref = file(params.ref)
process foo {
input:
set pairId, file(reads) from fastq_files
output:
file("${pairId}.bam") into bamFiles_ch
script:
"""
echo ${reads[0].toRealPath().getParent().baseName}
bwa-mem2 mem -t 8 ${ref} ${reads[0].toRealPath()} ${reads[1].toRealPath()} | samtools sort -#8 -o ${pairId}.bam
samtools index -#8 ${pairId}.bam
"""
}
process samToolsMerge {
publishDir "./aligned_minimap/", mode: 'copy', overwrite: 'false'
input:
file bamFile from bamFiles_ch.collect()
output:
file("**")
script:
"""
samtools merge ${params.sampleName}.bam ${bamFile}
samtools index -# 8 ${params.sampleName}.bam
"""
}
So need help to solve. Thanks in advance.

Answer 1

It looks like you've already built in a way to set your target sample name using:
params.sampleName="sample1"
params.fastq_path = "data/${params.sampleName}/*{1,2}.fq.gz"
To have the glob pattern match all samples, you could simply set the wildcard on the command line using:
nextflow run main.nf --sampleName '*'
Note the quotation marks above. If these are ignored, the glob star will be expanded by your shell before it is passed to your Nextflow command.
The short answer is that you need some easy way to extract the sample name from the parent directory. Then you need some way to group the coordinate-sorted BAMs by the sample name. Below, I've used the new Nextflow DSL 2 but it's not strictly necessary. I just find the new DSL 2 code a lot easier to read and debug. Below is just an example, and you'll need to adapt it to suit your exact use case, but that said, it should do very similar things. It uses a special groupKey so that we can dynamically specify the expected number of elements in each tuple prior to calling the groupTuple operator. This lets us stream the collected values as soon as possible so that each sample can 'merge' when all of it's readgroups have been aligned. Without this, all input readgroups would need to finish alignment before the merge could begin.
Contents of nextflow.config:
process {
shell = [ '/bin/bash', '-euo', 'pipefail' ]
}
Contents of main.nf:
nextflow.enable.dsl=2
params.ref_fasta = "GRCh38.primary_assembly.genome.chr22.fa.gz"
params.fastq_files = "data/*/*.read{1,2}.fastq.gz"
process bwa_index {
conda 'bwa-mem2'
input:
path fasta
output:
path "${fasta}.{0123,amb,ann,bwt.2bit.64,pac}"
"""
bwa-mem2 index "${fasta}"
"""
}
process bwa_mem2 {
tag { [sample, readgroup].join(':') }
conda 'bwa-mem2 samtools'
input:
tuple val(sample), val(readgroup), path(reads)
path bwa_index
output:
tuple val(sample), val(readgroup), path("${readgroup}.bam{,.bai}")
script:
def idxbase = bwa_index.first().baseName
def out_files = [ "${readgroup}.bam", "${readgroup}.bam.bai" ].join('##idx##')
def (r1, r2) = reads
"""
bwa-mem2 mem \\
-R '#RG\\tID:${readgroup}\\tSM:${sample}' \\
-t ${task.cpus} \\
"${idxbase}" \\
"${r1}" \\
"${r2}" |
samtools sort \\
--write-index \\
-# ${task.cpus} \\
-o "${out_files}"
"""
}
process samtools_merge {
tag { sample }
conda 'samtools'
input:
tuple val(sample), path(indexed_bam_files)
output:
tuple val(sample), path("${sample}.bam{,.bai}")
script:
def out_files = [ "${sample}.bam", "${sample}.bam.bai" ].join('##idx##')
def input_bam_files = indexed_bam_files
.findAll { it.name.endsWith('.bam') }
.collect { /"${it}"/ }
.join(' \\\n'+' '*8)
"""
samtools merge \\
--write-index \\
-o "${out_files}" \\
${input_bam_files}
"""
}
workflow {
ref_fasta = file( params.ref_fasta )
bwa_index( ref_fasta )
Channel.fromFilePairs( params.fastq_files ) \
| map { readgroup, reads ->
def (sample_name) = reads*.parent.baseName as Set
tuple( sample_name, readgroup, reads )
} \
| groupTuple() \
| map { sample, readgroups, reads ->
tuple( groupKey(sample, readgroups.size()), readgroups, reads )
} \
| transpose() \
| set { sample_readgroups }
bwa_mem2( sample_readgroups, bwa_index.out )
sample_readgroups \
| join( bwa_mem2.out, by: [0,1] ) \
| map { sample_key, readgroup, reads, indexed_bam ->
tuple( sample_key, indexed_bam )
} \
| groupTuple() \
| map { sample_key, indexed_bam_files ->
tuple( sample_key.toString(), indexed_bam_files.flatten() )
} \
| samtools_merge
}
Run Like:
nextflow run -ansi-log false main.nf
Results:
N E X T F L O W ~ version 21.04.3
Launching `main.nf` [zen_gautier] - revision: dcde9efc8a
Creating Conda env: bwa-mem2 [cache /home/steve/working/stackoverflow/69702077/work/conda/env-8cc153b2eb20a5374bf435019a61c21a]
[63/73c96b] Submitted process > bwa_index
Creating Conda env: bwa-mem2 samtools [cache /home/steve/working/stackoverflow/69702077/work/conda/env-5c358e413a5318c53a45382790eecbd4]
[52/6a92d3] Submitted process > bwa_mem2 (HBR:HBR_Rep2_ERCC-Mix2_Build37-ErccTranscripts-chr22)
[8b/535b21] Submitted process > bwa_mem2 (UHR:UHR_Rep3_ERCC-Mix1_Build37-ErccTranscripts-chr22)
[dc/03d949] Submitted process > bwa_mem2 (UHR:UHR_Rep1_ERCC-Mix1_Build37-ErccTranscripts-chr22)
[e4/bfd08b] Submitted process > bwa_mem2 (HBR:HBR_Rep1_ERCC-Mix2_Build37-ErccTranscripts-chr22)
[d5/e2aa27] Submitted process > bwa_mem2 (UHR:UHR_Rep2_ERCC-Mix1_Build37-ErccTranscripts-chr22)
[c2/23ce8a] Submitted process > bwa_mem2 (HBR:HBR_Rep3_ERCC-Mix2_Build37-ErccTranscripts-chr22)
Creating Conda env: samtools [cache /home/steve/working/stackoverflow/69702077/work/conda/env-912cee20caec78e112a5718bb0c00e1c]
[28/006c03] Submitted process > samtools_merge (HBR)
[3b/51311c] Submitted process > samtools_merge (UHR)