snakemake 6.0.5: Input a list of folders and multiple files from each folder (to merge Lanes) - bioinformatics

Question

Good day. I have some directories (shown in bold below) each having some .fastq files for different lanes.
CND1/ UD_LOO3_R1.fastq.gz UD_LOO4_R1.fastq.gz
CND2/ XD_L001_R1.fastq.gz XD_L004_R1.fastq.gz
Inside each directory, i want to create a merged fastq file that will be named as : sample_R1.fastq.gz. For instance, CND1/UD_R1.fastq.gz and CND2/XD_R1.fastq.gz etc and so on. To this end, i created the following snakemake workflow.
from collections import defaultdict
dirs,samp,lane = glob_wildcards("dir}/{sample}_L{lane}_R1.fastq.gz")
dirs, fls = glob_wildcards("{dir}/{files}_R1.fastq.gz")
D = defaultdict(list)
for x,y in zip(dirs,fls):
D[x].append(y+'_R1.fastq.gz')
rule ML:
message:
"Merge all Lanes for Fragment R1"
input:
expand( "{dir}/{files}",zip,dir=D.keys(),files=D.values() )
output:
expand( "{dir}/{s}_R1.fastq.gz",zip,dir=dirs,s=set(samp) )
shell:
"echo {input} && echo {output} "
#"zcat {input} >> {output}"
In the code above, dict D contains directories as keys and list of fastq as values.
{ 'CND2': ['XD_L001_R1.fastq.gz', 'XD_L004_R1.fastq.gz'], 'CND1': ['UD_LOO3_R1.fastq.gz', 'UD_LOO4_R1.fastq.gz'] }
Doing a dry-run, snakemake complains me of missing files as follows
Missing input files for rule ML:
CND2/['XD_L001_R1.fastq.gz', 'XD_L004_R1.fastq.gz']
CND1/['UD_LOO3_R1.fastq.gz', 'UD_LOO4_R1.fastq.gz']
I want to understand what is the correct way to provide both a directory and a list of files as input together. Any help is greatly appreciated.
Thanks.

Answer 1

The error clearly explains the issue. As a result of the expand function, your input is two files:
CND2/['XD_L001_R1.fastq.gz', 'XD_L004_R1.fastq.gz']
CND1/['UD_LOO3_R1.fastq.gz', 'UD_LOO4_R1.fastq.gz']
If you need the input to be four files like that:
CND1/UD_LOO3_R1.fastq.gz
CND1/UD_LOO4_R1.fastq.gz
CND2/XD_L001_R1.fastq.gz
CND2/XD_L004_R1.fastq.gz
you need to flatten your dictionary:
inputs = [(dir, file) for dir, files in D.items() for file in files]
rule ML:
input:
expand( "{dir}/{files}", zip, dir=[row[0] for row in inputs], files=[row[1] for row in inputs])
or alternatively:
inputs = [(dir, file) for dir, files in D.items() for file in files]
rule ML:
input:
expand( "{filename}", filename=[f"{dir}/{file}" for dir, file in inputs])
Overall you are overcomplicating the problem. The same can be done without this ugly juggling the lists of tuples. glob_wildcards("{filename_base}_R1.fastq.gz") should return you more convenient representation.