Can Snakemake run cluster jobs on a user-defined subset of data? - cluster-computing

Question

I was wondering whether there is any possibility to create cluster jobs with snakemake which will run a batch of tasks.
Assuming the following input:
image_00.tif
image_01.tif
...
image_99.tif
I would like to create the following output:
meas_image_00.txt
meas_image_02.txt
...
meas_image_99.txt
But instead of creating 100 jobs, I would like to submit 5 jobs with 20 input files each, creating 20 output files.
I thought about the group flag but I am not sure whether this is really doing what I want.
The only other way I can imagine is to additionally create a batch file as output and ignore that there will be per image outputs. The expected files then would be something like this:
batch_1.summary
batch_2.summary
batch_3.summary
batch_4.summary
batch_5.summary
But I would loose control if the meas_image*.txt are created at all.
Does anybody have an advice how to solve that?
Edit:
I played around with an example using rules generated by loops. I just concentrated on creating output in batches for now but something goes wrong. This is my snakefile:
data=list(range(0,100))
samples = ["image_" + str(i).zfill(2) for i in data]
batch_size = 5
def chunks(lst, n):
"""Yield successive n-sized chunks from lst."""
for i in range(0, len(lst), n):
yield lst[i:i + n]
def chunklist():
chunk_dict = {}
for i, chunk in enumerate(chunks(samples, batch_size)):
chunk_dict[i] = chunk
return chunk_dict
cd = chunklist()
rule all:
input: 'rule_3.out'
for i,chunk in cd.items():
rule: # batch rule
name: "my_rule_{}".format(i)
output: expand("{sample}.txt", sample=chunk)
group: 'batch_group'
run:
print(output)
print(chunk)
for file in chunk:
print("Creating {} of {}".format(file, chunk))
touch('{}.txt'.format(file))
for i in cd:
rule: # batch rule
name: "batch_rule_{}".format(i)
input: expand('{sample}.txt', sample=cd[i])
group: 'batch_group'
output: touch('{}.group'.format(i))
rule rule3:
input:
files=expand('{sample}.txt', sample=samples),
batches=expand('{i}.group', i=range(len(samples) // batch_size - 1)) # should be ceil in general
output:
'rule_3.out'
shell: 'cat {input.files} > {output}'
And this is the problem, I get:
rule my_rule_10:
output: image_50.txt, image_51.txt, image_52.txt, image_53.txt, image_54.txt
jobid: 12
reason: Missing output files: image_51.txt, image_54.txt, image_53.txt, image_52.txt, image_50.txt
resources: tmpdir=/tmp
image_50.txt image_51.txt image_52.txt image_53.txt image_54.txt
['image_95', 'image_96', 'image_97', 'image_98', 'image_99']
Creating image_95 of ['image_95', 'image_96', 'image_97', 'image_98', 'image_99']
Creating image_96 of ['image_95', 'image_96', 'image_97', 'image_98', 'image_99']
Creating image_97 of ['image_95', 'image_96', 'image_97', 'image_98', 'image_99']
Creating image_98 of ['image_95', 'image_96', 'image_97', 'image_98', 'image_99']
Creating image_99 of ['image_95', 'image_96', 'image_97', 'image_98', 'image_99']
It's strange to me as output is generated by chunk but then differs from the variable chunk...
I have no clue what goes wrong. This example should be executable for everybody. I would be happy to get help.