Invalid threads definition: entries have to be defined as RULE=THREADS pairs (with THREADS being a positive integer). Unparseable value - cluster-computing

Question

Did you notice that set-threads do not work with a recent version of snakemake? It looks long but you just have to copy/paste. Here is a MRE:
mkdir snakemake-test && cd snakemake-test
touch snakeFile
mkdir profile && touch profile/config.yaml && touch profile/status-sacct.sh && chmod +x profile/status-sacct.sh
mkdir envs && touch envs/environment1.yaml && touch envs/environment2.yaml
In envs/environment1.yaml:
channels:
- bioconda
- conda-forge
dependencies:
- snakemake-minimal=7.3.8
- pandas=1.4.2
- peppy=0.31.2
- eido=0.1.4
In envs/environment2.yaml:
channels:
- bioconda
- conda-forge
dependencies:
- snakemake-minimal=6.15.1
- pandas=1.4.2
- peppy=0.31.2
- eido=0.1.4
In snakeFile:
onstart:
print("\t Creating jobs output subfolders...\n")
shell("mkdir -p jobs/downloadgenome")
GENOME = "mm39"
PREFIX = "Mus_musculus.GRCm39"
rule all:
input:
expand("data/fasta/{genome}/{prefix}.dna.chromosome.1.fa", genome=GENOME, prefix=PREFIX)
rule downloadgenome:
output:
"data/fasta/{genome}/{prefix}.dna.chromosome.1.fa"
params:
genomeLinks = "http://ftp.ensembl.org/pub/release-106/fasta/mus_musculus/dna/Mus_musculus.GRCm39.dna.chromosome.1.fa.gz"
threads: 4
shell:
"""
wget {params.genomeLinks}
gunzip {wildcards.prefix}.dna.chromosome.1.fa.gz
mkdir -p data/fasta/{wildcards.genome}
mv {wildcards.prefix}.dna.chromosome.1.fa data/fasta/{wildcards.genome}
"""
In profile/config.yaml:
snakefile: snakeFile
latency-wait: 60
printshellcmds: True
max-jobs-per-second: 1
max-status-checks-per-second: 10
jobs: 400
jobname: "{rule}.{jobid}"
cluster: "sbatch --output=\"jobs/{rule}/slurm_%x_%j.out\" --error=\"jobs/{rule}/slurm_%x_%j.log\" --cpus-per-task={threads} --ntasks=1 --parsable" # --parsable added for handling the timeout exception
cluster-status: "./profile/status-sacct.sh" # Use to handle timeout exception, do not forget to chmod +x
set-threads:
- downloadgenome=2
In profile/status-sacct.sh:
#!/usr/bin/env bash
# Check status of Slurm job
jobid="$1"
if [[ "$jobid" == Submitted ]]
then
echo smk-simple-slurm: Invalid job ID: "$jobid" >&2
echo smk-simple-slurm: Did you remember to add the flag --parsable to your sbatch call? >&2
exit 1
fi
output=`sacct -j "$jobid" --format State --noheader | head -n 1 | awk '{print $1}'`
if [[ $output =~ ^(COMPLETED).* ]]
then
echo success
elif [[ $output =~ ^(RUNNING|PENDING|COMPLETING|CONFIGURING|SUSPENDED).* ]]
then
echo running
else
echo failed
fi
Now build the conda environments:
cd envs
conda env create -p ./smake --file environment1.yaml
conda env create -p ./smake2 --file environment2.yaml
cd ..
If you run the whole thing with smake2 (snakemake snakemake-minimal=6.15.1) it indeeds run the job with 2 CPUs:
conda activate envs/smake2
snakemake --profile profile/
conda deactivate
rm -r data
rm -r jobs
If you do the same thing with smake (snakemake-minimal=7.3.8), it will crash with the error: Invalid threads definition: entries have to be defined as RULE=THREADS pairs (with THREADS being a positive integer). Unparseable value: '{downloadgenome :'.
conda activate envs/smake
snakemake --profile profile/
more jobs/downloadgenome/*log
I tried many things without success to solve the problem...

Answer 1

This was indeed a bug and has been fixed in PR 1615.