Snakemake slowing down exponentially due to large amount of files being processed - performance

Question

I am currently writing a pipeline, which generates positive RNA sequences, shuffles them and afterwards analyses the positive sequences and the shuffled (negative) sequences. For example, I want to generate 100 positve sequences and shuffle EACH of those sequences 1000 times with three different algorithms. For this purpose, I utilize two wildcards (pos_index and pred_index) ranging from 0 to 100 and 0 to 1000, respectively. As a last step all files are analysed by yet another three different tools.
Now to my problem: The building process of the DAG takes literally hours, which is followed from an even slower execution of the actual pipeline. When it starts, it executes a batch of 32 jobs (because I allocated 32 cores for snakemake) and then takes 10 to 15 minutes to execute the next batch (due to some file checks, i guess). The complete execution of the pipeline would take approximately 2 months.
Below is a simplified example of my snakefile. Is there any way, I can optimise this in a way, so snakemake and its overhead is not the bottleneck anymore?
ITER_POS = 100
ITER_PRED = 1000
SAMPLE_INDEX = range(0, ITER_POS)
PRED_INDEX = range(0, ITER_PRED)
SHUFFLE_TOOLS = ["1", "2", "3"]
PRED_TOOLS = ["A", "B", "C"]
rule all:
input:
# Expand for negative sample analysis
expand("predictions_{pred_tool}/neg_sample_{shuffle_tool}_{sample_index}_{pred_index}.txt",
pred_tool = PRED_TOOLS,
shuffle_tool = SHUFFLE_TOOLS,
sample_index = SAMPLE_INDEX,
pred_index = PRED_INDEX),
# Expand for positive sample analysis
expand("predictions_{pred_tool}/pos_sample_{sample_index}.txt",
pred_tool = PRED_TOOLS,
sample_index = SAMPLE_INDEX)
# GENERATION
rule generatePosSample:
output: "samples/pos_sample_{sample_index}.clu"
shell: "sequence_generation.py > {output}"
# SHUFFLING
rule shufflePosSamples1:
input: "samples/pos_sample_{sample_index}.clu"
output: "samples/neg_sample_1_{sample_index}_{pred_index}.clu"
shell: "sequence_shuffling.py {input} > {output}"
rule shufflePosSamples2:
input: "samples/pos_sample_{sample_index}.clu"
output: "samples/neg_sample_2_{sample_index}_{pred_index}.clu"
shell: "sequence_shuffling.py {input} > {output}"
rule shufflePosSamples3:
input: "samples/pos_sample_{sample_index}.clu"
output: "samples/neg_sample_3_{sample_index}_{pred_index}.clu"
shell: "sequence_shuffling.py {input} > {output}"
# ANALYSIS
rule analysePosSamplesA:
input: "samples/pos_sample_{sample_index}.clu"
output: "predictions_A/pos_sample_{sample_index}.txt"
shell: "sequence_analysis_A.py {input} > {output}"
rule analysePosSamplesB:
input: "samples/pos_sample_{sample_index}.clu"
output: "predictions_B/pos_sample_{sample_index}.txt"
shell: "sequence_analysis_B.py {input} > {output}"
rule analysePosSamplesC:
input: "samples/pos_sample_{sample_index}.clu"
output: "predictions_C/pos_sample_{sample_index}.txt"
shell: "sequence_analysis_C.py {input} > {output}"
rule analyseNegSamplesA:
input: "samples/neg_sample_{shuffle_tool}_{sample_index}_{pred_index}.clu"
output: "predictions_A/neg_sample_{shuffle_tool}_{sample_index}_{pred_index}.txt"
shell: "sequence_analysis_A.py {input} > {output}"
rule analyseNegSamplesB:
input: "samples/neg_sample_{shuffle_tool}_{sample_index}_{pred_index}.clu"
output: "predictions_B/neg_sample_{shuffle_tool}_{sample_index}_{pred_index}.txt"
shell: "sequence_analysis_B.py {input} > {output}"
rule analyseNegSamplesC:
input: "samples/neg_sample_{shuffle_tool}_{sample_index}_{pred_index}.clu"
output: "predictions_C/neg_sample_{shuffle_tool}_{sample_index}_{pred_index}.txt"
shell: "sequence_analysis_C.py {input} > {output}"

Answer 1

Even though I did not really have a large amount of files being processed, and also I did not experience a slowed down execution time, I did experience also a significant slow down of my DAG computation step.
Hence, I want to share my solution:
In cases where you refer in your input to an output of another rule, simply make usage of snakemake's in-build functionality of rule dependencies and rule referencing:
### Bad example
rule bad_example_rule:
input:
"output_from_previous_rule.txt"
output:
"output.txt"
shell:
"touch {output[0]}"
### Solution
rule solution_example_rule:
input:
rules.previous_rule_name.output[0]
output:
"output.txt"
shell:
"touch {output[0]}"
I do not know why, but for me it speeded up the DAG build process by at least x100.