I have written a pipeline in Snakemake. It's an ATAC-seq pipeline (bioinformatics pipeline to analyze genomics data from a specific experiment). Basically, until merging alignment step I use {sample_id} wildcard, to later switch to {sample} wildcard (merging two or more sample_ids into one sample).
working DAG here (for simplicity only one sample shown; orange and blue {sample_id}s are merged into one green {sample}
Tha all rule looks as follows:
configfile: "config.yaml"
SAMPLES_DICT = dict()
with open(config['SAMPLE_SHEET'], "r+") as fil:
next(fil)
for lin in fil.readlines():
row = lin.strip("\n").split("\t")
sample_id = row[0]
sample_name = row[1]
if sample_name in SAMPLES_DICT.keys():
SAMPLES_DICT[sample_name].append(sample_id)
else:
SAMPLES_DICT[sample_name] = [sample_id]
SAMPLES = list(SAMPLES_DICT.keys())
SAMPLE_IDS = [sample_id for sample in SAMPLES_DICT.values() for sample_id in sample]
rule all:
input:
# FASTQC output for RAW reads
expand(os.path.join(config['FASTQC'], '{sample_id}_R{read}_fastqc.zip'),
sample_id = SAMPLE_IDS,
read = ['1', '2']),
# Trimming
expand(os.path.join(config['TRIMMED'],
'{sample_id}_R{read}_val_{read}.fq.gz'),
sample_id = SAMPLE_IDS,
read = ['1', '2']),
# Alignment
expand(os.path.join(config['ALIGNMENT'], '{sample_id}_sorted.bam'),
sample_id = SAMPLE_IDS),
# Merging
expand(os.path.join(config['ALIGNMENT'], '{sample}_sorted_merged.bam'),
sample = SAMPLES),
# Marking Duplicates
expand(os.path.join(config['ALIGNMENT'], '{sample}_sorted_md.bam'),
sample = SAMPLES),
# Filtering
expand(os.path.join(config['FILTERED'],
'{sample}.bam'),
sample = SAMPLES),
expand(os.path.join(config['FILTERED'],
'{sample}.bam.bai'),
sample = SAMPLES),
# multiqc report
"multiqc_report.html"
message:
'\n#################### ATAC-seq pipeline #####################\n'
'Running all necessary rules to produce complete output.\n'
'############################################################'
I know it's too messy, I should only leave the necessary bits, but here my understanding of snakemake fails cause I don't know what I have to keep and what I should delete.
This is working, to my knowledge exactly as I want.
However, I added a rule:
rule hmmratac:
input:
bam = os.path.join(config['FILTERED'], '{sample}.bam'),
index = os.path.join(config['FILTERED'], '{sample}.bam.bai')
output:
model = os.path.join(config['HMMRATAC'], '{sample}.model'),
gappedPeak = os.path.join(config['HMMRATAC'], '{sample}_peaks.gappedPeak'),
summits = os.path.join(config['HMMRATAC'], '{sample}_summits.bed'),
states = os.path.join(config['HMMRATAC'], '{sample}.bedgraph'),
logs = os.path.join(config['HMMRATAC'], '{sample}.log'),
sample_name = '{sample}'
log:
os.path.join(config['LOGS'], 'hmmratac', '{sample}.log')
params:
genomes = config['GENOMES'],
blacklisted = config['BLACKLIST']
resources:
mem_mb = 32000
message:
'\n######################### Peak calling ########################\n'
'Peak calling for {output.sample_name}\n.'
'############################################################'
shell:
'HMMRATAC -Xms2g -Xmx{resources.mem_mb}m '
'--bam {input.bam} --index {input.index} '
'--genome {params.genome} --blacklist {params.blacklisted} '
'--output {output.sample_name} --bedgraph true &> {log}'
And into the rule all, after filtering, before multiqc, I added:
# Peak calling
expand(os.path.join(config['HMMRATAC'], '{sample}.model'),
sample = SAMPLES),
Relevant config.yaml fragments:
# Path to blacklisted regions
BLACKLIST: "/mnt/data/.../hg38.blacklist.bed"
# Path to chromosome sizes
GENOMES: "/mnt/data/.../hg38_sizes.genome"
# Path to filtered alignment
FILTERED: "alignment/filtered"
# Path to peaks
HMMRATAC: "peaks/hmmratac"
This is the error* I get (It goes on for every input and output of the rule). *Technically it's a warning but it halts execution of snakemake so I am calling it an error.
File path alignment/filtered//mnt/data/.../hg38.blacklist.bed.bam contains double '/'. This is likely unintended. It can also lead to inconsistent results of the file-matching approach used by Snakemake.
WARNING:snakemake.logging:File path alignment/filtered//mnt/data/.../hg38.blacklist.bed.bam contains double '/'. This is likely unintended. It can also lead to inconsistent results of the file-matching approach used by Snakemake.
It isn't actually ... - I just didn't feel safe providing an absolute path here.
For a couple of days, I have struggled with this error. Looked through the documentation, listened to the introduction. I understand that the above description is far from perfect (it is huge bc I don't even know how to work it down to provide minimal reproducible example...) but I am desperate and hope you can be patient with me.
Any suggestions as to how to google it, where to look for an error would be much appreciated.
Technically it's a warning but it halts execution of snakemake so I am calling it an error.
It would be useful to post the logs from snakemake to see if snakemake terminated with an error and if so what error.
However, in addition to Eric C.'s suggestion to use wildcards.sample instead of {sample} as file name, I think that this is quite suspicious:
alignment/filtered//mnt/data/.../hg38.blacklist.bed.bam
/mnt/ is usually at the root of the file system and you are prepending to it a relative path (alignment/filtered). Are you sure it is correct?
I'm working with a Ruby script that reads a .json file.
Here is the JSON file:
{
"feed.xml": "93d5b140dd2b4779edef0347ac835fb1",
"index.html": "1cbe25936e392161bad6074d65acdd91",
"md5.json": "655d7c1dbf83a271f348a50a44ba4f6a",
"test.sh": "9be192b1b5a9978cb3623737156445fd",
"index.html": "c064e204040cde216d494776fdcfb68f",
"main.css": "21b13d87db2186d22720e8c881a78580",
"welcome-to-jekyll.html": "01d7c7d66bdeecd9cd69feb5b4b4184d"
}
It is completely valid, and is checked for its existence before trying to read from it. Example:
if File.file?("md5.json")
puts "MD5s exists"
mddigests = File.open("md5.json", "r")
puts "MD5s" + mddigests.read
items = JSON.parse(mddigests.read) <--- Where it all goes wrong.
puts items["feed.xml"]
Everything works up until that point:
MD5s exists
MD5s{
"feed.xml": "93d5b140dd2b4779edef0347ac835fb1",
"index.html": "1cbe25936e392161bad6074d65acdd91",
"md5.json": "655d7c1dbf83a271f348a50a44ba4f6a",
"test.sh": "9be192b1b5a9978cb3623737156445fd",
"index.html": "c064e204040cde216d494776fdcfb68f",
"main.css": "21b13d87db2186d22720e8c881a78580",
"welcome-to-jekyll.html": "01d7c7d66bdeecd9cd69feb5b4b4184d"
}
common.rb:156:in `initialize': A JSON text must at least contain two octets! (JSON::ParserError)
I've searched and tried a lot of different things, to no avail. I'm stumped. Thanks!
You have a duplicate call to read() at the point that it all goes wrong. Replace the second call to read() with the variable mddigests and all should be fine.
This code should work like you'd expect:
if File.file?("md5.json")
puts "MD5s exists"
mddigests = File.open("md5.json", "r")
digests = mddigests.read
puts "MD5s" + digests
items = JSON.parse(digests) #<--- This should work now!
puts items["feed.xml"]
end
The reason is that the file pointer is moved after the first read(), and by the second read(), it's at the end of file, hence the message requiring at least 2 octets.
I'm experimenting with grape and Ruby by trying to make a Yo API callback function.
I can get simple examples up and running like this . . .
resource :loc do
get ':loc' do
params.to_yaml
end
end
How would I go about extracting username and x and y coordinates into separate ruby variable given a callback with the following format?
http://yourcallbackurl.com/yourendpoint?username=THEYOER&location=42.360091;-71.094159
When the location data is screwed up . . .
--- !ruby/hash:Hashie::Mash
username: sfsdfsdf
location: '42.360091'
"-71.094159":
route_info: !ruby/object:Grape::Route
options:
:prefix:
:version: v1
:namespace: "/loc"
:method: GET
:path: "/:version/loc/:loc(.:format)"
:params:
loc: ''
:compiled: !ruby/regexp /\A\/(?<version>v1)\/loc\/(?<loc>[^\/.?]+)(?:\.(?<format>[^\/.?]+))?\Z/
version: v1
loc: toto
format: txt
This is how Rack::Utils works. Default params separators are "&" and ";" (its totally legal according to HTTP standard). So you have to parse query string by yourself here.
location = Rack::Utils.parse_nested_query(env['QUERY_STRING'], '&')['location']
coordinates = location.split(';')
UPD: typo with hash key fixed.
I have a very basic question, am looking at some rails code similar to following but not able to interpret it, what are the REST urls and corresponding actions inferred from this? Can someone please help understand as I did not find any examples of similar routes.
map.resources :myresources do |item|
item.resources :v, :controller => 'my_controller' do |v|
v.resource :abc
end
end
Thanks in advance!!
This is actually the old routing style.
Now you can just write the code this way:
resources :myresources do
resources :v, :controller => "my_controller" do
resource :abc
end
end
With this code you will get these routes:
myresource_v_abc POST /myresources/:myresource_id/v/:v_id/abc(.:format) abcs#create
new_myresource_v_abc GET /myresources/:myresource_id/v/:v_id/abc/new(.:format) abcs#new
edit_myresource_v_abc GET /myresources/:myresource_id/v/:v_id/abc/edit(.:format) abcs#edit
GET /myresources/:myresource_id/v/:v_id/abc(.:format) abcs#show
PUT /myresources/:myresource_id/v/:v_id/abc(.:format) abcs#update
DELETE /myresources/:myresource_id/v/:v_id/abc(.:format) abcs#destroy
myresource_v_index GET /myresources/:myresource_id/v(.:format) my_controller#index
POST /myresources/:myresource_id/v(.:format) my_controller#create
new_myresource_v GET /myresources/:myresource_id/v/new(.:format) my_controller#new
edit_myresource_v GET /myresources/:myresource_id/v/:id/edit(.:format) my_controller#edit
myresource_v GET /myresources/:myresource_id/v/:id(.:format) my_controller#show
PUT /myresources/:myresource_id/v/:id(.:format) my_controller#update
DELETE /myresources/:myresource_id/v/:id(.:format) my_controller#destroy
myresources GET /myresources(.:format) myresources#index
POST /myresources(.:format) myresources#create
new_myresource GET /myresources/new(.:format) myresources#new
edit_myresource GET /myresources/:id/edit(.:format) myresources#edit
myresource GET /myresources/:id(.:format) myresources#show
PUT /myresources/:id(.:format) myresources#update
DELETE /myresources/:id(.:format) myresources#destroy
Setuping a staticMatic project using /index.html:
#slug = current_page.gsub(/\.html/, '')
returns "/index(.html)", but should be /index
Changing term corrects: - #slug = current_page.gsub("/", "").gsub(".html", "") as found in:
https://github.com/adamstac/staticmatic-bootstrap/blob/master/src/helpers/application_helper.rb
To delete the beginning "/" after you've stripped the html simply execute this (which will do both in one command):
current_page.gsub(/\.html/, '').gsub(/\//,''))