How can I paste to codepad.org from the commandline using curl?
here's a Python script
import urllib
import urllib2
url = 'http://codepad.org'
content=open("myscript.py").read()
values = {'lang' : 'Python',
'code' : content,
'submit':'Submit'}
data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
the_page = response.read()
for href in the_page.split("</a>"):
if "Link:" in href:
ind=href.index('Link:')
found = href[ind+5:]
for i in found.split('">'):
if '<a href=' in i:
print "The link: " ,i.replace('<a href="',"").strip()
output
$ python python.py
The link: http://codepad.org/W0G8HIzM
Yes, you can do it with curl. Assuming your code is Python and in myfile.python, you can do it like this:
$ curl -d "lang=Python&submit=Submit" --data-urlencode code#myfile.py codepad.org
(Edited to make it work.)
You can also use reval:
reval test.py
reval -l ruby -e 'p 2+2'
reval prog.hs -p # make private
Related
I am trying to download some files with snakemake. The files (http://snpeff.sourceforge.net/SnpSift.html#dbNSFP) I would like to download are on a google site/drive and my usual wget approach does not work. I found a bash function that does the job (https://www.zachpfeffer.com/single-post/wget-a-Google-Drive-file):
function gdrive_download () { CONFIRM=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate "https://docs.google.com/uc?export=download&id=$1" -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p') wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$CONFIRM&id=$1" -O $2 rm -rf /tmp/cookies.txt }
gdrive_download 120aPYqveqPx6jtssMEnLoqY0kCgVdR2fgMpb8FhFNHo test.txt
I have tested this function with my ids in a plain bash script and was able to download all the files. To add a bit to the complexity, I must use a workplace template, and incorporate the function into it.
rule dl:
params:
url = 'ftp://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_{genome}/{afile}'
output:
'data/{genome}/{afile}'
params:
id1 = '0B7Ms5xMSFMYlOTV5RllpRjNHU2s',
f1 = 'dbNSFP.txt.gz'
shell:
"""CONFIRM=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate "https://docs.google.com/uc?export=download&id={{params.id1}}" -O- | sed -rn "s/.*confirm=([0-9A-Za-z_]+).*/\1\n/p") && wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$CONFIRM&id={{params.id1}}" -O {{params.f1}} && rm -rf /tmp/cookies.txt"""
#'wget -c {params.url} -O {output}'
rule checksum:
input:
i = 'data/{genome}/{afile}'
output:
o = temp('tmp/{genome}/{afile}.md5')
shell:
'md5sum {input} > {output}'
rule file_size:
input:
i = 'data/{genome}/{afile}'
output:
o = temp('tmp/{genome}/{afile}.size')
shell:
'du -csh --apparent-size {input} > {output}'
rule file_info:
"""md5 checksum and file size"""
input:
md5 = 'tmp/{genome}/{afile}.md5',
s = 'tmp/{genome}/{afile}.size'
output:
o = temp('tmp/{genome}/info/{afile}.csv')
run:
with open(input.md5) as f:
md5, fp = f.readline().strip().split()
with open(input.s) as f:
size = f.readline().split()[0]
with open(output.o, 'w') as fout:
print('filepath,size,md5', file=fout)
print(f"{fp},{size},{md5}", file=fout)
rule manifest:
input:
expand('tmp/{genome}/info/{suffix}.csv', genome=('GRCh37','GRCh38'), suffix=('dbNSFP.txt.gz', 'dbNSFP.txt.gz.tbi'))
#expand('tmp/{genome}/info/SnpSift{suffix}.csv', genome=('GRCh37','GRCh38'), suffix=('dbNSFP.txt.gz', 'dbNSFP.txt.gz.tbi'))
output:
o = 'MANIFEST.csv'
run:
pd.concat([pd.read_csv(afile) for afile in input]).to_csv(output.o, index=False)
There are four downloadable files for which I have ids (I only show one in params), however I don't know how to call the bash functions as written by ZPfeffer for all the ids I have with snakemake. Additionally, when I run this script, there are several errors, the most pressing being
sed: -e expression #1, char 31: unterminated `s' command
I am far from a snakemake expert, any assistance on how to modify my script to a) call the functions with 4 different ids, b) remove the sed error, and c) verify whether this is the correct url format (currently url = 'https://docs.google.com/uc?export/{afile}) will be greatly appreciated.
You would want to use raw string literal so that snakemake doesn't escape special characters, such as backslash in sed command. For example (notice r in front of shell command):
rule foo:
shell:
r"sed d\s\"
You could use --printshellcmds or -p to see how exactly shell: commands get resolved by snakemake.
Here is how I "solved" it:
import pandas as pd
rule dl:
output:
'data/{genome}/{afile}'
shell:
"sh download_snpsift.sh"
rule checksum:
input:
i = 'data/{genome}/{afile}'
output:
o = temp('tmp/{genome}/{afile}.md5')
shell:
'md5sum {input} > {output}'
rule file_size:
input:
i = 'data/{genome}/{afile}'
output:
o = temp('tmp/{genome}/{afile}.size')
shell:
'du -csh --apparent-size {input} > {output}'
rule file_info:
"""md5 checksum and file size"""
input:
md5 = 'tmp/{genome}/{afile}.md5',
s = 'tmp/{genome}/{afile}.size'
output:
o = temp('tmp/{genome}/info/{afile}.csv')
run:
with open(input.md5) as f:
md5, fp = f.readline().strip().split()
with open(input.s) as f:
size = f.readline().split()[0]
with open(output.o, 'w') as fout:
print('filepath,size,md5', file=fout)
print(f"{fp},{size},{md5}", file=fout)
rule manifest:
input:
expand('tmp/{genome}/info/{suffix}.csv', genome=('GRCh37','GRCh38'), suffix=('dbNSFP.txt.gz', 'dbNSFP.txt.gz.tbi'))
output:
o = 'MANIFEST.csv'
run:
pd.concat([pd.read_csv(afile) for afile in input]).to_csv(output.o, index=False)
And here is the bash script.
function gdrive_download () {
CONFIRM=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate "https://docs.google.com/uc?export=download&id=$1" -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')
wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$CONFIRM&id=$1" -O $2
rm -rf /tmp/cookies.txt
}
gdrive_download 0B7Ms5xMSFMYlSTY5dDJjcHVRZ3M data/GRCh37/dbNSFP.txt.gz
gdrive_download 0B7Ms5xMSFMYlOTV5RllpRjNHU2s data/GRCh37/dbNSFP.txt.gz.tbi
gdrive_download 0B7Ms5xMSFMYlbTZodjlGUDZnTGc data/GRCh38/dbNSFP.txt.gz
gdrive_download 0B7Ms5xMSFMYlNVBJdFA5cFZRYkE data/GRCh38/dbNSFP.txt.gz.tbi
Given the following string:
[2018-07-30 12:32:26] local.INFO: {"type":"canonical-log-line","http_method":"POST","http_path":"\/foo","http_status":200,"duration":1.720768928527832,"ts":"2018-07-30 12:32:26"}
How would you select only the JSON object? eg:
{"type":"canonical-log-line","http_method":"POST","http_path":"\/foo","http_status":200,"duration":1.720768928527832,"ts":"2018-07-30 12:32:26"}
I would use sed.
string='[2018-07-30 12:32:26] local.INFO: {"type":"canonical-log-line","http_method":"POST","http_path":"\/foo","http_status":200,"duration":1.720768928527832,"ts":"2018-07-30 12:32:26"}'
echo $string | sed 's/^.*{/{/'
This should work (if you can pipe the output into a python variable and use python) as long as your string only contains 1 pair of open/closing brackets:
import json
import subprocess
#command below is the bash command you want to run:
output = subprocess.check_output(command, shell=True).decode()
# overriding output as i dont know what bash command you're running
output = '[2018-07-30 12:32:26] local.INFO: {"type":"canonical-log-line","http_method":"POST","http_path":"\/foo","http_status":200,"duration":1.720768928527832,"ts":"2018-07-30 12:32:26"} '
json_str = output[output.index('{'): output.index('}')+1]
json_obj = json.loads(json_str)
print(json_obj['type'])
I have tried this code:
from docutils.core import publish_string
text = publish_string(open(file_path, 'r').read(), writer_name='html')
But it says:
<p>Unknown directive type "toctree".</p>
So it won't work with some specific sphinx directives.
What is the easiest way to do same stuff for sphinx RST files?
upd.
Seems like it must be:
sphinx-build -b singlehtml -D extensions='sphinx.ext.autodoc' -D master_doc='index' -C /mypath/docs .
How can I call that from Python code instead of console?
Here is what i wanted to do:
import sphinx
args = ". -b singlehtml -D extensions=sphinx.ext.autodoc -D master_doc=index -C /tmp/doc /tmp/out"
sphinx.main(args.split())
result = open('/tmp/out/index.html', 'r')
here is another example:
# sphinx version = 2.3.1
# Python version = 3.7.3
from sphinx.cmd.build import main as sphinx_main
from pathlib import Path
from os import startfile
import sys
master_doc = 'index'
source_suffix = '.rst'
output_file = 'tmp/dist'
html_theme = 'nature'
build_format = 'html' # singlehtml, ...
args = f". -b {build_format} -D extensions=sphinx.ext.autodoc " \
f"-D master_doc={master_doc} " \
f"-D source_suffix={source_suffix} " \
f"-D html_theme={html_theme} " \
f"-C {output_file} "
sys.path.append(str(Path('.').absolute()))
sphinx_main(args.split())
startfile(Path(output_file).joinpath(master_doc+'.html'))
more parameters >
https://www.sphinx-doc.org/en/master/man/sphinx-build.html
https://www.sphinx-doc.org/en/master/usage/configuration.html
I know this is a duplicate, but my question was not answered in any other threads. the output of sudo cpanm WWW::Mechanize is to long to put in tread. pastebin: 3BYUtSss
I tried executing a perl script, and I get this error:
Can't locate WWW/Mechanize.pm in #INC (#INC contains: /opt/local/lib/perl5/site_perl/5.16.3/darwin-thread-multi-2level /opt/local/lib/perl5/site_perl/5.16.3 /opt/local/lib/perl5/vendor_perl/5.16.3/darwin-thread-multi-2level /opt/local/lib/perl5/vendor_perl/5.16.3 /opt/local/lib/perl5/5.16.3/darwin-thread-multi-2level /opt/local/lib/perl5/5.16.3 /opt/local/lib/perl5/site_perl /opt/local/lib/perl5/vendor_perl .) at io.pl line 5.
In case you need it, here is my perl script's contents:
#!/usr/bin/env perl
use warnings;
use strict;
use WWW::Mechanize;
my $mech = WWW::Mechanize->new();
my ($get,$host,$title);
while (<>) {
if (m|^GET (\S+) |) {
$get = $1;
} elsif ( m|^Host: (\S+)\.| ) {
$host = $1;
} else {
# Unrecognized line...reset
$get = $host = $title = '';
}
if ($get and $host) {
my ($title) = $get =~ m|^.*\/(.+?)$|; # default title
my $url = 'http://' . $host . $get;
$mech->get($url);
if ($mech->success) {
# HTML may have title, images will not
$title = $mech->title() || $title;
}
print "Title: $title\n";
print "URL: $url\n";
print "\n";
$get = $host = $title = '';
}
}
These look to be the key lines in the output from cpanm, down at the bottom.
! Installing the dependencies failed: Installed version (3.59) of CGI is not in range '4.08'
! Bailing out the installation for WWW-Mechanize-1.75.
Looks like you need to install a higher version of the CGI distribution.
The key lines in the cpanm output are:
Building and testing CGI-4.21 ... FAIL
! Installing CGI failed. See /Users/skylerspaeth/.cpanm/work/1440436409.90704/build.log for details. Retry with --force to force install it.
So look in /Users/skylerspaeth/.cpanm/work/1440436409.90704/build.log and see what the problem is. If that log is no longer there, you may need to run cpanm again, which will generate another build.log.
You find the key lines in cpanm output by searching for "fail". Usually, it'll point you at a build.log file for further details.
I am trying to replace piece of shell curl in my ruby with something more native like 'open-uri', but failing and getting: '401 Authorization Required'
I'm trying to replace this:
status = system("curl -Is -w '%{http_code}\\n' --digest -u #{usr}:#{psw} https://#{source}/ -o /dev/null")
With this:
require 'open-uri'
status = open("https://#{source}/", :http_basic_authentication=>[usr, psw])
But still getting 401. Any idea?
Thank you
If you hit any redirects, this could be your problem:
if redirect
...
if options.include? :http_basic_authentication
# send authentication only for the URI directly specified.
options = options.dup
options.delete :http_basic_authentication
end
end