I have a hand of commands which each works perfectly in the console when isolated. To ease my work, I collected them into a makefile:
topojsoning: levels.json
topojson --id-property none -p name=elev -o final.json levels.json
geojsoning: contours.shp
ogr2ogr -f GeoJSON -where "elev < 10000" levels.json contours.shp
shaping: crop.tif
gdal_contour -a elev -fl -1000 -500 -200 -50 0 50 100 200 500 1000 2000 4000 6000 crop.tif contours.shp
boxing: ETOPO1_Ice_g_geotiff.tif
gdal_translate -projwin -005.48 051.30 10.00 041.00 ETOPO1_Ice_g_geotiff.tif crop.tif
# ulx uly lrx lry // W N E S // -005.48 051.30 10.00 041.00
unzip: ETOPO1.zip
unzip ETOPO1.zip
touch ETOPO1_Ice_g_geotiff.tif
download:
curl -o ETOPO1.zip 'http://www.ngdc.noaa.gov/mgg/global/relief/ETOPO1/data/ice_surface/grid_registered/georeferenced_tiff/ETOPO1_Ice_g_geotiff.zip'
clean:
rm `ls | grep -v 'zip' | grep -v 'Makefile' `
Yet, when It is within my makefile, I get the following error :
make: *** No rule to make target ? `levels.json', needed by `topojsoning'. Stop.
What does this error means ? How to make it works ? Did I made a small typo ?
There is the full fix:
# topojsoning:
final.json: levels.json
topojson --id-property none -p name=elev -o final.json levels.json
# geojsoning:
levels.json: contours.shp
ogr2ogr -f GeoJSON -where "elev < 10000" levels.json contours.shp
# shaping:
contours.shp: crop.tif
gdal_contour -a elev -fl -1000 -500 -200 -50 0 50 100 200 500 1000 2000 4000 6000 crop.tif contours.shp
# boxing:
crop.tif: ETOPO1_Ice_g_geotiff.tif
gdal_translate -projwin -005.48 051.30 10.00 041.00 ETOPO1_Ice_g_geotiff.tif crop.tif
# ulx uly lrx lry // W N E S // -005.48 051.30 10.00 041.00
# unzip:
ETOPO1_Ice_g_geotiff.tif: ETOPO1.zip
unzip ETOPO1.zip
touch ETOPO1_Ice_g_geotiff.tif
# download:
ETOPO1.zip:
curl -o ETOPO1.zip 'http://www.ngdc.noaa.gov/mgg/global/relief/ETOPO1/data/ice_surface/grid_registered/georeferenced_tiff/ETOPO1_Ice_g_geotiff.zip'
clean:
rm `ls | grep -v 'zip' | grep -v 'Makefile' `
I was misusing the makefile syntaxe. I was starting each step by a process name (i.e. topojsoning, bad) :
processName: sourcefile
command
should start with the target file (final.json, good):
targetfile: sourcefile
command
Related
I am trying to copy all video files recursively from a locally synced FUSE file system (GDrive, Insync, Cryptomator, Ubuntu 20.04), recreating the source directory structure in the destination for any video that is moved - only. Although it is working for some video others are left behind with the same file extension as some that were successfully moved.
Can I adjust my script somehow to make this work properly please? I was also using --ignore-existing but I removed it and have been testing with an empty source directory just in case that helped but it did not.
#!/bin/bash
# Define source and destination directories
src_dir="/home/user/cloud"
dest_dir="/home/user/testmove"
# Use rsync to move files and create directories recursively
rsync --progress --ignore-existing -avm -f'+ *[mM][pP]3$' -f'+ *[fF][lL][aA][cC]$' -f'+ *[mM][4][aA]$' -f'+ *[wW][aA][vV]$' -f'+ *[aA][iI][fF][fF]$' -f'+ *[pP][cC][mM]$' -f'+ *[aA][aA][cC]$' -f'+ *[oO][gG][gG]$' -f'+ *[aA][lL][aA][cC]$' -f'+ */' -f'- *' "$src_dir"/ "$dest_dir"/ --prune-empty-dirs
I found using the $ stopped any files being copied but I am testing following which seems to work, and Eric's much more comprehensive and flexible version from his answer below:
#!/bin/bash
# Define source and destination directories
src_dir="/home/user/cloud"
dest_dir="/home/user/testmove"
# Use rsync to copy files and create directories recursively
rsync -ahPv \
-f'+ *.[mM][pP]3' \
-f'+ *.[mM][4][aA]' \
-f'+ *.[wW][aA][vV]' \
-f'+ *.[oO][gG][gG]' \
-f'+ *.[wW][mM][aA]' \
-f'+ *.[fF][lL][aA][cC]' \
-f'+ *.[aA][iI][fF][fF]' \
-f'+ *.[pP][cC][mM]' \
-f'+ *.[aA][aA][cC]' \
-f'+ *.[aA][lL][aA][cC]' \
-f'+ */' \
-f'- *' \
"$src_dir"/ "$dest_dir" --prune-empty-dirs --stats -n
# To delete empty directories left in source when transferring:
# find "$src_dir" -depth -type d -empty -delete
There were some elements of the command structure which were conflicting, preventing the desired restrictions, and overlooking the inclusions. The following form provides the desired output.
Note that I have some hardcoded values that you need to remove.
#!/bin/bash
audio=1
video=0
DryRun=""
while [ $# -gt 0 ]
do
case $1 in
--source ) src_dir="$2" ; shift ; shift ;;
--target ) dest_dir="$2" ; shift ; shift ;;
--audio ) audio=1 ; video=0 ; shift ;;
--video ) audio=0 ; video=1 ; shift ;;
--dry ) DryRun="--dry-run --stats" ; shift ;;
* ) echo -e "\n Invalid option used on command line. Only valid options: [ --source {src_dir} | --target {dest_dir} | --audio | --video | --dry ]\n Bye!\n" ; exit 1 ;;
esac
done
# Define source and destination directories
#src_dir="/home/user/cloud"
src_dir="`pwd`/Output"
#dest_dir="/home/user/testmove"
dest_dir="`pwd`/Dupl"
# Define rules for include or exclude
cat >"EXCLUDES.filters" <<"EnDoFiNpUt"
- *
EnDoFiNpUt
#filters="--filter \"merge AUDIO_TYPES.filters\" " ### THIS FORM DOES NOT WORK !!!
if [ ${audio} -eq 1 ]
then
### NOTE: Trailing "$" to match end of string is implicit, unless there is a trailing "*"
cat >"AUDIO_TYPES.filters" <<"EnDoFiNpUt"
+ */
+ *\.[mM][pP][3]
+ *\.[fF][lL][aA][cC]
+ *\.[mM][4][aA]
+ *\.[wW][aA][vV]
+ *\.[aA][iI][fF][fF]
+ *\.[pP][cC][mM]
+ *\.[aA][aA][cC]
+ *\.[oO][gG][gG]
+ *\.[aA][lL][aA][cC]
EnDoFiNpUt
filters="--include-from=AUDIO_TYPES.filters --exclude-from=EXCLUDES.filters"
fi
if [ ${video} -eq 1 ]
then
### NOTE: Trailing "$" to match end of string is implicit, unless there is a trailing "*"
cat >"VIDEO_TYPES.filters" <<"EnDoFiNpUt"
+ */
+ *\.[mM][pP][4]
+ *\.[aA][vV][iI]
+ *\.[mM][kK][vV]
+ *\.[fF][lL][vV]
+ *\.[wW][mM][vV]
+ *\.[mM][oO][vV]
+ *\.[Aa][Vv][Cc][Hh][Dd]
+ *\.[wW][eE][bB][mM]
+ *\.[hH][2][6][4]
+ *\.[mM][pP][eE][gG][4]
+ *\.[Aa][Vv][Cc][Hh][Dd]
EnDoFiNpUt
filters="--include-from=VIDEO_TYPES.filters --exclude-from=EXCLUDES.filters"
fi
# Use rsync to move files and create directories recursively
### NOTE: I prefer long-form options for understanding code at first glance; shortform doesn't do that for me.
#rsync -rlptgoDvP ### shortform for universal options
rsync \
--verbose \
--recursive \
--links \
--owner \
--group \
--perms \
--times \
--devices \
--specials \
--prune-empty-dirs \
--partial \
--progress \
${DryRun}\
${filters} \
"${src_dir}/" "${dest_dir}/"
Log of session output:
building file list ...
3 files to consider
created directory /0__WORK/Dupl
./
DEF.mP4
0 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=1/3)
abc.mp4
0 100% 0.00kB/s 0:00:00 (xfr#2, to-chk=0/3)
sent 196 bytes received 126 bytes 644.00 bytes/sec
total size is 0 speedup is 0.00
Session log for version of script modified per requestor's (saltyeggs) reformulated command (also works, differently, but as expected):
building file list ...
3 files to consider
created directory /0__WORK/Dupl
./
DEF.mP4
abc.mp4
Number of files: 3 (reg: 2, dir: 1)
Number of created files: 3 (reg: 2, dir: 1)
Number of deleted files: 0
Number of regular files transferred: 2
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 128
Total bytes received: 94
sent 128 bytes received 94 bytes 444.00 bytes/sec
total size is 0 speedup is 0.00 (DRY RUN)
I have an file , my file name is lookup.txt. Sometimes this file content is 100 lines , so everything scroll while printing. Thats why i like to print it on screen in 2 or more columns.
my bash doesnt support commands like pr or columns.Need to be in old fashion shell.
Content of lookup file is:
1-Top Scores
2-Last Scores
3-Username
4-Birthday
5-Last Meal today
6-Coronavirus in Italy
7-Lets Stay Home
8-Winner
9-Germany and Belgium
10-Alfa
11-Bravo
12-Charlie and delta
For printing in columns i use this command found here on stackoverflow:
cat lookup.txt | xargs printf '%-24s\n' | sed '$p;N;s/\n//;$p;N;s/\n//'
and this is the print in columns:
1-Top Scores 2-Last
Scores 3-Username 4-Birthday
5-Last Meal today
6-Coronavirus in Italy
7-Lets Stay Home
8-Winner 9-Germany and
Belgium 10-Alfa 11-Bravo
12-Charlie and delta
What i need is to let sed not split the words into different columns but just the line.
1-Top Scores 6-Coronavirus in Italy 11-Bravo
2-Last Scores 7-Lets Stay Home 12-Charlie and delta
3-Username 8-Winner
4-Birthday 9-Germany and Belgium
5-Last Meal today 10-Alfa
if the 3 columns are to width then x chars then print in 2 columns (this last is optional)
thanks in advance
Available tools:
/bin:
ash gunzip ping
base64 gzip ping6
bash hostname ps
bash.bash ionice ps.procps
busybox kill pwd
busybox.nosuid kill.procps rm
busybox.suid kmod rmdir
cat ln run-parts
chattr login sed
chgrp login.shadow setserial
chmod ls sh
chown lsattr sleep
cp lsmod stat
cttyhack lsmod.kmod stty
date mkdir su
dd mknod su.shadow
df mktemp sync
dmesg more tar
dnsdomainname mount touch
echo mount.util-linux true
editor mountpoint true.coreutils
egrep mountpoint.sysvinit umount
fake-hwclock mpstat uname
false mv usleep
false.coreutils netstat vi
fatattr nice vi.sh
fgrep pidof watch
getopt pidof.procps watch.procps
grep pidof.sysvinit zcat
/sbin:
arp ifup mkfs.ext4 runlevel.sysvinit
blkid init mkswap setconsole
blkid.util-linux init.sysvinit modinfo shutdown
bootlogd inotifyd modinfo.kmod shutdown.sysvinit
bridge insmod modprobe start-stop-daemon
depmod insmod.kmod modprobe.kmod stb-hwclock
depmod.kmod ip mount.cifs sulogin
devmem ip.iproute2 mount.fuse sulogin.util-linux
e2fsck iwconfig mount.ntfs swapoff
e2label iwgetid mount.ntfs-3g swapon
flock iwlist nologin switch_root
flock.util-linux iwpriv nologin.shadow sysctl
fsck.ext2 iwspy poweroff sysctl.procps
fsck.ext3 killall5 poweroff.sysvinit syslogd
fsck.ext4 klogd reboot telinit
fstab-decode ldconfig reboot.sysvinit tunctl
fstrim loadkmap rmmod tune2fs
getty logread rmmod.kmod udhcpc
halt losetup route uevent
halt.sysvinit lsmod routef vigr
hdparm mdev routel vigr.shadow
ifcfg mke2fs rtmon vipw
ifconfig mkfs.ext2 rtpr vipw.shadow
ifdown mkfs.ext3 runlevel
/usr/bin:
[ realpath
[[ renice
alsamixer reset
ar resize
awk rpcclient
basename rtmpdump
bdpoll scp
bsdcat scsi_logging_level
bunzip2 scsi_mandat
bzcat scsi_readcap
c_rehash scsi_ready
chage scsi_satl
cheetah scsi_start
cheetah-analyze scsi_stop
cheetah-compile scsi_temperature
chfn sdparm
chfn.shadow seq
chrt setsid
chsh sexp-conv
chsh.shadow sg
cifsdd sg_compare_and_write
clear sg_copy_results
cmp sg_dd
crontab sg_decode_sense
curl sg_emc_trespass
cut sg_format
dbclient sg_get_config
dbus-cleanup-sockets sg_get_lba_status
dbus-daemon sg_ident
dbus-launch sg_inq
dbus-monitor sg_logs
dbus-run-session sg_luns
dbus-send sg_map
dbus-update-activation-environment sg_map26
dbus-uuidgen sg_modes
dc sg_opcodes
diff sg_persist
dirname sg_prevent
dlist_test sg_raw
dos2unix sg_rbuf
dotlockfile sg_rdac
du sg_read
easy_install sg_read_attr
easy_install-2.7 sg_read_block_limits
eject sg_read_buffer
enigma2 sg_read_long
enigma2.sh sg_readcap
env sg_reassign
expiry sg_referrals
expr sg_rep_zones
faillog sg_requests
find sg_reset
flip sg_reset_wp
free sg_rmsn
free.procps sg_rtpg
fuser sg_safte
fusermount sg_sanitize
get_device sg_sat_identify
get_driver sg_sat_phy_event
get_module sg_sat_read_gplog
gpasswd sg_sat_set_features
grab sg_scan
groups sg_senddiag
groups.shadow sg_ses
gst-inspect-1.0 sg_ses_microcode
gst-launch-1.0 sg_start
gst-stats-1.0 sg_stpg
gst-typefind-1.0 sg_sync
head sg_test_rwbuf
hotplug_e2_helper sg_timestamp
id sg_turs
killall sg_unmap
last sg_verify
last.sysvinit sg_vpd
lastb sg_wr_mode
lastb.sysvinit sg_write_buffer
lastlog sg_write_long
ldbadd sg_write_same
ldbdel sg_write_verify
ldbedit sg_xcopy
ldbmodify sg_zone
ldbrename sginfo
ldbsearch sgm_dd
less sgp_dd
llmnrd sha1sum
lockfile-check sha256sum
lockfile-create sha3sum
lockfile-remove sha512sum
lockfile-touch shellinaboxd
logger showiframe
logname shuf
lsof skill
lspci skill.procps
lsusb slabtop
mail-lock smbcacls
mail-touchlock smbclient
mail-unlock smbcquotas
md5sum smbget
mesg smbspool
mesg.sysvinit smbtar
mid3cp smbtree
mid3iconv snice
mid3v2 snice.procps
mkfifo sort
moggsplit ssh
mutagen-inspect strings
mutagen-pony systool
nc tail
ndg_httpclient taskset
nettle-hash tee
nettle-lfib-stream telnet
nettle-pbkdf2 test
newgidmap time
newgrp timeout
newgrp.shadow tload
newuidmap toix
nmap toms
nohup top
nslookup top.procps
ntfs-3g tr
ntfs-3g.probe traceroute
ntpdate-sync traceroute6
od truncate
odhcp6c tty
ofgwrite ulockmgr_server
ofgwrite_bin uniq
ofgwrite_test unix2dos
openssl unlink
opkg unxz
opkg-check-config unzip
passwd update-alternatives
passwd.shadow uptime
patch uptime.procps
pgrep users
pgrep.procps utmpdump
pilconvert.py utmpdump.sysvinit
pildriver.py vlock
pilfile.py vmstat
pilfont.py volname
pilprint.py w
pkcs1-conv wall
pkill wall.sysvinit
pkill.procps wc
pmap wget
pmap.procps which
printf who
pwdx whoami
pwdx.procps whois
python wpa_passphrase
python-config xargs
python2 xmlcatalog
python2-config xmllint
python2.7 xzcat
python2.7-config yes
readlink
/usr/sbin:
addgroup grpconv pwck
adduser grpunconv pwconv
alsactl hddtemp pwunconv
automount i2cdetect rdate
avahi-daemon i2cdump rdev
chgpasswd i2cget readprofile
chpasswd i2cset rfkill
chpasswd.shadow ifplugd run-postinsts
chroot inetd setlogcons
crond logoutd sfdisk
delgroup mkfs.ubifs telnetd
deluser mtd_debug ubiattach
dropbear mtdinfo ubiblock
dropbearconvert mtdpart ubicrc32
dropbearkey nanddump ubidetach
dropbearmulti nandtest ubiformat
ether-wake nandwrite ubimkvol
ethtool newusers ubinfo
fbset nl-class-add ubinize
flash_erase nl-class-delete ubirename
flash_eraseall nl-class-list ubirmvol
flash_lock nl-classid-lookup ubirsvol
flash_otp_dump nl-cls-add ubiupdatevol
flash_otp_info nl-cls-delete unix_chkpwd
flash_otp_lock nl-cls-list unix_update
flash_otp_write nl-link-list update-ca-certificates
flash_unlock nl-pktloc-lookup update-rc.d
flashcp nl-qdisc-add useradd
ftpd nl-qdisc-delete userdel
genl-ctrl-list nl-qdisc-list usermod
groupadd ntpd vsftpd
groupdel ntpdate wpa_action
groupmems odhcp6c-update wpa_cli
groupmod parted wpa_supplicant
grpck partprobe
root#xp1000max:~#
Not claiming complete credit, you can modify this andlrc's answer on In bash how can I split a column in several column of fixed dimension to your requirement.
Instead of appending the columns with a whitespace, you can use sprintf() to custom print your column strings. Tested on GNU awk
awk '
BEGIN {
# Numbers of rows to print
n=5;
}
{
# Add to array with key = 0, 1, 2, 3, 4, 0, 1, 2, ..
l[(NR-1)%n] = sprintf("%s%-24s", l[(NR-1)%n], $0)
};
END {
# print the array
for (i = 0; i < length(l); i++) {
print l[i];
}
}
' file
Try this:
cat lookup.txt | tr '\n' '\0' | xargs -0 -n 2 printf '%-24s %-24s\n'
The tr command will convert newlines to nul chars.
Then we tell xargs to take nul-separated args and pass two at a time to the command.
I am trying to download some files with snakemake. The files (http://snpeff.sourceforge.net/SnpSift.html#dbNSFP) I would like to download are on a google site/drive and my usual wget approach does not work. I found a bash function that does the job (https://www.zachpfeffer.com/single-post/wget-a-Google-Drive-file):
function gdrive_download () { CONFIRM=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate "https://docs.google.com/uc?export=download&id=$1" -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p') wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$CONFIRM&id=$1" -O $2 rm -rf /tmp/cookies.txt }
gdrive_download 120aPYqveqPx6jtssMEnLoqY0kCgVdR2fgMpb8FhFNHo test.txt
I have tested this function with my ids in a plain bash script and was able to download all the files. To add a bit to the complexity, I must use a workplace template, and incorporate the function into it.
rule dl:
params:
url = 'ftp://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_{genome}/{afile}'
output:
'data/{genome}/{afile}'
params:
id1 = '0B7Ms5xMSFMYlOTV5RllpRjNHU2s',
f1 = 'dbNSFP.txt.gz'
shell:
"""CONFIRM=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate "https://docs.google.com/uc?export=download&id={{params.id1}}" -O- | sed -rn "s/.*confirm=([0-9A-Za-z_]+).*/\1\n/p") && wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$CONFIRM&id={{params.id1}}" -O {{params.f1}} && rm -rf /tmp/cookies.txt"""
#'wget -c {params.url} -O {output}'
rule checksum:
input:
i = 'data/{genome}/{afile}'
output:
o = temp('tmp/{genome}/{afile}.md5')
shell:
'md5sum {input} > {output}'
rule file_size:
input:
i = 'data/{genome}/{afile}'
output:
o = temp('tmp/{genome}/{afile}.size')
shell:
'du -csh --apparent-size {input} > {output}'
rule file_info:
"""md5 checksum and file size"""
input:
md5 = 'tmp/{genome}/{afile}.md5',
s = 'tmp/{genome}/{afile}.size'
output:
o = temp('tmp/{genome}/info/{afile}.csv')
run:
with open(input.md5) as f:
md5, fp = f.readline().strip().split()
with open(input.s) as f:
size = f.readline().split()[0]
with open(output.o, 'w') as fout:
print('filepath,size,md5', file=fout)
print(f"{fp},{size},{md5}", file=fout)
rule manifest:
input:
expand('tmp/{genome}/info/{suffix}.csv', genome=('GRCh37','GRCh38'), suffix=('dbNSFP.txt.gz', 'dbNSFP.txt.gz.tbi'))
#expand('tmp/{genome}/info/SnpSift{suffix}.csv', genome=('GRCh37','GRCh38'), suffix=('dbNSFP.txt.gz', 'dbNSFP.txt.gz.tbi'))
output:
o = 'MANIFEST.csv'
run:
pd.concat([pd.read_csv(afile) for afile in input]).to_csv(output.o, index=False)
There are four downloadable files for which I have ids (I only show one in params), however I don't know how to call the bash functions as written by ZPfeffer for all the ids I have with snakemake. Additionally, when I run this script, there are several errors, the most pressing being
sed: -e expression #1, char 31: unterminated `s' command
I am far from a snakemake expert, any assistance on how to modify my script to a) call the functions with 4 different ids, b) remove the sed error, and c) verify whether this is the correct url format (currently url = 'https://docs.google.com/uc?export/{afile}) will be greatly appreciated.
You would want to use raw string literal so that snakemake doesn't escape special characters, such as backslash in sed command. For example (notice r in front of shell command):
rule foo:
shell:
r"sed d\s\"
You could use --printshellcmds or -p to see how exactly shell: commands get resolved by snakemake.
Here is how I "solved" it:
import pandas as pd
rule dl:
output:
'data/{genome}/{afile}'
shell:
"sh download_snpsift.sh"
rule checksum:
input:
i = 'data/{genome}/{afile}'
output:
o = temp('tmp/{genome}/{afile}.md5')
shell:
'md5sum {input} > {output}'
rule file_size:
input:
i = 'data/{genome}/{afile}'
output:
o = temp('tmp/{genome}/{afile}.size')
shell:
'du -csh --apparent-size {input} > {output}'
rule file_info:
"""md5 checksum and file size"""
input:
md5 = 'tmp/{genome}/{afile}.md5',
s = 'tmp/{genome}/{afile}.size'
output:
o = temp('tmp/{genome}/info/{afile}.csv')
run:
with open(input.md5) as f:
md5, fp = f.readline().strip().split()
with open(input.s) as f:
size = f.readline().split()[0]
with open(output.o, 'w') as fout:
print('filepath,size,md5', file=fout)
print(f"{fp},{size},{md5}", file=fout)
rule manifest:
input:
expand('tmp/{genome}/info/{suffix}.csv', genome=('GRCh37','GRCh38'), suffix=('dbNSFP.txt.gz', 'dbNSFP.txt.gz.tbi'))
output:
o = 'MANIFEST.csv'
run:
pd.concat([pd.read_csv(afile) for afile in input]).to_csv(output.o, index=False)
And here is the bash script.
function gdrive_download () {
CONFIRM=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate "https://docs.google.com/uc?export=download&id=$1" -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')
wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$CONFIRM&id=$1" -O $2
rm -rf /tmp/cookies.txt
}
gdrive_download 0B7Ms5xMSFMYlSTY5dDJjcHVRZ3M data/GRCh37/dbNSFP.txt.gz
gdrive_download 0B7Ms5xMSFMYlOTV5RllpRjNHU2s data/GRCh37/dbNSFP.txt.gz.tbi
gdrive_download 0B7Ms5xMSFMYlbTZodjlGUDZnTGc data/GRCh38/dbNSFP.txt.gz
gdrive_download 0B7Ms5xMSFMYlNVBJdFA5cFZRYkE data/GRCh38/dbNSFP.txt.gz.tbi
I want to export one certain page from a pdf document to an image and automatically fill the page number in the file name. When I run the following code:
gs \
-sDEVICE=jpeg \
-o outfile-%03.jpeg \
-dFirstPage=12 \
-dLastPage=12 \
wo.pdf
I get: outfile-001.jpeg instead of outfile-012.jpeg.
I've wrote a bash script for the job:
function extract_nth_page(){
printf -v j "outfile-%05g.png" $1
echo $j
gs -q -dNOPAUSE -sDEVICE=png16m -r600 -dFirstPage=$1 -dLastPage=$1 -sOutputFile=$j $2 -c quit
return 0
}
# Extracts page number 42 from myFile.pdf to outfile-00042.png
extract_nth_page 42 myFile.pdf
Given a GIS raster with elevation data, How to design a topographic map in D3js ?
Is there any example of relief / topographic maps of cropped lands made using D3js ?
Not working: I explored the posibility of .tif > gdal_contour.py > .shp > topojson > d3js without success.
I use a makefile which contain all my commands. As my area of interest (France) is a crop of land areas, the gdal_contour.py approach generates broken isolines which does NOT create closed polygons. Also, the SVG end result fails. The only example of D3 topographic map I know is about Iceland, which, as an island, avoid this issue: cropping the country out of the world doesn't result in broken isolines.
nb: This project is part of the #Wikipedia #wikimaps project.
Topographic map now on D3js, with full makefile workflow ! See http://bl.ocks.org/hugolpz/6279966 (<= older code, compare to here on SO)
0. Requirements:
Geographic area: You may customize your geographic area of interest by editing one line within each of the 2 files : makefile#boxing and html#Geo-frame_borders with your own decimal coordinates fo W,N,E,S borders, something like:
var WNES = { "target": "France", "W": -5.3, "N":51.6, "E": 10.2, "S": 41.0 };
Software: make, curl, unzip, gdal (include ogr, gdal_calc.py, gdal_polygonize.py), nodejs, topojson. Helpful: touch. The makefile then manage to download the sources, process them, and output a single topojson file that the D3js code provided can use.
1. Save into folder name: /topo_map/topo.mk
# topojsoning:
final.json: levels.json
topojson --id-property none --simplify=0.5 -p name=elev -o final.json -- levels.json
# simplification approach to explore further. Feedbacks welcome.
# shp2jsoning:
levels.json: levels.shp
ogr2ogr -f GeoJSON -where "elev < 10000" levels.json levels.shp
# merge
levels.shp: level0001.shp level0050.shp level0100.shp level0200.shp level0500.shp level1000.shp level2000.shp level3000.shp level4000.shp level5000.shp
ogr2ogr levels.shp level0001.shp
ogr2ogr -update -append levels.shp level0050.shp
ogr2ogr -update -append levels.shp level0100.shp
ogr2ogr -update -append levels.shp level0200.shp
ogr2ogr -update -append levels.shp level0500.shp
ogr2ogr -update -append levels.shp level1000.shp
ogr2ogr -update -append levels.shp level2000.shp
ogr2ogr -update -append levels.shp level3000.shp
ogr2ogr -update -append levels.shp level4000.shp
ogr2ogr -update -append levels.shp level5000.shp
# Polygonize slices:
level0001.shp: level0001.tif
gdal_polygonize.py level0001.tif -f "ESRI Shapefile" level0001.shp level_0001 elev
level0050.shp: level0050.tif
gdal_polygonize.py level0050.tif -f "ESRI Shapefile" level0050.shp level_0050 elev
level0100.shp: level0100.tif
gdal_polygonize.py level0100.tif -f "ESRI Shapefile" level0100.shp level_0100 elev
level0200.shp: level0200.tif
gdal_polygonize.py level0200.tif -f "ESRI Shapefile" level0200.shp level_0200 elev
level0500.shp: level0500.tif
gdal_polygonize.py level0500.tif -f "ESRI Shapefile" level0500.shp level_0500 elev
level1000.shp: level1000.tif
gdal_polygonize.py level1000.tif -f "ESRI Shapefile" level1000.shp level_1000 elev
level2000.shp: level2000.tif
gdal_polygonize.py level2000.tif -f "ESRI Shapefile" level2000.shp level_2000 elev
level3000.shp: level3000.tif
gdal_polygonize.py level3000.tif -f "ESRI Shapefile" level3000.shp level_3000 elev
level4000.shp: level4000.tif
gdal_polygonize.py level4000.tif -f "ESRI Shapefile" level4000.shp level_4000 elev
level5000.shp: level5000.tif
gdal_polygonize.py level5000.tif -f "ESRI Shapefile" level5000.shp level_5000 elev
# Raster slicing:
level0001.tif: crop.tif
gdal_calc.py -A crop.tif --outfile=level0001.tif --calc="1*(A>0)" --NoDataValue=0
level0050.tif: crop.tif
gdal_calc.py -A crop.tif --outfile=level0050.tif --calc="50*(A>50)" --NoDataValue=0
level0100.tif: crop.tif
gdal_calc.py -A crop.tif --outfile=level0100.tif --calc="100*(A>100)" --NoDataValue=0
level0200.tif: crop.tif
gdal_calc.py -A crop.tif --outfile=level0200.tif --calc="200*(A>200)" --NoDataValue=0
level0500.tif: crop.tif
gdal_calc.py -A crop.tif --outfile=level0500.tif --calc="500*(A>500)" --NoDataValue=0
level1000.tif: crop.tif
gdal_calc.py -A crop.tif --outfile=level1000.tif --calc="1000*(A>1000)" --NoDataValue=0
level2000.tif: crop.tif
gdal_calc.py -A crop.tif --outfile=level2000.tif --calc="2000*(A>2000)" --NoDataValue=0
level3000.tif: crop.tif
gdal_calc.py -A crop.tif --outfile=level3000.tif --calc="3000*(A>3000)" --NoDataValue=0
level4000.tif: crop.tif
gdal_calc.py -A crop.tif --outfile=level4000.tif --calc="4000*(A>4000)" --NoDataValue=0
level5000.tif: crop.tif
gdal_calc.py -A crop.tif --outfile=level5000.tif --calc="5000*(A>5000)" --NoDataValue=0
# boxing:
crop.tif: ETOPO1_Ice_g_geotiff.tif
gdal_translate -projwin -5.3 41.0 10.2 51.6 ETOPO1_Ice_g_geotiff.tif crop.tif
# ulx uly lrx lry // W S E N
# unzip:
ETOPO1_Ice_g_geotiff.tif: ETOPO1.zip
unzip ETOPO1.zip
touch ETOPO1_Ice_g_geotiff.tif
# download:
ETOPO1.zip:
curl -o ETOPO1.zip 'http://www.ngdc.noaa.gov/mgg/global/relief/ETOPO1/data/ice_surface/grid_registered/georeferenced_tiff/ETOPO1_Ice_g_geotiff.zip'
clean:
rm `ls | grep -v 'zip' | grep -v 'Makefile'`
# Makefile v4b (#hugo_lz)
2. Create data by running the makfile :
cd ./topo_map
make -f ./topo.mk
3. D3js & HTML code with auto-focus:
<!-- language: html -->
<style>
svg { border: 5px solid #333; background-color: #C6ECFF;}
/* TOPO */
path.Topo_1 { fill:#ACD0A5; stroke: #0978AB; stroke-width: 1px; }
path.Topo_50 {fill: #94BF8B; }
path.Topo_100 {fill: #BDCC96; }
path.Topo_200 {fill: #E1E4B5; }
path.Topo_500 {fill: #DED6A3; }
path.Topo_1000 {fill:#CAB982 ; }
path.Topo_2000 {fill: #AA8753; }
path.Topo_3000 {fill: #BAAE9A; }
path.Topo_4000 {fill: #E0DED8 ; }
path.Topo_5000 {fill: #FFFFFF ; }
.download {
background: #333;
color: #FFF;
font-weight: 900;
border: 2px solid #B10000;
padding: 4px;
margin:4px;
}
</style>
<body>
<script src="http://code.jquery.com/jquery-2.0.2.min.js"></script>
<script src="http://d3js.org/d3.v3.min.js"></script>
<script src="http://d3js.org/topojson.v1.min.js"></script>
<script>
// 1. -------------- SETTINGS ------------- //
// Geo-frame_borders in decimal ⁰: France
var WNES = { "W": -5.3, "N":51.6, "E": 10.2, "S": 41.0 };
// Geo values of interest :
var latCenter = (WNES.S + WNES.N)/2,
lonCenter = (WNES.W + WNES.E)/2,
geo_width = (WNES.E - WNES.W),
geo_height= (WNES.N - WNES.S);
// HTML expected frame dimensions
var width = 600,
height = width * (geo_height / geo_width);
// Projection: projection, reset scale and translate
var projection = d3.geo.equirectangular()
.scale(1)
.translate([0, 0]);
// SVG injection:
var svg = d3.select("body").append("svg")
.attr("width", width)
.attr("height", height);
// Path
var path = d3.geo.path()
.projection(projection)
.pointRadius(4);
// Data (getJSON: TopoJSON)
d3.json("final.json", showData);
// 2. ---------- FUNCTION ------------- //
function showData(error, fra) {
var Levels = topojson.feature(fra, fra.objects.levels);
// Focus area box compute for derive scale & translate.
// [[left, bottom], [right, top]] // E W N S
var b = path.bounds(Levels),
s = 1 / Math.max((b[1][0] - b[0][0]) / width, (b[1][1] - b[0][1]) / height),
t = [(width - s * (b[1][0] + b[0][0])) / 2, (height - s * (b[1][1] + b[0][1])) / 2];
// Projection update
projection
.scale(s)
.translate(t);
//Append Topo polygons
svg.append("path")
.datum(Levels)
.attr("d", path)
svg.selectAll(".levels")
.data(topojson.feature(fra, fra.objects.levels).features)
.enter().append("path")
.attr("class", function(d) { return "Topo_" + d.properties.name; })
.attr("data-elev", function(d) { return d.properties.name; })
.attr("d", path)
}
</script>
<br />
<div>
<a class="download ac-icon-download" href="javascript:javascript: (function () { var e = document.createElement('script'); if (window.location.protocol === 'https:') { e.setAttribute('src', 'https://raw.github.com/NYTimes/svg-crowbar/gh-pages/svg-crowbar.js'); } else { e.setAttribute('src', 'http://nytimes.github.com/svg-crowbar/svg-crowbar.js'); } e.setAttribute('class', 'svg-crowbar'); document.body.appendChild(e); })();"><!--⤋--><big>⇩</big> Download</a> -- Works on Chrome. Feedback me for others web browsers ?
</div>
<br />
</body>
</html>
4. Result will be precisely such: (applied to your area of interest)
If you publish map(s) online please share up the link :)
Note: encouraging +1 welcome.
If anyone is looking for an update, here's the build code I got running as of today. Required me to manually download the .zip file and move it into the topo_map directory, and then a few changes (noted in bold):
# topojsoning (USE GEO2TOPO not TOPOJSON):
final.json: levels.json
geo2topo --id-property none --simplify=0.5 -p name=elev -o final.json -- levels.json
# simplification approach to explore further. Feedbacks welcome.
# shp2jsoning:
levels.json: levels.shp
ogr2ogr -f GeoJSON -where "elev < 10000" levels.json levels.shp
# merge
levels.shp: level0001.shp level0050.shp level0100.shp level0200.shp level0500.shp level1000.shp level2000.shp level3000.shp level4000.shp level5000.shp
ogr2ogr levels.shp level0001.shp
ogr2ogr -update -append levels.shp level0050.shp
ogr2ogr -update -append levels.shp level0100.shp
ogr2ogr -update -append levels.shp level0200.shp
ogr2ogr -update -append levels.shp level0500.shp
ogr2ogr -update -append levels.shp level1000.shp
ogr2ogr -update -append levels.shp level2000.shp
ogr2ogr -update -append levels.shp level3000.shp
ogr2ogr -update -append levels.shp level4000.shp
ogr2ogr -update -append levels.shp level5000.shp
# Polygonize slices:
level0001.shp: level0001.tif
gdal_polygonize.py level0001.tif -f "ESRI Shapefile" level0001.shp level_0001 elev
level0050.shp: level0050.tif
gdal_polygonize.py level0050.tif -f "ESRI Shapefile" level0050.shp level_0050 elev
level0100.shp: level0100.tif
gdal_polygonize.py level0100.tif -f "ESRI Shapefile" level0100.shp level_0100 elev
level0200.shp: level0200.tif
gdal_polygonize.py level0200.tif -f "ESRI Shapefile" level0200.shp level_0200 elev
level0500.shp: level0500.tif
gdal_polygonize.py level0500.tif -f "ESRI Shapefile" level0500.shp level_0500 elev
level1000.shp: level1000.tif
gdal_polygonize.py level1000.tif -f "ESRI Shapefile" level1000.shp level_1000 elev
level2000.shp: level2000.tif
gdal_polygonize.py level2000.tif -f "ESRI Shapefile" level2000.shp level_2000 elev
level3000.shp: level3000.tif
gdal_polygonize.py level3000.tif -f "ESRI Shapefile" level3000.shp level_3000 elev
level4000.shp: level4000.tif
gdal_polygonize.py level4000.tif -f "ESRI Shapefile" level4000.shp level_4000 elev
level5000.shp: level5000.tif
gdal_polygonize.py level5000.tif -f "ESRI Shapefile" level5000.shp level_5000 elev
# Raster slicing:
level0001.tif: crop.tif
gdal_calc.py -A crop.tif --outfile=level0001.tif --calc="1*(A>0)" --NoDataValue=0
level0050.tif: crop.tif
gdal_calc.py -A crop.tif --outfile=level0050.tif --calc="50*(A>50)" --NoDataValue=0
level0100.tif: crop.tif
gdal_calc.py -A crop.tif --outfile=level0100.tif --calc="100*(A>100)" --NoDataValue=0
level0200.tif: crop.tif
gdal_calc.py -A crop.tif --outfile=level0200.tif --calc="200*(A>200)" --NoDataValue=0
level0500.tif: crop.tif
gdal_calc.py -A crop.tif --outfile=level0500.tif --calc="500*(A>500)" --NoDataValue=0
level1000.tif: crop.tif
gdal_calc.py -A crop.tif --outfile=level1000.tif --calc="1000*(A>1000)" --NoDataValue=0
level2000.tif: crop.tif
gdal_calc.py -A crop.tif --outfile=level2000.tif --calc="2000*(A>2000)" --NoDataValue=0
level3000.tif: crop.tif
gdal_calc.py -A crop.tif --outfile=level3000.tif --calc="3000*(A>3000)" --NoDataValue=0
level4000.tif: crop.tif
gdal_calc.py -A crop.tif --outfile=level4000.tif --calc="4000*(A>4000)" --NoDataValue=0
level5000.tif: crop.tif
gdal_calc.py -A crop.tif --outfile=level5000.tif --calc="5000*(A>5000)" --NoDataValue=0
# boxing:
crop.tif: ETOPO1_Ice_g_geotiff.tif
gdal_translate -projwin -84.9 47.0 -69.9 33.7 ETOPO1_Ice_g_geotiff.tif crop.tif
# ulx uly lrx lry // W N E S <- Coordinate order
# unzip:
ETOPO1_Ice_g_geotiff.tif: ETOPO1.zip
unzip ETOPO1.zip
touch ETOPO1_Ice_g_geotiff.tif
# download:
#ETOPO1.zip:
# curl -o ETOPO1.zip 'http://www.ngdc.noaa.gov/mgg/global/relief/ETOPO1/data/ice_surface/grid_registered/georeferenced_tiff/ETOPO1_Ice_g_geotiff.zip'
clean:
rm `ls | grep -v 'zip' | grep -v 'Makefile'`
# Makefile v4b (#Lopez_lz)