How to pass parameters to a command in bash when run line by line? - bash

I have a file of parameters which I want to feed into a command in bash. Please note that I write the command below but the question isn't about the command, it's about how to pass the parameters from this file into the command.
The file, myfile.txt, looks like this:
3 rs523534 62297313 63097313
4 rs6365365 375800230 376600230
8 rs75466 63683994 64483994
I have a script to read each line and feed it into a command:
while read -r line; do
plink --bfile mydata \
--chr echo $line | awk '{print $1}' \
--from-bp echo $line | awk '{print $3}' \
--to-bp echo $line | awk '{print $4}' \
--r2 \
--ld-snp echo $line | awk '{print $2}' \
--ld-window-kb 100000000 \
--ld-window 100000000 \
--ld-window-r2 0 \
--out echo "processing/5_locuszoom/$line" | sed 's/ /_/g'
done < "myfile.txt"
This doesn't work:
awk: fatal: cannot open file `--to-bp' for reading (No such file or directory)
awk: fatal: cannot open file `--from-bp' for reading (No such file or directory)
awk: fatal: cannot open file `--r2' for reading (No such file or directory)
awk: fatal: cannot open file `--ld-window-kb' for reading (No such file or directory)
But if I run it manually, e.g.
plink --bfile mydata \
--chr 3 \
--from-bp 62297313 \
--to-bp 63097313 \
--r2 \
--ld-snp rs523534 \
--ld-window-kb 100000000 \
--ld-window 100000000 \
--ld-window-r2 0 \
--out 3_rs523534_62297313_63097313
It works fine.
Is there a way that I can better feed the variable information into the command so that it works?

If you want to run a command inline to another command, you need to use a command substitution. So instead of --chr echo $line | awk '{print $1}' \ for example, you'd write --chr $( echo $line | awk '{print $1}' ) \.
But in this case it's not necessary, since read can already split the data for you.
#!/bin/bash
while read -r arg1 arg2 arg3 arg4 ; do
<do things with your args here>
done <"myfile.txt"
read will split each line based on the contents of IFS and populate each name you give it with the corresponding token from the split line.

Probably what you want is, in pure bash
#!/bin/bash
while read -ra args; do
out="${args[*]}"
plink --bfile mydata \
--chr "${args[0]}" \
--from-bp "${args[2]}" \
--to-bp "${args[3]}" \
--r2 \
--ld-snp "${args[1]}" \
--ld-window-kb 100000000 \
--ld-window 100000000 \
--ld-window-r2 0 \
--out "processing/5_locuszoom/${out// /_}"
done < "myfile.txt"
Use of awk and sed for this task is superfluous.

Related

How to `rm` files as awk action?

This one-liner:
sudo df /tmp \
| grep '/tmp' \
| expand - \
| cut -d " " -f 12 \
| sed 's/%//' \
| awk '{ if ($1<50)
$("sudo rm -rf /path/to/trash/files/*")
}'
seems to have no effect, while this:
sudo df /tmp \
| grep '/tmp' \
| expand - \
| cut -d " " -f 12 \
| sed 's/%//' \
| awk '{ if ($1<50)
print $1
}'
prints the percentage of disk used for tmp.
(The end goal is to flip the comparison around to ($1>50), but for testing I'm trying <.)
You don't really need so many commands in pipeline with awk.
You can just use:
df /tmp | awk 'NR>1 && $5+0 > 50 {system ("date")}'
Here change date command to something else that you need to run there.

adding && \ to each line on a text file except last line

I'm trying to add && \ on end of each line on a text file except the last line.
Sample input:
ps
mkdir repo
cd repo/
touch file1.txt
Expected output:
ps && \
mkdir repo && \
cd repo/ && \
touch file1.txt
First attempt
I tried this, but it outputs && \ on each line including the final line:
awk '{print $0"&& \\"}' RS="\r*\n\r*"
Second attempt
I tried using sed:
sed '1s/^//;$!s/$/"&&" \\/;$s/$//'
This seems to add extra newlines:
ps
&& \
mkdir repo
&& \
cd repo/
&& \
touch file1.txt
You could use sed for something that simple:
printf "line 1\nLine 2\nLine 3\n" | sed '$ ! s/$/ \&\& \\ /'
Output
line 1 && \
Line 2 && \
Line 3

Need help escaping from awk quotations in bash script

I have an alias in my bashrc file that outputs current folder contents and system available storage, updated continuously by the watch function.
alias wtch='watch -n 0 -t "du -sch * -B 1000000 2>/dev/null | sort -h && df -h -B 1000000| head -2 | awk '{print \$4}'"'
The string worked fine until I put in the awk part. I know I need to escape the single quotation marks, while still staying in the double quotation marks and the $4 but I haven't been able to get it to work. What am I doing wrong?
This is the error I get
-bash: alias: $4}": not found
Since the quoting for the alias is making it tough, you could just make it a function instead:
wtch() {
watch -n 0 -t "du -sch * -B 1000000 2>/dev/null | sort -h && df -h -B 1000000| head -2 | awk '{print $4}'"
}
This is a lot like issue 2 in the BashFAQ/050
Also, a minor thing but you can skip the head process at the end and just have awk do it, even exiting after the second row like
wtch() {
watch -n 0 -t "du -sch * -B 1000000 2>/dev/null | sort -h && df -h -B 1000000| awk '{print $4} NR >= 3 {exit}'"
}
In this case you can use cut instead of awk. And you'll have the same effect.
alias wtch="watch -n 0 -t 'du -sch * -B 1000000 2>/dev/null | sort -h && df -h -B 1000000| head -2 | cut -d\ -f4'"
Explaining cut:
-d option defines a delimiter
-d\ means that my delimiter is space
-f selects a column
-f4 gives you the fourth column

bash command substitution force to foreground

I have this:
echo -e "\n\n"
find /home/*/var/*/logs/ \
-name transfer.log \
-exec awk -v SUM=0 '$0 {SUM+=1} END {print "{} " SUM}' {} \; \
> >( sed '/\b0\b/d' \
| awk ' BEGIN {printf "\t\t\tTRANSFER LOG\t\t\t\t\t#OF HITS\n"}
{printf "%-72s %-s\n", $1, $2}
' \
| (read -r; printf "%s\n" "$REPLY"; sort -nr -k2)
)
echo -e "\n\n"
When run on a machine with bash 4.1.2 always returns correctly except I get all 4 of my new lines at the top.
When run on a machine with bash 3.00.15 it gives all 4 of my new lines at the top, returns the prompt in the middle of the output, and never completes just hangs.
I would really like to fix this for both versions as we have a lot of machines running both.
Why make life so difficult and unintelligible? Why not simplify?
TXFRLOG=$(find /home..... transfer.log)
awk .... ${TXFRLOG}
The answer I found was to use a while read
echo -e "\n\n"; \
printf "\t\t\tTRANSFER LOG\t\t\t\t\t#OF HITS\n"; \
while read -r line; \
do echo "$line" |sed '/\b0\b/d' | awk '{printf "%-72s %-s\n", $1, $2}'; \
done < <(find /home/*/var/*/logs/ -name transfer.log -exec awk -v SUM=0 '$0 {SUM+=1} END{print "{} " SUM}' {} \;;) \
|sort -nr -k2; \
echo -e "\n\n"

How to replace or escape <tab> characters with \t in bash script and being able to output single quotes?

In the goal to create a file from a one line (bash) command, the goal is to output the contents of any text file - in this example a bash script - and wrap each line inside a command that is able to output that same line when pasted in a Terminal window.
Example source input file:
Line 1
Line 2
Line 3
Example desired output:
echo 'Line 1';echo 'Line 2';echo 'Line 3';
Note: whether printf, echo or another command is used to create the output, doesn't matter as long as the source is human readable.
One hurdle were the single quotes, that would not be recreated. Therefore use the form $'string', which are treated specially. The word expands to string, with backslash-escaped characters replaced as specified by the ANSI C standard.
Another requirement is to re-create tab characters from the old file in the new file. Therefore the wish is to replace <\tab> characters with \t.
Our tries to do this with sed or tr fail. How to replace tabs with their escape \t counterpart and still being able to output lines with original quotes?
Input file /Library/Scripts/BootRepairMount.sh contains:
$ cat /Library/Scripts/BootRepairMount.sh
#!/bin/bash
sleep 18
for OUTPUT in $(diskutil list | grep ': Apple_HFS' | awk '{ print $NF }')
do
if [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then
echo "$OUTPUT is not mounted, repair and mount"
diskutil repairVolume $OUTPUT
diskutil mount $OUTPUT
fi
done
The best shell one line command we could create is:
$ oldifs=$IFS;printf '\n';printf '{';while IFS= read -r p;do [[ "$p" == *"'"* ]] && echo -n "echo $'$p';" || echo -n "echo '$p';"; done < /Library/Scripts/BootRepairMount.sh | tr '\t' '\134\164';printf '}';printf '\n\n';IFS=$oldifs
Which returns this faulty output:
{echo '#!/bin/bash';echo 'sleep 18';echo $'for OUTPUT in $(diskutil list | grep ': Apple_HFS' | awk '{ print $NF }')';echo 'do';echo '\if [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then';echo '\\echo "$OUTPUT is not mounted, repair and mount"';echo '\\diskutil repairVolume $OUTPUT';echo '\\diskutil mount $OUTPUT';echo '\fi';echo 'done';}
Desired output is:
{echo '#!/bin/bash';echo 'sleep 18';echo $'for OUTPUT in $(diskutil list | grep ': Apple_HFS' | awk '{ print $NF }')';echo 'do';echo '\tif [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then';echo '\t\techo "$OUTPUT is not mounted, repair and mount"';echo '\t\tdiskutil repairVolume $OUTPUT';echo '\t\tdiskutil mount $OUTPUT';echo '\tfi';echo 'done';}
Bash one line command version 2
$ oldifs=$IFS;printf '\n';printf '{';while IFS= read -r p;do [[ "$p" == *"'"* ]] && printf 'printf $'\''%q'\'';' "$p" || printf 'printf '\''%q'\'';' "$p"; done < /Library/Scripts/BootRepairMount.sh;printf '}';printf '\n\n';IFS=$oldifs
returns output that is heavy escaped:
{printf '\#\!/bin/bash';printf 'sleep\ 18';printf $'for\ OUTPUT\ in\ \$\(diskutil\ list\ \|\ grep\ \':\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Apple_HFS\'\ \|\ awk\ \'\{\ print\ \$NF\ \}\'\)';printf 'do';printf '$'\tif [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then'';printf '$'\t\techo "$OUTPUT is not mounted, repair and mount"'';printf '$'\t\tdiskutil repairVolume $OUTPUT'';printf '$'\t\tdiskutil mount $OUTPUT'';printf '$'\tfi'';printf 'done';}
that never gets unescaped back to its original values in Mac OS X 10.7.5.
printf '\#\!/bin/bash';
outputs:
\#\!/bin/bash
As well as:
echo -e '\#\!/bin/bash'
does output the unescaped value
\#\!/bin/bash
-e is not a valid command switch for the Mac OS X 10.7.5 echo command, according to its man page.
bash's builtin command printf has %q format code that handles this:
printf '\n{ '; while IFS= read -r p; do printf "echo %q; " "$p"; done < /Library/Scripts/BootRepairMount.sh; printf '}\n\n'
Unfortunately, it doesn't always choose quoting/escaping modes that're easy to read. Specifically, it tends to prefer escaping individual metacharacters (e.g. spaces) rather than enclosing them in quotes:
{ echo \#\!/bin/bash; echo sleep\ 18; echo for\ OUTPUT\ in\ \$(diskutil\ list\ \|\ grep\ \':\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Apple_HFS\'\ \|\ awk\ \'{\ print\ \$NF\ }\'); echo do; echo $'\tif [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then'; echo $'\t\techo "$OUTPUT is not mounted, repair and mount"'; echo $'\t\tdiskutil repairVolume $OUTPUT'; echo $'\t\tdiskutil mount $OUTPUT'; echo $'\tfi'; echo done; }
If I understand right you want paste one long line to the Terminal.app and want get the "source code" of original script. So, need a script what will generate the one-line script.
Maybe a bit unusual solution, but it is easy and simple.
here is the test script called test.sh (instead of your BootReapirMount.sh)
for i in {1..10}
do
date
done
Here is the generator script mkecho.sh
#!/bin/bash
[[ ! -f "$1" ]] && echo "Need filename" && exit 1
asc=$(gzip < "$1" | base64)
echo "base64 -D <<<'$asc'| gzip -d"
Now, run:
bash mkecho.sh test.sh
you will get the next:
base64 -D <<<'H4sIAASwqFEAA0vLL1LIVMjMU6g21NMzNKjlSsnn4kxJLEkFMvJSuQBZFmY0HwAAAA=='| gzip -d
If you copy and paste the above into the terminal, it will will display the original test.sh
Variant2
If you want directly execute the script, you should modify the mkecho.sh to the next mkeval.sh
#!/bin/bash
[[ ! -f "$1" ]] && echo "Need filename" && exit 1
asc=$(gzip < "$1" | base64)
echo -n 'eval "$(base64 -D <<<"'
echo -n $asc
echo -n '" | gzip -d)"'
echo
When run
bash mkeval.sh test.sh
will get
eval "$(base64 -D <<<"H4sIAASwqFEAA0vLL1LIVMjMU6g21NMzNKjlSsnn4kxJLEkFMvJSuQBZFmY0HwAAAA==" | gzip -d)"
and finally when you copy and paste it into the terminal, you run the test.sh and will get:
Fri May 31 16:25:08 CEST 2013
... 8 lined deleted...
Fri May 31 16:25:08 CEST 2013
Warning: because the script is NOT TESTED for every possible conditions, nor for redirects and so on - I really don't recommending using the eval verision.
sed 's/\\t/\\/g'
$ echo 'ffsd \tif [[ -z $' | sed 's/\\t/\\/g'
ffsd \if [[ -z $

Resources