I have wrote script for massive transfer in imapsync:
#!/bin/sh
echo
{ while IFS=';' read u1 p1 u2 p2; do
echo "==== Starting transfer for $u2 ===="
imapsync \
--host1 "src.host.com" --user1 "$u1" --password1 "$p1" \
--noauthmd5 \
--sep1 "." --prefix1 "" \
--host2 "desc.host.com" --user2 "$u2" --password2 "$p2" \
--sep2 "." --prefix2 "" \
--folder "INBOX" \
> ./logs/"$u1".log
echo "==== Ended transfer for $u2 ===="
done ; } < cred.txt
And that's how looks cred.txt
src#email.com;password!;dest#email.com;3kXxFQBm7u
(empty line)
Now, when I trying to run this script, I'm getting:
==== Starting transfer for dest#email.com ====
2 BAD CR sent without LF
...propagated at /usr/bin/imapsync line 759.
==== Ended transfer for dest#email.com ====
but if I'm enter second password staticaly (code will looks like below) - it's working, why?
#!/bin/sh
echo
{ while IFS=';' read u1 p1 u2 p2; do
echo "==== Starting transfer for $u2 ===="
imapsync \
--host1 "src.host.com" --user1 "$u1" --password1 "$p1" \
--noauthmd5 \
--sep1 "." --prefix1 "" \
--host2 "desc.host.com" --user2 "$u2" --password2 "3kXxFQBm7u" \
--sep2 "." --prefix2 "" \
--folder "INBOX" \
> ./logs/"$u1".log
echo "==== Ended transfer for $u2 ===="
done ; } < cred.txt
Since the problem is only with the password at the end of the line, your file probably has CRLF line breaks because it was created on a Windows system. Fix it with
dos2unix cred.txt
In case you are not having dos2unix in your system and you are not root too then you could use following commands to remove these carriage characters on same too:
tr -d '\r' < Input_file > temp_file && mv temp_file Input_file
OR
awk '{gsub(/\r/,"")} 1' Input_file > temp_file && mv temp_file Input_file
Add an extra ; at the end of each line in cred.txt like this:
src#email.com;password!;dest#email.com;3kXxFQBm7u;
Related
I have a file of parameters which I want to feed into a command in bash. Please note that I write the command below but the question isn't about the command, it's about how to pass the parameters from this file into the command.
The file, myfile.txt, looks like this:
3 rs523534 62297313 63097313
4 rs6365365 375800230 376600230
8 rs75466 63683994 64483994
I have a script to read each line and feed it into a command:
while read -r line; do
plink --bfile mydata \
--chr echo $line | awk '{print $1}' \
--from-bp echo $line | awk '{print $3}' \
--to-bp echo $line | awk '{print $4}' \
--r2 \
--ld-snp echo $line | awk '{print $2}' \
--ld-window-kb 100000000 \
--ld-window 100000000 \
--ld-window-r2 0 \
--out echo "processing/5_locuszoom/$line" | sed 's/ /_/g'
done < "myfile.txt"
This doesn't work:
awk: fatal: cannot open file `--to-bp' for reading (No such file or directory)
awk: fatal: cannot open file `--from-bp' for reading (No such file or directory)
awk: fatal: cannot open file `--r2' for reading (No such file or directory)
awk: fatal: cannot open file `--ld-window-kb' for reading (No such file or directory)
But if I run it manually, e.g.
plink --bfile mydata \
--chr 3 \
--from-bp 62297313 \
--to-bp 63097313 \
--r2 \
--ld-snp rs523534 \
--ld-window-kb 100000000 \
--ld-window 100000000 \
--ld-window-r2 0 \
--out 3_rs523534_62297313_63097313
It works fine.
Is there a way that I can better feed the variable information into the command so that it works?
If you want to run a command inline to another command, you need to use a command substitution. So instead of --chr echo $line | awk '{print $1}' \ for example, you'd write --chr $( echo $line | awk '{print $1}' ) \.
But in this case it's not necessary, since read can already split the data for you.
#!/bin/bash
while read -r arg1 arg2 arg3 arg4 ; do
<do things with your args here>
done <"myfile.txt"
read will split each line based on the contents of IFS and populate each name you give it with the corresponding token from the split line.
Probably what you want is, in pure bash
#!/bin/bash
while read -ra args; do
out="${args[*]}"
plink --bfile mydata \
--chr "${args[0]}" \
--from-bp "${args[2]}" \
--to-bp "${args[3]}" \
--r2 \
--ld-snp "${args[1]}" \
--ld-window-kb 100000000 \
--ld-window 100000000 \
--ld-window-r2 0 \
--out "processing/5_locuszoom/${out// /_}"
done < "myfile.txt"
Use of awk and sed for this task is superfluous.
I'm trying to add && \ on end of each line on a text file except the last line.
Sample input:
ps
mkdir repo
cd repo/
touch file1.txt
Expected output:
ps && \
mkdir repo && \
cd repo/ && \
touch file1.txt
First attempt
I tried this, but it outputs && \ on each line including the final line:
awk '{print $0"&& \\"}' RS="\r*\n\r*"
Second attempt
I tried using sed:
sed '1s/^//;$!s/$/"&&" \\/;$s/$//'
This seems to add extra newlines:
ps
&& \
mkdir repo
&& \
cd repo/
&& \
touch file1.txt
You could use sed for something that simple:
printf "line 1\nLine 2\nLine 3\n" | sed '$ ! s/$/ \&\& \\ /'
Output
line 1 && \
Line 2 && \
Line 3
In the code below I am attempting to assign variables to the two yad values Radius and Amount.
This can be done with awk by printing the yad values to file but I want to avoid this if I can.
The string (that is, both yad values) is assigned a variable and trimmed of characters, as required, using sed. However, the script stops at this line;
radius=$(sed 's|[amount*,]||g')
Two questions
is there a better way of tackling this; and
why is the script not completing? I have not been able to figure out the syntax.
EDIT: don't need the loop and working on the sed syntax
#!/bin/bash
#ifs.sh
values=`yad --form --center --width=300 --title="Test" --separator=' ' \
--button=Skip:1 \
--button=Apply:0 \
--field="Radius":NUM \
'0!0..30!1!0' \
--field="Amount":NUM \
'0!0..5!0.01!2'`
radius=$(echo "$values" | sed 's|[amount*,]||g')
amount=$(echo "$values" | sed 's/.a://')
if [ $? = 1 ]; then
echo " " >/dev/null 2>&1; else
echo "Radius = $radius"
echo "Amount = $amount"
fi
exit
Alternatives
# with separator
# radius="${values%????????}"
# amount="${values#????????}"
# without separator
# radius=$(echo "$values" | sed s'/........$//')
# amount=$(echo "$values" | sed 's/^........//')
It's easier than you think:
$ values=( $(echo '7.000000 0.100000 ') )
$ echo "${values[0]}"
7.000000
$ echo "${values[1]}"
0.100000
Replace $(echo '7.000000 0.100000 ') with yad ... so the script would be:
values=( $(yad --form --center --width=300 --title="Test" --separator=' ' \
--button=Skip:1 \
--button=Apply:0 \
--field="Radius":NUM \
'0!0..30!1!0' \
--field="Amount":NUM \
'0!0..5!0.01!2') )
if [ $? -eq 0 ]; then
echo "Radius = ${values[0]}"
echo "Amount = ${values[1]}"
fi
EDIT: Changed answer based on #Ed Morton
#!/bin/bash
#ifs.sh
values=($(yad --form --center --width=300 --title="Test" --separator=' ' \
--button=Skip:1 \
--button=Apply:0 \
--field="Radius":NUM \
'0!0..30!1!0' \
--field="Amount":NUM \
'0!0..5!0.01!2'))
if [ $? -eq 0 ]; then
radius="${values[0]}"
amount="${values[1]}"
fi
exit
bash -x Output
+ '[' 0 -eq 0 ']'
+ radius=7.000000
+ amount=1.000000
+ exit
In bash we can set an environment variable from a sequence of commands using read and a pipe to a subprocess. But I'm having trouble detecting errors in my processing in one edge case - a part of the subprocess pipeline producing some output before erroring.
A simplified example which takes an input file, looks for a line starting with "foo" and sets var to the first word on that line is:
set -e
set -o pipefail
set -o nounset
die() {
echo $1 > /dev/stderr
exit 1
}
read -r var rest < <( \
cat data.txt \
| grep foo \
|| die "PIPELINE" \
) || die "OUTER"
echo "var=$var"
Running this with data.txt like
blah
zap foo awesome
bang foo
will output
var=zap
Running this on a data.txt file that doesn't contain foo outputs (to stderr)
DEAD: PIPELINE
DEAD: OUTER
This is all as expected.
We can introduce another no-op stage like cat at the end of the process
...
read -r var rest < <( \
cat data.txt \
| grep foo \
| cat \
|| die "PIPELINE" \
) || die "OUTER"
...
and everything continues to work.
But if the additional stage is paste -s -d' ' and the input does not contain "foo" the output is
var=
DEAD: PIPELINE
Which seems to show that the pipeline errors, but read succeeds with an empty line. (It looks like paste -s -d' ' outputs a line of output even when its input is empty.)
Is there a simple way to detect this failure of the pipeline, and cause the main script to error out?
I guess I could check that the variable is not empty - but this is a simplified version - I'm actually using sed and paste to join multiple lines to set multiple variables, like
read -r v1 v2 v3 rest < <( \
cat data.txt \
| grep "^foo=" \
| sed -e 's/foo=//' \
| paste -s -d' ' \
|| die "PIPELINE"
) || die "OUTER"
You could use another grep to see if the output of paste contained something:
read -r var rest < <( \
cat data.txt \
| grep foo \
| paste -s -d' ' \
| grep . \
|| die "PIPELINE" \
) || die "OUTER"
In the end I went with two different solutions depending on the context.
The first was to pipe the results to a temporary file. This will process the entire file before performing the read, and thus any failures in the pipe will cause the script to fail.
cat data.txt \
| grep "^foo=" \
| sed -e 's/foo=//' \
| paste -s -d' ' \
> $TMP/result.txt
|| die "PIPELINE"
read -r var rest < $TMP/result.txt || die "OUTER"
The second was to just test that the variables were set. While this meant
there was a bunch of duplication that I wanted to avoid, it seemed the most bullet-proof solution.
read -r var rest < <( cat data.txt \
| grep "^foo=" \
| sed -e 's/foo=//' \
| paste -s -d' ' \
|| die "PIPELINE"
) || die "OUTER"
[ ! -z "$var" ] || die "VARIABLE NOT SET"
In the goal to create a file from a one line (bash) command, the goal is to output the contents of any text file - in this example a bash script - and wrap each line inside a command that is able to output that same line when pasted in a Terminal window.
Example source input file:
Line 1
Line 2
Line 3
Example desired output:
echo 'Line 1';echo 'Line 2';echo 'Line 3';
Note: whether printf, echo or another command is used to create the output, doesn't matter as long as the source is human readable.
One hurdle were the single quotes, that would not be recreated. Therefore use the form $'string', which are treated specially. The word expands to string, with backslash-escaped characters replaced as specified by the ANSI C standard.
Another requirement is to re-create tab characters from the old file in the new file. Therefore the wish is to replace <\tab> characters with \t.
Our tries to do this with sed or tr fail. How to replace tabs with their escape \t counterpart and still being able to output lines with original quotes?
Input file /Library/Scripts/BootRepairMount.sh contains:
$ cat /Library/Scripts/BootRepairMount.sh
#!/bin/bash
sleep 18
for OUTPUT in $(diskutil list | grep ': Apple_HFS' | awk '{ print $NF }')
do
if [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then
echo "$OUTPUT is not mounted, repair and mount"
diskutil repairVolume $OUTPUT
diskutil mount $OUTPUT
fi
done
The best shell one line command we could create is:
$ oldifs=$IFS;printf '\n';printf '{';while IFS= read -r p;do [[ "$p" == *"'"* ]] && echo -n "echo $'$p';" || echo -n "echo '$p';"; done < /Library/Scripts/BootRepairMount.sh | tr '\t' '\134\164';printf '}';printf '\n\n';IFS=$oldifs
Which returns this faulty output:
{echo '#!/bin/bash';echo 'sleep 18';echo $'for OUTPUT in $(diskutil list | grep ': Apple_HFS' | awk '{ print $NF }')';echo 'do';echo '\if [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then';echo '\\echo "$OUTPUT is not mounted, repair and mount"';echo '\\diskutil repairVolume $OUTPUT';echo '\\diskutil mount $OUTPUT';echo '\fi';echo 'done';}
Desired output is:
{echo '#!/bin/bash';echo 'sleep 18';echo $'for OUTPUT in $(diskutil list | grep ': Apple_HFS' | awk '{ print $NF }')';echo 'do';echo '\tif [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then';echo '\t\techo "$OUTPUT is not mounted, repair and mount"';echo '\t\tdiskutil repairVolume $OUTPUT';echo '\t\tdiskutil mount $OUTPUT';echo '\tfi';echo 'done';}
Bash one line command version 2
$ oldifs=$IFS;printf '\n';printf '{';while IFS= read -r p;do [[ "$p" == *"'"* ]] && printf 'printf $'\''%q'\'';' "$p" || printf 'printf '\''%q'\'';' "$p"; done < /Library/Scripts/BootRepairMount.sh;printf '}';printf '\n\n';IFS=$oldifs
returns output that is heavy escaped:
{printf '\#\!/bin/bash';printf 'sleep\ 18';printf $'for\ OUTPUT\ in\ \$\(diskutil\ list\ \|\ grep\ \':\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Apple_HFS\'\ \|\ awk\ \'\{\ print\ \$NF\ \}\'\)';printf 'do';printf '$'\tif [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then'';printf '$'\t\techo "$OUTPUT is not mounted, repair and mount"'';printf '$'\t\tdiskutil repairVolume $OUTPUT'';printf '$'\t\tdiskutil mount $OUTPUT'';printf '$'\tfi'';printf 'done';}
that never gets unescaped back to its original values in Mac OS X 10.7.5.
printf '\#\!/bin/bash';
outputs:
\#\!/bin/bash
As well as:
echo -e '\#\!/bin/bash'
does output the unescaped value
\#\!/bin/bash
-e is not a valid command switch for the Mac OS X 10.7.5 echo command, according to its man page.
bash's builtin command printf has %q format code that handles this:
printf '\n{ '; while IFS= read -r p; do printf "echo %q; " "$p"; done < /Library/Scripts/BootRepairMount.sh; printf '}\n\n'
Unfortunately, it doesn't always choose quoting/escaping modes that're easy to read. Specifically, it tends to prefer escaping individual metacharacters (e.g. spaces) rather than enclosing them in quotes:
{ echo \#\!/bin/bash; echo sleep\ 18; echo for\ OUTPUT\ in\ \$(diskutil\ list\ \|\ grep\ \':\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Apple_HFS\'\ \|\ awk\ \'{\ print\ \$NF\ }\'); echo do; echo $'\tif [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then'; echo $'\t\techo "$OUTPUT is not mounted, repair and mount"'; echo $'\t\tdiskutil repairVolume $OUTPUT'; echo $'\t\tdiskutil mount $OUTPUT'; echo $'\tfi'; echo done; }
If I understand right you want paste one long line to the Terminal.app and want get the "source code" of original script. So, need a script what will generate the one-line script.
Maybe a bit unusual solution, but it is easy and simple.
here is the test script called test.sh (instead of your BootReapirMount.sh)
for i in {1..10}
do
date
done
Here is the generator script mkecho.sh
#!/bin/bash
[[ ! -f "$1" ]] && echo "Need filename" && exit 1
asc=$(gzip < "$1" | base64)
echo "base64 -D <<<'$asc'| gzip -d"
Now, run:
bash mkecho.sh test.sh
you will get the next:
base64 -D <<<'H4sIAASwqFEAA0vLL1LIVMjMU6g21NMzNKjlSsnn4kxJLEkFMvJSuQBZFmY0HwAAAA=='| gzip -d
If you copy and paste the above into the terminal, it will will display the original test.sh
Variant2
If you want directly execute the script, you should modify the mkecho.sh to the next mkeval.sh
#!/bin/bash
[[ ! -f "$1" ]] && echo "Need filename" && exit 1
asc=$(gzip < "$1" | base64)
echo -n 'eval "$(base64 -D <<<"'
echo -n $asc
echo -n '" | gzip -d)"'
echo
When run
bash mkeval.sh test.sh
will get
eval "$(base64 -D <<<"H4sIAASwqFEAA0vLL1LIVMjMU6g21NMzNKjlSsnn4kxJLEkFMvJSuQBZFmY0HwAAAA==" | gzip -d)"
and finally when you copy and paste it into the terminal, you run the test.sh and will get:
Fri May 31 16:25:08 CEST 2013
... 8 lined deleted...
Fri May 31 16:25:08 CEST 2013
Warning: because the script is NOT TESTED for every possible conditions, nor for redirects and so on - I really don't recommending using the eval verision.
sed 's/\\t/\\/g'
$ echo 'ffsd \tif [[ -z $' | sed 's/\\t/\\/g'
ffsd \if [[ -z $