I'm trying to add && \ on end of each line on a text file except the last line.
Sample input:
ps
mkdir repo
cd repo/
touch file1.txt
Expected output:
ps && \
mkdir repo && \
cd repo/ && \
touch file1.txt
First attempt
I tried this, but it outputs && \ on each line including the final line:
awk '{print $0"&& \\"}' RS="\r*\n\r*"
Second attempt
I tried using sed:
sed '1s/^//;$!s/$/"&&" \\/;$s/$//'
This seems to add extra newlines:
ps
&& \
mkdir repo
&& \
cd repo/
&& \
touch file1.txt
You could use sed for something that simple:
printf "line 1\nLine 2\nLine 3\n" | sed '$ ! s/$/ \&\& \\ /'
Output
line 1 && \
Line 2 && \
Line 3
Related
I have a file of parameters which I want to feed into a command in bash. Please note that I write the command below but the question isn't about the command, it's about how to pass the parameters from this file into the command.
The file, myfile.txt, looks like this:
3 rs523534 62297313 63097313
4 rs6365365 375800230 376600230
8 rs75466 63683994 64483994
I have a script to read each line and feed it into a command:
while read -r line; do
plink --bfile mydata \
--chr echo $line | awk '{print $1}' \
--from-bp echo $line | awk '{print $3}' \
--to-bp echo $line | awk '{print $4}' \
--r2 \
--ld-snp echo $line | awk '{print $2}' \
--ld-window-kb 100000000 \
--ld-window 100000000 \
--ld-window-r2 0 \
--out echo "processing/5_locuszoom/$line" | sed 's/ /_/g'
done < "myfile.txt"
This doesn't work:
awk: fatal: cannot open file `--to-bp' for reading (No such file or directory)
awk: fatal: cannot open file `--from-bp' for reading (No such file or directory)
awk: fatal: cannot open file `--r2' for reading (No such file or directory)
awk: fatal: cannot open file `--ld-window-kb' for reading (No such file or directory)
But if I run it manually, e.g.
plink --bfile mydata \
--chr 3 \
--from-bp 62297313 \
--to-bp 63097313 \
--r2 \
--ld-snp rs523534 \
--ld-window-kb 100000000 \
--ld-window 100000000 \
--ld-window-r2 0 \
--out 3_rs523534_62297313_63097313
It works fine.
Is there a way that I can better feed the variable information into the command so that it works?
If you want to run a command inline to another command, you need to use a command substitution. So instead of --chr echo $line | awk '{print $1}' \ for example, you'd write --chr $( echo $line | awk '{print $1}' ) \.
But in this case it's not necessary, since read can already split the data for you.
#!/bin/bash
while read -r arg1 arg2 arg3 arg4 ; do
<do things with your args here>
done <"myfile.txt"
read will split each line based on the contents of IFS and populate each name you give it with the corresponding token from the split line.
Probably what you want is, in pure bash
#!/bin/bash
while read -ra args; do
out="${args[*]}"
plink --bfile mydata \
--chr "${args[0]}" \
--from-bp "${args[2]}" \
--to-bp "${args[3]}" \
--r2 \
--ld-snp "${args[1]}" \
--ld-window-kb 100000000 \
--ld-window 100000000 \
--ld-window-r2 0 \
--out "processing/5_locuszoom/${out// /_}"
done < "myfile.txt"
Use of awk and sed for this task is superfluous.
I got this script which reads a delimited part of my .gitignore file and remove all files after the given mark # #:
# https://stackoverflow.com/questions/55527923/how-to-stop-makefile-from-expanding-my-shell-output
RAW_GITIGNORE_CONTENTS := $(shell while read -r line; do printf "$$line "; done < ".gitignore")
GITIGNORE_CONTENTS := $(shell echo "$(RAW_GITIGNORE_CONTENTS)" | sed -E $$'s/[^\#]+\# //g')
# https://stackoverflow.com/questions/4210042/exclude-directory-from-find-command
DIRECTORIES_TO_CLEAN := $(shell /bin/find -not -path "./**.git**" -not -path "./pictures**" -type d)
clean:
# https://stackoverflow.com/questions/10586153/split-string-into-an-array-in-bash
# https://stackoverflow.com/questions/11289551/argument-list-too-long-error-for-rm-cp-mv-commands
readarray -td' ' GARBAGE_DIRECTORIES <<<"$(DIRECTORIES_TO_CLEAN) "; \
unset 'GARBAGE_DIRECTORIES[-1]'; \
declare -p GARBAGE_DIRECTORIES; \
readarray -td' ' GARBAGE_EXTENSIONS <<<"$(GITIGNORE_CONTENTS) "; \
unset 'GARBAGE_EXTENSIONS[-1]'; \
declare -p GARBAGE_EXTENSIONS; \
for filename in "$${GARBAGE_DIRECTORIES[#]}"; \
do \
arraylength="$${#GARBAGE_EXTENSIONS[#]}"; \
printf 'Cleaning %s extensions on %s\n' "$${arraylength}" "$$filename"; \
for extension in "$${GARBAGE_EXTENSIONS[#]}"; \
do \
[[ ! -z "$$filename" ]] || continue; \
[[ ! -z "$$extension" ]] || continue; \
full_expression="$${filename}/$${extension}" ;\
printf '%s\n' "$$full_expression"; \
rm -v "$$full_expression"; \
done; \
done;
Running it with the following .gitignore file:
*.txt
*.var
# Comment #
*.aux
The rm command is not expanding the wildcards and keeps telling me rm: cannot remove './*.aux': No such file or directory and do not remove the *.aux files from the ./ directory.
Update
After asked on a comment by #Beta, I simplified the Makefile to this:
GITIGNORE_CONTENTS := "*.aux" "*.lof"
DIRECTORIES_TO_CLEAN := "./setup/cache" "./setup/cache/chapters"
clean:
readarray -td' ' GARBAGE_DIRECTORIES <<<"$(DIRECTORIES_TO_CLEAN) "; \
unset 'GARBAGE_DIRECTORIES[-1]'; \
declare -p GARBAGE_DIRECTORIES; \
readarray -td' ' GARBAGE_EXTENSIONS <<<"$(GITIGNORE_CONTENTS) "; \
unset 'GARBAGE_EXTENSIONS[-1]'; \
declare -p GARBAGE_EXTENSIONS; \
for filename in "$${GARBAGE_DIRECTORIES[#]}"; \
do \
arraylength="$${#GARBAGE_EXTENSIONS[#]}"; \
printf 'Cleaning %s extensions on %s\n' "$${arraylength}" "$$filename"; \
for extension in "$${GARBAGE_EXTENSIONS[#]}"; \
do \
[[ ! -z "$$filename" ]] || continue; \
[[ ! -z "$$extension" ]] || continue; \
full_expression="$${filename}/$${extension}" ;\
printf '%s\n' "$$full_expression"; \
rm -vf "$$full_expression"; \
done; \
done;
Which results on this output after running it:
$ make
readarray -td' ' GARBAGE_DIRECTORIES <<<""./setup/cache" "./setup/cache/chapters" "; \
unset 'GARBAGE_DIRECTORIES[-1]'; \
declare -p GARBAGE_DIRECTORIES; \
readarray -td' ' GARBAGE_EXTENSIONS <<<""*.aux" "*.lof" "; \
unset 'GARBAGE_EXTENSIONS[-1]'; \
declare -p GARBAGE_EXTENSIONS; \
for filename in "${GARBAGE_DIRECTORIES[#]}"; \
do \
arraylength="${#GARBAGE_EXTENSIONS[#]}"; \
printf 'Cleaning %s extensions on %s\n' "${arraylength}" "$filename"; \
for extension in "${GARBAGE_EXTENSIONS[#]}"; \
do \
[[ ! -z "$filename" ]] || continue; \
[[ ! -z "$extension" ]] || continue; \
full_expression="${filename}/${extension}" ;\
printf '%s\n' "$full_expression"; \
rm -vf "$full_expression"; \
done; \
done;
declare -a GARBAGE_DIRECTORIES=([0]="./setup/cache" [1]="./setup/cache/chapters")
declare -a GARBAGE_EXTENSIONS=([0]="*.aux" [1]="*.lof")
Cleaning 2 extensions on ./setup/cache
./setup/cache/*.aux
./setup/cache/*.lof
Cleaning 2 extensions on ./setup/cache/chapters
./setup/cache/chapters/*.aux
./setup/cache/chapters/*.lof
More simplification
I reduced to the more simple version it could be:
clean:
rm -v "./setup/cache/*.aux";
Running this, also do not remove the files:
$ make
rm -v "./setup/cache/*.aux";
rm: cannot remove './setup/cache/*.aux': No such file or directory
make: *** [Makefile:3: clean] Error 1
$ ls ./setup/cache/*.aux
./setup/cache/main.aux
On above, after running ls, you can see the file still exists and it is there.
I managed to fix it by changing:
rm -vf "$$full_expression"; \
To:
rm -vf $${full_expression}; \
I have wrote script for massive transfer in imapsync:
#!/bin/sh
echo
{ while IFS=';' read u1 p1 u2 p2; do
echo "==== Starting transfer for $u2 ===="
imapsync \
--host1 "src.host.com" --user1 "$u1" --password1 "$p1" \
--noauthmd5 \
--sep1 "." --prefix1 "" \
--host2 "desc.host.com" --user2 "$u2" --password2 "$p2" \
--sep2 "." --prefix2 "" \
--folder "INBOX" \
> ./logs/"$u1".log
echo "==== Ended transfer for $u2 ===="
done ; } < cred.txt
And that's how looks cred.txt
src#email.com;password!;dest#email.com;3kXxFQBm7u
(empty line)
Now, when I trying to run this script, I'm getting:
==== Starting transfer for dest#email.com ====
2 BAD CR sent without LF
...propagated at /usr/bin/imapsync line 759.
==== Ended transfer for dest#email.com ====
but if I'm enter second password staticaly (code will looks like below) - it's working, why?
#!/bin/sh
echo
{ while IFS=';' read u1 p1 u2 p2; do
echo "==== Starting transfer for $u2 ===="
imapsync \
--host1 "src.host.com" --user1 "$u1" --password1 "$p1" \
--noauthmd5 \
--sep1 "." --prefix1 "" \
--host2 "desc.host.com" --user2 "$u2" --password2 "3kXxFQBm7u" \
--sep2 "." --prefix2 "" \
--folder "INBOX" \
> ./logs/"$u1".log
echo "==== Ended transfer for $u2 ===="
done ; } < cred.txt
Since the problem is only with the password at the end of the line, your file probably has CRLF line breaks because it was created on a Windows system. Fix it with
dos2unix cred.txt
In case you are not having dos2unix in your system and you are not root too then you could use following commands to remove these carriage characters on same too:
tr -d '\r' < Input_file > temp_file && mv temp_file Input_file
OR
awk '{gsub(/\r/,"")} 1' Input_file > temp_file && mv temp_file Input_file
Add an extra ; at the end of each line in cred.txt like this:
src#email.com;password!;dest#email.com;3kXxFQBm7u;
I'm having difficulties with makefiles.
So in a recipe, I'm making a file (with a name and a .ujc extension) in a for loop and would like to have a text file at the end which contains all the created files. Purpose is to feed it to an application.
For example, in a semi high-level example,
List= [Class1,Class2,Class3]
foreach(Class C in List) {
#do operations on C > outputs a ClassX.ujc file
# add name of file to a text file named "list_of_files"
}
At the end I should have a text file, list_of_files.txt, which contains the following string:
Class1.ujc Class2.ujc Class3.ujc
As a reference, the code I have at the moment (and which does a bit of the stuff above but does not work is) is:
pc: $(APP)
$(foreach C, $(shell echo $(CLASS) | tr ',' ' '), \
make -C BUILDENV CLASS=$(C) BUILD=just_filelist OUTPUT=filelist.txt SKIPSELF=yes && \
../classCvt/classCvt <./Applications/$(C).class> ./Applications/$(C).ujc && \
cat app_file_list.txt | xargs echo ./Applications/$(C).ujc >app_file_list.txt && \
) true
time -p ./$(APP) `cat app_file_list.txt` `cat filelist.txt`
The internal make does make a filelist which is fed to the app, but I'd also like to feed the app_file_list but its construction goes totally wrong.
Probably simple, but I'm not getting there.
Edit:
The code below does what I want:
pc: $(APP)
rm -f cat app_file_list.txt
$(foreach C, $(shell echo $(CLASS) | tr ',' ' '), \
make -C BUILDENV CLASS=$(C) BUILD=just_filelist OUTPUT=filelist.txt SKIPSELF=yes && \
../classCvt/classCvt <./Applications/$(C).class> ./Applications/$(C).ujc && \
cat app_file_list.txt | echo ./Applications/$(C).ujc >>app_file_list.txt && \
) true
time -p ./$(APP) `cat app_file_list.txt` `cat filelist.txt`
Notable mistake I made was the xargs.
(Also in the post)
The solution turned out to be not-so-difficult. I needed to remove the xargs command and do the correct operation (i.e., >> instead of >) in the 'cat app_file_list.txt | etc...' line.
The code below does what I want:
pc: $(APP)
rm -f cat app_file_list.txt
$(foreach C, $(shell echo $(CLASS) | tr ',' ' '), \
make -C BUILDENV CLASS=$(C) BUILD=just_filelist OUTPUT=filelist.txt SKIPSELF=yes && \
../classCvt/classCvt <./Applications/$(C).class> ./Applications/$(C).ujc && \
cat app_file_list.txt | echo ./Applications/$(C).ujc >>app_file_list.txt && \
) true
time -p ./$(APP) `cat app_file_list.txt` `cat filelist.txt`
Notable mistake I made was the xargs which caused strings to repeat into the .txt file.
In the goal to create a file from a one line (bash) command, the goal is to output the contents of any text file - in this example a bash script - and wrap each line inside a command that is able to output that same line when pasted in a Terminal window.
Example source input file:
Line 1
Line 2
Line 3
Example desired output:
echo 'Line 1';echo 'Line 2';echo 'Line 3';
Note: whether printf, echo or another command is used to create the output, doesn't matter as long as the source is human readable.
One hurdle were the single quotes, that would not be recreated. Therefore use the form $'string', which are treated specially. The word expands to string, with backslash-escaped characters replaced as specified by the ANSI C standard.
Another requirement is to re-create tab characters from the old file in the new file. Therefore the wish is to replace <\tab> characters with \t.
Our tries to do this with sed or tr fail. How to replace tabs with their escape \t counterpart and still being able to output lines with original quotes?
Input file /Library/Scripts/BootRepairMount.sh contains:
$ cat /Library/Scripts/BootRepairMount.sh
#!/bin/bash
sleep 18
for OUTPUT in $(diskutil list | grep ': Apple_HFS' | awk '{ print $NF }')
do
if [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then
echo "$OUTPUT is not mounted, repair and mount"
diskutil repairVolume $OUTPUT
diskutil mount $OUTPUT
fi
done
The best shell one line command we could create is:
$ oldifs=$IFS;printf '\n';printf '{';while IFS= read -r p;do [[ "$p" == *"'"* ]] && echo -n "echo $'$p';" || echo -n "echo '$p';"; done < /Library/Scripts/BootRepairMount.sh | tr '\t' '\134\164';printf '}';printf '\n\n';IFS=$oldifs
Which returns this faulty output:
{echo '#!/bin/bash';echo 'sleep 18';echo $'for OUTPUT in $(diskutil list | grep ': Apple_HFS' | awk '{ print $NF }')';echo 'do';echo '\if [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then';echo '\\echo "$OUTPUT is not mounted, repair and mount"';echo '\\diskutil repairVolume $OUTPUT';echo '\\diskutil mount $OUTPUT';echo '\fi';echo 'done';}
Desired output is:
{echo '#!/bin/bash';echo 'sleep 18';echo $'for OUTPUT in $(diskutil list | grep ': Apple_HFS' | awk '{ print $NF }')';echo 'do';echo '\tif [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then';echo '\t\techo "$OUTPUT is not mounted, repair and mount"';echo '\t\tdiskutil repairVolume $OUTPUT';echo '\t\tdiskutil mount $OUTPUT';echo '\tfi';echo 'done';}
Bash one line command version 2
$ oldifs=$IFS;printf '\n';printf '{';while IFS= read -r p;do [[ "$p" == *"'"* ]] && printf 'printf $'\''%q'\'';' "$p" || printf 'printf '\''%q'\'';' "$p"; done < /Library/Scripts/BootRepairMount.sh;printf '}';printf '\n\n';IFS=$oldifs
returns output that is heavy escaped:
{printf '\#\!/bin/bash';printf 'sleep\ 18';printf $'for\ OUTPUT\ in\ \$\(diskutil\ list\ \|\ grep\ \':\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Apple_HFS\'\ \|\ awk\ \'\{\ print\ \$NF\ \}\'\)';printf 'do';printf '$'\tif [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then'';printf '$'\t\techo "$OUTPUT is not mounted, repair and mount"'';printf '$'\t\tdiskutil repairVolume $OUTPUT'';printf '$'\t\tdiskutil mount $OUTPUT'';printf '$'\tfi'';printf 'done';}
that never gets unescaped back to its original values in Mac OS X 10.7.5.
printf '\#\!/bin/bash';
outputs:
\#\!/bin/bash
As well as:
echo -e '\#\!/bin/bash'
does output the unescaped value
\#\!/bin/bash
-e is not a valid command switch for the Mac OS X 10.7.5 echo command, according to its man page.
bash's builtin command printf has %q format code that handles this:
printf '\n{ '; while IFS= read -r p; do printf "echo %q; " "$p"; done < /Library/Scripts/BootRepairMount.sh; printf '}\n\n'
Unfortunately, it doesn't always choose quoting/escaping modes that're easy to read. Specifically, it tends to prefer escaping individual metacharacters (e.g. spaces) rather than enclosing them in quotes:
{ echo \#\!/bin/bash; echo sleep\ 18; echo for\ OUTPUT\ in\ \$(diskutil\ list\ \|\ grep\ \':\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Apple_HFS\'\ \|\ awk\ \'{\ print\ \$NF\ }\'); echo do; echo $'\tif [[ -z $(df -lnh | grep /dev/$OUTPUT) ]]; then'; echo $'\t\techo "$OUTPUT is not mounted, repair and mount"'; echo $'\t\tdiskutil repairVolume $OUTPUT'; echo $'\t\tdiskutil mount $OUTPUT'; echo $'\tfi'; echo done; }
If I understand right you want paste one long line to the Terminal.app and want get the "source code" of original script. So, need a script what will generate the one-line script.
Maybe a bit unusual solution, but it is easy and simple.
here is the test script called test.sh (instead of your BootReapirMount.sh)
for i in {1..10}
do
date
done
Here is the generator script mkecho.sh
#!/bin/bash
[[ ! -f "$1" ]] && echo "Need filename" && exit 1
asc=$(gzip < "$1" | base64)
echo "base64 -D <<<'$asc'| gzip -d"
Now, run:
bash mkecho.sh test.sh
you will get the next:
base64 -D <<<'H4sIAASwqFEAA0vLL1LIVMjMU6g21NMzNKjlSsnn4kxJLEkFMvJSuQBZFmY0HwAAAA=='| gzip -d
If you copy and paste the above into the terminal, it will will display the original test.sh
Variant2
If you want directly execute the script, you should modify the mkecho.sh to the next mkeval.sh
#!/bin/bash
[[ ! -f "$1" ]] && echo "Need filename" && exit 1
asc=$(gzip < "$1" | base64)
echo -n 'eval "$(base64 -D <<<"'
echo -n $asc
echo -n '" | gzip -d)"'
echo
When run
bash mkeval.sh test.sh
will get
eval "$(base64 -D <<<"H4sIAASwqFEAA0vLL1LIVMjMU6g21NMzNKjlSsnn4kxJLEkFMvJSuQBZFmY0HwAAAA==" | gzip -d)"
and finally when you copy and paste it into the terminal, you run the test.sh and will get:
Fri May 31 16:25:08 CEST 2013
... 8 lined deleted...
Fri May 31 16:25:08 CEST 2013
Warning: because the script is NOT TESTED for every possible conditions, nor for redirects and so on - I really don't recommending using the eval verision.
sed 's/\\t/\\/g'
$ echo 'ffsd \tif [[ -z $' | sed 's/\\t/\\/g'
ffsd \if [[ -z $