Related
I have the following script:
#!/bin/bash
…code setting array ids, etc…
for i in "${!ids[#]}" ; do
echo "#${ids[i]}_${pos[i]}_${wild[i]}_${sub[i]}"
curl -sS "http://www.uniprot.org/uniprot/"${ids[i]}".fasta";
done |
sed '/^>/ d' |
sed -r 's/[#]+/>/g' |
perl -npe 'chomp if ($.!=1 && !s/^>/\n>/)' > $id.pph.fasta
However the results will not store in the file. I can output the result to the terminal and store in file by doing:
./myscript > result.txt
However I want to do this within the script and output to file outside the loop.
Add
exec 1>result.txt
to the top of the script, and all output will be redirected.
Here is a variation of your script:
#!/bin/sh
for i in ${!ids[*]}
do
echo ">${ids[i]}_${pos[i]}_${wild[i]}_${sub[i]}"
curl -Ss www.uniprot.org/uniprot/${ids[i]}.fasta
done |
awk '
/>/ {if (z++) printf RS; print; printf RS; getline; next}
1
END {printf RS}
' ORS= > $id.pph.fasta
I have a file test.txt like below spaces in between each record
service[1.1],parttion, service[1.2],parttion, service[1.3],parttion, service[2.1],parttion, service2[2.2],parttion,
Now I want to rearrange it as below into a output.txt
COMPOSITES=parttion/service/1.1,parttion/service/1.2,parttion/service/1.3,parttion/service/2.1,parttion/service/2.2
I've tried:
final_str=''
COMPOSITES=''
# Re-arranging the composites and preparing the composite property file
while read line; do
partition_val="$(echo $line | cut -d ',' -f 2)"
composite_temp1_val="$(echo $line | cut -d ',' -f 1)"
composite_val="$(echo $composite_temp1_val | cut -d '[' -f 1)"
version_temp1_val="$(echo $composite_temp1_val | cut -d '[' -f 2)"
version_val="$(echo $version_temp1_val | cut -d ']' -f 1)"
final_str="$partition_val/$composite_val/$version_val,"
COMPOSITES=$COMPOSITES$final_str
done <./temp/test.txt
We start with the file:
$ cat test.txt
service[1.1],parttion, service[1.2],parttion, service[1.3],parttion, service[2.1],parttion, service2[2.2],parttion,
We can rearrange that file as follows:
$ awk -F, -v RS=" " 'BEGIN{printf "COMPOSITES=";} {gsub(/[[]/, "/"); gsub(/[]]/, ""); if (NF>1) printf "%s%s/%s",NR==1?"":",",$2,$1;}' test.txt
COMPOSITES=parttion/service/1.1,parttion/service/1.2,parttion/service/1.3,parttion/service/2.1,parttion/service2/2.2
The same command split over multiple lines is:
awk -F, -v RS=" " '
BEGIN{
printf "COMPOSITES=";
}
{
gsub(/[[]/, "/")
gsub(/[]]/, "")
if (NF>1) printf "%s%s/%s",NR==1?"":",",$2,$1
}
' test.txt
Here's what I came up with.
awk -F '[],[]' -v RS=" " 'BEGIN{printf("COMPOSITES=")}/../{printf("%s/%s/%s,",$4,$1,$2);}' test.txt
Broken out for easier reading:
awk -F '[],[]' -v RS=" " '
BEGIN {
printf("COMPOSITES=");
}
/../ {
printf("%s/%s/%s,",$4,$1,$2);
}' test.txt
More detailed explanation of the script:
-F '[],[]' - use commas or square brackets as field separators
-v RS=" " - use just the space as a record separator
'BEGIN{printf("COMPOSITES=")} - starts your line
/../ - run the following code on any line that has at least two characters. This avoids the empty field at the end of a line terminating with a space.
printf("%s/%s/%s,",$4,$1,$2); - print the elements using a printf() format string that matches the output you specified.
As concise as this is, the format string does leave a trailing comma at the end of the line. If this is a problem, it can be avoided with a bit of extra code.
You could also do this in sed, if you like writing code in line noise.
sed -e 's:\([^[]*\).\([^]]*\).,\([^,]*\), :\3/\1/\2,:g;s/^/COMPOSITES=/;s/,$//' test.txt
Finally, if you want to avoid external tools like sed and awk, you can do this in bash alone:
a=($(<test.txt))
echo -n "COMPOSITES="
for i in "${a[#]}"; do
i="${i%,}"
t="${i%]*}"
printf "%s/%s/%s," "${i#*,}" "${i%[*}" "${t#*[}"
done
echo ""
This slurps the contents of test.txt into an array, which means your input data must be separated by whitespace, per your example. It then adds the prefix, then steps through the array, using Parameter Expansion to massage the data into the fields you need. The last line (echo "") is helpful for testing; you may want to eliminate it in practice.
I'm using the following awk command:
my_command | awk -F "[[:space:]]{2,}+" 'NR>1 {print $2}' | egrep "^[[:alnum:]]"
which successfully returns my data like this:
fileName1
file Name 1
file Nameone
f i l e Name 1
So as you can see some file names have spaces. This is fine as I'm just trying to echo the file name (nothing special). The problem is calling that specific row within a loop. I'm trying to do it this way:
i=1
for num in $rows
do
fileName=$(my_command | awk -F "[[:space:]]{2,}+" 'NR==$i {print $2}' | egrep "^[[:alnum:]])"
echo "$num $fileName"
$((i++))
done
But my output is always null
I've also tried using awk -v record=$i and then printing $record but I get the below results.
f i l e Name 1
EDIT
Sorry for the confusion: rows is a variable that list ids like this 11 12 13
and each one of those ids ties to a file name. My command without doing any parsing looks like this:
id File Info OS
11 File Name1 OS1
12 Fi leNa me2 OS2
13 FileName 3 OS3
I can only use the id field to run a the command that I need, but I want to use the File Info field to notify the user of the actual File that the command is being executed against.
I think your $i does not expand as expected. You should quote your arguments this way:
fileName=$(my_command | awk -F "[[:space:]]{2,}+" "NR==$i {print \$2}" | egrep "^[[:alnum:]]")
And you forgot the other ).
EDIT
As an update to your requirement you could just pass the rows to a single awk command instead of a repeatitive one inside a loop:
#!/bin/bash
ROWS=(11 12)
function my_command {
# This function just emulates my_command and should be removed later.
echo " id File Info OS
11 File Name1 OS1
12 Fi leNa me2 OS2
13 FileName 3 OS3"
}
awk -- '
BEGIN {
input = ARGV[1]
while (getline line < input) {
sub(/^ +/, "", line)
split(line, a, / +/)
for (i = 2; i < ARGC; ++i) {
if (a[1] == ARGV[i]) {
printf "%s %s\n", a[1], a[2]
break
}
}
}
exit
}
' <(my_command) "${ROWS[#]}"
That awk command could be condensed to one line as:
awk -- 'BEGIN { input = ARGV[1]; while (getline line < input) { sub(/^ +/, "", line); split(line, a, / +/); for (i = 2; i < ARGC; ++i) { if (a[1] == ARGV[i]) {; printf "%s %s\n", a[1], a[2]; break; }; }; }; exit; }' <(my_command) "${ROWS[#]}"
Or better yet just use Bash instead as a whole:
#!/bin/bash
ROWS=(11 12)
while IFS=$' ' read -r LINE; do
IFS='|' read -ra FIELDS <<< "${LINE// +( )/|}"
for R in "${ROWS[#]}"; do
if [[ ${FIELDS[0]} == "$R" ]]; then
echo "${R} ${FIELDS[1]}"
break
fi
done
done < <(my_command)
It should give an output like:
11 File Name1
12 Fi leNa me2
Shell variables aren't expanded inside single-quoted strings. Use the -v option to set an awk variable to the shell variable:
fileName=$(my_command | awk -v i=$i -F "[[:space:]]{2,}+" 'NR==i {print $2}' | egrep "^[[:alnum:]])"
This method avoids having to escape all the $ characters in the awk script, as required in konsolebox's answer.
As you already heard, you need to populate an awk variable from your shell variable to be able to use the desired value within the awk script so thi:
awk -F "[[:space:]]{2,}+" 'NR==$i {print $2}' | egrep "^[[:alnum:]]"
should be this:
awk -v i="$i" -F "[[:space:]]{2,}+" 'NR==i {print $2}' | egrep "^[[:alnum:]]"
Also, though, you don't need awk AND grep since awk can do anything grep van do so you can change this part of your script:
awk -v i="$i" -F "[[:space:]]{2,}+" 'NR==i {print $2}' | egrep "^[[:alnum:]]"
to this:
awk -v i="$i" -F "[[:space:]]{2,}+" '(NR==i) && ($2~/^[[:alnum:]]/){print $2}'
and you don't need a + after a numeric range so you can change {2,}+ to just {2,}:
awk -v i="$i" -F "[[:space:]]{2,}" '(NR==i) && ($2~/^[[:alnum:]]/){print $2}'
Most importantly, though, instead of invoking awk once for every invocation of my_command, you can just invoke it once for all of them, i.e. instead of this (assuming this does what you want):
i=1
for num in rows
do
fileName=$(my_command | awk -v i="$i" -F "[[:space:]]{2,}" '(NR==i) && ($2~/^[[:alnum:]]/){print $2}')
echo "$num $fileName"
$((i++))
done
you can do something more like this:
for num in rows
do
my_command
done |
awk -F '[[:space:]]{2,}' '$2~/^[[:alnum:]]/{print NR, $2}'
I say "something like" because you don't tell us what "my_command", "rows" or "num" are so I can't be precise but hopefully you see the pattern. If you give us more info we can provide a better answer.
It's pretty inefficient to rerun my_command (and awk) every time through the loop just to extract one line from its output. Especially when all you're doing is printing out part of each line in order. (I'm assuming that my_command really is exactly the same command and produces the same output every time through your loop.)
If that's the case, this one-liner should do the trick:
paste -d' ' <(printf '%s\n' $rows) <(my_command |
awk -F '[[:space:]]{2,}+' '($2 ~ /^[::alnum::]/) {print $2}')
Here is the actual file data:
abc
def
ghi
jkl
mno
And the required output should be in this format:
'abc','def','ghi','jkl','mno'
The command what I used to do this gives output as:
abc,def,ghi,jkl,mno
The command is as follows:
sed -n 's/[0-3]//;s/ //;p' Split_22_05_2013 | \
awk -v ORS= '{print $0" ";if(NR%4==0){print "\n"}}'
In response to sudo_O's comment I add an awk less solution in pure bash. It does not exec any program at all. Of course instead of <<XXX ... XXX (here-is-the-document) stuff one could add <filename.
set c=""
while read w; do
echo -e "$c'$w'\c"
c=,
done<<XXX
abc
def
ghi
jkl
mno
XXX
Output:
'abc','def','ghi','jkl','mno'
An even shorter version:
printf -v out ",'%s'" $(<infile)
echo ${out:1}
Without the horrifying pipe snakes You can try something like this:
awk 'NR>1{printf ","}{printf "\x27%s\x27",$0}' <<XXX
abc
def
ghi
jkl
mno
XXX
Output:
'abc','def','ghi','jkl','mno'
Or an other version which reads the whole input as one line:
awk -vRS="" '{gsub("\n","\x27,\x27");print"\x27"$0"\x27"}'
Or a version which lets awk uses the internal variables more
awk -vRS="" -F"\n" -vOFS="','" -vORS="'" '{$1=$1;print ORS $0}'
The $1=$1; is needed to tell to awk to repack $0 using the new field and record separators (OFS, ORS).
$ cat test.txt
abc
def
ghi
jkl
mno
$ cat test.txt | tr '\n' ','
abc,def,ghi,jkl,mno,
$ cat test.txt | awk '{print "\x27" $1 "\x27"}' | tr '\n' ','
'abc','def','ghi','jkl','mno',
$ cat test.txt | awk '{print "\x27" $1 "\x27"}' | tr '\n' ',' | sed 's/,$//'
'abc','def','ghi','jkl','mno'
The last command can be shortened to avoid UUOC:
$ awk '{print "\x27" $1 "\x27"}' test.txt | tr '\n' ',' | sed 's/,$//'
'abc','def','ghi','jkl','mno'
Using sed alone:
sed -n "/./{s/^\|\$/'/g;H}; \${x;s/\n//;s/\n/,/gp};" test.txt
Edit: Fixed, it should also work with or without empty lines now.
$ cat file
abc
def
ghi
jkl
mno
$ cat file | tr '\n' ' ' | awk -v q="'" -v OFS="','" '$1=$1 { print q $0 q }'
'abc','def','ghi','jkl','mno'
Replace '\n' with ' ' -> (tr '\n\ ' ')
Replace each separator (' ' space) with (',' quote-comma-quote) ->
(-v OFS="','")
Add quotes to the begin and end of line -> (print q $0 q)
This can be done pretty briefly with sed and paste:
<infile sed "s/^\|\$/'/g" | paste -sd,
Or more portably (I think, cannot test right now):
sed "s/^\|\$/'/g" infile | paste -s -d , -
$ sed "s/[^ ][^ ]*/'&',/g" input.txt | tr -d '\n'
'abc','def','ghi','jkl','mno',
To clean the last ,, throw in a
| sed 's/,$//'
awk 'seen == 1 { printf("'"','"'%s", $1);} seen == 0 {seen = 1; printf("'"'"'%s", $1);} END { printf("'"'"'\n"); }'
In slightly more readable format (suitable for awk -f):
# Print quote-terminator, separator, quote-start, thing
seen == 1 { printf("','%s", $1); }
# Set the "print separator" flag, print quote-start thing
seen == 0 { seen = 1; printf("'%s", $1}; }
END { printf("'\n"); } # Print quote-end
perl -l54pe 's/.*/\x27$&\x27/' file
I'm using a few commands to cat a few files, like this:
cat somefile | grep example | awk -F '"' '{ print $2 }' | xargs cat
It nearly works, but my issue is that I'd like to add a newline after each file.
Can this be done in a one liner?
(surely I can create a new script or a function that does cat and then echo -n but I was wondering if this could be solved in another way)
cat somefile | grep example | awk -F '"' '{ print $2 }' | while read file; do cat $file; echo ""; done
Using GNU Parallel http://www.gnu.org/software/parallel/ it may be even faster (depending on your system):
cat somefile | grep example | awk -F '"' '{ print $2 }' | parallel "cat {}; echo"
awk -F '"' '/example/{ system("cat " $2 };printf "\n"}' somefile