I have a function "checkExist" that takes in a command and executes it based on whether or not the output file already exists. I pass a command like this, where file1 and file2 are just the names of output files that the commands create, so if they already exist it will ask if you want to overwrite, else it will skip:
checkExist file1 file2 command1 command2
In actual use like this:
checkExist 1.txt 2.txt "echo $1 | awk '$5 <= 10 {print $3, $4}'" "echo $2 | awk '$5 <= 10 {print $3, $4}'"
$1 and $2 above are inputs to the script "smartfilter.sh" containing the function checkExist within. So they are just file inputs.
Later in the checkExist function if the user types 'Y/y' to overwrite, or the files don't already exist then it will
eval $3 &
eval $4 &
wait
And I get an error like so:
awk: >= 10 {print , }
awk: ^ syntax error
awk: >= 10 {print , }
awk: ^ syntax error
awk: >= 10 {print , }
awk: ^ syntax error
awk: cmd. line:1: >= 10 {print , }
awk: cmd. line:1: ^ unexpected newline or end of string
I know it is to do with the single quotations ' around the awk and eval not parsing them correctly. I have tried \' but that doesn't work either. Is there a proper way to do this?
checkExist 1.txt 2.txt "echo $1 | awk '\$5 <= 10 {print \$3, \$4}'" "echo $2 | awk '\$5 <= 10 {print \$3, \$4}'"
Related
Please let me know the mistake in the following command
for i in file*; do printf "$i processing" && \
for j in {3..10}; do printf ">=$j\t" && k=$(< "$i" wc -l) ; \
awk 'BEGIN {FS="\t"} {if ($7 >= '$j') {print $0} }' $i| \
awk 'END{print (NR/"$k")*100}' ; done ; done
file1 processong
>=3 awk: cmd. line:1: (FILENAME=- FNR=126052) fatal: division by zero attempted
>=4 awk: cmd. line:1: (FILENAME=- FNR=118562) fatal: division by zero attempted
>=5 awk: cmd. line:1: (FILENAME=- FNR=113376) fatal: division by zero attempted
>=6 awk: cmd. line:1: (FILENAME=- FNR=109501) fatal: division by zero attempted
>=7 awk: cmd. line:1: (FILENAME=- FNR=106388) fatal: division by zero attempted
>=8 awk: cmd. line:1: (FILENAME=- FNR=103781) fatal: division by zero attempted
>=9 awk: cmd. line:1: (FILENAME=- FNR=101547) fatal: division by zero attempted
>=10 awk: cmd. line:1: (FILENAME=- FNR=99552) fatal: division by zero attempted
Thanks
The reason you obtain division by zero is due to the following line:
awk 'END{print (NR/"$k")*100}'
You attempt to use a shell variable $k which you assigned as k=$(< "$i" wc -l). However, you use it in a single quoted string, so bash does not perform the variable substitution as you imagine. It only does this in double-quoted strings. More details here: How do I use shell variables in an awk script?
So your mystery line should look like:
for i in file*; do
printf "$i processing" && \
for j in {3..10}; do
printf ">=$j\t" && k=$(< "$i" wc -l)
awk -v j="$j" 'BEGIN {FS="\t"} {if ($7 >= j) {print $0} }' $i \
| awk -v k="$k" 'END{print (NR/k)*100}'
done
done
However, this command can be cleaned up a lot:
no need for compound commands using &&. (printf will not fail)
Don't use printf "string", but rather printf -- "%s" "string"
when using awk, you are not in need of anything of the form like grep, wc, sed, ...
for i in file*; do
printf -- "%s" "$i processing"
for j in {3..10}; do
printf -- "%s" ">=$j\t"
awk -v j="$j" 'BEGIN {FS="\t"}($7>j) { print; c++ }END{print (c/NR)*100}' "$i"
done
done
Using GNU awk, you can even reduce this entire code-block into a single-awk, but this is beyond the scope of this question.
I don’t know why I can not give only 0 to awk in a direct statement, e.g. if I want to output the square of a number:
$ echo 4 | awk '$0=$1*$1'
16
$ echo 3 | awk '$0=$1*$1'
9
$ echo 0 | awk '$0=$1*$1'
Why do I get nothing on the last try?
PS. it works if I write $1 in a bracketed statement:
$ echo 0 | awk '{print $1*$1}'
0
No, awk does not ignore a line with 0.
However, your awk command: $0=$1*$1 does not do what you think.
By default awk prints $0 if there is an statement that evaluates to true (not zero).
So, this will always print $0:
awk '1'
And this will never print $0:
awk '0'
To do what you want: to always print $0 after it has been re-calculated, you need to do:
awk '{$0=$1*$1; print}'
And so:
$ echo "0" | awk '{$0=$1*$1; print}'
0
$ echo "2" | awk '{$0=$1*$1; print}'
4
Or, without changing the value of $0, do:
$ echo "2" | awk '{print $0*$0}'
Or (shorter but less readable):
$ echo "2" | awk '{$0=$0*$0}1'
4
And, even shorter:
$ echo "4" | awk '{$0*=$0}1'
16
This last awk script is actually composed of two command lines:
awk '
<default pattern> { $0*=$0 }
1 { <default action> }
'
Which become, replacing the action by print and the condition by all:
awk ' /.*/{$0*=$0}
1 {print $0}'
Both lines are applied to all input lines. For all lines $0 is changed, and for all input lines a print $0 is executed.
I am writing a script that's picking up two values from a file and then subtracting them.But I am unable to do substraction as it is throwing error.
res1= awk 'FNR == '${num1}' {print $1}' /home/shell/test.txt
res2= awk 'FNR == '${num2}' {print $1}' /home/shell/test.txt
res= $((res2 - res1))
echo $res
I also tried expr = `expr $res2 -$res1` but it didn't work. please help me so as to get the desired result.
your assignments for res1/res2 are wrong. It should be
res1=$(awk 'FNR == '${num1}' {print $1}' /home/shell/test.txt)
However, you can do it all in awk
$ num1=5; num2=2; awk -v n1=${num1} -v n2=${num2} 'FNR==n1{r1=$1;f1=1}
FNR==n2{r2=$1;f2=1}
f1&&f2{print r1-r2; exit}' <(seq 5)
3
This is because there is one space char after each = sign: res1= awk
Remove the spaces and use $( command ) to execute a command and gather its output.
Give a try to this:
res1="$(awk -v num=${num1} 'FNR == num {print $1}' /home/shell/test.txt)"
res2="$(awk -v num=${num2} 'FNR == num {print $1}' /home/shell/test.txt)"
res=$(( res2 - res1 ))
printf "%d\n" ${res}
I had read in another answer that it is preferred to pass variable's value to awk script using -v var_name=value, rather than concatenating strings.
Still newish to the site, but here goes...
Basically I'm storing events in multiple files, with each event being a line and each line containing dates ($1), start($2) and stop($3) times and several other pieces of data. I use two double underscores ("__") as Field Separators. I've been using a variety of shell scripts to manage the data, and I was using awk to calculate stats and I'm having trouble invoking the date function so I can get a total by day of the week. After much tinkering and scanning of discussion boards I got to this:
ls /home/specified/folder/MBRS.db/* |
xargs -n 1 -I % awk -F"__" '$6 == "CLOSED" && $1 >= "'$backDATE'" { print $0 }' % |
awk 'BEGIN{FS="__"}{specDATE=system("date --date="$1" +%a")} specDATE == "Tue" {print $2" "$3}'
or
ls /home/lingotech/Einstein/data/MBRS.db/* |
xargs -n 1 -I % awk -F"__" '$6 == "CLOSED" && $1 >= "'$backDATE'" { print $0 }' % |
awk 'BEGIN{FS="__"}system("date --date="$1" +%a") == "Mon" {print $2" "$3}'`
Instead of outputting the start and stop times, I'm getting a list of all the different days of the week for each entry.
I've tried more variations of the date usage than I care to admit, including:
for y in Sun Mon Tue Wed Thu Fri Sat; do
for directory in $( ls /home/specified/directory/MBRS.db/* | xargs -n 1 ); do
printf "."
[[ $( cat $directory | awk -F"__" '$6 == "CLOSED" && $1 >= "'$backDATE'" { print $1 }' | xargs -n 1 -I z date +%a -d z ) == "$y" ]] && echo BLAH
done
done
Some helpful explanation of what I'm screwing up would be much appreciated. Thanks in advance. Oh and I'm storing the date in YYMMDD format but that doesn't seem to be an issue for ubuntu server's version of 'date'.
I don't know about all the rest of it (too much text for my reading tastes!) but wrt the answer you posted, this part of it:
awk 'BEGIN{FS="__"} NF == 10 && $1 >= "'$backDATE'" && $4 == "'$x'" && $6 == "CLOSED" {while ( "date +%a -d "$1"" | getline newDAY){if (newDAY == "'$y'") print $2" "$3}}' /home/absolute/path/*
assuming it does what you want would be written as:
awk -v backDATE="$backDATE" -v x="$x" -v y="$y" '
BEGIN { FS="__" }
(NF == 10) && ($1 >= backDATE) && ($4 == x) && ($6 == "CLOSED") {
cmd = "date +%a -d \"" $1 "\""
while ( (cmd | getline newDAY) > 0 ) {
if (newDAY == y) {
print $2, $3
}
}
close(cmd)
}
' /home/absolute/path/*
wrt why use awk variables instead of letting shell variables expand to become part of the body of a shell script, the answer is robustness and simplicity.
This is letting a shell variable expand to become part of the body of an awk script:
$ x="hello world"
$ awk 'BEGIN{ print '$x' }'
awk: cmd. line:1: BEGIN{ print hello
awk: cmd. line:1: ^ unexpected newline or end of string
$ awk 'BEGIN{ print "'$x'" }'
awk: cmd. line:1: BEGIN{ print "hello
awk: cmd. line:1: ^ unterminated string
awk: cmd. line:1: BEGIN{ print "hello
awk: cmd. line:1: ^ syntax error
$ awk 'BEGIN{ print "'"$x"'" }'
hello world
$ x="hello
world"
$ awk 'BEGIN{ print "'"$x"'" }'
awk: cmd. line:1: BEGIN{ print "hello
awk: cmd. line:1: ^ unterminated string
awk: cmd. line:1: BEGIN{ print "hello
awk: cmd. line:1: ^ syntax error
and this is using an awk variable initialized with the value of a shell variable:
$ x="hello world"
$ awk -v x="$x" 'BEGIN{ print x }'
hello world
$ x="hello
world"
$ awk -v x="$x" 'BEGIN{ print x }'
hello
world
See the difference?
As for why store the command in a variable - because you have to close it after you use it and it must be spelled exactly the same way in the close command as it was when you opened the pipe. Compare:
cmd = "date +%a -d \"" $1 "\""
cmd | getline
close(cmd)
vs:
"date +%a -d \"" $1 "\"" | getline
close("date +%a -d \"" $l "\"")
and take an extremely close second look to spot the bug in the 2nd version.
Ok, so I ended up using this:
>backDATE=150000;
> for x in $listOFsites; do
> for y in Sun Mon Tue Wed Thu Fri Sat; do
> totalHOURS=$( awk 'BEGIN{FS="__"} NF == 10 && $1 >= "'$backDATE'" && $4 == "'$x'" && $6 == "CLOSED" {while ( ( "date +%a -d \""$1"\"" | getline newDAY) > 0 ){if (newDAY == "'$y'") print $2" "$3}}' /home/absolute/path/* | xargs -I % /home/custom/duration/calc % | paste -sd+ | bc ); printf ".";
> done
> done
I had to use date inside the single quotes (so that I could pass $1 to it) rather than outside (using -F"__" -v newDAY=...), but inside the single quotes getting the output of system() is problematic. After seeing:How can I pass variables from awk to a shell command? I finally saw the while (cmd | get line x) format which was the crux of my issue. Props to Ed Morton
I spent on this 2 hours and get nothing. I want to get $1 and $2 as a first command line input of shell script, but I couldn't manage this. And $3 and $0 would be columns in awk. I try different methods but nothing works for me.
awk -F':' -v "limit=1000" '{ if ( $3 >=limit ) gsub("~/$1/",~/$2/); print \$0}' file.txt
the cleanest method is to explicitly pass the values from shell to awk with awk's -v option:
awk -F: -v limit=1000 -v patt="~/$1/" -v repl="~/$2/" '
$3 >=limit {gsub(patt,repl); print}
' file.txt
When your awk line is part of a script file, and you want to use $1 and $2 from the script in your awk command, you should temporary stop the literal string with a single quote and start it again.
awk -F':' -v "limit=1000" '{ if ( $3 >=limit ) gsub("~/'$1'/",~/'$2'/); print $0}' file.txt
You didn't post any sample input or expected output so this is a guess but you probably want something like this:
awk -F':' -v limit=1000 -v arg1="$1" -v arg2="$2" '$3 >= limit{gsub("~/" arg1 "/","~/" arg2 "/"); print}' file.txt