I'm trying to execute bash script but I'm got stuck.
data
$ cat test.txt
cat,command,for
cp,command,for
ls,command,to
script
#!/bin/bash
CUT_FILE=test.txt
TRN_GUID="1bcd1adf-2016-443b-9f00-2e4ce20726d7"
LCTN_ID="8002"
LCTN_NAME="TEST FILE"
LCTN_ADDR1="This is test"
cat $CUT_FILE | awk -F ',' '{ print '$TRN_GUID','$LCTN_ID','$LCTN_NAME','$LCTN_ADDR1',$1,$2 }'
output
-bash-3.2# sh test4
awk: cmd. line:1: { print 1bcd1adf-2016-443b-9f00-2e4ce20726d7,8002,TEST
awk: cmd. line:1: ^ unexpected newline or end of string
desired output
1bcd1adf-2016-443b-9f00-2e4ce20726d7,8002,"TEST FILE","This is test",cat,command
Any ideas?
Syntax Error
You are closing your script using the single quotes.
awk -F ',' '{ print '$TRN_GUID','$LCTN_ID','$LCTN_NAME','$LCTN_ADDR1',$1,$2 }'
Can you see, using the syntax highlighting, that the script gets closed after print '? You should use double quotation marks inside your script to prevent this, like so:
awk -F ',' '{ print "$TRN_GUID", "$LCTN_ID", "$LCTN_NAME", "$LCTN_ADDR1", $1, $2 }'
Do you notice the difference in syntax highlighting?
Undesired output
You cannot access bash variables inside awk, to accomplish this you should pass them as variables. To do that, we simply use the -v option of awk. To illustrate this point:
awk -v trn_guid="$TRN_GUID" -v lctn_id="LCTN_ID" -v lctn_name="$LCTN_NAME" -v lctn_addr1="$LCTN_ADDR1" -v arg1="$1" -v arg2="$2" -F ',' '{ print trn_guid, lctn_id, lctn_name, lctn_addr1, arg1, arg2 }'
The syntax is simply -v variable="value". You can put anything in the value, including bash variables. The variable will be accessible in the awk script using the name you chose.
You don't need awk for this. Just use it like this in pure BASH:
CUT_FILE="test.txt"
TRN_GUID="1bcd1adf-2016-443b-9f00-2e4ce20726d7"
LCTN_ID="8002"
LCTN_NAME="TEST FILE"
LCTN_ADDR1="This is test"
while read -r line; do
echo "$TRN_GUID,$LCTN_ID,$LCTN_NAME,$LCTN_ADDR1,${line%,*}"
done < "$CUT_FILE"
Output:
1bcd1adf-2016-443b-9f00-2e4ce20726d7,8002,TEST FILE,This is test,cat,command
1bcd1adf-2016-443b-9f00-2e4ce20726d7,8002,TEST FILE,This is test,cp,command
1bcd1adf-2016-443b-9f00-2e4ce20726d7,8002,TEST FILE,This is test,ls,command
Related
I'm writing a simple shell command using awk, as follow:
input_folder='/home/Desktop/files'
results_folder='/home/results'
for entry in $input_folder/*
do
re=$(samtools view -H $entry | grep -P '^#SQ' | cut -f 3 -d ':' | awk '{sum+=$1} END {print sum}')
echo -e "$(samtools depth $entry | awk '{sum+=$3} END { print $(sum/$re)}')\t/$entry" >> $results_folder/Results.txt
done
the result in variable re is a number but using the result of re into the second command print $(sum/$re)}' give me this error
awk: cmd. line:1: (FILENAME=- FNR=312843568) fatal: division by zero attempted
I tried not to put $ with the variable but also the same error.
Any help with that please?
Change the awk part to:
awk -v re="$re" '{sum+=$3} END { if(re) print sum/re; else print "oo";}'
You have to use -v to transfer the variable into awk.
And also it's better to check if re is zero.
I used oo to represent Infinity symbol.
I am not clear why you are sending output of echo command to awk. YOur actual awk command should be to avoid your error (in which it tells that you are dividing it by zero). Try changing your awk program to following once?
awk -v re="$re" '{sum+=$3} END {if(re){print (sum/re)} else {print "Please check seems value of re is ZERO else you will get an error from awk program}}'
Inplacing bash variable into awk will do the job:
awk '{sum+=$3} END { print(sum/'${re}') }'
You'd better also check ${re} in bash for non zero check before passing to awk.
Hi I have a shell command like this.
s3=$(awk 'BEGIN{ print "S3 bucket path" }
/Executing command\(queryId/{ sub(/.*queryId=[^[:space:]]+: /,""); q=$0 }
/s3:\/\//{ print "," $10 }' OFS=',' hive-server2.log)
The output of the above command like this.
echo $s3
2018-02-21T17:58:22,
2018-02-21T17:58:26,
2018-02-21T18:05:33,
2018-02-21T18:05:34
I want to select the last line only. I need the last output like this.
2018-02-21T18:05:34
I tried like this.
awk -v $s3 '{print $(NF)}'
Not working.Any help will be appreciated.
In general, command | tail -n 1 prints the last line of the output from command. However, where command is of the form awk '... { ... print something }' you can refactor to awk '... { ... result = something } END { print result }' to avoid spawning a separate process just to discard the other output. (Conversely, you can replace awk '/condition/ { print something }' | head -n 1 with awk '/condition/ { print something; exit }'.)
If you already have the result in a shell variable s3 and want to print just the last line, a parameter expansion echo "${s3##*$'\n'}" does that. The C-style string $'\n' to represent a newline is a Bash extension, and the parameter expansion operator ## to remove the longest matching prefix isn't entirely portable either, so you should make sure the shebang line says #!/bin/bash, not #!/bin/sh
Notice also that $s3 without quotes is an error unless you specifically require the shell to perform whitespace tokenization and wildcard expansion on the value. You should basically always use double quotes around variables except in a couple of very specific scenarios.
Your Awk command would not work for two reasons; firstly, as explained in the previous paragraph, you are setting s3 to the first token of the variable, and the second is your Awk script (probably a syntax error). In more detail, you are basically running
awk -v s3=firstvalue secondvalue thirdvalue '{ print $(NF) }'
^ value ^ script to run ^ names of files ...
where you probably wanted to say
awk -v s3=$'firstvalue\nsecondvalue\nthirdvalue' '{ print $(NF) }'
But even with quoting, your script would set v to something but then tell Awk to (ignore the variable and) process standard input, which on the command line leaves it reading from your terminal. A fixed script might look like
awk 'END { print }' <<<"$s3"
which passes the variable as standard input to Awk, which prints the last line. The <<<value "here string" syntax is also a Bash extension, and not portable to POSIX sh.
much simple way is
command | grep "your filter" | tail -n 1
or directly
command | tail -n 1
You could try this:
echo -e "This is the first line \nThis is the second line" | awk 'END{print}'
another approach can be, processing the file from the end and exiting after first match.
tac file | awk '/match/{print; exit}'
Hi you can do it just by adding echo $s3 | sed '$!d'
s3=$(awk 'BEGIN{ print "S3 bucket path" }/Executing command\(queryId/{ sub(/.*queryId=[^[:space:]]+: /,""); q=$0 } /s3:\/\//{ print "," $10 }' OFS=',' hive-server2.log)
echo $s3 | sed '$!d'
It will simply print:-
2018-02-21T18:05:34
Hope this will help you.
I have a csv file like the following example:
fieldname1: "This is field1", "id":55, fieldname2: "This is field2", "id":66
I would like to replace the fourth field from ""id":66" to ""id":72" using the awk command. I have tried it the following way but am getting a syntax error:
awk -F, '{${4}="\"id\":999";}1' OFS=, rule.txt
The error is:
awk: {${4}="\"id\":999";}1
awk: ^ syntax error
awk: {${4}="\"id\":999";}1
awk: ^ syntax error
awk: cmd. line:1: {${4}="\"id\":999";}1
awk: cmd. line:1: ^ unexpected newline or end of string
Any suggestions for correct way of doing this?
You just need to say $4 instead of ${4}:
$ awk -F, '{$4="\"id\":999";}1' OFS=, file
# ^^
fieldname1: "This is field1", "id":55, fieldname2: "This is field2","id":999
If you want to give the value via a variable, use -v value="$bash_var" as usual:
$ awk -F, -v val=999 '{$4="\"id\":" val;}1' OFS=, file
# ^^^^^^^^^^ ^^^^^
fieldname1: "This is field1", "id":55, fieldname2: "This is field2","id":999
Note that ${ } is used in Bash to avoid confusion when using a variable $hello being confused with $hello_you when saying eg echo "$hello_you" -> in that case, you would say echo "${hello}_you" to define the scope of the name of the variable.
But in awk such thing shouldn't be necessary because you enclose the string part in double quotes:
$ awk 'BEGIN {a=23; print a"_b"}'
23_b
I'm writing a script that involves generating Awk programs and running them via awk $(...), as in
[lynko#hephaestus] ~ % awk $(echo 'BEGIN { print "hello!" }')
The generated program is going to be more complicated in the end, but first I want to make sure this is possible. In the past I've done
[lynko#hephaestus] ~ % program=$(echo 'BEGIN { print "hello" }')
[lynko#hephaestus] ~ % awk "$program"
hello!
where the grouping is unsurprising. But the first example (under GNU awk, which gives a more helpful error message than mawk which is default on my other machine) gives
[lynko#hephaestus] ~ % awk $(echo 'BEGIN { print "hello!" }')
awk: cmd. line:1: BEGIN blocks must have an action part
presumably because this is executed as awk BEGIN { print "hello!" } rather than awk 'BEGIN { print "hello!" }'. Is there a way I can force $(...) to remain as one group? I'd rather not use "$()" since I'd have to escape all the double-quotes in the program generator.
I'm running Bash 4.2.37 and mawk 1.3.3 on Crunchbang Waldorf.
Put quotes around it. You don't need to escape the double quotes inside it:
awk "$(echo 'BEGIN { print "hello!" }')"
I'm also wondering why you are using an echo statement. Awk doesn't need one.
awk 'BEGIN { print "Awk SQUAWK!" }'
That will work perfectly.
I am writing a shell script which needs to pull values out of a text file which looks like this:
app.full.name /warfilelocation/ warfilename
My shell script will be iterating over a list of application names and pulling out either the location or name using AWK. I have tested doing this on the command line using the following:
awk "\$1 ~/app.full.name/ { print $2 }" applications.txt
which returns what I would expect however when i put this in a shell script I start having issues.
I have a function that looks like this:
function get_location() {
local application=$1
awk "\$1 ~/^$application/ { print \$2 }" applications.txt
}
But when i call this function i get the following error:
awk: $1 ~/^app.full.name
awk: ^ unterminated regexp
awk: cmd. line:1: app.full.name
awk: cmd. line:1: ^ syntax error
awk: cmd. line:2: app.full.name/ { print $2 }
awk: cmd. line:2: ^ syntax error
Does anyone have any ideas what I am doing wrong here. I presume I am not escaping the variable correct but no matter what i try it doesnt seem to work.
Thanks in advance
Use this approach to make awk recognize shell variables:
awk -v "v1=$VAR1" -v "v2=$VAR2" '{print v1, v2}' input_file
Update
$ cat input
tinky-winky
dipsy
laa-laa
noo-noo
po
$ teletubby='po'
$ awk -v "regexp=$teletubby" '$0 ~ regexp' input
po
Note that anything could go into the shell-variable,
even a full-blown regexp, e.g ^d.*y. Just make sure to use single-quotes
to prevent the shell from doing any expansion.
The error messages seem to indicate that there is a stray newline at the end of $application, which gives the "line 2" error messages.
see this: using awk match() function
kent$ app=app.ful
kent$ echo "app.full.name /warfilelocation/ warfilename"|awk -v a=$app '{if(match($1,a))print $2}'
/warfilelocation/
It's hard to tell without knowing exactly the value of $application, but it seems like you have a strange character in $application, such as a " or a / or something like that.
$ export application=foo/bar
$ awk "\$1 ~/^$application/ { print \$1 }"
gawk: cmd. line:1: $1 ~/^foo/bar/ { print $1 }
gawk: cmd. line:1: ^ parse error
I would look at the exact value that you have in $application, and if it contains a /, escape it.
One way to do this would be to use:
$ export application=`echo foo/bar | sed -e 's;/;\\\\/;g'`
$ awk "\$1 ~/^$application/ { print \$1 }"