I have a file VALIDATION_CONFIG_FILE.cfg which contains the records below:
ES_VDF_1|1
DE_VDF_1|2
ES_VDF_1|7
When I am using the grep command below by using variable then the command is returning ES_VDF_1 output. As per my understanding, command should not give any results. When I use the same command without using variables (use values directly) then command is returning no results, which is as expected. So what is the problem with variables which I am using?
FEED_ID_1_7="HU_VDF_1"
FEED_ID_2_7="ES_VDF_1"
FEED_ID_3_7="PT_VDF_2"
awk -F'|' '{ if($2=="7") print $1; }' VALIDATION_CONFIG_FILE.cfg |
grep -E -v '${FEED_ID_1_7}|${FEED_ID_2_7}|${FEED_ID_3_7}'
Output: ES_VDF_1
awk -F'|' '{ if($2=="7") print $1; }' VALIDATION_CONFIG_FILE.cfg |
grep -E -v 'ES_VDF_1|HU_VDF_1|PT_VDF_2'
Output: nothing
The problem you are seeing is that single quotes in Bash do not interpolate variables, whereas double quotes do.
For example with a variable imaginatively called "VARIABLE":
alex#yuzu:~$ export VARIABLE="foo"
If you echo it with double quotes, it is interpolated and the value of the variable is used:
alex#yuzu:~$ echo "$VARIABLE"
foo
But if you use single quotes the literal string '$VARIABLE' is used instead:
alex#yuzu:~$ echo '$VARIABLE'
$VARIABLE
The same goes for your grep.
grep -E -v '${FEED_ID_1_7}|${FEED_ID_2_7}|${FEED_ID_3_7}'
Should be:
grep -E -v "${FEED_ID_1_7}\|${FEED_ID_2_7}\|${FEED_ID_3_7}"
For example:
alex#yuzu:~$ echo "foo" | grep -E "$VARIABLE|$HOME|$USER"
foo
alex#yuzu:~$ echo "foo" | grep -E '$VARIABLE|$HOME|$USER'
[ no output ]
This is happening due to quotes.
Single quotes won't interpolate anything, but double quotes will do. Replace single quotes to double quotes with variables like below :
awk -F'|' '{ if($2=="7") print $1; }' VALIDATION_CONFIG_FILE.cfg |
grep -E -v "${FEED_ID_1_7}|${FEED_ID_2_7}|${FEED_ID_3_7}"
Refer bash manual for more details
Adding to Kaoru/Nishu Tayal's answer, you can make it safer further by using normal text search with fgrep and multiple -e:
fgrep -v -e "${FEED_ID_1_7}" -e "${FEED_ID_2_7}" -e "${FEED_ID_3_7}"
This would help prevent misinterpretations just in case special characters would be added to the values of variables.
If you don't have fgrep try grep -F.
Related
Say I have strings that look like this:
$ a='/o\\'
$ echo $a
/o\
$ b='\//\\\\/'
$ echo $b
\//\\/
I'd like a shell script (ideally a one-liner) to replace / occurrences by \ and vice-versa.
Suppose the command is called invert, it would yield (in a shell prompt):
$ invert $a
\o/
$ invert $b
/\\//\
For example using sed, it seems unavoidable to use a temporary character, which is not great, like so:
$ echo $a | sed 's#/#%#g' | sed 's#\\#/#g' | sed 's#%#\\#g'
\o/
$ echo $b | sed 's#/#%#g' | sed 's#\\#/#g' | sed 's#%#\\#g'
/\\//\
For some context, this is useful for proper printing of git log --graph --all | tac (I like to see newer commits at the bottom).
tr is your friend:
% echo 'abc' | tr ab ba
bac
% echo '/o\' | tr '\\/' '/\\'
\o/
(escaping the backslashes in the output might require a separate step)
I think this can be done with (g)awk:
$ echo a/\\b\\/c | gawk -F "/" 'BEGIN{ OFS="\\" } { for(i=1;i<=NF;i++) gsub(/\\/,"/",$i); print $0; }'
a\/b/\c
$ echo a\\/b/\\c | gawk -F "/" 'BEGIN{ OFS="\\" } { for(i=1;i<=NF;i++) gsub(/\\/,"/",$i); print $0; }'
a/\b\/c
$
-F "/" This defines the separator, The input will be split in "/", and should no longer contain a "/" character.
for(i=1;i<=NF;i++) gsub(/\\/,"/",$i);. This will replace, in all items in the input, the backslash (\) for a slash (/).
If you want to replace every instance of / with \, you can uses the y command of sed, which is quite similar to what tr does:
$ a='/o\'
$ echo "$a"
/o\
$ echo "$a" | sed 'y|/\\|\\/|'
\o/
$ b='\//\\/'
$ echo "$b"
\//\\/
$ echo "$b" | sed 'y|/\\|\\/|'
/\\//\
If you are strictly limited to GNU AWK you might get desired result following way, let file.txt content be
\//\\\\/
then
awk 'BEGIN{FPAT=".";OFS="";arr["/"]="\\";arr["\\"]="/"}{for(i=1;i<=NF;i+=1){if($i in arr){$i=arr[$i]}};print}' file.txt
gives output
/\\////\
Explanation: I inform GNU AWK that field is any single character using FPAT built-in variable and that output field separator (OFS) is empty string and create array where key-value pair represent charactertobereplace-replacement, \ needs to be escaped hence \\ denote literal \. Then for each line I iterate overall all fields using for loop and if given field hold character present in array arr keys I do exchange it for corresponding value, after loop I print line.
(tested in gawk 4.2.1)
I have the path of "file1 Nov 2018.txt" stored in variable "var". Then I use this shell variable inside the awk script
to generate another script (this is a small example). The issue is the path and the filename have spaces and even I put the variable between double quotes ""
and within awk I put between single quotes '' is not working either. I get the error "No such file or directory"
How to handle this path that has spaces?
The script is like this:
var="/mydrive/d/data 2018/Documents Nov/file1 Nov 2018.txt"
z=$(awk -v a="$var" 'BEGIN{str = "cat " 'a' ; print str}')
eval "$z"
I get these errors:
$ eval "$z"
cat: /mydrive/d/data: No such file or directory
cat: 2018/Documents: No such file or directory
cat: Nov/file1: No such file or directory
cat: Nov: No such file or directory
cat: 2018.txt: No such file or directory
Thanks for any help.
The single-quote escape sequence comes in handy here. Note that 047 is the value in octal for the ASCII ' character, and awk allows you to use \nnn within a string to include any character using its octal value.
$ cat 'foo bar.txt'
a b c
1 2 3
$ var="foo bar.txt"
$ echo "$var"
foo bar.txt
$ z=$(awk -v a="$var" 'BEGIN{print "cat \047" a "\047"}')
$ eval "$z"
a b c
1 2 3
Maybe it's a bit nicer with printf:
$ awk -v a="$var" 'BEGIN{ printf "cat \047%s\047\n", a }'
cat 'foo bar.txt'
The problem is coming from the fact that the single quote has special meaning to the shell, so it's not surprising that there's a clash when single quotes are also being used in your awk program, when that program is on the command line.
This can be avoided by putting the awk program in its own file:
$ cat a.awk
BEGIN { printf "cat '%s'\n", a }
$ awk -v a="$var" -f a.awk
cat 'foo bar.txt'
remove the single quotes around a and add escaped double quotes instead.
$ echo success > "a b"
$ var="a b"; z=$(awk -v a="$var" 'BEGIN{print "cat \"" a "\""}');
$ eval "${z}"
success
however, most likely you're doing some task unnecessarily complex.
$ cat > path\ to/test
foo
$ z=$(awk -v a="$var" 'BEGIN{gsub(/ /,"\\ ",a); str = "cat " a ; print str}')
$ echo "$z"
cat path\ to/test
$ eval "$z"
foo
The key (in this solution) being: gsub(/ /,"\\ ",a) ie. escaping the spaces with a \ (\\ due to awk).
With bash's printf %q "$var" you can correctly escape any string for later use in eval - even linebreaks will be handled correctly. However, the resulting string may contain special symbols like \ that could be interpreted by awk when assigning variables with awk -v var="$var". Therefore, better pass the variable via stdin:
path='/path/with spaces/and/special/symbols/like/*/?/\/...'
cmd=$(printf %q "$path" | awk '{print "cat "$0}')
eval "$cmd"
In this example the generated command $cmd is
cat /path/with\ spaces/and/special/symbols/like/\*/\?/\\/...
In a bash script, I need to do this:
cat<<EOF> ins.exe
grep 'pattern' file | awk '{print $2}' > results
EOF
The problem is that $2 is interpreted as a variable and the file ins.exe ends up containing
"grep 'pattern' file | awk '{print }' > results", without the $2.
I've tried using
echo "grep 'pattern' file | awk '{print $2}' > results" >> ins.exe
But it's the same problem.
How can I fix this?
Just escape the $:
cat<<EOF> ins.exe
awk '/pattern/ { print \$2 }' file > results
EOF
No need to pipe grep to awk, by the way.
With bash, you have another option as well, which is to use <<'EOF'. This means that no expansions will occur within the string.
Trying to pass a variable into awk from user input:
Have tried variations of awk -v with errors stating 'awk: invalid -v option', even though the option is listed in man files.
#! /bin/bash
read -p "Enter ClassID:" CLASS
read -p "Enter FacultyName:" FACULTY
awk '/FacultyName/ {print}' data-new.csv > $FACULTY.csv
awk -vclass=${CLASS} '/class/ {print}' data-new.csv >> $FACULTY.csv
echo Class is $CLASS
echo Faculty member is $FACULTY
Some versions of awk require a space between the -v and the variable assignment. Also, you should put the bash variable in double-quotes to prevent unwanted parsing by the shell (e.g. word splitting, wildcard expansion) before it's passed to awk. Finally, in awk /.../ is a constant regular expression (i.e. /class/ will search for the string "class", not the value of the variable "class"). With all of this corrected, here's the awk command that I think will do what you want:
awk -v class="${CLASS}" '$0 ~ class {print}' data-new.csv >> $FACULTY.csv
Now, is there any reason you're using this instead of:
grep "$CLASS" data-new.csv >> $FACULTY.csv
Your script is not clear to me, but these all work:
CLASS=ec123
echo | awk -vclass=$CLASS '{print class}'
echo | awk -vclass=${CLASS} '{print class}'
I'm trying to pass a variable to nawk in a bash script, but it's not actually printing the $commentValue variable's contents. Everything works great, except the last part of the printf statement. Thanks!
echo -n "Service Name: "
read serviceName
echo -n "Comment: "
read commentValue
for check in $(grep "CURRENT SERVICE STATE" $nagiosLog |grep -w "$serviceName" | nawk -F": " '{print $2}' |sort -u ) ; do
echo $check | nawk -F";" -v now=$now '{ printf( "[%u]=ACKNOWLEDGE_SVC_PROBLEM;"$1";"$2";2;1;0;admin;$commentValue"\n", now)}' >> $nagiosCommand
done
$commentValue is inside an invocation to nawk, so it is considered as a variable in nawk, not a variable in bash. Since you do not have such a variable in nawk, you won't get anything there. You should first pass the variable "inside" nawk using the -v switch just like you did for the now variable; i.e.:
... | nawk -F";" -v now=$now -v "commentValue=$commentValue"
Note the quotes - they are required in case $commentValue contains whitespace.