Difference between single and double quotes in awk - bash

I have this awk statement:
glb_library="my_library"
awk "
/^Direct Dependers of/ { next }
/^---/ { next }
/^$glb_library:/ { ver=\$0; next }
{ gsub(/[[:space:]]/, '', \$0); print ver':'\$0 }
" file
Basically, I have enclosed the awk code in double quotes so that the shell variable glb_library is expanded. I have made sure to escape the $ character to prevent the shell from expanding $0. Followed the guidance from here.
awk gives me this error:
awk: syntax error at source line 5
context is
{ gsub(/[[:space:]]/, >>> ' <<<
I want to understand:
Is it legal to use single quotes inside awk? Why is '' not a null string like "" is?
Does awk treat single and double quotes differently?
My code worked after I escaped the single quotes with backslashes and used \"\" to represent the null string instead of ''.

Never enclose any script in double quotes or you're sentencing yourself to backslash-hell. This is the syntax for what you're trying to do:
glb_library="my_library"
awk -v glb_library="$glb_library" '
/^Direct Dependers of/ { next }
/^---/ { next }
$0 ~ "^"glb_library":" { ver=$0; next }
{ gsub(/[[:space:]]/, ""); print ver":"$0 }
' file

Based on the comments above by awk experts and some research, I am posting this answer:
awk strings are enclosed in double quotes, not single quotes; more precisely: single quotes are not string delimiters in awk, unlike shell
awk attaches no special meaning to single quotes and they need to be enclosed in double quotes if used in string literals
it is best to use single quotes to wrap awk statements on command line, unlike OP's code that's using double quotes (Ed pointed this out clearly)
Further clarification:
"" is the null string in awk, not ''
to use single quotes in an awk string literal, enclose them in double quotes, as in "Ed's answers are great!"
other techniques followed while handling single quotes in awk are:
a) use a variable, as in awk -v q="'" '{ print q }' ...
b) use octal or hex notation, as in awk '{ print "\047"$0"\047" }' ...
Relevant documentation here.

A pragmatic summary:
As Ed Morton's helpful answer sensibly recommends:
Always use single quotes to enclose your awk script as a whole ('...'), which ensures that there's no confusion over what the shell interprets up front, and what awk ends up seeing.
To define strings inside an awk script, always use double quotes ("...").
" is the only string delimiter awk recognizes.
"..." strings are non-interpolating (you cannot embed variable references), but they do recognize control-character sequences such as \n and \t.
A single quote (') has no syntactic meaning inside an awk script, but, - if you're using '...' for your overall script, as recommended - you cannot use a literal ' inside of it anyway, because the shell's single-quoted strings do not permit embedded ' chars.
If you do need to use a literal single quote (') in your awk script, you have three choices:
Pass a variable that defines it, and use awk's string concatenation, based on directly adjoining string literals and variable references:
awk -v q=\' 'BEGIN { print "I" q "m good." }' # -> I'm good
Use an escape sequence inside "..."; for maximum portability and disambiguation, use an octal escape sequence (\047), not a hex one (\x27):
awk 'BEGIN { print "I\047m good." }' # -> I'm good
Use '\'' (sic) to "escape" embedded ' chars. (technically, 3 distinct single-quoted shell string literals are being concatenated)Thanks, snr:awk 'BEGIN { print "I'\''m good" }' # -> I'm good

Related

Replacing a string with the value of a variable using shell script

I want to replace a string in a file with the value of a variable. I a string lvl in the template file prm.prm which needs to be replaced by the value of SLURM_ARRAY.
I tried using
sed -i 's/lvl/${SLURM_ARRAY}/' prm.prm
This replaces the string lvl with ${SLURM_ARRAY} and not its value. How can I rectify this?
Every character between single quotes is used literally.
You could use double quotes instead as follows:
sed -i "s/lvl/${SLURM_ARRAY}/" prm.prm
However, your code now suffers from a code injection bug. There are characters (e.g. /) that will cause problems if found in the value of SLURM_ARRAY. To avoid this, these characters needs to be escaped.
quotemeta() { printf %s "$1" | sed 's/\([^a-zA-Z0-9]\)/\\\1/g'; }
sed -i "s/lvl/$( quotemeta "$SLURM_ARRAY" )/" prm.prm
However, it would be best to avoid generating programs from the shell. But that would require avoiding sed since it doesn't provide the necessary tools. For example, a couple of Perl solutions:
perl -i -spe's/lvl/$s/' -- -s="$SLURM_ARRAY" prm.prm
S="$SLURM_ARRAY" perl -i -pe's/lvl/$ENV{S}/' prm.prm
Replace pattern with the value of an environment variable, with no extra interpolation of the contents:
Using perl:
export SLURM_ARRAY
perl -pe's/lvl/$ENV{SLURM_ARRAY}/g' prm.prm
Using awk:
export SLURM_ARRAY
awk '
{
if (match($0, /lvl/)) {
printf "%s", substr($0, 1, RSTART - 1)
printf "%s", ENVIRON["SLURM_ARRAY"]
print substr($0, RSTART + RLENGTH)
}
else {
print
}
}
' prm.prm
There's also SLURM_ARRAY=$SLURM_ARRAY perl ...etc or similar, to set the environment of a single process.
It can also be done with the variable as an argument. With both perl and awk you can access and modify the ARGV array. For awk you have to reset it so it's not processed as a file. The perl version looks like perl -e 'my $r = $ARGV[0]; while (<STDIN>) {s/lvl/$r/g; print}' "$SLURM_ARRAY" < prm.prm. It looks even better as perl -spe's/lvl/$r/g' -- -r="$SLURM_ARRAY". Thanks to ikegami.
For awk, I should say that the reason for not using awk -v r=foo is the expansion of C escapes. You could also read the value from a file (or bash process substitution).

How to concatenate string variables that contain space? [duplicate]

How can I escape double quotes inside a double string in Bash?
For example, in my shell script
#!/bin/bash
dbload="load data local infile \"'gfpoint.csv'\" into table $dbtable FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY \"'\n'\" IGNORE 1 LINES"
I can't get the ENCLOSED BY '\"' with double quote to escape correctly. I can't use single quotes for my variable, because I want to use variable $dbtable.
Use a backslash:
echo "\"" # Prints one " character.
A simple example of escaping quotes in the shell:
$ echo 'abc'\''abc'
abc'abc
$ echo "abc"\""abc"
abc"abc
It's done by finishing an already-opened one ('), placing the escaped one (\'), and then opening another one (').
Alternatively:
$ echo 'abc'"'"'abc'
abc'abc
$ echo "abc"'"'"abc"
abc"abc
It's done by finishing already opened one ('), placing a quote in another quote ("'"), and then opening another one (').
More examples: Escaping single-quotes within single-quoted strings
Keep in mind that you can avoid escaping by using ASCII codes of the characters you need to echo.
Example:
echo -e "This is \x22\x27\x22\x27\x22text\x22\x27\x22\x27\x22"
This is "'"'"text"'"'"
\x22 is the ASCII code (in hex) for double quotes and \x27 for single quotes. Similarly you can echo any character.
I suppose if we try to echo the above string with backslashes, we will need a messy two rows backslashed echo... :)
For variable assignment this is the equivalent:
a=$'This is \x22text\x22'
echo "$a"
# Output:
This is "text"
If the variable is already set by another program, you can still apply double/single quotes with sed or similar tools.
Example:
b="Just another text here"
echo "$b"
Just another text here
sed 's/text/"'\0'"/' <<<"$b" #\0 is a special sed operator
Just another "0" here #this is not what i wanted to be
sed 's/text/\x22\x27\0\x27\x22/' <<<"$b"
Just another "'text'" here #now we are talking. You would normally need a dozen of backslashes to achieve the same result in the normal way.
Bash allows you to place strings adjacently, and they'll just end up being glued together.
So this:
echo "Hello"', world!'
produces
Hello, world!
The trick is to alternate between single and double-quoted strings as required. Unfortunately, it quickly gets very messy. For example:
echo "I like to use" '"double quotes"' "sometimes"
produces
I like to use "double quotes" sometimes
In your example, I would do it something like this:
dbtable=example
dbload='load data local infile "'"'gfpoint.csv'"'" into '"table $dbtable FIELDS TERMINATED BY ',' ENCLOSED BY '"'"'"' LINES "'TERMINATED BY "'"'\n'"'" IGNORE 1 LINES'
echo $dbload
which produces the following output:
load data local infile "'gfpoint.csv'" into table example FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY "'\n'" IGNORE 1 LINES
It's difficult to see what's going on here, but I can annotate it using Unicode quotes. The following won't work in Bash – it's just for illustration:
dbload=‘load data local infile "’“'gfpoint.csv'”‘" into ’“table $dbtable FIELDS TERMINATED BY ',' ENCLOSED BY '”‘"’“' LINES ”‘TERMINATED BY "’“'\n'”‘" IGNORE 1 LINES’
The quotes like “ ‘ ’ ” in the above will be interpreted by bash. The quotes like " ' will end up in the resulting variable.
If I give the same treatment to the earlier example, it looks like this:
echo “I like to use” ‘"double quotes"’ “sometimes”
Store the double quote character in a variable:
dqt='"'
echo "Double quotes ${dqt}X${dqt} inside a double quoted string"
Output:
Double quotes "X" inside a double quoted string
Check out printf...
#!/bin/bash
mystr="say \"hi\""
Without using printf
echo -e $mystr
Output: say "hi"
Using printf
echo -e $(printf '%q' $mystr)
Output: say \"hi\"
Make use of $"string".
In this example, it would be,
dbload=$"load data local infile \"'gfpoint.csv'\" into table $dbtable FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY \"'\n'\" IGNORE 1 LINES"
Note (from the man page):
A double-quoted string preceded by a dollar sign ($"string") will cause the string to be translated according to the current locale. If the current locale is C or POSIX, the dollar sign is ignored. If the string is translated and replaced, the replacement is double-quoted.
For use with variables that might contain spaces in you Bash script, use triple quotes inside the main quote, e.g.:
[ "$(date -r """$touchfile""" +%Y%m%d)" -eq "$(date +%Y%m%d)" ]
Add "\" before double quote to escape it, instead of \
#! /bin/csh -f
set dbtable = balabala
set dbload = "load data local infile "\""'gfpoint.csv'"\"" into table $dbtable FIELDS TERMINATED BY ',' ENCLOSED BY '"\""' LINES TERMINATED BY "\""'\n'"\"" IGNORE 1 LINES"
echo $dbload
# load data local infile "'gfpoint.csv'" into table balabala FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY "''" IGNORE 1 LINES

Bash script double quote escaping

I'm trying to run a command in a bash script using a variable as a parameter, but I am failing to correctly escape the quotes.
This is what I have have:
vsnames=$(comm -13 --nocheck-order $cur_ha_stat_report $old_ha_stat_report | awk -F\| '{print $1}' | tr '\r\n' ' ')
create event type=0xfff00033 text="$vsnames" mh=$dev_mh &
Where vsnames is a string with words separated by spaces.
When I run the script, this is what I get:
+ vsnames='VS1 VS2 '
+ create event type=0xfff00033 'text=VS1 VS2 ' mh=xxx
How to I put the $vsnames variable between quotes?
I have tried using \" but it didn't work, bash just added a bunch of unwanted single-quotes.
Any tips?
The command I actually need is:
create event type=0xfff00033 text="VS1 VS2 " mh=xxx
With your variable vsnames being set to the 8-character-string VS1 VS2 (as we can see from the trace), doing a
create event type=0xfff00033 text=\""$vsnames"\" mh=xxx
would set the argv[2] of the create command to
text="VS1 VS2 "
Explanation: I have enclosed $vsnames into single quotes, to tell bash that we don't want to do word splitting here. Then I wrapped this between literal \" quotes, because you want create to see these quotes.

Nested dollar signs inside quotes

Trying to write a bash script containing nested dollar variables and I can't get it to work :
#!/bin/bash
sed '4s/.*/$(grep "remote.*$1" /home/txtfile)/' /home/target
The error says :
sed / -e expression #1, char 30: unkown option to 's'
The problem seems to come from $1 which need to be replaced by the parameter passed from the bash call and then the whole $(...) needs to be replaced by the command call so we replace the target line 4 by the string output.
Variable expansion and Command substitution won't be done when put inside single quotes, use double quotes instead:
sed "4s/.*/$(grep "remote.*$1" /home/txtfile)/" /home/target
Your approach is wrong, the right way to do what you want is just one command, something like this (depending on your possible $1 values and input file contents which you haven't shown us):
awk -v tgt='remote.*$1' '
NR==FNR { if ($0 ~ tgt) str = str $0 ORS; next }
FNR==4 { printf "%s", str; next }
{ print }
' /home/txtfile /home/target

How can I escape a double quote inside double quotes?

How can I escape double quotes inside a double string in Bash?
For example, in my shell script
#!/bin/bash
dbload="load data local infile \"'gfpoint.csv'\" into table $dbtable FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY \"'\n'\" IGNORE 1 LINES"
I can't get the ENCLOSED BY '\"' with double quote to escape correctly. I can't use single quotes for my variable, because I want to use variable $dbtable.
Use a backslash:
echo "\"" # Prints one " character.
A simple example of escaping quotes in the shell:
$ echo 'abc'\''abc'
abc'abc
$ echo "abc"\""abc"
abc"abc
It's done by finishing an already-opened one ('), placing the escaped one (\'), and then opening another one (').
Alternatively:
$ echo 'abc'"'"'abc'
abc'abc
$ echo "abc"'"'"abc"
abc"abc
It's done by finishing already opened one ('), placing a quote in another quote ("'"), and then opening another one (').
More examples: Escaping single-quotes within single-quoted strings
Keep in mind that you can avoid escaping by using ASCII codes of the characters you need to echo.
Example:
echo -e "This is \x22\x27\x22\x27\x22text\x22\x27\x22\x27\x22"
This is "'"'"text"'"'"
\x22 is the ASCII code (in hex) for double quotes and \x27 for single quotes. Similarly you can echo any character.
I suppose if we try to echo the above string with backslashes, we will need a messy two rows backslashed echo... :)
For variable assignment this is the equivalent:
a=$'This is \x22text\x22'
echo "$a"
# Output:
This is "text"
If the variable is already set by another program, you can still apply double/single quotes with sed or similar tools.
Example:
b="Just another text here"
echo "$b"
Just another text here
sed 's/text/"'\0'"/' <<<"$b" #\0 is a special sed operator
Just another "0" here #this is not what i wanted to be
sed 's/text/\x22\x27\0\x27\x22/' <<<"$b"
Just another "'text'" here #now we are talking. You would normally need a dozen of backslashes to achieve the same result in the normal way.
Bash allows you to place strings adjacently, and they'll just end up being glued together.
So this:
echo "Hello"', world!'
produces
Hello, world!
The trick is to alternate between single and double-quoted strings as required. Unfortunately, it quickly gets very messy. For example:
echo "I like to use" '"double quotes"' "sometimes"
produces
I like to use "double quotes" sometimes
In your example, I would do it something like this:
dbtable=example
dbload='load data local infile "'"'gfpoint.csv'"'" into '"table $dbtable FIELDS TERMINATED BY ',' ENCLOSED BY '"'"'"' LINES "'TERMINATED BY "'"'\n'"'" IGNORE 1 LINES'
echo $dbload
which produces the following output:
load data local infile "'gfpoint.csv'" into table example FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY "'\n'" IGNORE 1 LINES
It's difficult to see what's going on here, but I can annotate it using Unicode quotes. The following won't work in Bash – it's just for illustration:
dbload=‘load data local infile "’“'gfpoint.csv'”‘" into ’“table $dbtable FIELDS TERMINATED BY ',' ENCLOSED BY '”‘"’“' LINES ”‘TERMINATED BY "’“'\n'”‘" IGNORE 1 LINES’
The quotes like “ ‘ ’ ” in the above will be interpreted by bash. The quotes like " ' will end up in the resulting variable.
If I give the same treatment to the earlier example, it looks like this:
echo “I like to use” ‘"double quotes"’ “sometimes”
Store the double quote character in a variable:
dqt='"'
echo "Double quotes ${dqt}X${dqt} inside a double quoted string"
Output:
Double quotes "X" inside a double quoted string
Check out printf...
#!/bin/bash
mystr="say \"hi\""
Without using printf
echo -e $mystr
Output: say "hi"
Using printf
echo -e $(printf '%q' $mystr)
Output: say \"hi\"
Make use of $"string".
In this example, it would be,
dbload=$"load data local infile \"'gfpoint.csv'\" into table $dbtable FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY \"'\n'\" IGNORE 1 LINES"
Note (from the man page):
A double-quoted string preceded by a dollar sign ($"string") will cause the string to be translated according to the current locale. If the current locale is C or POSIX, the dollar sign is ignored. If the string is translated and replaced, the replacement is double-quoted.
For use with variables that might contain spaces in you Bash script, use triple quotes inside the main quote, e.g.:
[ "$(date -r """$touchfile""" +%Y%m%d)" -eq "$(date +%Y%m%d)" ]
Add "\" before double quote to escape it, instead of \
#! /bin/csh -f
set dbtable = balabala
set dbload = "load data local infile "\""'gfpoint.csv'"\"" into table $dbtable FIELDS TERMINATED BY ',' ENCLOSED BY '"\""' LINES TERMINATED BY "\""'\n'"\"" IGNORE 1 LINES"
echo $dbload
# load data local infile "'gfpoint.csv'" into table balabala FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY "''" IGNORE 1 LINES

Resources