I'm trying to extract a string from a group of strings stored in a variable, say foo, in bash:
foo="I foobar you"
using awk, but I got different results when I wrapped awk with single and double quotes:
$ echo $foo | awk '{print $2}'
$ foobar
$ echo $foo | awk "{print $2}"
$ I foobar you
Could anyone tell me why ' and " cause different results when wrapping awk?
It is because when using double-quotes the the shell tries to expand it as a positional parameter or a variable before passing it to awk. In your case it tries to expand $1 but finds no value for it, and so the expansion results to nothing and that's why you get the default print action in awk to print the whole line.
The reason why we single quote the awk commands is that we want a plain string and not modified/tampered by the shell by any means before processed by the Awk command itself which understands $1 as the first field (with default space delimiter)
According to shell quoting rules:
Single quotes (‘ ‘) protect everything between the opening and closing quotes. The shell does no interpretation of the quoted text, passing it on verbatim to the command.
As in your code {print $2} passed on as it is to awk as an action:
$ echo $foo | awk '{print $2}'
$ foobar
Double quotes (“ “) protect most things between the opening and closing quotes. The shell does at least variable and command substitution on the quoted text. Different shells may do additional kinds of processing on double-quoted text. Because certain characters within double-quoted text are processed by the shell, they must be escaped within the text. Of note are the characters ‘$’, ‘‘’, ‘\’, and ‘"’, all of which must be preceded by a backslash within double-quoted text if they are to be passed on literally to the program.
So you have to escape $ to get the value of second col as:
$ echo $foo | awk "{print \$2}"
$ foobar
otherwise awk is doing its default action of printing the whole line as in your case.
Fully agreed with #Ed Mortan regarding the best practice of quoting the shell variables. Although the double quotes are default, but if accidentally ….
$ echo "$foo" with double quotes
$ I foobar you
$ echo ‘$foo’ with single quotes
$ $foo
Remember, quoting rules are specific to shell. The above rules only apply to POSIX-complaint, Bourne-style shells (such as Bash, the GNU Bourne-Again Shell). It has nothing to do with awk, sed, grep, etc…
Related
I want to issue this command from the bash script
sed -e $beginning,$s/pattern/$variable/ file
but any possible combination of quotes gives me an error, only one that works:
sed -e "$beginning,$"'s/pattern/$variable/' file
also not good, because it do not dereferences the variable.
Does my approach can be implemented with sed?
Feel free to switch the quotes up. The shell can keep things straight.
sed -e "$beginning"',$s/pattern/'"$variable"'/' file
You can try this:
$ sed -e "$beginning,$ s/pattern/$variable/" file
Example
file.txt:
one
two
three
Try:
$ beginning=1
$ variable=ONE
$ sed -e "$beginning,$ s/one/$variable/" file.txt
Output:
ONE
two
three
There are two types of quotes:
Single quotes preserve their contents (> is the prompt):
> var=blah
> echo '$var'
$var
Double quotes allow for parameter expansion:
> var=blah
> echo "$var"
blah
And two types of $ sign:
One to tell the shell that what follows is the name of a parameter to be expanded
One that stands for "last line" in sed.
You have to combine these so
The shell doesn't think sed's $ has anything to do with a parameter
The shell parameters still get expanded (can't be within single quotes)
The whole sed command is quoted.
One possibility would be
sed "$beginning,\$s/pattern/$variable/" file
The whole command is in double quotes, i.e., parameters get expanded ($beginning and $variable). To make sure the shell doesn't try to expand $s, which doesn't exist, the "end of line" $ is escaped so the shell doesn't try anything funny.
Other options are
Double quoting everything but adding a space between $ and s (see Ren's answer)
Mixing quoting types as needed (see Ignacio's answer)
Methods that don't work
sed '$beginning,$s/pattern/$variable/' file
Everything in single quotes: the shell parameters are not expanded (doesn't follow rule 2 above). $beginning is not a valid address, and pattern would be literally replaced by $variable.
sed "$beginning,$s/pattern/$variable/" file
Everything in double qoutes: the parameters are expanded, including $s, which isn't supposed to (doesn't follow rule 1 above).
the following form worked for me from within script
sed $beg,$ -e s/pattern/$variable/ file
the same form will also work if executed from the shell
I'm almost certain the code I have here worked before. Here's a simplified version and what it produces:
a="atext"
b="btext"
var=$'${a}\n${b}\n'
printf "var=$var"
Which produces output:
var=${a}
${b}
The real code outputs var to file, but the variable expansions aren't happening for some reason.
If this can't work, can you suggest a nice alternative way, and why one uses $' '? Thanks.
GNU bash, version 4.3.42
$'' is a quoting type used to allow backslash escape sequences to describe literal strings with nonprintable characters and other such oddities. Thus, $'\n' evaluates to a single character -- a newline -- whereas '\n' and "\n" both evaluate to two characters, the first being a backslash and the second being an n.
If you want to have the exact behavior of your original code -- putting a literal newline between the results of two different expansions -- you can switch quote types partway through a string:
a="atext"
b="btext"
var="$a"$'\n'"$b"
printf '%s' "var=$var"
That is, right next to each other, with no spaces between:
"$a"
$'\n'
"$b"
This gives you $a and $b expanded, with a literal newline between them.
Why does this matter? Try the following:
$ a=atext
$ b=btext
$ var1="$a\n$b" # Assign with literal "\" and "n" characters
$ printf "$var1" # Here, printf changes the "\n" into the newline
atext
btext
$ printf '%s' "$var1" # ...but this form shows that the "\n" are really there
atext\nbtext
$ var2="$a"$'\n'"$b" # now, we put a single newline in the string
$ printf '%s' "$var2" # and now even accurate use of printf shows that newline
atext
btext
Just replace the single quotes with double quotes.
$ cat test
a="atext"
b="btext"
var=$"${a}\n${b}\n"
printf "var=$var"
$ sh test
var=atext
btext
For variable expansion you either need to use double quotes or no quotes. Single quotes negate expansion.
I am new in linux and AWK as well. I have a file in my home folder called testing.txt and i am trying to read the file using this awk command:
**arjun#arjun-Aspire-4741:~$ awk ´{print $1}´ testing.txt¨**
And I am getting this as output
**¨awk: ´{print
awk: ^ invalid char '�' in expression
arjun#arjun-Aspire-4741:~$ ¨**
The problem is that you've used forward ticks instead of quotes (in this case only single quotes are appropriate):
awk '{print $1}' testing.txt
instead of
awk ´{print $1}´ testing.txt
In shell, strings in double quotes " can contain expressions with special meaning (such as backticks, variables) which will be expanded before the string is processed as part of the full shell command. Strings in single quotes ' are fully escaped; to put it another way, the string is passed literally without any interpretation. That's why you should use single quotes when writing awk scripts, because the awk variable dereference operator $ is the same as in shell. There are no other valid string-delimiting characters*.
I initially thought you'd used backticks (thanks to Andras Deak for spotting my error).
Backticks have a special meaning in shell (equivalent to wrapping something in $(...)): execute this string as a command, and evaluate to its output (stdout). This is done before your main command is executed.
So, if I do
cat `echo myfile`
this turns into
cat myfile
which then executes.
You can read more about shell behaviour in a few places:
http://www.grymoire.com/Unix/Sh.html
http://www.tldp.org/LDP/Bash-Beginners-Guide/html/index.html
* ignoring that spaces are also technically string-delimiters
I'm trying to use a variable in a grep regex. I'll just post an example of the failure and maybe someone can suggest how to make the variable be evaluated while running the grep command. I've tried ${var} as well.
$ string="test this"
$ var="test"
$ echo $string | grep '^$var'
$
Since my regex should match lines which start with "test", it should print the line echoed thru it.
$ echo $string
test this
$
You need to use double quotes. Single quotes prevent the shell variable from being interpolated by the shell. You use single quotes to prevent the shell from doing interpolation which you may have to do if your regular expression used $ as part of the pattern. You can also use a backslash to quote a $ if you're using double quotes.
Also, you may need to put your variable in curly braces ${var} in order to help separate it from the rest of the pattern.
Example:
$ string="test this"
$ var="test"
$ echo $string | grep "^${var}"
I am trying to escape backslash in AWK. This is a sample of what I am trying to do.
Say, I have a variable
$echo $a
hi
The following works
$echo $a | awk '{printf("\\\"%s\"",$1)'}
\"hi"
But, when I am trying to save the output of the same command to a variable using command substitution, I get the following error:
$ q=`echo $a | awk '{printf("\\\"%s\"",$1)'}`
awk: {printf("\\"%s\"",$1)}
awk: ^ backslash not last character on line
I am unable to understand why command substitution is breaking the AWK. Thanks a lot for your help.
Try this:
q=$(echo $a | awk '{printf("\\\"%s\"",$1)}')
Test:
$ a=hi
$ echo $a
hi
$ q=$(echo $a | awk '{printf("\\\"%s\"",$1)}')
$ echo $q
\"hi"
Update:
It will, it just gets a littler messier.
q=`echo $a | awk '{printf("\\\\\"%s\"",$1)}'`
Test:
$ b=hello
$ echo $b
hello
$ t=`echo $b | awk '{printf("\\\\\"%s\"",$1)}'`
$ echo $t
\"hello"
Reference
Quoting inside backquoted commands is somewhat complicated, mainy
because the same token is used to start and to end a backquoted
command. As a consequence, to nest backquoted commands, the backquotes
of the inner one have to be escaped using backslashes. Furthermore,
backslashes can be used to quote other backslashes and dollar signs
(the latter are in fact redundant). If the backquoted command is
contained within double quotes, a backslash can also be used to quote a
double quote. All these backslashes are removed when the shell reads
the backquoted command. All other backslashes are left intact.
The new $(...) avoids these troubles.
Don't get into bad habits with backticks, quoting and parsing shell variables to awk The correct way to do this is:
$ shell_var="hi"
$ awk -v awk_var="$shell_var" -v c='\' 'BEGIN{printf "%s%s\n",c,awk_var}'
\hi
$ res=$(awk -v awk_var="$shell_var" -v c='\' 'BEGIN{printf "%s%s\n",c,awk_var}')
$ echo "$res"
\hi