Is it possible to store an awk script inside a shell variable; for example:
export script="'{printf(\$2); printf("\"\\n\"");}'"
echo $script
'{printf($2); printf("\n");}'
The script functions properly when I call it directly as such:
awk '{printf($2); printf("\n");}' testFile.txt
prints proper output
When I try and pass the script as a shell variable, I run into issues.
awk $script testFile.txt
awk: syntax error at source line 1
context is
>>> ' <<<
missing }
awk: bailing out at source line 1
I get a slightly different error when I wrap the variable in double quotes
awk "$script" testFile.txt
awk: syntax error at source line 1
context is
>>> ' <<<
awk: bailing out at source line 1
I'm still learning exactly how shell expansions work, I would appreciate any suggestions about what I am missing here.
Error in your quoting
export script='{printf($2); printf("\n");}'
awk "${script}" YourFile
I am not sure about the proper answer to this, but a very ugly (and probably unstable depending on the $script contents) workaround would be:
echo $script | awk '{print "awk "$0" testFile.txt"}' | bash
This is just printing the contents of $script in an awk statement that is then executed by bash. I am not particularly proud of this, but maybe it helps!
When you type
awk '{printf($2); printf("\n");}' testFile.txt
awk only sees {printf($2); printf("\n");} -- the shell removes the quotes
(see Quote Removal in the bash manual)
Heed #NeronLeVelu's answer.
Related
I'm sure this is simple, but I'm new to bash scripts and the syntactical process here is beyond me. I can't seem to find the right search terms to find what I need. This script is really just a stepping stone to my final version.
Invocation: ./myscript.sh testFile
Script:
#!/bin/bash
file=$1
awk='{print $9}' # do not expand $9
awk="'/$file/$awk'" # DO expand file argument
echo "$awk" # prints '/graphic/{print $9}' (as expected)
echo "ls -l | awk $awk" # prints ls -l | awk '/graphic/{print $9}' (as expected)
test="$(ls -l | awk $awk)" # error
echo "$test"
Output:
'/testFile/{print $9}'
ls -l | awk '/testFile/{print $9}'
awk: syntax error at source line 1
context is
>>> ' <<<
missing }
awk: bailing out at source line 1
Even though I can copy and run the second echo'd line and it works successfully, the failure of the command leads me to believe this is not simple string concat but some crazier voodoo.
I've tried some other version as well like making a variable containing the whole command, but then I get even less expected output.
If I do test="$($awk)" I get
'/testFile/{print $9}'
ls -l | awk '/testFile/{print $9}'
ls: $9}': No such file or directory
ls: '/testFile/{print: No such file or directory
ls: awk: No such file or directory
ls: |: No such file or directory
If I do test=$(awk) I get
'/testFile/{print $9}'
ls -l | awk '/testFile/{print $9}'
usage: awk [-F fs] [-v var=value] [-f progfile | 'prog'] [file ...]
Since my Google queries basically only contain the words "bash command variable assignment", I can't get anything related to the nested variable expansion that I have here. I understand what it's doing based on the error, but I couldn't say why or how to fix it.
If someone could provide a fix as well as explain or point me to a resource explaining what's going on here, it would be greatly appreciated. Or maybe there's even another approach that would simplify the logic.
Thanks!
Change to:
test="$(ls -l | awk "$awk")" # error
awk requires the script to be a single argument. But when you expand a variable outside double quotes, the shell performs word splitting, so $awk is expanded into two arguments:
'{print
$9}'
The quotes keep the expansion as a single argument.
Also, take the single quotes out of
awk="'/$file/$awk'"
Single quotes are not processed after expanding a variable, so they'll be passed literally to awk. Putting double quotes around $awk achieves the result you were trying to get with these quotes.
I am new to shell scripting and I am encountering some issue with my codes. Currently, I have this line of code which is able to run perfectly on my terminal which will give me the result which I am looking for:
cat BookDB.txt | awk -F ":" '$1 ~ $Title'
However, when I try to implement this into my script, no result is shown. Anyone able to help me with this problem?
Is $Title supposed to be a shell variable? If so, the shell can't substitute it because the awk body is in single quotes. Use awk's -v option to pass shell variables into awk:
awk -F : -v "title=$Title" '$1 ~ title' BookDB.txt
See also https://en.wikipedia.org/wiki/Cat_(Unix)#Useless_use_of_cat
Can you help me with this script ?
#!/bin/bash
data=`date '+%Y %m %d'`
data2=" 7 0 0"
string=$data$data2
awk -v str=$string '{print mktime("str");}'
I´m getting this output:
awk: cmd. line:1: fatal: cannot open file `09' for reading (No such file or directory)
You should not quote str inside awk otherwise it is treated as a literal string. You must also quote the shell variables properly.
#!/bin/bash
data=$(date '+%Y %m %d')
data2=" 7 0 0"
string="$data$data2"
awk -v str="$string" 'BEGIN {print mktime(str)}'
#anubhava has the right answer. Here's why: after variable substitution, the shell sees this line in your code:
awk -v str=2014 01 09 7 0 0 '{print mktime("str");}'
So, you're passing "2014" as the "str" variable, your awk script is "01" (which is a valid awk program that will print every line), and passing the file arguments "09", "7", ... Hence the error "cannot open file `09' for reading"
Lesson: in shell scripting, unless you know when not to, always quote your "$variables".
You want this:
#!/bin/bash
data="`date '+%Y %m %d'`"
data2=" 7 0 0"
string="$data$data2"
awk -v str="$string" 'BEGIN {print mktime(str);exit;}'
Another alternative for the awk line is: (This has the advantage of working on older awk versions that lack the -v option, like Solaris 10 /usr/bin/awk (They tend to miss mktime though))
awk "BEGIN {print mktime(\"$string\");exit;}"
The BEGIN block is needed to make awk run the code at startup and not for every input line.
The exit is needed to make it quit before it waits for input. (It seems to work without it on GAWK)
The quotes are needed to pass the variable as a single entry... This overview of shell quoting is important reading for anyone attempting shell scripting...
The quotes in the awk code needs to be left out to get awk to use the variable str and not the string str.
You likely also want to replace the backticks (``) with $(), it is much easier to read in complicated, nested cases and supported in all modern shells (It seems like it might be in the POSIX standard, I haven't checked though...).. (It doesn't affect the code in this case though)
Here is my file.sh bash script:
#!/bin/bash
$(awk '/[ \t]+'$1'\/'$2'/' /etc/servicies) | awk '{print '$1'}';
if I run:
file.sh 21 tcp
it should print ftp. But I get that error:
./file.sh: line 2: ftp: command not found
Can anybody explain me a little bit why that error and how to fix it?
Thanks.
awk '/[ \t]+'$1'\/'$2'/ { print $1 }' /etc/services
Why do you use the $( ... ) construct? It is replaced by the output of the enclosed commands by bash, so your pipeline becomes something like
ftp 21/tcp xxx_master # ftp / XXX
which triggers the mentioned error.
Solution: Just remove the $( and ) and everything should work as you expected.
awk '$2==A && $3=B, NF=1' FPAT='[^ /]+' A=$1 B=$2 /etc/services
kinda hard to understand but...
$(whatever) executes its output...
So, assuming that
$(awk '/[ \t]+'$1'\/'$2'/' /etc/services)
the part inside the $() gets you as output
ftp blahblahblah
it will first try to execute it and only after pass the execution output to the second awk piping.
Just remove the $() syntax, in that way the output of the first awk will be sent to the later steps.
Btw, please, instead of doing
cat xxxx | dosomething
try doing
dosomething xxxxx
I have a shell script that constructs an awk program as a string then pass that string to awk. This is because I want to use values of shell variables in the awk program.
My code looks like this:
awk_prog="'{if (\$4~/$shell_var/) print \$1,\$2}'"
echo $awk_prog
awk $awk_prog $FILENAME
However, when I pass the string to awk, I always get the error:
'{if ($4~/regex/) print $1,$2}'
awk: '{if
awk: ^ invalid char ''' in expression
What does that error message mean? I tried the -F: switch but it does not help. How can I settle this issue?
Thank you.
This is caused by shell quoting. The following will work:
awk_prog="{ if (\$4 ~ /$shell_var/) print \$1, \$2 }"
echo "$awk_prog"
awk "$awk_prog" $FILENAME
When you run awk '{ print }' foo from the command line, the shell interprets and removes the quotes around the program so awk receives two arguments - the first is the program text and the second is the filename foo. Your example was sending awk the program text '{if ...}' which is invalid syntax as far as awk is concerned. The outer quotes should not be present.
In the snippet that I gave above, the shell uses the quotes in the awk_prog= line to group the contents of the string into a single value and then assigns it to the variable awk_prog. When it executes the awk "$awk_prog"... line, you have to quote the expansion of $awk_prog so awk receives the program text as a single argument.
There's another way to get your shell variable into awk -- use awk's -v option:
awk -v pattern="$shell_var" '$4 ~ pattern {print $1, $2}' "$FILENAME"
Use -v multiple times if you have several variables to pass to awk.
If you truly want to hold your awk program in a shell variable, build it up using printf:
awk_script="$( printf '$4 ~ /%s/ {print $1, $2}' "$shell_var" )"
awk "$awk_script" "$FILENAME"
Note the use of quotes in the printf command: single quotes around the template to protect the dollar signs you want awk to interpret, double quotes for shell variables.
Another (IMO simpler) solution which (I think) addresses what you are intuitively trying to do is simply to use eval. You want the shell to behave as if you had literally typed:
awk '{if ($4~/foo/) print $1,$2}' path
(where foo and path are the literal contents of $shell_var and $FILENAME). To make that happen, just slap an eval on the front of your last line (and perhaps quotes for good measure, but they aren't necessary in this case) so that your last line is:
eval "awk $awk_prog $FILENAME"