bash script - awk not processing variables passed as regex [duplicate] - bash

This question already has answers here:
How do I use shell variables in an awk script?
(7 answers)
Closed 5 years ago.
I am writing a bash script. The following awk command does not print anything out even when there is a match.
I have copied the awk line and it executes successfully in the interactive shell when I replace $SEARCH_QUERY with real data. This leads me to believe that awk is not translating the data in the variable $SEARCH_QUERY correctly.
Any ideas how to fix this or what bash topics I should read up on to understand why it is not working? Thank you
search_contacts()
{
echo "Enter contact Name:"
read SEARCH_QUERY
awk ' /$SEARCH_QUERY/ { print $0 }' contacts.db
}

In terms of why the original code was broken -- single quotes suppress parameter expansion, so awk was treating $SEARCH_QUERY as a regex itself, ie. looking for a line with the string SEARCH_QUERY after its end -- a regex that can't possibly ever match.
Thus, the naive approach would be to change quoting types, such that your intended substitution actually takes place:
# THIS IS DANGEROUSLY INSECURE; DON'T DO THIS EVER
awk "/$SEARCH_QUERY/" contacts.db
However, as per the comment, the above is dangerously insecure; if your SEARCH_QUERY value ended the regex (included a /), it could then run any arbitrary awk code it wanted. nonmatch/ || system("rm -rf ~") || /anything would do Very Bad Things.
In terms of the best practice -- data should be passed out-of-band from code:
awk -v search_query="$SEARCH_QUERY" '$0 ~ search_query' contacts.db

Related

Bash script Using a variable in awk [duplicate]

This question already has answers here:
How do I use shell variables in an awk script?
(7 answers)
Closed 6 years ago.
I have a variable LINE and want to use it with awk to pull out the contents of the line numbered LINE from table.txt and make that a new variable called NAME which is then used to grep another file.
NAME=`awk 'FNR==$LINE' < table.txt`
echo "this is $NAME"
Seems to be close, but not quite the syntax.
If I use:
NAME=`awk 'FNR==1' < table.txt`
echo "this is $NAME"
Then echo gives me the first line of table.txt, if I use 2 I get the 2nd line, 3 the 3rd line, then I stopped variations.
Thanks for any advice.
EDITed first post formatting faux pas.
You're looking for:
NAME=`awk -v line="$LINE" 'FNR==line' < table.txt`
but the backticks notation is obsolete so this is better:
NAME=$(awk -v line="$LINE" 'FNR==line' < table.txt)
and you should never use all-upper-case for variable names unless they are exported (in shell) to avoid clashing with builtin names so really it should be:
name=$(awk -v line="$line" 'FNR==line' < table.txt)
but whatever you're doing is almost certainly the wrong approach and should be done entirely within awk. Make sure you fully understand everything discussed in why-is-using-a-shell-loop-to-process-text-considered-bad-practice if you're considering using shell to manipulate text.
To complement Ed Morton's helpful awk-based answer:
If you only need to extract a single line by index, sed allows for a more concise solution that is also easier to optimize (note that I've changed the variable names to avoid all-uppercase variable names):
name=$(sed -n "$line {p;q;}")
-n tells sed not to print (possibly modified) input lines by default
$line, assuming this shell variable expands it to a positive integer (see caveat below), only matches the input line with that (1-based) index.
{p;q;}, prints (p) the matching line, then exits the overall script immediately (q) as an optimization (no need to read the remaining lines).
Note:
For more complex sed scripts it is NOT a good idea to use a double-quoted shell string with shell-variable expansion as the sed script, because understanding what is interpreted by the shell up front vs. what sed ends up seeing as a result can become confusing.
Heed Ed's point that you're likely better off solving your problem with awk alone, given that awk can do its own regex matching (probably no need for grep).

Bash quoting paranoid or necessary [duplicate]

This question already has answers here:
When to wrap quotes around a shell variable?
(5 answers)
Closed 3 years ago.
I wanted to know how vulnerable is bash by code injection. So I wrote a script as simple as this:
#!/bin/bash
grep $1 $2
and saved it as greptest.sh. The quotes around the variables were dropped intentionally for vulnerability test, so grep "$1" "$2" is the preferred way.
Then I created test.txt:
sadhuer
sadjfh Hello
cusad
Hello
fgdfg
First was to show its proper use.
$ ./greptest.sh 'Hello' 'test.txt'
Output as expected:
sadjfh Hello
Hello
Then the first attack:
$ ./greptest.sh 'Hello test.txt'
Outputs the same as above. So, obviously, it does something due to missing quotes within the script - altough $2 is empty! Next try with $2 not empty for proving my assumption that $2 will be interpreted as a further input file:
$ ./greptest.sh 'Hello test.txt' 'nonexistingfile.txt'
outputs:
test.txt:sadjfh Hello
test.txt:Hello
grep: nonexistingfile.txt: No such file or directory
Then the harder attack: Trying to execute an arbitrary command:
$ ./greptest.sh 'Hello test.txt' '; ls'
outputs:
test.txt:sadjfh Hello
test.txt:Hello
grep: ;: No such file or directory
grep: ls: No such file or directory
I did not expect this. I thought the variables were subsitituted to yield
grep Hello test.txt ; ls
which should result in listing the current directory. So, is missing these quotes just ugly and error prone or a serious security concern I should care about (given the values of these parameters come from an untrusted source)?
You don't need to worry about the parameters passed to your greptest.sh bash script, as you are passing the parameters directly as parameters to the called program (grep in this case).
You need to worry about whatever calls greptest.sh injecting parameters onto that command line.
See this post for information.
One security issue of unquoted variables (though not code injection) is glob expansion DoS – a variable without quotes not only undergoes word splitting, but also globbing!
If the variable contains a pattern like */*/../../ repeated several times, like */*/../../*/*/../../, */*/../../*/*/../../*/*/../../ and so on, then the number of files that expands to increases exponentially with the length of the pattern.
Not related to quoting, but if the attacker can influence $PATH, $LD_PRELOAD, etc, then he can make grep in your script mean anything.

looping over AWK commands doesn't work [duplicate]

This question already has answers here:
Variable interpolation in the shell
(3 answers)
Closed 7 years ago.
I have a huge dictionary file that contains each word in each line, and would like to split the files by the first character of the words.
a.txt --> only contains the words that start with a
I used this awk commands to successfully extract words that start with b.
awk 'tolower($0)~/^b/{print}' titles-sorted.txt > b.txt
Now I wanted to iterate this for all alphabets
for alphabet in {a..z}
do
awk 'tolower($0)~/^alphabet/{print}' titles-sorted.txt > titles-links/^alphabet.txt
done
But the result files contain no contents. What did I do wrong? I don't even know how to debug this. Thanks!
Because your awk program is in single quotes, there will not be any shell variable expansion. In this example:
awk 'tolower($0)~/^alphabet/{print}' titles-sorted.txt > titles-links/^alphabet.txt
...you are looking for the lines that begin with the literal string alphabet.
This would work:
awk "tolower(\$0)~/^$alphabet/{print}" titles-sorted.txt > titles-links/$alphabet.txt
Note several points:
We are using double quotes, which does not inhibit shell variable expansion.
We need to escape the $ in $0, otherwise the shell would expand that.
We need to replace alphabet with $alphabet, because that's how you refer to shell variables.
We need to replace ^alphabet with $alphabet in the filename passed to >.
You could also transform the shell variable into an awk variable with -v, and do this:
for alphabet in {a..z} ; do
awk -valphabet=$alphabet 'tolower($0)~"^"alphabet {print}' /usr/share/dict/words > words-$alphabet.txt
done

sed. passing a variable in sed command [duplicate]

This question already has answers here:
Replace a string in shell script using a variable
(12 answers)
Closed 7 years ago.
I want to use sed command in a loop passing a variable say a such that it searches for a and in the line it gets a it replaces "true" to "false".
I have a text file containing 3000 different names and another xml file containing 15000 lines. in the lines in which these 3000 entries are there i need to make changes.
I have written a code snippet but that is not giving expected output. Can anyone help. Thanks in advance.
for i in {1..3000}; do
a=`awk NR==$i'{print $1}' names.txt`
# echo $a
sed -e '/$\a/ s/true/false/' abc.xml > abc_new.xml
done
You have to replace single-quotes(') around sed's parameters with double-quotes("). In bash, single-quote won't allow variable expansion. Also, you might want to use sed's in-place edit (pass -i option) in your for loop.
So the one liner script will look like:
for a in `cat names.txt`; do sed -i.bak -e "/$a/s/true/false/" abc.xml ; done

how to store the awk command output value into some variable in ksh [duplicate]

This question already has answers here:
ksh storing result of a command to a variable
(5 answers)
Closed 8 years ago.
foundflag = awk -F" " 'FNR==NR{A[$arg1 OFS $PREVFILE];next}
!($arg1 OFS $PREVFILE in A){X++} END{if(!X){print "No diff."}
else {print "Diff found."}} $arg1 ${PREVFILE}
echo $foundflag
here am comparing two files and want to store awk command result into some variable please help in Ksh script.
I think what You need is:
foundflag=`awk ....`
remove spaces arounf =
put the command into reverse quotes (``)
Or use command substitution. The backticks/gravemarks have been deprecated in favor of $()
foundflag=$(awk....)
Also note where cannot be any space around the assignment operator (no space before and after)

Resources