This question already has answers here:
How do I use shell variables in an awk script?
(7 answers)
Closed 6 years ago.
I have a variable LINE and want to use it with awk to pull out the contents of the line numbered LINE from table.txt and make that a new variable called NAME which is then used to grep another file.
NAME=`awk 'FNR==$LINE' < table.txt`
echo "this is $NAME"
Seems to be close, but not quite the syntax.
If I use:
NAME=`awk 'FNR==1' < table.txt`
echo "this is $NAME"
Then echo gives me the first line of table.txt, if I use 2 I get the 2nd line, 3 the 3rd line, then I stopped variations.
Thanks for any advice.
EDITed first post formatting faux pas.
You're looking for:
NAME=`awk -v line="$LINE" 'FNR==line' < table.txt`
but the backticks notation is obsolete so this is better:
NAME=$(awk -v line="$LINE" 'FNR==line' < table.txt)
and you should never use all-upper-case for variable names unless they are exported (in shell) to avoid clashing with builtin names so really it should be:
name=$(awk -v line="$line" 'FNR==line' < table.txt)
but whatever you're doing is almost certainly the wrong approach and should be done entirely within awk. Make sure you fully understand everything discussed in why-is-using-a-shell-loop-to-process-text-considered-bad-practice if you're considering using shell to manipulate text.
To complement Ed Morton's helpful awk-based answer:
If you only need to extract a single line by index, sed allows for a more concise solution that is also easier to optimize (note that I've changed the variable names to avoid all-uppercase variable names):
name=$(sed -n "$line {p;q;}")
-n tells sed not to print (possibly modified) input lines by default
$line, assuming this shell variable expands it to a positive integer (see caveat below), only matches the input line with that (1-based) index.
{p;q;}, prints (p) the matching line, then exits the overall script immediately (q) as an optimization (no need to read the remaining lines).
Note:
For more complex sed scripts it is NOT a good idea to use a double-quoted shell string with shell-variable expansion as the sed script, because understanding what is interpreted by the shell up front vs. what sed ends up seeing as a result can become confusing.
Heed Ed's point that you're likely better off solving your problem with awk alone, given that awk can do its own regex matching (probably no need for grep).
Related
I'm writing a shell script that should be somewhat secure, i.e., does not pass secure data through parameters of commands and preferably does not use temporary files. How can I pass a variable to the standard input of a command?
Or, if it's not possible, how can I correctly use temporary files for such a task?
Passing a value to standard input in Bash is as simple as:
your-command <<< "$your_variable"
Always make sure you put quotes around variable expressions!
Be cautious, that this will probably work only in bash and will not work in sh.
Simple, but error-prone: using echo
Something as simple as this will do the trick:
echo "$blah" | my_cmd
Do note that this may not work correctly if $blah contains -n, -e, -E etc; or if it contains backslashes (bash's copy of echo preserves literal backslashes in absence of -e by default, but will treat them as escape sequences and replace them with corresponding characters even without -e if optional XSI extensions are enabled).
More sophisticated approach: using printf
printf '%s\n' "$blah" | my_cmd
This does not have the disadvantages listed above: all possible C strings (strings not containing NULs) are printed unchanged.
(cat <<END
$passwd
END
) | command
The cat is not really needed, but it helps to structure the code better and allows you to use more commands in parentheses as input to your command.
Note that the 'echo "$var" | command operations mean that standard input is limited to the line(s) echoed. If you also want the terminal to be connected, then you'll need to be fancier:
{ echo "$var"; cat - ; } | command
( echo "$var"; cat - ) | command
This means that the first line(s) will be the contents of $var but the rest will come from cat reading its standard input. If the command does not do anything too fancy (try to turn on command line editing, or run like vim does) then it will be fine. Otherwise, you need to get really fancy - I think expect or one of its derivatives is likely to be appropriate.
The command line notations are practically identical - but the second semi-colon is necessary with the braces whereas it is not with parentheses.
This robust and portable way has already appeared in comments. It should be a standalone answer.
printf '%s' "$var" | my_cmd
or
printf '%s\n' "$var" | my_cmd
Notes:
It's better than echo, reasons are here: Why is printf better than echo?
printf "$var" is wrong. The first argument is format where various sequences like %s or \n are interpreted. To pass the variable right, it must not be interpreted as format.
Usually variables don't contain trailing newlines. The former command (with %s) passes the variable as it is. However tools that work with text may ignore or complain about an incomplete line (see Why should text files end with a newline?). So you may want the latter command (with %s\n) which appends a newline character to the content of the variable. Non-obvious facts:
Here string in Bash (<<<"$var" my_cmd) does append a newline.
Any method that appends a newline results in non-empty stdin of my_cmd, even if the variable is empty or undefined.
I liked Martin's answer, but it has some problems depending on what is in the variable. This
your-command <<< """$your_variable"""
is better if you variable contains " or !.
As per Martin's answer, there is a Bash feature called Here Strings (which itself is a variant of the more widely supported Here Documents feature):
3.6.7 Here Strings
A variant of here documents, the format is:
<<< word
The word is expanded and supplied to the command on its standard
input.
Note that Here Strings would appear to be Bash-only, so, for improved portability, you'd probably be better off with the original Here Documents feature, as per PoltoS's answer:
( cat <<EOF
$variable
EOF
) | cmd
Or, a simpler variant of the above:
(cmd <<EOF
$variable
EOF
)
You can omit ( and ), unless you want to have this redirected further into other commands.
Try this:
echo "$variable" | command
If you came here from a duplicate, you are probably a beginner who tried to do something like
"$variable" >file
or
"$variable" | wc -l
where you obviously meant something like
echo "$variable" >file
echo "$variable" | wc -l
(Real beginners also forget the quotes; usually use quotes unless you have a specific reason to omit them, at least until you understand quoting.)
This question already has answers here:
Difference between single and double quotes in Bash
(7 answers)
sed command not replacing ip address via variables [duplicate]
(1 answer)
Closed 1 year ago.
I'm familiar with sed's ability to replace in-place in a file, as answered here: sed edit file in place.
I.e. sed -i 's/STRING_TO_REPLACE/STRING_TO_REPLACE_IT/g' filename
I'm having trouble expanding this concept to replacing a string-literal that takes the form of a shell-variable -- how does one do this?
To illustrate with an example:
Given file file.txt:
# file.txt
set(FOO ${FOO})
...and shell environment variable ${FOO}:
$ FOO=bar
$ echo ${FOO}
bar
...how can I use sed to replace string-literal "${FOO}" in file.txt with the value of shell-variable ${FOO} i.e. "bar"? I.e. I'd like the resulting content of file.txt to be:
# file.txt
set(FOO bar)
I have a mental block thinking past the obviously-incorrect sed -i 's/${FOO}/${FOO}/g' file.txt
I gravitate towards sed because of past experience, and might prefer a sed-centric answer for the same reason. But any solution is probably okay, but with a preference for POSIX-compliance, if shell-native. To be even more specific, this is going to be run in a docker container with a debian-10.3 base...so I suppose solutions that work with any tools included in that distro should be okay as well.
This question already has answers here:
How do I use shell variables in an awk script?
(7 answers)
Closed 5 years ago.
I am writing a bash script. The following awk command does not print anything out even when there is a match.
I have copied the awk line and it executes successfully in the interactive shell when I replace $SEARCH_QUERY with real data. This leads me to believe that awk is not translating the data in the variable $SEARCH_QUERY correctly.
Any ideas how to fix this or what bash topics I should read up on to understand why it is not working? Thank you
search_contacts()
{
echo "Enter contact Name:"
read SEARCH_QUERY
awk ' /$SEARCH_QUERY/ { print $0 }' contacts.db
}
In terms of why the original code was broken -- single quotes suppress parameter expansion, so awk was treating $SEARCH_QUERY as a regex itself, ie. looking for a line with the string SEARCH_QUERY after its end -- a regex that can't possibly ever match.
Thus, the naive approach would be to change quoting types, such that your intended substitution actually takes place:
# THIS IS DANGEROUSLY INSECURE; DON'T DO THIS EVER
awk "/$SEARCH_QUERY/" contacts.db
However, as per the comment, the above is dangerously insecure; if your SEARCH_QUERY value ended the regex (included a /), it could then run any arbitrary awk code it wanted. nonmatch/ || system("rm -rf ~") || /anything would do Very Bad Things.
In terms of the best practice -- data should be passed out-of-band from code:
awk -v search_query="$SEARCH_QUERY" '$0 ~ search_query' contacts.db
echo $fbname | awk -F'[__]' '{print $2 $A_name = $2}'
echo $A_name
I am trying to extract a name within the fbname variable like for example,
newlist_sammy_card.csv So I am trying to get the name sammy which is between the two underscores and assign it to a variable I can use for rest of the script.
The first line prints out sammy, which is what I need, but the second line does not.
Can anyone show me where I am not assinging the variable correctly?
There is a fundamental flaw in your understanding and reasoning. If you invoke awk in your script it is spawned as a program in its own individual right. Therefore all the variables that exist in your current shell are not available to awk and vice versa. As such you can not 'define' variables in awk that are then visible to your shell. What you should do is 'capture' the output of awk, by using the notation $(), and assign it to a variable. Consider this example:
var=$(awk '{print "test"}')
echo $var
This will output
test
Now in your case, we are actually facing an xy-problem. You want to extract sammy from the string newlist_sammy_card.csv and use that as a variable. One possible solution in pure bash is the following:
name="newlist_sammy_card.csv"
temp=${name#*_}
var=${temp%%_*}
echo $var
This will output
sammy
There's a LOT of ways to do what you're asking for, e.g. here's a couple more in addition to the other ideas you've received so far:
$ fbname='newlist_sammy_card.csv'
$ A_name=$(echo "$fbname" | cut -d_ -f2)
$ echo "$A_name"
sammy
or:
$ IFS=_
$ set -- $fbname
$ A_name="$2"
$ echo "$A_name"
sammy
but I wonder if you're approaching your problem completely the wrong way. We can't tell without more info on what you're trying to do though.
You can simply use bash:
str1="newlist_sammy_card.csv"
# replace 'newlist_' from front of the string
str2=${str1#*_}
# replace '_card.csv' from back of the string:
str2=${str2%%_*}
echo "$str2" # Output: sammy
Unfortunately it can't be done in a single run in bash. However it should still perform a lot better than launching any kind of external program.
Pankrates's answer explains the problem with the OP's approach well and offers a pure shell solution using shell parameter expansion.
Here's another pure shell solution, using a single command based on the read builtin:
Using bash, with a here-string:
IFS=_ read -r _ A_name _ <<<"$fbname"
POSIX-compliant equivalent, using a here-doc:
IFS=_ read -r _ A_name _ <<EOF
$fbname
EOF
If $fbname contains 'newlist_sammy_card.csv', $A_name will contain 'sammy' afterward.
IFS=_ tells read to split the input into tokens by _ instances.
Note that by directly prepending IFS=... to read, the effect of setting $IFS is localized to the read command - no need to restore the original $IFS later.
read -r _ A_name _ ... reads input unmodified (-r - no interpretation of backslash escape sequences)
Note that _ on either side of A_name is the - customary - name of a dummy variable that is used for input that is of no interest, but is needed to positionally extract the token of interest.
It is a mere coincidence in this case that the name of this dummy variable is the same as the $IFS character.
In this case: $_ receives the 1st field (before the first _ char. in the input), and is then overwritten with any remaining fields after the 2nd field, where the 2nd field is read into the variable of interest, $A_name.
This question already has answers here:
Bash and filenames with spaces
(6 answers)
Closed 8 years ago.
I'm writing a script to do variable substitution into a Java properties file, of the format name=value. I have a source file, source.env like this:
TEST_ENV_1=test environment variable one
TEST_ENV_2=http://test.environment.com/one
#this is a comment with an equal sign=blah
TEST_ENV_3=/var/log/test/env/2.log
My script will replace every occurence of TEST_ENV_1 in the file dest.env with "test environment variable one", and so on.
I'm trying to process a line at a time, and having problems because looping on output from a command like sed or grep tokenizes on white space rather than the entire line:
$ for i in `sed '/^ *#/d;s/#.*//' source.env`; do
echo $i
done
TEST_ENV_1=test
environment
variable
one
TEST_ENV_2=http://test.environment.com/one
TEST_ENV_3=/var/log/test/env/2.log
How do I treat them as lines? What I want to be able to do is split each line apart on the "=" sign and make a sed script with a bunch of substitution regex's based on the source.env file.
sed '/^ *#/d;s/#.*//' source.env | while read LINE; do
echo "$LINE"
done
An alternative is to change $IFS as per #Jim's answer. It's better to avoid backticks in this case as they'll cause the entire file to be read in at once, whereas piping the output of sed to while above will allow the file to be processed line by line without reading the whole thing in to memory.