I have found that the command expanded from variable won't run the latter parts in a pipe. For example, as following test.sh:
#!/bin/bash
y='echo hello man | awk "{print \$1}"'
$y
eval $y
y='echo hello'
$y
The output of the script is:
hello man | awk "{print \$1}"
hello
hello
The first $y only execute echo hello man but not execute awk "{print $1}" of the pipe. That's why?
My bash version is 4.3.48.
That's because the variable is expanded after the pipes and redirection is already done. So in that case | is just another argument to echo, not a pipe that's interpreted by the shell.
Recommended reading: http://mywiki.wooledge.org/BashFAQ/050
When executing the command
echo hello man | awk '{print $1}'
the shell will see the | and setup a pipeline, on one side it will run the command echo hello man and on the other awk '{print $1}'. The commands then get subjected to word splitting so you run the command echo with 2 arguments: hello and man. On the other side you run the command awk with a single argument (because of the quoting) "{print \$1}"
When that command is stored as a string though the shell first looks at the command $y and sees no redirection to do. It then expands $y and then does word splitting on it. It gets expanded to the same looking string, but now it's too late for redirection. So it gets split into the words echo, hello, man, |, awk, "{print, \$1}" (note that the argument to awk now gets split because the quotes inside the string are part of the string, not syntactical)
The first word in that list is echo so that's the command, and all the rest of the words are passed as arguments to it, which is why you see the output
hello man | awk "{print \$1}"
When you do the eval line, it takes that same string and tells bash to parse it as though it had been typed so the pipe once again becomes syntatical and causes the pipeline.
Because echo puts its arguments on the same line it's a little tougher sometimes to see what's happening, if we replace it with printf '%s\n' each argument will become its own line though:
$ y='printf %s\n hello man | awk "{print \$1}"'
$ $y
hello
man
|
awk
"{print
\$1}"
Related
I run the command
df -gP /data1 /data2 | grep -v File | awk '{print $1}' |
awk -F/dev/ '$0=$2' | tr '\n' '
on the AIX shell (ksh) and it prints the output below:
lv_data01 lv_data02 root#testhost:/
However, I would like the output to be printed this way. Could someone help?
lv_data01 lv_data02
Using grep … | awk … | awk … is not necessary; a single awk could do the whole job. So could sed and it might even be easier. I'd be tempted to deal with the spacing by using:
x=$(df … | sed …); echo $x
The tr command, once corrected, replaces newlines with spaces, so the prompt follows without a newline before it. The ; echo suggestion adds the missing newline; the echo $x suggestion (note no double quotes) does too.
As for the sed command:
sed -n '/File/!{ s/[[:space:]].*//; s%^.*/dev/%%p; }'
Don't print anything by default
If the line doesn't match File (doing the work of grep -v):
remove the first space (blank or tab) and everything after it (doing the work of awk '{print $1}')
replace everything up to /dev/ with nothing and print (doing the work of awk -F/dev/ '{$0=$2}')
The command substitution and capture, followed by echo, deals with spaces and newlines.
So, my suggested solution is:
x=$(df -gP /data1 /data2 | sed -n '/File/!{ s/[[:space:]].*//; s%^.*/dev/%%p; }'); echo $x
You could add unset x after the echo if you are going to be using this directly in the shell and not in a shell script. If it'll be encapsulated in a shell script, you don't have to worry about it.
I'm blithely assuming the output from df -gP won't contain a path such as this, with two occurrences of /dev:
/who/knows/dev/lv_data01/dev/bin
If that's a real problem, you can fix the sed script, but I don't think it will be. It's one thing the second awk script in the question handles differently.
I found this thread with two solutions for trimming whitespace: piping to xargs and defining a trim() function:
trim() {
local var="$*"
# remove leading whitespace characters
var="${var#"${var%%[![:space:]]*}"}"
# remove trailing whitespace characters
var="${var%"${var##*[![:space:]]}"}"
echo -n "$var"
}
I prefer the second because of one comment:
This is overwhelmingly the ideal solution. Forking one or more external processes merely to trim whitespace from a single string is fundamentally insane – particularly when most shells (including bash) already provide native string munging facilities out-of-the-box.
I am getting, for example, the wifi SSID on macOS by piping to awk (when I get comfortable with regular expressions in bash, I won't fork an awk process), which includes a leading space:
$ /System/Library/PrivateFrameworks/Apple80211.framework/Resources/airport -I | awk -F: '/ SSID/{print $2}'
<some-ssid>
$ /System/Library/PrivateFrameworks/Apple80211.framework/Resources/airport -I | awk -F: '/ SSID/{print $2}' | xargs
<some-ssid>
$ /System/Library/PrivateFrameworks/Apple80211.framework/Resources/airport -I | awk -F: '/ SSID/{print $2}' | trim
$ wifi=$(/System/Library/PrivateFrameworks/Apple80211.framework/Resources/airport -I | awk -F: '/ SSID/{print $2}')
$ trim "$wifi"
<some-ssid>
Why does piping to the trim function fail and giving it an argument work?
It is because your trim() function is expecting a positional argument list to process. The $* is the argument list passed to your function. For the case that you report as not working, you are connecting the read end of a pipe to the function inside which you need to fetch from the standard input file descriptor.
In such a case you need to read from standard input using read command and process the argument list, i.e. as
trim() {
# convert the input received over pipe to a a single string
IFS= read -r var
# remove leading whitespace characters
var="${var#"${var%%[![:space:]]*}"}"
# remove trailing whitespace characters
var="${var%"${var##*[![:space:]]}"}"
echo -n "$var"
}
for which you can now do
$ echo " abc " | trim
abc
or using a command substitution syntax to run the command that fetches the string, that you want to pass to trim() with your older definition.
trim "$(/System/Library/PrivateFrameworks/Apple80211.framework/Resources/airport -I | awk -F: '/ SSID/{print $2}')"
In this case, the shell expands the $(..) by running the command inside and replaces it with output of the commands run. So now the function sees trim <args> which it interprets as a positional argument and runs the string replacement functions directly on it.
I have this script that's designed to assign variables to commands that collect information about a system and then echo them back. This works very well for the first few commands, but the last one continues to return the value without "PRETTY_NAME=" stripped out of the output.
Is there some problem with this that I'm not seeing?
I have tried using grep to separate awk:
grep PRETTY_NAME /etc/*-release | awk -F '=' '{print $2}'
Using escaped quotes:
awk -F \"=\" '/PRETTY_NAME/ {print $2}' /etc/*-release
Whole block (edited somewhat for relevance)
declare -A CMDS=(
[primaryMacAddress]="cat /sys/class/net/$(ip route show default | awk '/default/ {print $5}')/address"
[primaryIpAddress]="hostname --ip-address"
[hostname]="hostname"
[osType]="awk -F '=' '/PRETTY_NAME/ {print $2}' /etc/*-release"
)
#This bit is actually nested in another function
for kpair in "${!CMDS[#]}" do
echo "$kpair=\"$( eval ${CMDS[$kpair]} )\""
done
Results when run from .sh file:
osType="PRETTY_NAME="Red Hat Enterprise Linux Server 7.4 (Maipo)""
expected:
osType=""Red Hat Enterprise Linux Server 7.4 (Maipo)""
When this command is run by itself, it seems to work as intended:
$ awk -F '=' '/PRETTY_NAME/ {print $2}' /etc/*-release
"Red Hat Enterprise Linux Server 7.4 (Maipo)"
Because your Awk command is specified in double quotes, interior dollar signs are subject to special treatment: the $2 is treated as a parameter substitution by your shell, and so the array element doesn't store the text $2 but rather its expansion. The Awk interpreter never sees the $2 syntax.
However, you have a second problem in your command dispatcher. Your eval command does not prevent word splitting:
eval ${CMDS[$kpair]}
you want this:
eval "${CMDS[$kpair]}"
without the quotes, your command is arbitrarily chopped into fields on whitespace. Then eval catenates the pieces together, using one space between them, and evaluates the resulting syntax. The difference can be demonstrated with the following example:
$ cmd="awk '/foo/ { print \$1\" \"\$2 }'"
$ echo 'foo a' | eval $cmd
foo a
$ echo 'foo a' | eval "$cmd"
foo a
We can just use echo to understand the issue:
$ echo $cmd
awk '/foo/ { print $1" "$2 }'
$ echo "$cmd"
awk '/foo/ { print $1" "$2 }'
The substitution of $cmd and the subsequent word splitting is done irrespective of any shell syntax that `cmd contains. We can see the pieces like this:
$ for x in $cmd ; do echo "<$x>" ; done
<awk>
<'/foo/>
<{>
<print>
<$1">
<"$2>
<}'>
When we execute eval $cmd, the above pieces are generated and re-combined by eval and evaluated. Needless to say, you don't want your command syntax to be chopped up and re-combined like this; who knows what sort of hidden bug will arise. It may be okay for the commands you have now, but as a generic command dispatch mechanism, it is flawed.
I have found that the command expanded from variable won't run the latter parts in a pipe. For example, as following test.sh:
#!/bin/bash
y='echo hello man | awk "{print \$1}"'
$y
eval $y
y='echo hello'
$y
The output of the script is:
hello man | awk "{print \$1}"
hello
hello
The first $y only execute echo hello man but not execute awk "{print $1}" of the pipe. That's why?
My bash version is 4.3.48.
That's because the variable is expanded after the pipes and redirection is already done. So in that case | is just another argument to echo, not a pipe that's interpreted by the shell.
Recommended reading: http://mywiki.wooledge.org/BashFAQ/050
When executing the command
echo hello man | awk '{print $1}'
the shell will see the | and setup a pipeline, on one side it will run the command echo hello man and on the other awk '{print $1}'. The commands then get subjected to word splitting so you run the command echo with 2 arguments: hello and man. On the other side you run the command awk with a single argument (because of the quoting) "{print \$1}"
When that command is stored as a string though the shell first looks at the command $y and sees no redirection to do. It then expands $y and then does word splitting on it. It gets expanded to the same looking string, but now it's too late for redirection. So it gets split into the words echo, hello, man, |, awk, "{print, \$1}" (note that the argument to awk now gets split because the quotes inside the string are part of the string, not syntactical)
The first word in that list is echo so that's the command, and all the rest of the words are passed as arguments to it, which is why you see the output
hello man | awk "{print \$1}"
When you do the eval line, it takes that same string and tells bash to parse it as though it had been typed so the pipe once again becomes syntatical and causes the pipeline.
Because echo puts its arguments on the same line it's a little tougher sometimes to see what's happening, if we replace it with printf '%s\n' each argument will become its own line though:
$ y='printf %s\n hello man | awk "{print \$1}"'
$ $y
hello
man
|
awk
"{print
\$1}"
I want to use awk in my bashscript, and this line clearly doesn't work:
line="foo bar"
echo $line | awk '{print $1}'
How do I escape $1, so it doesn't get replaced with the first argument of the script?
Your script (with single quotes around the awk script) will work as expected:
$ cat script-single
#!/bin/bash
line="foo bar"
echo $line | awk '{print $1}'
$ ./script-single test
foo
The following, however, will break (the script will output an empty line):
$ cat script-double
#!/bin/bash
line="foo bar"
echo $line | awk "{print $1}"
$ ./script-double test
Notice the double quotes around the awk program.
Because the double quotes expand the $1 variable, the awk command will get the script {print test}, which prints the contents of the awk variable test (which is empty). Here's a script that shows that:
$ cat script-var
#!/bin/bash
line="foo bar"
echo $line | awk -v test=baz "{print $1}"
$ ./script-var test
baz
Related reading: Bash Reference Manual - Quoting and Shell Expansions
As currently written, the $1 will not be replaced (since it's within single-quoted string, bash will not parse it)
If you write awk "{print $1}", bash will expand the $1 within the double-quoted string
Note that the variable expansion rules depend on the outermost level of quoting, so the $1 in "awk '{print $1}'" will be expanded