How to set variables in make from commands containing pipes and variables - makefile

I have the following make file:
deploy.runtime:
kubectl describe service hello-node -n default | grep "LoadBalancer Ingress:" | awk '{print $$3}'
$(eval MY_IP=$(kubectl describe service hello-node -n default | grep "LoadBalancer Ingress:" | awk '{print $$3}'))
#echo IP: $(MY_IP)
When I run this, the output I get is:
35.198.222.110
IP:
It seems like the bariable MY_IP is not being set. I've also tried running it with back ticks like this:
$(eval MY_IP=`kubectl describe service hello-node -n default | grep "LoadBalancer Ingress:" | awk '{print $$3}'`)
Which gives me the output:
IP: LoadBalancer Ingress: 35.198.222.110
I'm really confused about why awk doesn't seem to be getting the correct arguments. I'm sure it's something to do with escape arguments, but I cannot for the life of me see how.

I've found that I can fix this by adding a 'shell' into the command:
$(eval MY_IP=$(shell kubectl describe service hello-node -n default | grep "LoadBalancer Ingress:" | awk '{print $$3}'))

What you probably missed is that the eval parameter is expanded by make. So, in:
$(eval MY_IP=$(kubectl describe service hello-node -n default | grep "LoadBalancer Ingress:" | awk '{print $$3}'))
the $(kubectl describe...) is expanded by make as the empty string (unless you have a make variable named kubectl describe..., which is unlikely). The eval function thus assigns the empty string to the make variable MY_IP. It does not happen with back ticks but here you hit another problem: the MY_IP variable is assigned:
`kubectl describe... | awk '{print $3}'`
not the result of its evaluation by the shell. Note that the $$3 has been transformed in $3 by make when it expanded the eval parameter. And it is when executing:
echo IP: $(MY_IP)
that make expands (recursively) the recipe that becomes, step by step:
echo IP: `kubectl describe... | awk '{print $3}'`
and then:
echo IP: `kubectl describe... | awk '{print }'`
(unless you have a make variable named 3). This is what is passed to the shell, leading to what you see. Use $$$$3, instead, and it should work as you expect... except that the make variable MY_IP is not assigned the value you wanted.
The shell function, as you noticed, solves all this but you end up with a terrible mixture: a shell command, evaluated by make through the shell function, its result assigned to a make variable thanks to the eval make function, in the middle of a recipe which is a list of shell commands. I don't know what you want to do but there must be something simpler.
For instance, if you used a make variable just because you couldn't pass a shell variable from one line of your recipe to the next (normal, they are executed by two independent shell invocations), you could reduce your recipe to a single line (but with ; \ line continuation for better readability):
deploy.runtime:
#MY_IP=`kubectl describe... | awk '{print $$3}'`; \
echo IP: $$MY_IP
Here, MY_IP is a shell variable and it is still available to the echo command because it is part of the same recipe line. Note the double $ sign in $$3 and $$MY_IP to escape the first expansion by make.
If, instead, you really want to use a make variable, you could assign it as a regular make variable (using back ticks or the shell function, as you wish):
MY_IP = `kubectl describe... | awk '{print $$3}'`
deploy.runtime:
#echo IP: $(MY_IP)
or:
MY_IP = $(shell kubectl describe... | awk '{print $$3}')
deploy.runtime:
#echo IP: $(MY_IP)
Important note about this last solution:
The MY_IP = ... assignment is a recursive (delayed) assignment, instead of a simple (immediate) assignment MY_IP := .... This means that the shell command is not expanded and executed immediately when you invoke make. It is only when make needs the value of MY_IP. So, if the deploy.runtime recipe is the only place where there is a reference to the value of MY_IP, the kubectl... will be executed only if this recipe is executed. For instance, if you have another clean target which recipe does not use the value of MY_IP, and if you invoke make clean, the kubectl... will not be executed at all. The drawback is that if you have several places where make needs its value, the kubectl... will be executed several times.
If you use the simple assignment, instead:
MY_IP := $(shell kubectl describe... | awk '{print $$3}')
the kubectl command will be executed only once but it will be each time you invoke make, even if its value is not needed.
In case this is a problem, Mad Scientist has a very nice trick to combine the pros and cons of recursive and simple assignments:
MY_IP = $(eval MY_IP := $$(shell kubectl...))$(MY_IP)
If you are interested, read his post about this. How many $ you will need for the awk command is left as an exercise...

Related

Error in assigning awk variable to bash variable

Variable b has a string. Awk retrieves a substring which I want to assign to variable c. This is what I did:
#!/bin/bash
b=$(llsubmit multiple.cmd)
echo $b | c=$(awk '{
ret=match($0,".in.")
rwt=match($0,"\" has")
rqt=rwt-(ret+4)
subs=substr($0,(ret+4),rqt)
}')
... but I get a blank output for echo $c:
You can't pipe into an assignment.
c=$(echo "$b" | awk '{
ret=match($0,".in.")
rwt=match($0,"\" has")
rqt=rwt-(ret+4)
subs=substr($0,(ret+4),rqt)
}')
(Notice also the quoting around $b.)
But your Awk script looks rather complex. And it doesn't produce any output. Should it print something at the end? Without access to sample output from llsubmit this is mildly speculative, but I'm guessing something like this could work:
c=$(echo "b" | sed -n 's/.*\(\.in\.[^"]*\)" has .*/\1/p')
(Notice also the backslashes to make the dots match literally.)
You should properly then use double quotes in echo "$c" too (unless you are completely sure that the output cannot contain any shell metacharacters).
... And, of course, very often you don't want or need to store results in a variable in shell scripts if you can refactor your code into a pipeline. Perhaps you are really looking for something like
llsubmit multiple.cmd |
sed -n 's/.*\(\.in\.[^"]\)" has .*/p' |
while read -r job; do
: things with "$job"
done
It's hard to tell from your question since you didn't provide sample input and expected output but is this what you're trying to do:
$ b='foo .in.bar has'
$ c="${b% has*}"
$ c="${c#*.in.}"
$ echo "$c"
bar

awk shell variables not working

Hi I'm using GNU awk version 3.1.5 and I've specified 2 variables in a KSH script
PKNAME= ls -lt /var/db/pkg | tr -s " " | cut -d" " -f9
PKDATE= ls -lt /var/db/pkg/$PKNAME/ | tr -s " " | cut -d" " -f6-8
I'm trying to prove that I'm getting the correct output, by running a test using
echo bar
awk -F, -v pkname="$PKNAME" -v pkdate="$PKDATE" 'BEGIN{print pkname, pkdate, "foo"; exit}'
echo baz
The output from this results in 2 blank spaces and foo, like so
bar
foo
baz
I have tried, double quoting the variables, single quotes and back ticks. Also tried double quotes with back ticks.
Any ideas why the variables are not being executed? I'm fairly new to awk and appreciate any help! Thanks
I suppose it is possible that it is not possible to run a sub shell comand within an awk statement. Is this true?
This has nothing to do with awk. The problem is in the way you're assigning your variables. Your lines should be like this:
PKNAME=$(ls -lt /var/db/pkg | tr -s " " | cut -d" " -f9)
There can be no spaces around either side of an assignment in the shell.
At the moment, you're running the command ls -lt ... with a variable PKNAME temporarily assigned to an empty string. In subsequent commands the variable remains unset.
Your awk command should remain unchanged, i.e. the shell variables should be passed like -v pkname="$PKNAME". As an aside, it's generally considered bad practice to use uppercase variable names, as these should be reserved for internal use by the shell.

Reference to a bash variable whose name contains dot

I have a bash variable: agent1.ip with 192.168.100.137 as its value. When I refer to it in echo like this:
echo $agent1.ip
the result is:
.ip
How can I access the value?
UPDATE: my variables are:
Bash itself doesn't understand variable names with dots in them, but that doesn't mean you can't have such a variable in your environment. Here's an example of how to set it and get it all in one:
env 'agent1.ip=192.168.100.137' bash -c 'env | grep ^agent1\\.ip= | cut -d= -f2-'
Since bash.ip is not a valid identifier in bash, the environment string bash.ip=192.168.100.37 is not used to create a shell variable on shell startup.
I would use awk, a standard tool, to extract the value from the environment.
bash_ip=$(awk 'BEGIN {print ENVIRON["bash.ip"]}')
The cleanest solution is:
echo path.data | awk '{print ENVIRON[$1]}'
Try this:
export myval=`env | grep agent1.port | awk -F'=' '{print $2}'`;echo $myval
Is your code nested, and using functions or scripts that use ksh?
Dotted variable names are an advanced feature in ksh93. A simple case is
$ a=1
$ a.b=123
$ echo ${a.b}
123
$ echo $a
1
If you first attempt to assign to a.b, you'll get
-ksh: a.b=123: no parent
IHTH

makefile assign command to

I'm probably being stupid, but I cannot do the most basic variable assignment in my Makefile.
TEST = $(pwd);
all:
echo $(TEST)
When I run "make all" from Bash (v4.2.42) in FreeBSD (v9.1) I get:
echo
No idea what I'm screwing up. I've also tried to assign the variable using $(shell ...) with the same result.
If I use back ticks (`) then basic assignment works, but it doesn't store the result, it stores the command. This breaks in the following example Makefile:
SERVERIP = `ifconfig em0 | grep -E 'inet.[0-9]' | awk '{ print $$2}'`
all:
echo $(SERVERIP)
sed -e 's/%LISTENIP%/${SERVERIP}/g' test.conf > work/test.conf.tmp
The result is:
[pete#pete] ~/make
echo `ifconfig em0 | grep -E 'inet.[0-9]' | awk '{ print $2}'`
10.128.28.151
sed -e 's/%LISTENIP%/`ifconfig em0 | grep -E 'inet.[0-9]' | awk '{ print $2}'`/g' test.conf > work/test.conf.tmp
sed: 1: "s/%LISTENIP%/`ifconfig ...": unescaped newline inside substitute pattern
*** [all] Error code 1
You can see that basic variable assignment appears to work, but when you insert the result into sed, it inserts the whole command rather than the result, which breaks!
Any ideas?
Pete.
Maybe the answer is too late but I hope I can help (somebody else).
If you don't want to install GNU's make you can use the !=:
TEST!=pwd
all:
echo ${TEST}
will work. The explanation is very simple, please read carefully FreeBSD's man make, especially the "Variable assignment modifiers" inside the "VARIABLE ASSIGNMENTS" section:
Variable assignment modifiers
The five operators that can be used to assign values to variables are
as follows: ...
!= Expand the value and pass it to the shell for execution and
assign the result to the variable. Any newlines in the result
are replaced with spaces.
The makefile contains:
TEST = $(pwd);
all:
echo $(TEST)
The first line assigns the value in the make variable pwd plus a semicolon to make variable TEST. The rule then echoes that. You forgot, I think, that $(...) is significant to both make and shell. Assuming you want to echo the output of the pwd command, then use a double-dollar so make expands that to one dollar:
TEST = $$(pwd)
all:
echo $(TEST)
To get the output of a shell command into a make variable, you have to use GNU make extensions. There are two extensions in use: the := immediate assignment that evaluates the value of SERVERIP at the point of definition, rather than at the point of use; and $(shell ...cmd...) that is used to run a shell command and capture its output.
SERVERIP := $(shell ifconfig em0 | grep -E 'inet.[0-9]' | awk '{ print $$2}')
all:
echo ${SERVERIP}
sed -e 's/%LISTENIP%/${SERVERIP}/g' test.conf > work/test.conf.tmp
This should work. The original error, about a newline in the sed regular expression, is odd; the back-ticks should have removed the trailing newline from the output of the ifconfig pipeline. However, if there were in fact multiple lines returned, then you'd still have a newline inside, justifying the complaint from sed.
This generates the complaint:
SERVERIP = `printf "%s\n" a b c`
all:
echo $(SERVERIP)
sed -e "s/%LISTENIP%/${SERVERIP}/g" test.conf > work/test.conf.tmp
This works:
SERVERIP = `printf "%s\n" c`
all:
echo $(SERVERIP)
sed -e "s/%LISTENIP%/${SERVERIP}/g" test.conf > work/test.conf.tmp
Since you're using FreeBSD, it is quite possible that you don't have GNU make. In that case, you will need to:
Make sure the ifconfig command pipeline only produces one line of output.
Use make commands like this:
SERVERIP = `ifconfig em0 | grep -E 'inet.[0-9]' | awk '{ print $$2}'`
all:
echo ${SERVERIP}
sed -e "s/%LISTENIP%/${SERVERIP}/g" test.conf > work/test.conf.tmp
That should work with any version of make provided condition 1 is met.
This is a test makefile, similar to yours. The ifconfig command has been placed in a $(shell ...) command and the result is stored in SERVERIP.
TOP:= $(shell pwd)
SERVERIP:= $(shell ifconfig en1 | grep -E 'inet.[0-9]' | awk '{ print $$2}')
LISTENIP:=240.12.4.63
all:
echo $(SERVERIP)
sed -e 's/$(LISTENIP)/$(SERVERIP)/g' test.conf > work/test.conf.tmp

Get list of variables whose name matches a certain pattern

In bash
echo ${!X*}
will print all the names of the variables whose name starts with 'X'.
Is it possible to get the same with an arbitrary pattern, e.g. get all the names of the variables whose name contains an 'X' in any position?
Use the builtin command compgen:
compgen -A variable | grep X
This should do it:
env | grep ".*X.*"
Edit: sorry, that looks for X in the value too.
This version only looks for X in the var name
env | awk -F "=" '{print $1}' | grep ".*X.*"
As Paul points out in the comments, if you're looking for local variables too, env needs to be replaced with set:
set | awk -F "=" '{print $1}' | grep ".*X.*"
Easiest might be to do a
printenv |grep D.*=
The only difference is it also prints out the variable's values.
This will search for X only in variable names and output only matching variable names:
set | grep -oP '^\w*X\w*(?==)'
or for easier editing of searched pattern
set | grep -oP '^\w*(?==)' | grep X
or simply (maybe more easy to remember)
set | cut -d= -f1 | grep X
If you want to match X inside variable names, but output in name=value form, then:
set | grep -P '^\w*X\w*(?==)'
and if you want to match X inside variable names, but output only value, then:
set | grep -P '^\w*X\w*(?==)' | grep -oP '(?<==).*'
Enhancing Johannes Schaub - litb answer removing fork/exec in modern bash we could do
compgen -A variable -X '!*X*'
i.e an X in any position in the variable list.
env | awk -F= '{if($1 ~ /X/) print $1}'
To improve on Johannes Schaub - litb's answer:
There is a shortcut for -A variable and a flag to include a pattern:
compgen -v -X '!*SEARCHED*'
-v is a shortcut for -A variable
-X takes a pattern that must not be matched.
Hence -v -X '!*SEARCHED*' reads as:
variables that do not, not match "anything + SEARCHED + anything"
Which is equivalent to:
variables that do match "anything + SEARCHED + anything"
The question explicitly mentions "variables" but I think it's safe to say that many people will be looking for "custom declared things" instead.
But neither functions nor aliases are listed by -v.
If you are looking for variables, functions and aliases, you should use the following instead:
compgen -av -A function -X '!*SEARCHED*'
# equivalent to:
compgen -A alias -A variable -A function -X '!*SEARCHED*'
And if you only search for things that start with a PREFIX, compgen does that for you by default:
compgen -v PREFIX
You may of course adjust the options as needed, and the official doc will help you: https://www.gnu.org/software/bash/manual/html_node/Programmable-Completion-Builtins.html
to expand Phi's and Johannes Schaub - litb's answers for the following use case:
print contents of all environment variables whose names match a pattern as strings which can be reused in other (Bash) scripts, i.e. with all special characters properly escaped and the whole contents quoted
In case you have the following environment variables
export VAR_WITH_QUOTES=\"FirstName\ LastName\"\ \<firstname.lastname#example.com\>
export VAR_WITH_WHITESPACES="
a bc
"
export VAR_EMPTY=""
export VAR_WITH_QUOTES_2=\"\'
then the following snippet prints all VAR* environment variables in reusable presentation:
for var in $(compgen -A export -X '!VAR*'); do
printf "%s=%s\n" "$var" "${!var#Q}"
done
Snippet is is valid for Bash 4+.
The output is as follows, please note output for newlines, empty variables and variables which contain quotation characters:
VAR_EMPTY=''
VAR_WITH_QUOTES='"FirstName LastName" <firstname.lastname#example.com>'
VAR_WITH_QUOTES_2='"'\'''
VAR_WITH_WHITESPACES=$' \n\ta bc\n'
This also relates to the question Escape a variable for use as content of another script

Resources