I am using make's shell command to populate some variables, and on output it indicates that they are being set to the values I expect. However when I pass them to my make recipes they all show up empty. I suspect make is doing some odd parsing on the results. Eg:
MyTarget: $(MySources)
LINE='$(shell cat $< | grep GIMME_THE_LINE_I_WANT)'
CH9=$(shell echo $(LINE) | cut -b9)
echo $(CH9) # doesn't print anything
I checked my generator commands manually by setting SHELL=sh -XV and when I run identical commands I get the right values, it just looks like bash is 'zeroing' my variables. Any idea what's wrong?
There are several things going on here. The first is that when you have:
MyTarget: $(MySources)
LINE='$(shell cat $< | grep GIMME_THE_LINE_I_WANT)'
You are setting a shell variable called LINE, not a make variable. The build instructions for a target are all shell commands. So after the first line, the shell variable $LINE contains GIMME_THE_LINE_I_WANT. However...
...each line in the build instructions for a target runs in a separate shell process. So if you have:
mytarget:
MYVAR=foo
echo $$MYVAR
You'll see no output, because $MYVAR isn't set in the context of the second command. Also note the use of $$ here, because otherwise the $ would be interpreted by Make (that is, writing $MYVAR would actually be the make expression $M followed by the text YVAR). You can resolve this by logically joining your lines into a single shell script, like this:
mytarget:
MYVAR=foo; \
echo $$MYVAR
The \ is Makefile syntax that extends a single logical line over multiple physical lines, and of course ; is simply shell syntax for combining multiple commands on one line.
With all this in mind, we could rewrite your target like this:
MyTarget: $(MySources)
LINE=$$(cat $< | grep GIMME_THE_LINE_I_WANT); \
CH9=$$(echo $$LINE | cut -b9); \
echo $$CH9
Notice that since we are already running a shell script I'm not using Make's $(shell ...) construct, and that I'm making sure to escape all of the $ characters to ensure that the shell, not Make, is handling variable expansion.
Taking it just a little further, you don't need to use cat in that script; you could simply write:
MyTarget: $(MySources)
LINE=$$(grep GIMME_THE_LINE_I_WANT $<); \
CH9=$$(echo $$LINE | cut -b9); \
echo $$CH9
Or:
MyTarget: $(MySources)
CH9=$$(grep GIMME_THE_LINE_I_WANT $< | cut -b9); \
echo $$CH9
(NB: While not germane to this solution, it's although worth noting that each invocation of $(shell ...) is also run in a separate process, so a variable set in one won't be available in another.)
The make runs every command in its separate shell. So, the values are not carried over.
When in doubt, you could always debug it with -d option. Also, a site note, the debug option is very useful when you are trying to figure out why a rule did not fire the way you had intended it.
~> cat Makefile
MyTarget:
LINE="somevalue"
echo ${LINE}
~>
~>
~> make -d MyTarget | tail -10
LINE="somevalue"
Putting child 0x2004fbb0 (MyTarget) PID 4052 on the chain.
Live child 0x2004fbb0 (MyTarget) PID 4052
Reaping winning child 0x2004fbb0 PID 4052
echo
Live child 0x2004fbb0 (MyTarget) PID 4192
Reaping winning child 0x2004fbb0 PID 4192
Removing child 0x2004fbb0 PID 4192 from chain.
Successfully remade target file 'MyTarget'.
~>
Just clarifying for the people who downvoted and commented that my above solution doesn't work: First, I apologize for my laziness. May be I should have been clear. Let me try again.
The "make -d" is not a solution to OP's problem.
I tried to show OP how he/she could use debug option to solve a variety of problems that people come across while using makefiles (which, I admit, goes in a slight tangent than just solving the OP's problem at hand).
The above debug shows that the first command was executed in a shell with PID=4052 and the second command was executed in another shell with PID=4192 (which doesn't carry the value of that variable). Also it shows that using a variable with single dollar (${LINE}) just gives you a blank (because the makefile doesn't interpret it as a shell variable).
Again, to be clear: "make -d" is not a solution. Just combine the commands in one line, separated by commas, use double dollars; if the line is long, escape the new lines.
MyTarget:
LINE="somevalue"; \
echo $${LINE}
Related
My Goal
I'm writing a small Bash script, which uses entr, which is a utility to re-run arbitrary commands when it detects file-system events. My immediate goal is to pass entr a command which converts a given markdown file to HTML. entr will run this command every time the markdown file changes. A simplified but working script looks like:
# script 1
in="$1"
out="${in%.md}.html"
echo "$in" | entr pandoc "${in}" -o "${out}"
This works fine. The filename to be watched is supplied to entr on stdin. On detecting changes in that file, entr runs the command specified by its args. In this example that is pandoc, and all the args after it, to convert the markdown file to an HTML file.
For future reference, set -x shows that entr was invoked as we'd expect. (Throughout, lines starting with + show the output from set -x):
+ entr pandoc 'READ ME.md' -o 'READ ME.html'
The problem
I want to look-up the command given to entr depending on the file-type of the
given input file. So the file-conversion command ends up in a variable, and I want to use that variable as the command-line args to entr. But I can't get the quoting right.
Again, simplified:
# script 2
in="$1"
out="${in%.md}.html"
cmd="pandoc \"${in}\" -o \"${out}\""
echo "$in" | entr "$cmd"
(shellcheck.net detects no issues on the above)
This fails. Because "$cmd" in the final line is in quotes, the entirety of $cmd
is treated as a single arg to entr:
+ entr 'pandoc "READ ME.md" -o "READ ME.html"'
entr tries to interpret the whole thing as the name of an executable, which
it cannot find:
entr: exec pandoc "READ ME.md" -o "READ ME.html": No such file or directory
So how should I modify script 2, to use the content of $cmd as the args to
entr?
What have I tried?
Check that $cmd is being formed as I expect? If I echo "$cmd" right after
it is defined in script 2, it looks exactly how I'd hope:
pandoc "READ ME.md" -o "READ ME.html"
I tried messing around with alternate ways of constructing cmd, such as:
cmd='pandoc "'"${in}"'" -o "'"${out}"'"'
but variations like this produce identical values of $cmd, and identical
behavior as script2.
Try not quoting the use of $cmd?
Since the final line of script 2 erroneously treats the whole of "$cmd"
as a single arg, and we want it to split up the words into seprate args
instead, maybe removing the quotes and using a bare $cmd is a step in the
right direction?
echo "$in" | entr $cmd
Predictably enough though, this splits $cmd up on every space, even the
ones inside our double-quotes:
+ entr pandoc '"READ' 'ME.md"' -o '"READ' 'ME.html"'
This makes Pandoc try, and fail, to open a file called "READ:
pandoc: "READ: openBinaryFile: does not exist (No such file or directory)
Try constructing $cmd using printf?
I notice printf -v can store output in a variable. How about using that
instead of assiging to cmd?
printf -v cmd 'pandoc "%s" -o "%s"' "$in" "$out"
Predictably enough, this produces the same results as script2. I tried some
speculative variations, such as %q in the format string, or using $in
and $out directly in the format string, but didn't stumble on anything
that seemed to help.
Try using the ${var#Q} form of parameter expansion.
echo "$in" | entr ${cmd#Q}
Tried with and without double quotes around the use of ${cmd#q}. No joy,
I guess I'm misunderstanding what #Q is for.
+ entr ''\''pandoc' '"READ' 'ME.md"' -o '"READ' 'ME.html"'\'''
entr: exec 'pandoc: No such file or directory
Details
I'm using Bash v5.1.16, in Pop!_OS 22.04, derived from Ubuntu 22.04 (Jammy).
The current 'apt' version of entr (v5.1) in Ubuntu Jammy (22.04) is too old
for my needs (e.g. the -z flag doesn't work.) so I'm compiling my own from
the latest v5.3 source release.
I know there are a lot of questions about quoting in Bash, but I don't see any that seem to match this. Apologies if I'm wrong.
Assemble the command as an array, instead of a string.
I read somewhere that maybe $# might do what I need, so I put the parts of $cmd into an array:
in="$1"
out="${in%.md}.html"
cmd=(pandoc "$in" -o "$out")
echo "$in" | entr "${cmd[#]}"
This correctly quotes the items in ${cmd[#]} which require it (e.g. have spaces in.)
+ entr pandoc 'READ ME.md' -o 'READ ME.html'
So ‘entr’ successfully calls ‘pandoc’, which successfully converts the documents. It works! I confess I did not expect that.
This approach seems viable for other similar situations, not just when invoking entr.
So I have a solution. It doesn't seem completely ideal for my future plans. I had visions of these 'file conversion commands' being configurable, and hence defined in a text file somewhere, so that users (==me, probably) could override them and define their own, and I'm not fluent enough with Bash to be sure how to go about that when commands are defined as arrays instead of strings.
I can't help but feel I've overlooked something simpler.
Use a shell to interpret the value of "$cmd":
echo "$in" | entr sh -c "$cmd"
This approach seems viable for other similar situations, not just when invoking entr.
Similarly, entr has a -s option which invokes a shell for you (chosen using the first word in $SHELL):
echo "$in" | entr -s "$cmd"
These both work well, at the minor cost of spawning an extra shell process.
Consider the following Makefile.
$(shell touch /tmp/example.txt)
FILE := /tmp/example.txt
CONTENTS = $(shell cat $(FILE); bash -c 'echo [debugging id: $$RANDOM]')
.PHONY: all
all:
#cat $(FILE)
#echo '$$(CONTENTS):' $(CONTENTS)
bash -c 'echo file-contents-$$RANDOM' > $(FILE)
#cat $(FILE)
#echo '$$(CONTENTS):' $(CONTENTS) # This line outputs the old contents. Why?
It prints the contents of the file, overwrites with new contents and prints the contents again. It shows as (after second shots of make):
file-contents-1543
$(CONTENTS): file-contents-1543 [debugging id: 15172]
bash -c 'echo file-contents-$RANDOM' > /tmp/example.txt
file-contents-22441
$(CONTENTS): file-contents-1543 [debugging id: 151]
The old content is file-contents-1543 and new content is file-contents-22441 (the numbers are random), but the last line echo $(CONTENTS) does not print the new contents.
I think the command is actually called twice as debugging ids show but shell function in the lazy variable seems to be executed before writing the new contents to the file.
I expect that lazy variable in Makefile is evaluated every time it is referred, the echo $(CONTENTS) command always prints the latest file contents. What am I wrong?
By the way, I found that using CONTENTS = $$(cat $(FILE)) works as I expect. I will using this instead of shell function but is it ok?
I expect that lazy variable in Makefile is evaluated every time it is referred, the echo $(CONTENTS) command always prints the latest file contents. What am I wrong?
First of all, in make's slang these variables are called recursive, not lazy. And, yes, they get expanded (i.e. recursively substituted) each time they are referred with $(CONTENTS). Considering that $(eval...) and $(shell...) (as pretty much anything looking as $(...)) also went through the same (recursive) expansion procedure (albeit, with some "side-effects"), each expansion of such variable could also result in some sort of "evaluation" or "execution".
Next, the order of expansion in make is a bit specific. In particular, the recipes (i.e. the lines starting with [tab]) are expanded after the whole makefile was (pre-)processed, but before the first line of the recipe gets executed by shell. Which is the main source of your confusion, I suppose.
I found that using CONTENTS = $$(cat $(FILE)) works as I expect
$$ is a way to get a single literal $ after an expansion procedure. So $$(cat $(FILE)) when expanded becomes $(cat /tmp/example.txt) which is a legal syntax for command substitution in bash. This means it will work only as part of a bash command (recipe line). If that is what you want then it's okay.
Trying to compare the process list before and after running a bash script of tests. Having trouble, since ps returns 1, and I'm not sure how to compare the before and after when I have them.
Ideally, it would look something like this. Forgive the crude pseudo-code:
run-tests:
ps -x
export before=$?
# run tests and scripts
ps -x
export after=$?
# compare before and after
Suggests and advice appreciated.
I'm assuming you want to count the number of running processes before and after (your question wasn't overly clear on that). If so, you can pipe ps into wc:
export before=`ps --no-headers | wc -l`
-- EDIT ---
I reread the question, and it may be that you're looking for the actual processes that differ. If that's the case, then, you can capture the output in variables and compare those:
target:
# before=$$(ps --no-headers); \
run test; \
after=$$(ps --no-headers); \
echo "differing processes:"; \
comm -3 <(echo "$before") <(echo "$after")
A few quick notes on this: I concatenated all the lines using \'s as you mentioned you used makefiles, and the scope of a variable is only the recipe line in which it's defined. By concatenating the lines, the variables have a scope of the whole recipe.
I used double $$ as your original post suggested a makefile, and a makefile $$ will expand to a single $ in the bash code to be run.
Doing var=$(command) in bash assigns var the output of command
I used the <() convention which is specific to bash. This lets you treat the output of a command as file, without having to actually create a file. Notice that I put quotes around the variables -- this is required, otherwise the bash will ignore newlines when expanding the variable.
I have a rule like this in a makefile:
target : dependencies
rm -rd somedirectory
runcodecoverage.exe # this generates somefile
$(eval COVERAGE=$(shell grep "blah" somefile))
#echo $(COVERAGE)
When I run this file the first time (after make cleaning) the echo doesn't print anything. But the second and times after that, it prints the correct result. If I replace the $(eval ..) line with just grep "blah" somefile I get the result I want, so the problem must be in the use of $(eval) and $(shell). Why is this happening?
Edit: I solved this by adding a new dependency, so it now looks like this:
generatesomefile :
runcodecoverage.exe # this generates somefile
target : dependencies generatesomefile
rm -rd somedirectory
$(eval COVERAGE=$(shell grep "blah" somefile))
#echo $(COVERAGE)
It seems like $(eval) was being substituted with the result of grep as soon as the "target" target was entered, even though I wanted it to be run after runcodecoverage.exe had run. In this sense the answer I accepted as correct wasn't quite right - the docs say this on variable expansion:
Rule Definition
A rule is always expanded the same way, regardless of the form:
immediate : immediate ; deferred
deferred
The eval function doesn't work the way you think it does.
The first time you run Make, it expands the eval function and carries out the variable assignment before executing any rule. And since there is no somefile, the grep returns nothing and COVERAGE stays empty. When Make executes the rule, it passes the empty variable to the echo which duly reports nothing.
The second time you run Make, once again it expands the eval function, executes the grep on somefile (which was built in the first run), and stores the result in COVERAGE. Then it executes the rule and passes COVERAGE to the echo, which puts it up on your screen.
The $(eval ...) runs when your Makefile is parsed, not when the recipe is executed.
It's not clear to me what you expect to happen; setting a Make variable from within a recipe does not seem like a sustainable approach, but it depends a lot on where and how you need to use this variable. If all you need is to output the coverage from the same recipe, replace the last couple of lines with just
grep "blah" somefile
This question already has answers here:
How to use shell commands in Makefile
(2 answers)
Closed 9 months ago.
I would like to use a loop to find some files and rename them:
for i in `find $# -name *_cu.*`;do mv $i "$(echo $i|sed s/_cu//)"
done
This works in the shell. But how can I do this in a makefile recipe?
There are two main things you need to know when putting non-trivial shell fragments into make recipes:
Commands in the recipe are (of course!) executed one at a time, where command means "tab-prefixed line in the recipe", possibly spread over several makefile lines with backslashes.
So your shell fragment has to be written all on one (possibly backslashed) line. Moreover it's effectively presented to the shell as a single line (the backslashed-newlines are not plain newlines so are not used as command terminators by the shell), so must be syntactically correct as such.
Both shell variables and make variables are introduced by dollar signs ($#, $i), so you need to hide your shell variables from make by writing them as $$i. (More precisely, any dollar sign you want to be seen by the shell must be escaped from make by writing it as $$.)
Normally in a shell script you would write separate commands on separate lines, but here you effectively only get a single line so must separate the individual shell commands with semicolons instead. Putting all this together for your example produces:
foo: bar
for i in `find $# -name *_cu.*`; do mv $$i "$$(echo $$i|sed s/_cu//)"; done
or equivalently:
foo: bar
for i in `find $# -name *_cu.*`; do \
mv $$i "$$(echo $$i|sed s/_cu//)"; \
done
Notice that the latter, even though it's laid out readably on several lines, requires the same careful use of semicolons to keep the shell happy.
I found this useful, trying to use for loops to build multiple files:
PROGRAMS = foo bar other
.PHONY all
all: $(PROGRAMS)
$(PROGRAMS):
gcc -o $# $#.c
It will compile foo.c, bar.c, other.c into foor bar other executables
I spend good time on this and finally had it working. I had an easy solution using the global variable in makefile available for all targets, however I don`t want that so this is how I did it.
target:
$(eval test_cont=$(shell sh -c "docker ps | grep test" | awk '{print $$1}'))
for container in $(test_cont);do \
docker cp ssh/id_rsa.pub $${container}:/root/.ssh/authorized_keys; \
docker exec -it $${container} chown root.root /root/.ssh/authorized_keys; \
done