Say I have the following function defined in my .bashrc:
foo (){
touch $1
}
I want to call this function in a makefile like so:
%.png : %.tex
bash -i -c foo $<
But this complains about a missing operand. (The error is "touch: missing file operand", so it is calling touch...). I've seen some solutions to similar issues, but they aren't applicable since I need the automatic variable $< (or rather, its value) to be passed as an argument to the shell script.
I know I could work around this problem (copy the function wholesale into the makefile, for example) but I'm curious to see how I could solve my problem within the makefile.
Your problem is probably because you don't have enough quoting. The shell -c option takes its script as a single argument.
Consider:
$ bash -c echo hi
<prints nothing>
Versus:
$ bash -c 'echo hi'
hi
Related
Considering that every command is run in its own shell, what is the best way to run a multi-line bash command in a makefile? For example, like this:
for i in `find`
do
all="$all $i"
done
gcc $all
You can use backslash for line continuation. However note that the shell receives the whole command concatenated into a single line, so you also need to terminate some of the lines with a semicolon:
foo:
for i in `find`; \
do \
all="$$all $$i"; \
done; \
gcc $$all
But if you just want to take the whole list returned by the find invocation and pass it to gcc, you actually don't necessarily need a multiline command:
foo:
gcc `find`
Or, using a more shell-conventional $(command) approach (notice the $ escaping though):
foo:
gcc $$(find)
As indicated in the question, every sub-command is run in its own shell. This makes writing non-trivial shell scripts a little bit messy -- but it is possible! The solution is to consolidate your script into what make will consider a single sub-command (a single line).
Tips for writing shell scripts within makefiles:
Escape the script's use of $ by replacing with $$
Convert the script to work as a single line by inserting ; between commands
If you want to write the script on multiple lines, escape end-of-line with \
Optionally start with set -e to match make's provision to abort on sub-command failure
This is totally optional, but you could bracket the script with () or {} to emphasize the cohesiveness of a multiple line sequence -- that this is not a typical makefile command sequence
Here's an example inspired by the OP:
mytarget:
{ \
set -e ;\
msg="header:" ;\
for i in $$(seq 1 3) ; do msg="$$msg pre_$${i}_post" ; done ;\
msg="$$msg :footer" ;\
echo msg=$$msg ;\
}
The ONESHELL directive allows to write multiple line recipes to be executed in the same shell invocation.
all: foo
SOURCE_FILES = $(shell find . -name '*.c')
.ONESHELL:
foo: ${SOURCE_FILES}
FILES=()
for F in $^; do
FILES+=($${F})
done
gcc "$${FILES[#]}" -o $#
There is a drawback though : special prefix characters (‘#’, ‘-’, and ‘+’) are interpreted differently.
https://www.gnu.org/software/make/manual/html_node/One-Shell.html
Of course, the proper way to write a Makefile is to actually document which targets depend on which sources. In the trivial case, the proposed solution will make foo depend on itself, but of course, make is smart enough to drop a circular dependency. But if you add a temporary file to your directory, it will "magically" become part of the dependency chain. Better to create an explicit list of dependencies once and for all, perhaps via a script.
GNU make knows how to run gcc to produce an executable out of a set of .c and .h files, so maybe all you really need amounts to
foo: $(wildcard *.h) $(wildcard *.c)
What's wrong with just invoking the commands?
foo:
echo line1
echo line2
....
And for your second question, you need to escape the $ by using $$ instead, i.e. bash -c '... echo $$a ...'.
EDIT: Your example could be rewritten to a single line script like this:
gcc $(for i in `find`; do echo $i; done)
Consider the following Makefile.
$(shell touch /tmp/example.txt)
FILE := /tmp/example.txt
CONTENTS = $(shell cat $(FILE); bash -c 'echo [debugging id: $$RANDOM]')
.PHONY: all
all:
#cat $(FILE)
#echo '$$(CONTENTS):' $(CONTENTS)
bash -c 'echo file-contents-$$RANDOM' > $(FILE)
#cat $(FILE)
#echo '$$(CONTENTS):' $(CONTENTS) # This line outputs the old contents. Why?
It prints the contents of the file, overwrites with new contents and prints the contents again. It shows as (after second shots of make):
file-contents-1543
$(CONTENTS): file-contents-1543 [debugging id: 15172]
bash -c 'echo file-contents-$RANDOM' > /tmp/example.txt
file-contents-22441
$(CONTENTS): file-contents-1543 [debugging id: 151]
The old content is file-contents-1543 and new content is file-contents-22441 (the numbers are random), but the last line echo $(CONTENTS) does not print the new contents.
I think the command is actually called twice as debugging ids show but shell function in the lazy variable seems to be executed before writing the new contents to the file.
I expect that lazy variable in Makefile is evaluated every time it is referred, the echo $(CONTENTS) command always prints the latest file contents. What am I wrong?
By the way, I found that using CONTENTS = $$(cat $(FILE)) works as I expect. I will using this instead of shell function but is it ok?
I expect that lazy variable in Makefile is evaluated every time it is referred, the echo $(CONTENTS) command always prints the latest file contents. What am I wrong?
First of all, in make's slang these variables are called recursive, not lazy. And, yes, they get expanded (i.e. recursively substituted) each time they are referred with $(CONTENTS). Considering that $(eval...) and $(shell...) (as pretty much anything looking as $(...)) also went through the same (recursive) expansion procedure (albeit, with some "side-effects"), each expansion of such variable could also result in some sort of "evaluation" or "execution".
Next, the order of expansion in make is a bit specific. In particular, the recipes (i.e. the lines starting with [tab]) are expanded after the whole makefile was (pre-)processed, but before the first line of the recipe gets executed by shell. Which is the main source of your confusion, I suppose.
I found that using CONTENTS = $$(cat $(FILE)) works as I expect
$$ is a way to get a single literal $ after an expansion procedure. So $$(cat $(FILE)) when expanded becomes $(cat /tmp/example.txt) which is a legal syntax for command substitution in bash. This means it will work only as part of a bash command (recipe line). If that is what you want then it's okay.
I've been learning make and am struggling to figure something out. I have some rules with this general structure.
FILE = "myfile.txt"
test :
YOUR = $(subst my,your,$(FILE));\
cat $(FILE) $(YOUR)
I would expect the end result to be running the command:
cat myfile.txt yourfile.txt
Instead I get the following...
YOUR = "yourfile.txt";\
cat "myfile.txt"
/bin/sh: YOUR: command not found
make: *** [test] Error 1
If instead of using the subst function, I just do YOUR="yourfile" in the makefile, everything looks fine. Any suggestions or have I missed something pretty fundamental? I should add that I'm using tabs and not spaces to start the lines for the commands within the rule.
FILE = "myfile.txt"
test :
$(eval YOUR = $(subst my,your,$(FILE)))
cp $(FILE) $(YOUR)
You have to use the eval function in the recipe (Define make variable at rule execution time)
You need to distinguish between what make executes and what the shell executes. Your line with YOUR = starts with a tab and is part of the actions of a rule, so it is executed by the shell, which can't find a program YOUR to execute with some arguments.
Place the expansion outside the rule:
YOUR = $(subst my,your,$(FILE))
test:
cat $(FILE) $(YOUR)
Note that shell assignments require no space around the equals sign, and use ${} rather than $() to reference variables: YOUR=${FILE/my/your} in Bash (and if written in a make rule, you'd need $$ in place of $ so that the shell sees a single dollar sign and make does not try the variable expansion that it doesn't understand). The shell uses $() to execute the command contained within, and the result is often captured in a variable: YOUR=$(echo "${FILE}" | sed 's/my/your/').
If you only need the variable in the shell recipe and not in the make context then you don't need to bother playing with eval (which are hoisted) and can just assign to shell variables instead.
For example:
FILE = "myfile.txt"
test :
YOUR='$(subst my,your,$(FILE))';\
cat $(FILE) "$${YOUR}"
I have a bash shell script which I usually source into my shell, with lots of environment variables defined, which are not exported. I do not want to:
Export the variables, because this would make the exportable environment too big, and eventually make the whole system slower (it must be exported when running every command from the shell)
Redefine those variables in the makefile (DRY)
I would like to source the same shell script into the environment of the makefile, so that I can access those variables. Is this possible? How can I do that? Ideally I would do in the makefile:
source setup-env.sh
There is not source command for makefiles, but maybe something equivalent? Any special hack I can use to simulate the same effect?
As per the additional question in the comment, here is one way to effectively mark the whole environment as exported:
for var in $(compgen -v); do export $var; done
compgen -v simply outputs all variable names, as per the bash manual, section 8.7 Programmable Completion Builtins. Then we simply loop over this list and export each one.
Credit to https://stackoverflow.com/a/16337687/2113226 - compgen is new to me.
There are two ways I can think of to integrate this into your make workflow:
- Shell script wrapper
Simply write a shell script which sources your setup-env.sh, exports all variables as above, then calls make itself. Something like:
#!/bin/bash
./source setup-env.sh
for var in $(compgen -v); do export $var; done
make $#
- Recursive make
It may be that you don't want a shell script wrapper, and want to directly invoke make for whatever reason. You can do this all in one Makefile which calls itself recursively:
$(info MAKELEVEL=$(MAKELEVEL) myvar=$(myvar))
ifeq ($(MAKELEVEL), 0)
all:
bash -c "source ./setup-env.sh; \
for var in \$$(compgen -v); do export \$$var; done; \
$(MAKE) $#"
else
all: myprog
myprog:
echo "Recipe for myprog. myvar=$(myvar)"
endif
Output for this Makefile is:
$ make
MAKELEVEL=0 myvar=
bash -c "source ./setup-env.sh; \
for var in \$(compgen -v); do export \$var; done; \
make all"
MAKELEVEL=1 myvar=Hello World
make[1]: Entering directory `/home/ubuntu/makesource'
echo "Recipe for myprog. myvar=Hello World"
Recipe for myprog. myvar=Hello World
make[1]: Leaving directory `/home/ubuntu/makesource'
$
We check the GNU Make builtin variable MAKELEVEL to see what level of recursion we are at. if the level is 0, then we recursively call make for all targets, but first source ./setup-env.sh and export all variables. If the recursion level is anything else, we just do the normal makefile stuff, but you see that the variables you need are now available. This is highlighted by the $(info ) line at the top of the Makefile, which shows the recursion level, and the value (or not) of myvar.
Notes:
We have to use bash -c because compgen is strictly a bash builtin, and not available in Posix mode - i.e. when make invokes the shell as sh -c by default.
The $ in the first all: recipe need to be escaped very carefully. The $$ escapes the $ from being expanded by make, and the \$$ escapes the $ from being expanded by the implicit sh
There is plenty of literature arguing that "Recursive make is considered harmful". E.g. http://aegis.sourceforge.net/auug97.pdf
Considering that every command is run in its own shell, what is the best way to run a multi-line bash command in a makefile? For example, like this:
for i in `find`
do
all="$all $i"
done
gcc $all
You can use backslash for line continuation. However note that the shell receives the whole command concatenated into a single line, so you also need to terminate some of the lines with a semicolon:
foo:
for i in `find`; \
do \
all="$$all $$i"; \
done; \
gcc $$all
But if you just want to take the whole list returned by the find invocation and pass it to gcc, you actually don't necessarily need a multiline command:
foo:
gcc `find`
Or, using a more shell-conventional $(command) approach (notice the $ escaping though):
foo:
gcc $$(find)
As indicated in the question, every sub-command is run in its own shell. This makes writing non-trivial shell scripts a little bit messy -- but it is possible! The solution is to consolidate your script into what make will consider a single sub-command (a single line).
Tips for writing shell scripts within makefiles:
Escape the script's use of $ by replacing with $$
Convert the script to work as a single line by inserting ; between commands
If you want to write the script on multiple lines, escape end-of-line with \
Optionally start with set -e to match make's provision to abort on sub-command failure
This is totally optional, but you could bracket the script with () or {} to emphasize the cohesiveness of a multiple line sequence -- that this is not a typical makefile command sequence
Here's an example inspired by the OP:
mytarget:
{ \
set -e ;\
msg="header:" ;\
for i in $$(seq 1 3) ; do msg="$$msg pre_$${i}_post" ; done ;\
msg="$$msg :footer" ;\
echo msg=$$msg ;\
}
The ONESHELL directive allows to write multiple line recipes to be executed in the same shell invocation.
all: foo
SOURCE_FILES = $(shell find . -name '*.c')
.ONESHELL:
foo: ${SOURCE_FILES}
FILES=()
for F in $^; do
FILES+=($${F})
done
gcc "$${FILES[#]}" -o $#
There is a drawback though : special prefix characters (‘#’, ‘-’, and ‘+’) are interpreted differently.
https://www.gnu.org/software/make/manual/html_node/One-Shell.html
Of course, the proper way to write a Makefile is to actually document which targets depend on which sources. In the trivial case, the proposed solution will make foo depend on itself, but of course, make is smart enough to drop a circular dependency. But if you add a temporary file to your directory, it will "magically" become part of the dependency chain. Better to create an explicit list of dependencies once and for all, perhaps via a script.
GNU make knows how to run gcc to produce an executable out of a set of .c and .h files, so maybe all you really need amounts to
foo: $(wildcard *.h) $(wildcard *.c)
What's wrong with just invoking the commands?
foo:
echo line1
echo line2
....
And for your second question, you need to escape the $ by using $$ instead, i.e. bash -c '... echo $$a ...'.
EDIT: Your example could be rewritten to a single line script like this:
gcc $(for i in `find`; do echo $i; done)