I want to declare a variable using a for from bash in a makefile. That's what I tried:
SRCS="path/to/foo.c path/to/boo.c path/to/bar.c"
OBJS=$(for file in $SRCS; do TEMP="$TEMP $(basename $file .c).o"; done; echo $TEMP)
This command:
for file in $SRCS; do TEMP="$TEMP $(basename $file .c).o"; done
works on bash when echoing TEMP. But OBJS in makefile is empty. What am I missing here?
Desired output:
foo.o boo.o bar.o
First of all there's better ways to do this without using bash's for. You could use the make constructs to generate these lists. See the documentation on $(notdir), $(addprefix...) and $(addsuffix ...)
However, to answer your question on this particular example:
First, you are putting quotes around sources. Make does not interpret quotes as special characters, and thus SRCS will expand to "path/to/foo.c path/to/boo.c path/to/bar.c" (including the quotes). This will mess up your for loop later on.
The next thing is the reference to $SRCS -- make will interpret this as $S followed by the literal RCS (which is not what you want). You have to use braces around multi0letter variables in bash as so -- $(SRCS)
Next, TEMP=$TEMP.... When make sees $TEMP, it will immediately attempt to expand it. Because TEMP is not set to anything in the make context, it will expand to a null string -- this is before it invokes the bash shell... What you wanted to do in this case is use $$TEMP, which make will expand to $TEMP.
The following does what you want: note the $(info) lines are just for debugging.
SRCS:=path/to/foo.c path/to/boo.c path/to/bar.c
OBJS:=$(shell for file in $(SRCS); do TEMP="$$TEMP $$(basename $$file .c).o"; done; echo $$TEMP)
$(info for file in $(SRCS); do TEMP="$$TEMP $$(basename $$file .c).o"; done; echo $$TEMP)
$(info OBJS=$(OBJS))
If you wanted to do this in makefile without bash or sh, you might try:
OBJS:=$(notdir $(SRCS:.c=.o))
Note: this is gnu-make specific syntax and may not work on other makes.
Related
Considering that every command is run in its own shell, what is the best way to run a multi-line bash command in a makefile? For example, like this:
for i in `find`
do
all="$all $i"
done
gcc $all
You can use backslash for line continuation. However note that the shell receives the whole command concatenated into a single line, so you also need to terminate some of the lines with a semicolon:
foo:
for i in `find`; \
do \
all="$$all $$i"; \
done; \
gcc $$all
But if you just want to take the whole list returned by the find invocation and pass it to gcc, you actually don't necessarily need a multiline command:
foo:
gcc `find`
Or, using a more shell-conventional $(command) approach (notice the $ escaping though):
foo:
gcc $$(find)
As indicated in the question, every sub-command is run in its own shell. This makes writing non-trivial shell scripts a little bit messy -- but it is possible! The solution is to consolidate your script into what make will consider a single sub-command (a single line).
Tips for writing shell scripts within makefiles:
Escape the script's use of $ by replacing with $$
Convert the script to work as a single line by inserting ; between commands
If you want to write the script on multiple lines, escape end-of-line with \
Optionally start with set -e to match make's provision to abort on sub-command failure
This is totally optional, but you could bracket the script with () or {} to emphasize the cohesiveness of a multiple line sequence -- that this is not a typical makefile command sequence
Here's an example inspired by the OP:
mytarget:
{ \
set -e ;\
msg="header:" ;\
for i in $$(seq 1 3) ; do msg="$$msg pre_$${i}_post" ; done ;\
msg="$$msg :footer" ;\
echo msg=$$msg ;\
}
The ONESHELL directive allows to write multiple line recipes to be executed in the same shell invocation.
all: foo
SOURCE_FILES = $(shell find . -name '*.c')
.ONESHELL:
foo: ${SOURCE_FILES}
FILES=()
for F in $^; do
FILES+=($${F})
done
gcc "$${FILES[#]}" -o $#
There is a drawback though : special prefix characters (‘#’, ‘-’, and ‘+’) are interpreted differently.
https://www.gnu.org/software/make/manual/html_node/One-Shell.html
Of course, the proper way to write a Makefile is to actually document which targets depend on which sources. In the trivial case, the proposed solution will make foo depend on itself, but of course, make is smart enough to drop a circular dependency. But if you add a temporary file to your directory, it will "magically" become part of the dependency chain. Better to create an explicit list of dependencies once and for all, perhaps via a script.
GNU make knows how to run gcc to produce an executable out of a set of .c and .h files, so maybe all you really need amounts to
foo: $(wildcard *.h) $(wildcard *.c)
What's wrong with just invoking the commands?
foo:
echo line1
echo line2
....
And for your second question, you need to escape the $ by using $$ instead, i.e. bash -c '... echo $$a ...'.
EDIT: Your example could be rewritten to a single line script like this:
gcc $(for i in `find`; do echo $i; done)
Consider the following Makefile.
$(shell touch /tmp/example.txt)
FILE := /tmp/example.txt
CONTENTS = $(shell cat $(FILE); bash -c 'echo [debugging id: $$RANDOM]')
.PHONY: all
all:
#cat $(FILE)
#echo '$$(CONTENTS):' $(CONTENTS)
bash -c 'echo file-contents-$$RANDOM' > $(FILE)
#cat $(FILE)
#echo '$$(CONTENTS):' $(CONTENTS) # This line outputs the old contents. Why?
It prints the contents of the file, overwrites with new contents and prints the contents again. It shows as (after second shots of make):
file-contents-1543
$(CONTENTS): file-contents-1543 [debugging id: 15172]
bash -c 'echo file-contents-$RANDOM' > /tmp/example.txt
file-contents-22441
$(CONTENTS): file-contents-1543 [debugging id: 151]
The old content is file-contents-1543 and new content is file-contents-22441 (the numbers are random), but the last line echo $(CONTENTS) does not print the new contents.
I think the command is actually called twice as debugging ids show but shell function in the lazy variable seems to be executed before writing the new contents to the file.
I expect that lazy variable in Makefile is evaluated every time it is referred, the echo $(CONTENTS) command always prints the latest file contents. What am I wrong?
By the way, I found that using CONTENTS = $$(cat $(FILE)) works as I expect. I will using this instead of shell function but is it ok?
I expect that lazy variable in Makefile is evaluated every time it is referred, the echo $(CONTENTS) command always prints the latest file contents. What am I wrong?
First of all, in make's slang these variables are called recursive, not lazy. And, yes, they get expanded (i.e. recursively substituted) each time they are referred with $(CONTENTS). Considering that $(eval...) and $(shell...) (as pretty much anything looking as $(...)) also went through the same (recursive) expansion procedure (albeit, with some "side-effects"), each expansion of such variable could also result in some sort of "evaluation" or "execution".
Next, the order of expansion in make is a bit specific. In particular, the recipes (i.e. the lines starting with [tab]) are expanded after the whole makefile was (pre-)processed, but before the first line of the recipe gets executed by shell. Which is the main source of your confusion, I suppose.
I found that using CONTENTS = $$(cat $(FILE)) works as I expect
$$ is a way to get a single literal $ after an expansion procedure. So $$(cat $(FILE)) when expanded becomes $(cat /tmp/example.txt) which is a legal syntax for command substitution in bash. This means it will work only as part of a bash command (recipe line). If that is what you want then it's okay.
I have a bash shell script which I usually source into my shell, with lots of environment variables defined, which are not exported. I do not want to:
Export the variables, because this would make the exportable environment too big, and eventually make the whole system slower (it must be exported when running every command from the shell)
Redefine those variables in the makefile (DRY)
I would like to source the same shell script into the environment of the makefile, so that I can access those variables. Is this possible? How can I do that? Ideally I would do in the makefile:
source setup-env.sh
There is not source command for makefiles, but maybe something equivalent? Any special hack I can use to simulate the same effect?
As per the additional question in the comment, here is one way to effectively mark the whole environment as exported:
for var in $(compgen -v); do export $var; done
compgen -v simply outputs all variable names, as per the bash manual, section 8.7 Programmable Completion Builtins. Then we simply loop over this list and export each one.
Credit to https://stackoverflow.com/a/16337687/2113226 - compgen is new to me.
There are two ways I can think of to integrate this into your make workflow:
- Shell script wrapper
Simply write a shell script which sources your setup-env.sh, exports all variables as above, then calls make itself. Something like:
#!/bin/bash
./source setup-env.sh
for var in $(compgen -v); do export $var; done
make $#
- Recursive make
It may be that you don't want a shell script wrapper, and want to directly invoke make for whatever reason. You can do this all in one Makefile which calls itself recursively:
$(info MAKELEVEL=$(MAKELEVEL) myvar=$(myvar))
ifeq ($(MAKELEVEL), 0)
all:
bash -c "source ./setup-env.sh; \
for var in \$$(compgen -v); do export \$$var; done; \
$(MAKE) $#"
else
all: myprog
myprog:
echo "Recipe for myprog. myvar=$(myvar)"
endif
Output for this Makefile is:
$ make
MAKELEVEL=0 myvar=
bash -c "source ./setup-env.sh; \
for var in \$(compgen -v); do export \$var; done; \
make all"
MAKELEVEL=1 myvar=Hello World
make[1]: Entering directory `/home/ubuntu/makesource'
echo "Recipe for myprog. myvar=Hello World"
Recipe for myprog. myvar=Hello World
make[1]: Leaving directory `/home/ubuntu/makesource'
$
We check the GNU Make builtin variable MAKELEVEL to see what level of recursion we are at. if the level is 0, then we recursively call make for all targets, but first source ./setup-env.sh and export all variables. If the recursion level is anything else, we just do the normal makefile stuff, but you see that the variables you need are now available. This is highlighted by the $(info ) line at the top of the Makefile, which shows the recursion level, and the value (or not) of myvar.
Notes:
We have to use bash -c because compgen is strictly a bash builtin, and not available in Posix mode - i.e. when make invokes the shell as sh -c by default.
The $ in the first all: recipe need to be escaped very carefully. The $$ escapes the $ from being expanded by make, and the \$$ escapes the $ from being expanded by the implicit sh
There is plenty of literature arguing that "Recursive make is considered harmful". E.g. http://aegis.sourceforge.net/auug97.pdf
I have a variable containing list of files separated with string _NEWLINE_. I need to output that variable into a file so that each file is in a separate line. The trick is that it needs to works on FreeBSD and Solaris.
This is what I am trying now:
echo "lib/alarms-1.2/priv/snmp_conf/agent.conf: lib/alarms/priv/snmp_conf/agent.conf_NEWLINE_lib/alarms-1.2/priv/snmp_conf/agent.conf.src: lib/alarms/priv/snmp_conf/agent.conf.src_NEWLINE_lib/alarms-1.2/priv/snmp_conf/community.conf: lib/alarms/priv/snmp_conf/community.conf" | sed 's|_NEWLINE_|\'$'\n|g'
This works on FreeBSD and in shell on Solaris. But when run in GNUmakefile on Solaris I am getting this (notice $ at the end of each line):
lib/alarms-1.2/priv/snmp_conf/agent.conf: lib/alarms/priv/snmp_conf/agent.conf$
lib/alarms-1.2/priv/snmp_conf/agent.conf.src: lib/alarms/priv/snmp_conf/agent.conf.src$
lib/alarms-1.2/priv/snmp_conf/community.conf: lib/alarms/priv/snmp_conf/community.conf$
If I remove \'$' from sed then it works on Solaris but doesn't on FreeBSD. Maybe there is a way of telling which version to use depending on which system the makefile is executed?
EDIT:
Thanks to the solution proposed by bobbogo I created an exemplary makefile that provides the desired outcome and seems to be working on both FreeBSD and Solaris:
one-line := lib/alarms-1.2/priv/snmp_conf/agent.conf: lib/alarms/priv/snmp_conf/agent.conf_NEWLINE_lib/alarms-1.2/\
priv/snmp_conf/agent.conf.src: lib/alarms/priv/snmp_conf/agent.conf.src_NEWLINE_lib/alarms-1.2/priv/snmp_conf/comm\
unity.conf: lib/alarms/priv/snmp_conf/community.conf
many-lines := { echo '$(subst _NEWLINE_,' && echo ',${one-line})'; }
.PHONY: all
all:
$(shell $(many-lines) > test.txt)
If this is GNU make, then do it all in make.
one-line := lib/alarms-1.2/priv/snmp_conf/agent.conf: lib/alarms/priv/snmp_conf/agent.conf_NEWLINE_lib/alarms-1.2/priv/snmp_conf/agent.conf.src: lib/alarms/priv/snmp_conf/agent.conf.src_NEWLINE_lib/alarms-1.2/priv/snmp_conf/community.conf: lib/alarms/priv/snmp_conf/community.conf
define \n
endef
many-lines := $(subst _NEWLINE_,${\n},${one-line})
Now ${many-lines} has just what you want. Annoyingly, it's quite hard to use in shell lines. If you do this:
tgt:
echo '${many-lines}'
make will invoke a separate shell for each line. The first shell invocation will get an un-paired ' and exit with an error.
.ONESHELL:
tgt:
echo '${many-lines}'
will work in an invasive sort of way. The proper fix is to ensure each line of ${many-lines} has valid sh syntax. Some mouthfull like:
echolines = $(subst ${\n},'${\n}echo ',echo '${many-lines}')
.PHONY: aa
aa:
$(call echolines,${many-lines})
Sheesh.
Tried many different solutions, including defining \n as mentioned in this answer: Add a newline in Makefile 'foreach' loop
The real problem is inconsistent implementation of the echo command across platforms, and the fact that by default make invokes shell commands using sh, which itself is quite inflexible.
I found a better way thanks to this answer: "echo -e" when called directly and when called via a shell-script
The better way is to use printf instead of echo
Construct the string with \n instead of _NEWLINE_ to separate parts that go into separate lines in the output file:
some_string = lib/alarms-1.2/priv/snmp_conf/target_params.conf: lib/alarms/priv/snmp_conf/target_params.conf\nlib/alarms-1.2/priv/snmp_conf/community.conf: lib/alarms/priv/snmp_conf/community.conf\n
and then in the makefile print it simply as this:
#printf "$(some_string)" >> $(some_file)
Works on both, FreeBSD and Solaris.
Disclaimer: I have no experience with Solaris or FreeBSD... here goes anyway.
In make, you can use $(patsubst pattern,replacement,text) to substitute a pattern.
try this...
FILENAMES := "lib/alarms-1.2/priv/snmp_conf/agent.conf: lib/alarms/priv/snmp_conf/agent.conf_NEWLINE_lib/alarms-1.2/priv/snmp_conf/agent.conf.src: lib/alarms/priv/snmp_conf/agent.conf.src_NEWLINE_lib/alarms-1.2/priv/snmp_conf/community.conf: lib/alarms/priv/snmp_conf/community.conf"
.PHONY: all
all:
#echo $(patsubst _NEWLINE_,${\n},$(FILENAMES))
As an alternative, I think your first approach will work, if you just double the $ to "escape" it:
sed 's|_NEWLINE_|\'$$'\n|g'
Considering that every command is run in its own shell, what is the best way to run a multi-line bash command in a makefile? For example, like this:
for i in `find`
do
all="$all $i"
done
gcc $all
You can use backslash for line continuation. However note that the shell receives the whole command concatenated into a single line, so you also need to terminate some of the lines with a semicolon:
foo:
for i in `find`; \
do \
all="$$all $$i"; \
done; \
gcc $$all
But if you just want to take the whole list returned by the find invocation and pass it to gcc, you actually don't necessarily need a multiline command:
foo:
gcc `find`
Or, using a more shell-conventional $(command) approach (notice the $ escaping though):
foo:
gcc $$(find)
As indicated in the question, every sub-command is run in its own shell. This makes writing non-trivial shell scripts a little bit messy -- but it is possible! The solution is to consolidate your script into what make will consider a single sub-command (a single line).
Tips for writing shell scripts within makefiles:
Escape the script's use of $ by replacing with $$
Convert the script to work as a single line by inserting ; between commands
If you want to write the script on multiple lines, escape end-of-line with \
Optionally start with set -e to match make's provision to abort on sub-command failure
This is totally optional, but you could bracket the script with () or {} to emphasize the cohesiveness of a multiple line sequence -- that this is not a typical makefile command sequence
Here's an example inspired by the OP:
mytarget:
{ \
set -e ;\
msg="header:" ;\
for i in $$(seq 1 3) ; do msg="$$msg pre_$${i}_post" ; done ;\
msg="$$msg :footer" ;\
echo msg=$$msg ;\
}
The ONESHELL directive allows to write multiple line recipes to be executed in the same shell invocation.
all: foo
SOURCE_FILES = $(shell find . -name '*.c')
.ONESHELL:
foo: ${SOURCE_FILES}
FILES=()
for F in $^; do
FILES+=($${F})
done
gcc "$${FILES[#]}" -o $#
There is a drawback though : special prefix characters (‘#’, ‘-’, and ‘+’) are interpreted differently.
https://www.gnu.org/software/make/manual/html_node/One-Shell.html
Of course, the proper way to write a Makefile is to actually document which targets depend on which sources. In the trivial case, the proposed solution will make foo depend on itself, but of course, make is smart enough to drop a circular dependency. But if you add a temporary file to your directory, it will "magically" become part of the dependency chain. Better to create an explicit list of dependencies once and for all, perhaps via a script.
GNU make knows how to run gcc to produce an executable out of a set of .c and .h files, so maybe all you really need amounts to
foo: $(wildcard *.h) $(wildcard *.c)
What's wrong with just invoking the commands?
foo:
echo line1
echo line2
....
And for your second question, you need to escape the $ by using $$ instead, i.e. bash -c '... echo $$a ...'.
EDIT: Your example could be rewritten to a single line script like this:
gcc $(for i in `find`; do echo $i; done)