The Automake Manual says
15.2.3 Parallel Test Harness
You can set the TEST_LOGS variable. By default, this variable is computed at
make run time from the value of TESTS as described above. For example, you
can use the following:
set x subset*.log; shift
env TEST_LOGS="foo.log $*" make -e check
What does setting TEST_LOGS do? It is read only in the Makefile file and write only in the shell.
TEST_LOGS contains the name of all .log files. I get that. What happens if the user changes one or all the names. Suppose Makefile looks something like:
TESTS = foo.exe
Then we have
TEST_LOGS = foo.log
And the user sets this to
env TEST_LOGS=bar.log make -e check
What on earth does this mean?
And the examples with TEST_EXTENSIONS and duplicate base names (test cases
with extensions removed, like TESTS = test.sh test.pl and TEST_EXTENSIONS =
.pl .sh) grows even more complex.
Any idea?
And what does
set x subset*.log; shift
have to do with the example. It looks like a fragment of some script erroneously put it the example. Certainly 'shift' doesn't apply here.
Related
I'm trying to include an external preprocess script to my current Makefile flow.
The external script location is /home/preprocessscript.sh, and content is:
cd ${path}; rm -rf *
Makefile content is:
path=/home/rubbish
clean:
cat ${preprocessscript} >> clean.sh
I execute the command by:
make clean preprocessscript=/home/preprocessscript.sh
./clean.sh
The problem is clean.sh doesn't get path. Its content will be:
cd ; rm -rf *
How do I pass the make variable to preprocessscript?
The problem is clean.sh doesn't get path. Its content will be:
cd ; rm -rf *
I'm sorry about the contents of your home directory.
Anyway, no. The cat command will output the contents of the original file verbatim. It will not interpret them in any way, and in particular, it will not attempt to expand shell-style variable references within.
Supposing that you have faithfully represented your situation, what you actually see is that when you run the resulting clean.sh as a script, the ${path} within expands to nothing. This is not particularly surprising, because nowhere in what you've presented is an environment variable named path defined. You have defined a make variable of that name, but that's not the same thing, and it anyway would not have an effect persisting past the end of the make run to when the generated script is executed.
I guess the idea is that you want to use the original file as a template to generate content for clean.sh. There are lots of ways to do that sort of thing, but the one closest to your current attempt would probably be to output (shell) variable assignments into the generated script, like so:
path = /home/rubbish
clean:
echo "path='$(path)'" >> clean.sh
cat ${preprocessscript} >> clean.sh
The generated script lines would then be
path='/home/rubbish'
cd ${path}; rm -rf *
Note also that there appear to be several other issues with the general approach presented in the question, but I am focusing my comments on the specific question asked.
You can export a make variable to the shell before running your command and use envsubst to expand the variables in the file as so:
path=/home/rubbish
clean:
export path=${path}; envsubst < ${preprocessscript} >> clean.sh
notice that the export has to be on the same recipe line as the command. You can also pipe the output of a command to envsubst: cat ${preprocessscript} | envsubst >> clean.sh. This might be useful if you're trying to do something more complicated like only print a few lines, etc.
I have a script "set_env.py" that outputs the following uppon execution:
export MY_VAR_1=some_value
export MY_VAR_2=some_other_value
I cannot change this script, it is supplied in my current environment.
Further I have a Makefile that looks like this:
SHELL := /bin/bash
all: set_env
env | grep MY_VAR
set_env:
eval ./set_env.py
With this makefile I expected the grep to list my two variables, however it seems the environment is not set.
I suspect this is because make creates a sub-environment for each line and so the variables set on the first line will not be available in the second.
So the question is, how would I go about exporting the environment from the untouchable script in my makefile?
Actually, the output of the python is valid make.
One option then is to read the output of the python directly into the makefile.
The only fly in the ointment is that $(shell) doesn't cut the mustard.
include Environment.mk
PHONY: test
test:
env | grep MY_VAR
Environment.mk:
./set_env.py >$#-tmp
mv $#-tmp $#
How does this work?
The first thing that make tries to do is to ensure the makefile itself is up-to-date.
Since we have told it to include Environment.mk,
make must ensure that is up-to-date too.
Make finds a rule for Environment.mk
The python is run, creating Environment.mk
Environment.mk is read in, creating two make variables with the export attribute
The makefile is now up-to-date, so make proceeds on to the target (test in this case)
Make runs test's recipe, exporting any variables with the export attribute.
No recursion, but you should ensure the python always spits out make compatible syntax.
EDIT
As #raspy points out, this is not the whole story.
As it stands,
once Environment.mk has been created,
it will never be regenerated.
If set_env.py ever generates different output,
you should tell make what conditions these are by adding dependencies.
If set_env.py takes a trivial time to run,
I advise a simple .PHONY.
That way it will run every time you run make,
and Environment.mk will never be stale.
.PHONY: Environment.mk
Recursive make is your friend I'm afraid.
.PHONY: all
all:
eval $$(./set_env.py) && ${MAKE} test
.PHONY: test
test:
env | grep MY_VAR
There are a few moving parts here.
make all executes the shell command eval $(./set_env.py) && make test
The shell first does the command substitution
$(./set_env.py) is replaced by the export commands
The export commands are passed to the (shell) eval command
The environment variables are defined, but only for this shell invocation
The eval succeeds, so the execution passes to the command after the &&
Make runs recursively, but this second make has an augmented environment
I'm looking how to export from a Makefile environment variables to be exposed in the userland environment so exporting these variables from the Makefile should be accessible from the user shell.
I have tried make's export but as I understand and have tried does not export to outside of Makefile.
The idea of this is to populate Docker Compose environment variables in a elegant way and have these variables ready to use in the user shell also.
This is a fragment of what I've tried with make's export:
include docker.env
export $(shell sed -n '/=/p' docker.env)
SHELL := /bin/bash
run:
#docker-compose -f my-service.yml up -d
According with ArchWiki, each process of Bash...
Each process stores their environment in the /proc/$PID/environ file.
so once Make execute a source, export or any other command to set a new environment variable it will be applied only for that process.
As workaround I've written in the bash startup file so the variables will be in the global environment as soon as a new bash shell is loaded:
SHELL := /bin/bash
RC := ~/.bashrc
ENV := $(shell sed -n '/=/p' docker.env)
test:
#$(foreach e,$(ENV),echo $(e) >> $(RC);) \
EDIT completely reworked the answer after the OP explained in a comment that he wants the environment variables to be defined for any user shell.
If your goal is to have a set of environment variables defined for any user shell (I assume this means interactive shell), you can simply add these definitions to the shell's startup file (.bashrc for bash). From GNU make manual:
Variables in make can come from the environment in which make is run.
Every environment variable that make sees when it starts up is
transformed into a make variable with the same name and value.
However, an explicit assignment in the makefile, or with a command
argument, overrides the environment. (If the ‘-e’ flag is specified,
then values from the environment override assignments in the makefile.
See Summary of Options. But this is not recommended practice.)
Example:
$ cat .bashrc
...
export FOOBAR=foobar
export BARFOO="bar foo"
...
$ cat Makefile
all:
#printf '$$(FOOBAR)=%s\n' '$(FOOBAR)'
#printf 'FOOBAR='; printenv FOOBAR
#printf '$$(BARFOO)=%s\n' '$(BARFOO)'
#printf 'BARFOO='; printenv BARFOO
$ make
$(FOOBAR)=foobar
FOOBAR=foobar
$(BARFOO)=bar foo
BARFOO=bar foo
If you want to keep these definitions separate, you can just source the file from .bashrc:
$ cat docker.env
export FOOBAR=foobar
export BARFOO="bar foo"
$ cat .bashrc
...
source <some-path>/docker.env
...
And finally, if you don't want to add the export bash command to your file, you can parse the file in your .bashrc:
$ cat docker.env
FOOBAR=foobar
BARFOO="bar foo"
$ cat .bashrc
...
while read -r line; do
eval "export $$line"
done < <(sed -n '/=/p' <some-path>/docker.env)
...
Of course, there are some constraints for the syntax of your docker.env file (no unquoted special characters, no spaces in variable names, properly quoted values...) If your syntax is not bash-compatible it is time to ask another question about parsing this specific syntax and converting it into bash-compatible syntax.
Make cannot change the calling shell's environment without its cooperation. Of course, if you are in control, you can make the calling shell cooperate.
In broad terms, you could replace the make command with a shell alias or function which runs the real make and also sets the environment variables from the result. I will proceed to describe in more detail one way to implement this.
Whether you call this alias or function of yours make or e.g. compose is up to you really. To wrap the real make is marginally harder -- inside the function, you need to say command make, because just make would cause an infinite loop with the alias or function calling itself recursively -- so I will demonstrate this. Let's define a function (aliases suck);
make () {
# run the real make, break out on failure
command make "$#" || return
# if there is no env for us to load, we are done
test -f ./docker.env || return 0
# still here? load it
. ./docker.env
}
If you want even stricter control, maybe define a variable in the function and check inside the Makefile that the variable is set.
$(ifneq '${_composing}','function_make')
$(error Need to use the wrapper function to call make)
$(endif)
The error message is rather bewildering if you haven't read this discussion, so maybe it needs to be improved, and/or documented in a README or something. You would change the make line in the function above into
_composing='function_make' \
command make "$#" || return
The syntax var=value cmd args sets the variable var to the string value just for the duration of running the command line cmd args; it then returns to its previous state (unset, or set to its previous value).
For this particular construction, the name of the variable just needs to be reasonably unique and transparent to a curious human reader; and the value is also just a reasonably unique and reasonably transparent string which the function and the Makefile need to agree on.
Depending on what you end up storing in the environment, this could introduce complications if you need this mechanism for multiple Makefiles. Running it in directory a and then switching to a similar directory b will appear to work, but uses the a things where the poor puny human would expect the b things. (If the variables you set contain paths, relative paths fix this scenario, but complicate others.)
Extending this to a model similar to Ruby's rvm or Python's virtualenv might be worth exploring; they typically add an indicator to the shell prompt to remind you which environment is currently active, and have some (very modest) safeguards in place to warn you when your current directory and the environment disagree.
Another wart: Hardcoding make to always load docker.env is likely to produce unwelcome surprises one day. Perhaps hardcode a different file name which is specific to this hook - say, .compose_post_make_hook? It can then in turn contain something like
. ./docker.env
in this particular directory.
I have several dirs with files stamp1.txt and stamp0.txt, and i want override cat command. I need it for example to suppress 'stamp1' files from archiving into library.
So i wrote little filter program called 'realname' and bash script to override original cat command.
function cat() {
local e=""
for s in $#
do
if realname $s; then
e=$e" "$s;
fi
done
command cat $e;
}
So command:
cat dir1/stamp1.txt dir2/stamp0.txt
will be converted to
cat dir2/stamp0.txt
And this example works just fine
ar cruv some_lib.a `cat dir1/stamp1.txt dir2/stamp0.txt`
But when i run some makefile to build some software - inside this process used original cat nor overrided.
How to override cat or any other command in way to get it work for make process without changing makefile(makefile is 3rdparty software and i don't want patch it every time when upgrade is needed)?
You can't do that with a shell function, because a shell function exists only in the local shell. It's not passed to programs like make. Also, GNU make always invokes /bin/sh by default, not /bin/bash, and your shell function above is written in bash syntax, so putting it in your ~/.bashrc will have no impact.
You could run:
$ make SHELL=/bin/bash
and add that shell function to your ~/.bashrc and that might work.
The only other thing you can do (assuming that your third party makefile invokes cat directly and doesn't use a variable like $(CAT) instead) is to create a cat shell script (not a function) and put it on your PATH before /bin and /usr/bin when you invoke make. Something like:
$ mkdir tmp
$ vi tmp/cat
...add commands...
$ chmod 755 tmp/cat
$ PATH=$(pwd)/tmp:$PATH make ...
Of course when you do this you can't use command cat ... in your script, you'll have to use a fully-qualified path like /bin/cat ...
Short version: you cannot (and you probably should not want to).
Longer version: You've defined a shell function. You could actually even export that shell function into the environment export -f cat. But unless the Makefile was saying bash -c cat ... instead of just cat (or other reference to the same effect), it would not give you the behavior you wanted.
If you really insisted... and the Makefile did not use hard coded path (e.g. /bin/cat). You could write your own cat, place it somewhere and make this location precede other possible hits for cat (just put it up front).
There is also some chance (look in the make file) it uses a variable (e.g. CAT) to know what to call, so you could just provide your own definition if that was the case.
In any case though. I would discourage you from using workaround like these because the actual behavior of the machinery gets obfuscated by doing so. There is something declared here... and something else in the environment giving it a different meaning. Which is a very common source of mistakes and eventually non-obvious (harder to resolve) bugs.
Example/clarification to the function bit. I have a Makefile:
all:
#echo foo
And define and export a function "overriding" echo. echo() { /bin/echo "$#" bar; } ; export -f echo. I run make and get:
$ make
foo
Because make just looks for echo in PATH (tries to exec and once it finds it, it runs it). If I changed it to have bash step in between, the exported function would kick in, but... that's an usual way to use commands in make and you'd have to edit the Makefile which you did not want:
all:
#bash -c 'echo foo'
This would yield you the result you wanted:
$ make
foo bar
The other option I've mentioned. I've put behavior of that function into a script /tmp/bin/echo reading:
#!/bin/bash
/bin/echo "$#" bar
And I've modified the PATH env var export PATH=/tmp/bin:$PATH. Now even with the first form of the Makefile:
all:
#echo foo
I get:
$ make
foo bar
But if the Makefile that is given says /bin/echo instead, I'd have no such luck. You could still change the binary... or change its behavior by forcing a shared library preload... but sounds quote extreme and fully exposes why this really might not be the best direction to take it.
So, to compile my executable, I need to have the library locations set up correctly. The problem is, the setup comes from a bunch of scripts that do the env variable exporting, and what needs to be set up may change (beyond my control) so I need to use those scripts instead of copying their functionality. To compile in regular command line, I need to do something like:
setup library1
setup library2
source some_other_setup_script.bash
g++ blah.c
# setup is a executable on my system that run some scripts
How would I write a makefile that accomplishes that? As far as I tried, the env variable exporting does not carry over (i.e. "export VAR=remember; echo $VAR" won't work)
You can also add environment variables properly with the machinery of GNU make, like so:
export TEST:="Something Good!"
test:
echo $$TEST
This (I think) has different semantics from:
TEST2:="Something not quite so useful?"
test2:
echo ${TEST2}
Which (again, I think) does the substitution within make before passing along to the shell. Note that the export command doesn't work within a target block, just unindented as an immediately executed command.
If variable exporting is not working the way it does on your command line, that suggests that Make is choosing a shell different from the one you're using, with different syntax for handling variables (export VAR=remember; echo $VAR works fine for me). Make uses /bin/sh by default, but you can override this with the SHELL variable, which Make does not import from the environment. I suggest setting SHELL (in the Makefile) to whatever you're using in your environment and trying the export VAR=remember experiment again.
Ultimately you will need to define the variable and execute the compiler in a shell list or even a script, rather than in separate make commands. There are a couple of refinements you could add, however. You could tell make about the script:
maintarget: script.sh blah.c
source script.sh; g++ blah.c
script.sh:
setup include script here
Another thing would be to just execute all that stuff in the same shell
maintarget: blah.c
run this; run that; run the other thing; g++ blah.c
I believe all make versions will run a ; list in the same shell, but you can always force a subshell with (list) or by calling specifically a shell script as a compiler command wrapper.
Don't forget to have the appropriate targets depend on your scripts themselves. BTW, some make versions (pmake aka bsd make) can execute a command when defining a make variable, and all versions of make then exports those. But I don't think gmake can do that.
You could write another shell script that executes all those commands, then prints out variable assignments that make can use. Run the script, pipe its output to a file, then include that file from your Makefile. For example:
Makefile:
all:
echo $(FOO)
test.mk: test.sh
./$< > $#
include test.mk
test.sh
echo "FOO=1"
Running "make" in the directory containing this Makefile produces:
make: Entering directory `/home/luser/build/mktest'
Makefile:7: test.mk: No such file or directory
./test.sh > test.mk
make: Leaving directory `/home/luser/build/mktest'
make: Entering directory `/home/luser/build/mktest'
echo 1
1
make: Leaving directory `/home/luser/build/mktest'
make creates test.mk by running the shell script, then includes it. test.mk contains the output of test.sh, and is parsed as a Makefile. See http://www.gnu.org/software/make/manual/make.html#Include for more details.
We use a variant of this in Mozilla's client.mk to let you define options in a "mozconfig" file:
http://mxr.mozilla.org/mozilla-central/source/client.mk#138
Restatement: How do I get a shell variable into a make file?
Something like:
MYVAR := $(shell echo $(MYVAR)) <any_makefile_additions_here>
So, this defines MYVAR inside a MAKEFILE when an environment variable named MYVAR is also set.
It might be of interest, that, in order to override an option that is already defined in a makefile, make supports (I am referring to GNU Make 3.82, but other version probably too) the option -e.
Example:
Makefile:
CC=gcc
...
Run make:
CC=gcc-4.7
make -e
will use gcc-4.7 instead of gcc.