I've got a Makefile recipe which runs a Python script. However, before and after the script I want to write some information to the screen describing what is being done. I could put these print statements into the Python script but that's a workaround and I'd like to understand why this doesn't work. My Makefile looks like:
/data/interim/opt_smoothing.csv: $(shell find /data/raw/evi_data -type f) src/data/determine_optimal_smoothing.py
$(info Determining optimal smoothing) && python src/data/determine_optimal_smoothing.py && $(info Optimal smoothing calculation complete)
I was under the impression that putting these && would chain these commands together and have them execute one after each other, but this doesn't appear to work. When I try to make this file I get the error:
root#61276deb5c1a:/code# make /data/interim/opt_smoothing.csv
Determining optimal smoothing
Optimal smoothing calculation complete
&& python src/data/determine_optimal_smoothing.py &&
/bin/sh: 1: Syntax error: "&&" unexpected
make: *** [Makefile:10: /data/interim/opt_smoothing.csv] Error 2
When I include these 3 things on separate lines it works except the calculation complete message occurs before the script even finishes. What is the right way to chain these things together so they execute sequentially in the same shell?
You can't use make's info function for this. Make functions are run by make, not by the shell, as part of the expansion of the script in preparation to send it to the shell. So, they are run before the shell is invoked. Second, they expand to the empty string.
So for the recipe line:
$(info foo) && python bar && $(info baz)
make will expand the line which causes foo and baz to be printed, then it will invoke the resulting string in the shell, which has &&, like this:
foo
baz
/bin/sh -c '&& python bar &&`
which is clearly invalid.
If you want to have the shell print things you have to use a shell command to do it, such as echo, not a make function.
Related
Beginner in bash and makefiles here. I have a course where we need to create a makefile where each rule calls one of the already compiled programs. All of the compiled programs take a command line argument. As the arguments can be quite large and mostly consists of the same character in a row (for example AAAAAAA) I made a script that uses python to print the argument. Example:
#!/bin/bash
python -c 'print "A"*1000 + "Q"*200'
I am wondering how to create the rule in the makefile so that the output of the above script will be passed as the command line argument. Essentially like this:
test:
./schoolprogram ./myprogram.sh
So when make test is executed then ./schoolprogram should be run with the argument 1000 A's followed by 200 Q's and not the literal string "./myprogram.sh".
I don't know why you have a script that does nothing but invoke python; why not just run python directly?
In any event, this isn't really a makefile question it's mostly a shell question. When writing makefile recipes, the best way is to get the command you want to run working at your shell prompt, then take that same command (with one small tweak) and put it into your makefile.
In the shell, you can use either $(...) or backticks (an older style) to run a command and replace it with the output of that command. So you can run this:
$ ./schoolprogram $(./myprogram.sh)
or more easily:
$ ./schoolprogram $(python -c 'print "A"*1000 + "Q"*200')
Now when you take a command and put it into a makefile recipe, the thing you have to remember is that the $ character is special to make, so if you want a $ to be passed to your command you have to escape it by doubling it, $$. So you'd write:
test:
./schoolprogram $$(./myprogram.sh)
or equivalently:
test:
./schoolprogram $$(python -c 'print "A"*1000 + "Q"*200')
Using the code below, I am able to redirect the program's standard error output to an error.log file, and call a specific function based on whether or not the program exits peacefully. However in addition to this, I would like to organize the stderr output using the fold command, to keep the lines under a max length of 80 characters. How can I achieve this, whilst retaining the current functionality?
Code:
define func
./myProgram 2> error.log && \
$(call successCommand) || \
$(call failureCommand)
endf
This can't work as you have it here. Make doesn't contain an internal shell interpreter: make invokes a separate shell process. So, you can't jump back and forth between "shell code" and "make code".
Before make invokes the shell it will expand all make variables and functions. Then once the expansion is complete, the resulting string is passed to the shell and the shell runs the entire thing. Then once the shell is complete and exits, make looks at the exit code to see if the command succeeded or not.
So in your case, both $(call successCommand) and $(call failureCommand) will be expanded first, before the shell is invoked to determine whether ./myProgram succeeded or not.
Since you didn't give us any idea of what the successCommand and failureCommand do, there's not much help we can give beyond, if the test runs in the shell then the condition AND the results also need to run in the shell, and you can't use make functions like $(call ...).
I have a script "set_env.py" that outputs the following uppon execution:
export MY_VAR_1=some_value
export MY_VAR_2=some_other_value
I cannot change this script, it is supplied in my current environment.
Further I have a Makefile that looks like this:
SHELL := /bin/bash
all: set_env
env | grep MY_VAR
set_env:
eval ./set_env.py
With this makefile I expected the grep to list my two variables, however it seems the environment is not set.
I suspect this is because make creates a sub-environment for each line and so the variables set on the first line will not be available in the second.
So the question is, how would I go about exporting the environment from the untouchable script in my makefile?
Actually, the output of the python is valid make.
One option then is to read the output of the python directly into the makefile.
The only fly in the ointment is that $(shell) doesn't cut the mustard.
include Environment.mk
PHONY: test
test:
env | grep MY_VAR
Environment.mk:
./set_env.py >$#-tmp
mv $#-tmp $#
How does this work?
The first thing that make tries to do is to ensure the makefile itself is up-to-date.
Since we have told it to include Environment.mk,
make must ensure that is up-to-date too.
Make finds a rule for Environment.mk
The python is run, creating Environment.mk
Environment.mk is read in, creating two make variables with the export attribute
The makefile is now up-to-date, so make proceeds on to the target (test in this case)
Make runs test's recipe, exporting any variables with the export attribute.
No recursion, but you should ensure the python always spits out make compatible syntax.
EDIT
As #raspy points out, this is not the whole story.
As it stands,
once Environment.mk has been created,
it will never be regenerated.
If set_env.py ever generates different output,
you should tell make what conditions these are by adding dependencies.
If set_env.py takes a trivial time to run,
I advise a simple .PHONY.
That way it will run every time you run make,
and Environment.mk will never be stale.
.PHONY: Environment.mk
Recursive make is your friend I'm afraid.
.PHONY: all
all:
eval $$(./set_env.py) && ${MAKE} test
.PHONY: test
test:
env | grep MY_VAR
There are a few moving parts here.
make all executes the shell command eval $(./set_env.py) && make test
The shell first does the command substitution
$(./set_env.py) is replaced by the export commands
The export commands are passed to the (shell) eval command
The environment variables are defined, but only for this shell invocation
The eval succeeds, so the execution passes to the command after the &&
Make runs recursively, but this second make has an augmented environment
I found many answers here and elsewhere on the topic, but none that worked. Please help me out here.
I need to set some environment variables, which is partly done in some scripts, called from a master script, partly directly. Here is a minimal Makefile that shows the unwanted behaviour:
FC := ifort
SHELL := /bin/bash
some_target: load_ifort
$(FC) file.f
load_ifort:
source /usr/local2/bin/ifort-compilervars.sh ia32
export LM_LICENSE_FILE=/usr/local2/misc/intel2013/flexlm/server.lic
if I call make, I get an "ifort: command not found" error. If I execute the two comamnds by hand on the command line before calling make, ifort is found and everything is good.
What am I missing???
Each line in a recipe gets executed in a separate subshell. So you create one shell which sources the .sh file, then exits and forgets everything, then another shell which starts with a clean slate.
The straightforward solution in your case would be to collect all these commands in a single variable. I have factored out the LM_LICENSE_FILE assignment because that can be done in Make directly, but you could include that in the FC variable as well.
LM_LICENSE_FILE := /usr/local2/misc/intel2013/flexlm/server.lic
export LM_LICENSE_FILE
FC := source /usr/local2/bin/ifort-compilervars.sh ia32; \
ifort
some_target:
$(FC) file.f
If the shell commands can be straightforwardly run by Make as well, you could include them, or perhaps translate the sh file into Make commands by a simple script.
Another option would be to create a simple wrapper in your PATH; maybe call it fc:
#!/bin/sh
. /usr/local2/bin/ifort-compilervars.sh ia32
ifort "$#"
then just use fc where you currently have $(FC). (If the ifort-compilervars.sh file contains Bash constructs, in spite of the name, you should change the shebang to #!/bin/bash.)
As a rule, only one-liner shell commands "work". From the comment about "bash", it seems likely you are using GNU make. In your example, the word "source" is not found in the GNU make manual's index. (If you found this in a working example, it would be helpful to start from that). There are two types of variables of interest:
makefile variables, which live in the make program
environment variables, which are "exported"
The latter would include $PATH, which is used to find programs. For updating that, you do need shell commands. But (lacking some special provision in the make program), exported variables from a shell script are not passed up into the make program and made available for the next line of the makefile.
You could reorganize the makefile to provide a rule which combines the source command and other initialization into a shell command which then recurs (carrying those variables along) into a subprocess which would then do the compiles. Something like
build:
sh -c "source /usr/local2/bin/ifort-compilervars.sh ia32; \
export LM_LICENSE_FILE=/usr/local2/misc/intel2013/flexlm/server.lic; \
$(MAKE) some_target"
some_target: load_ifort
$(FC) file.f
I have a Makefile from which I want to call another external bash script to do another part of the building. How would I best go about doing this?
Just like calling any other command from a makefile:
target: prerequisites
shell_script arg1 arg2 arg3
Regarding your further explanation:
.PHONY: do_script
do_script:
shell_script arg1 arg2 arg3
prerequisites: do_script
target: prerequisites
Perhaps not the "right" way to do it like the answers already provided, but I came across this question because I wanted my makefile to run a script I wrote to generate a header file that would provide the version for a whole package of software. I have quite a bit of targets in this package, and didn't want to add a brand new prerequisite to them all. Putting this towards the beginning of my makefile worked for me
$(shell ./genVer.sh)
which tells make to simply run a shell command. ./genVer.sh is the path (same directory as the makefile) and name of my script to run. This runs no matter which target I specify (including clean, which is the downside, but ultimately not a huge deal to me).
Each of the actions in the makefile rule is a command that will be executed in a subshell. You need to ensure that each command is independent, since each one will be run inside a separate subshell.
For this reason, you will often see line breaks escaped when the author wants several commands to run in the same subshell:
targetfoo:
command_the_first foo bar baz
command_the_second wibble wobble warble
command_the_third which is rather too long \
to fit on a single line so \
intervening line breaks are escaped
command_the_fourth spam eggs beans
Currently using Makefile, I can easily call the bash script like this:
dump:
./script_dump.sh
And call:
make dump
This also works like mentioned in the other answer:
dump:
$(shell ./script_dump.sh)
But the downside is that you don't get the shell commands from the console unless you store in a variable and echo it.