I have a Makefile, trying to loop over a series of strings in a recipe and make them lower case.
My goal: I have a series of commands I would like to run on different files with the appropriate suffix.
# files in directory: test_file1.txt test_file2.txt test_file3.txt
MYLIST = file1 file2 file3
recipe:
for name in $(MYLIST) ; do \
$(eval FILENAME=`echo $($name) | tr A-Z a-z`) \
echo "Final name : test_${FILENAME}.txt" ; \
done
My problem, FILENAME always resolves to blank:
File name: test_.txt
I hope to see:
File name: test_file1.txt
File name: test_file2.txt
File name: test_file3.txt
You cannot mix make functions and shell commands like this. Make works like this: when it decides that your target is out of date and it wants to run the recipe, first it will expand the entire recipe string. Second it sends that expanded string to the shell to run.
So in your case, the $(eval ...) (which is a make operation) is expanded one time, then the resulting string is passed to the shell to run. The shell runs the for loop and all that stuff.
You have to use shell variables here to store values obtained by running your shell for loop. You cannot use make variables or make functions.
In general if you ever think about using $(shell ...) or $(eval ...) inside a recipe, you are probably going down the wrong road.
Related
I have a make target, that first calls a CAE tool which generates reports. When this is done, the target calls a python script that shall take the content of the CAE reports (or more specific some grep'ed lines of the reports) as argument.
A minimum example is
target1:
date > ./bar.txt
echo $(shell cat ./bar.txt)
Problem is, that make expands the $(shell cat ./bar.txt) before the first command has been called and bar.txt has been updated. So for this minimum example, the echo prints the content of bar.txt before the update (the date from the previous target run).
(
I know that I simply could write this example in another way without variables and the shell function call, this is just for the sake of showing the problem where I call a tool that takes an argument from a shell call. So actually I want to do sth like this:
target1:
cae_tool_call
report_eval.py -text "$(shell cat $(generated_report) | grep 'foo')"
where cae_tool_call generates the generated_report. And this -text "argument" does not resolve the argument without an explicit call of the shell function.
)
I already tried with actual shell variables (instead of make variables), double escapes, immediate vs deferred variables but have no working solution yet. Any ideas?
#######################################
Edit to show some unexpected behavior:
I have this python script argument_example.py
import argparse
def main():
parser = argparse.ArgumentParser()
parser.add_argument("-r", "--reporttext", help="text string", required=True)
args=parser.parse_args()
if args.reporttext:
print(args.reporttext)
main()
It just prints the text given with argument -r.
And I have these two make targets:
####################################
#this does not work
REPORTNAME := ./bar.txt
variable_report:
date > $REPORTNAME
python3 ./argument_example.py --reporttext "`(cat $REPORTNAME)`"
####################################
#this works
static_report:
date > ./bar.txt
python3 ./argument_example.py --reporttext "`(cat ./bar.txt)`"
When calling variable_report, the python scripts prints the outdated bar.txt content. When calling static_report, the python script prints the updated content.
make recipes are already shell scripts. Never use the shell make function inside a recipe. In your first simple example use:
target1:
date > bar.txt
cat bar.txt
In your other example use:
generated_report := name-of-generated-report
target1:
cae_tool_call
report_eval.py -text "`cat $(generated_report) | grep 'foo'`"
Or even better:
generated_report := name-of-generated-report
target1:
cae_tool_call
report_eval.py -text "`grep 'foo' $(generated_report)`"
I have a variable AAA which is in this format
AAA='BBB=1 CCC=2 DDD=3'
How can I use this to set environment variables BBB, CCC and DDD in a command I run (without permanently exporting them to the shell)? That is, I want to use to use the above to do something identical to:
# this works correctly: node is run with AAA, BBB, and CCC in its environment
BBB=1 CCC=2 DDD=3 node index.js
...however, when I try:
# this does not work: AAA=1 is run as a command, so it causes "command not found"
$AAA node index.js
...it tries to run BBB=1 as a command, instead of parsing the assignment as a variable to set in node's environment.
If you can, use a different format.
There are several better options:
An array.
envvars=( AAA=1 BBB=2 CCC=3 )
env "${envvars[#]}" node.js index.js
A NUL-delimited stream (the ideal format to use to save environment variables in a file -- this is the format your operating system uses for /proc/self/environ, for example).
Saving to a file:
```
printf '%s\0' 'foo=bar' \
'baz=qux' \
$'evil=$(rm -rf importantDir)\'$(rm- rf importantDir)\'\nLD_LIBRARY_PATH=/tmp/evil.so' \
> envvars
```
...or, even more simply (on Linux):
```
# save all your environment variables (as they existed at process startup)
cp /proc/self/environ envvars
```
Restoring from that file, and using it:
```
mapfile -d '' vars <envvars
env "${vars[#]}" node.js
```
But whatever you do, don't use eval
Remember that evil environment variable I set above? It's a good example of a variable that poorly-written code can't set correctly. If you try to run it through eval, it deletes all your files. If you try to read it from a newline-delimited (not NUL-delimited) file, it sets another LD_LIBRARY_PATH variable that tells your operating system to load a shared library from an untrusted/untrustworthy location. But those are just a few cases. Consider also:
## DO NOT DO THIS
AAA='BBB=1 CCC=2 DDD="value * with * spaces"'
eval $AAA node foo.js
Looks simple, right? Except that what eval does with it is not simple at all:
First, before eval is started, your parameters and globs are expanded. Let's say your current directory contains files named 1 and 2:
'eval' 'BBB=1' 'CCC=2' 'DDD="value' '1' '2' 'with' '1' '2' 'spaces"' 'node' 'foo.js'
Then, eval takes all the arguments it's passed, and gloms them all together into a single string.
eval "BBB=1 CCC=2 DDD="value 1 2 with 1 2 spaces" node foo.js
Then that string is parsed from the very beginning
...which means that if instead of having a file named 1 you had a file named $(rm -rf ~) (a perfectly valid filename!), you would have a very, very bad day.
I have a text file called OPTIONS.txt storing all flags of Makefile:
arg1=foo arg2="-foo -bar"
I want to pass all flags in this file to make. However,
make `cat OPTIONS.txt`
fails with make: invalid option -- 'a'. It seems that shell interprets it as:
make arg1=foo arg2="-foo -bar"
^argv[1] ^argv[2] ^argv[3]
Is there any way to make it interpreted as:
make arg1=foo arg2="-foo -bar"
^argv[1] ^--------argv[2]
Since you control the options file, store the options one per line:
arg1=foo
arg2="-foo -bar"
Then in the shell, you'll read the file into an array, one element per line:
readarray -t opts < OPTIONS.txt
Now you can invoke make and keep the options whole:
make "${opts[#]}"
If you want the shell to interpret quotes after backtick expansion you need to use eval, like this:
eval make `cat OPTIONS.txt`
however just be aware that this evaluates everything, so if you have quoted content outside of the backticks you'll get the same issue:
eval make `cat OPTIONS.txt` arg4="one two"
will give an error. You'd have to double-quote the arg4, something like this:
eval make `cat OPTIONS.txt` arg4='"one two"'
In general it's tricky to do stuff like this from the command line, outside of scripts.
ETA
The real problem here is that we don't have a set of requirements. Why do you want to put these into a file, and what kind of things are you adding; are they only makefile variable assignments, or are there other make options here as well such as -k or similar?
IF the OP controls (can change) the format of the file AND the file contains content only used by make AND the OP doesn't care about the variables being command line assignments vs. regular assignments AND there are only variable assignments and not other options, then they can just (a) put each variable assignment on its own line, (b) remove all quotes, and (c) use include OPTIONS.txt from inside the makefile to "import" them.
I read this question: Makefile: $subst in dependency list, but I still can't make my shell script work correctly.
I have a makefile with a line with the contents:
##public_detailed#|test_create|Syntax: commoncmdsyntax test_create test_name=<test-name>
A target runs a multiline bash script, where the commoncmdsyntax must be replaced by a string containing words and spaces.
In the script, I use cut to assign to a variable desc the following string:
Syntax: commoncmdsyntax test_create test_name=<test-name>
The problem is that commoncmdsyntax is not replaced by new text here:
$(subst commoncmdsyntax,new text,$$desc)
I also tried to replace it by a single word, like XX, but it also does not work.
The subst function (as in $(subst commoncmdsyntax,new text,$$desc)) is a Make function, so Make will perform the substitution before running any rule and therefore before your script assigns a value to desc. So even if secondary expansion worked the way you seem to think it will, this approach would still fail.
If you want to perform a substitution within something made by a shell script (in a recipe), the sensible way is to do so within the recipe:
echo $dest | sed 's/commoncmdsyntax/new text/'
We can give you a more detailed solution if you give us a minimal complete example of the problem.
I have a list of objects in a Makefile variable called OBJECTS which is too big for the command buffer. Therefore I'm using the following method to create a file listing the objects (to pass to ar):
objects.lst:
$(foreach OBJ,$(OBJECTS),$(shell echo "$(OBJ)">>$#))
While this works it is extremely slow (on Cygwin at least) and I don't like relying on shell commands and redirection.
Additionlly foreach is not intended for this purpose - it is evaluated before any commands are run which means I can't for example rm -f objects.lst before appending.
Is there a better way? I don't want to use incremental archiving as that causes problems with multiple jobs.
The only thing I can think of is parsing the Makefile with a separate script to read the object list or storing the object list in a separate file. Both solutions have their own problems though.
Try something like:
OBJECTS:=a b c d
objects.lst:
echo > $# <<EOF $(OBJECTS)
i.e. make use of the <<EOF functionality that is built into the shell. It does not have any max-length limitations.
In the following example I also replaced echo with a simple Perl script to split the arguments onto new lines but this is the jist of it..
objects.lst:
echo $(wordlist 1,99,$(OBJECTS))>$#
echo $(wordlist 100,199,$(OBJECTS))>>$#
echo $(wordlist 200,299,$(OBJECTS))>>$#
echo $(wordlist 300,399,$(OBJECTS))>>$#
...
How about something like this:
OBJECTS_AM=$(filter a% b% c% d% e% f% g% h% i% j% k% l% m%,$(OBJECTS))
OBJECTS_NZ=$(filter-out a% b% c% d% e% f% g% h% i% j% k% l% m%,$(OBJECTS))
objects.lst:
$(shell echo "$(OBJECTS_AM)">$#)
$(shell echo "$(OBJECTS_NZ)">>$#)
You might need to split it one or two more times, but it's not that bad, especially as the distribution of file names doesn't change all that often.
Here's a patch to gnu make that lets you directly write a variable into a file.
It creates a new 'writefile' function, similar to the existing 'info' function, except it takes a filename argument and writes to the file:
https://savannah.gnu.org/bugs/?35384