I am writing a Makefile in which I have a function which is called in foreach.
In this function, I want to use awk command in which I would like to use $3.
However, I do not know how to correctly escape $.
Here is the simplified code. (the input.txt have three columns and the third column is a value)
define MyFunction
%/filtered$(1).txt: %/input.txt
cat $$*/input.txt | awk '{ if ($$3 < $(1)) print }'
endef
$(foreach threshold,1 2 3,$(eval $(call MyFunction,$(threshold))))
When I execute the above code, I get an error
awk: syntax error at source line 1
I have tried to modify the number of $s, or escaping with backslash \. It did not help...
How can I solve this problem?
Your problem is that the call function will evaluate the macro once, then when the recipe is run it will be evaluated again, then the awk command needs to see the $. So you have to escape it twice, using $$$$:
define MyFunction
%/filtered$(1).txt: %/input.txt
cat $$*/input.txt | awk '{ if ($$$$3 < $(1)) print }'
endef
If you want to see what make is parsing a good trick is to replace your eval with info:
$(foreach threshold,1 2 3,$(info $(call MyFunction,$(threshold))))
$(foreach threshold,1 2 3,$(eval $(call MyFunction,$(threshold))))
Related
I wrote in Makefile
define deploy_meta
$(shell git log -n 2 --oneline | awk '{print "commit"NR ": " $0}')
commit: nogit-$(timestamp)
tag: nogit-$(timestamp)
deployed-from: $(shell hostname)
deployed-by: $(USER)
deploy-date: $(shell date -u '+%Y%m%d%H%M%S')
endef
but if gives me
$cat .deploy
commit1: commit2:
commit: nogit-1669806282
tag: nogit-1669806282
...
Command itself
git log -n 2 --oneline | awk '{print "commit"NR ": " $0}'
works fine and gives two lines. It is evident, that make feels it, since it prints two "commit#" words. But it doesn't print content. Why?
The main problem is simple enough: Make expands $0 here (to nothing). You need $$0.
If you want two lines, though, $(shell) is also part of your problem here:
The shell function provides for make the same facility that backquotes ('`') provide in most shells: it does command expansion. This means that it takes as an argument a shell command and expands to the output of the command. The only processing make does on the result is to convert each newline (or carriage-return / newline pair) to a single space. If there is a trailing (carriage-return and) newline it will simply be removed.
(emphasis mine). You'll need a completely different approach, e.g., use $(shell) to produce a file, then include that file, for instance.
I'm looking to record the exact commands used to build artifacts within a makefile. I'd like this to be stored in a file for later consumption. I am running into issues due to quotes. Basically, what I want is:
define record_and_run_recipe
#echo '$(2)' > $1
$2
endef
all:
$(call record_and_run_recipe,out.cmd,\
#echo 'hello world "$$1"' )
cat out.cmd
I would like this to output (exactly)
#echo 'hello world "$1"'
Of course, the quotes end up matching with the quotes in the expansion of the variable, and this messes everything up. (I get #echo hello world instead). Bash doesn't like '\'' either, so I can't simply do $(2:'=\'). I also seem to have issues with , characters...
I'm not looking to debug the entire makefile, just dump a couple of recipes. I'm wondering if anyone has a robust way of accomplishing this.
As I said in my comment above, you can use GNU make's $(info ...) function. It's not exactly clear from your example above what you want to do; why are you trying to put the output into a file, then cat it? Is that important?
If you can't use info, the canonical way to handle quoting in shell is to surround the string with single quotes, then replace every single quote with '\''. You say "bash doesn't like" that, but I don't know what that means. Normally you'd do something like:
define record_and_run_recipe
#echo '$(subst ','\'',$2)' > $1
$2
endef
As far as commas you will absolutely have a problem with commas if you want to use the $(call ...) function. The only way to avoid that is to put the string into a variable, like:
output = foo, bar
... $(call blah,$(output))
to "hide" the comma from call.
Makefile:
.PHONY: all
SHELL:=/usr/bin/env bash
all:
$(eval x=$(shell cat file))
#echo "$x"
File:
foo
bar
Output:
foo bar
How do I get the contents of the file into the make variable without losing the newlines?
You can't do this with shell, as described in its documentation.
If you have a sufficiently new version of GNU make, you can use the file function however.
Make converts newlines from shell outputs to spaces (see here):
The shell function performs the same function that backquotes (‘`’)
perform in most shells: it does command expansion. This means that it
takes as an argument a shell command and evaluates to the output of
the command. The only processing make does on the result is to convert
each newline (or carriage-return / newline pair) to a single space. If
there is a trailing (carriage-return and) newline it will simply be
removed.
So, you cannot preserve spaces from the $(shell) command directly. That being said, make does allow multiline variables using define -- but beware, attempting to use such variables is problematic. Consider:
define x
foo
bar
endef
all:
#echo "$x"
Make expands the $x in place, and you end up with:
all:
#echo " foo
bar"
(where the newline is considered the end of the recipe line..).
Depending on what you want this for, you may be able to get around this is using a bash variable:
all:
#x=$$(cat file); \
echo $$x
Or potentially storing your output in a file, and referencing that when necessary.
all:
eval (cat file >> output.txt)
cat output.txt
(and yes, the last one is convoluted as written, but I'm not sure what you're trying to do, and this allows the output of your command to be persistent across recipe lines).
If the file contents are ensured not to contain any binary data, and if you're willing to to extra processing each time you access the variable, then you could:
foo:=$(shell cat file | tr '\n' '\1')
all:
#echo "$(shell echo "$(foo)" | tr '\1' '\n')"
Note that you cannot use nulls \0, and I suspect that probably means there's a buffer overflow bug in my copy of Make.
Does anyone know how to use a here-document redirection on a recipe?
test:
sh <<EOF
echo I Need This
echo To Work
ls
EOF
I can't find any solution trying the usual backslash method (which basically ends with a command in a single line).
Rationale:
I have a set of multi-line recipes that I want to proxy through another command (e.g., sh, docker).
onelinerecipe := echo l1
define twolinerecipe :=
echo l1
echo l2
endef
define threelinerecipe :=
echo l1
echo l2
echo l3
endef
# sh as proxy command and proof of concept
proxy := sh
test1:
$(proxy) <<EOF
$(onelinerecipe)
EOF
test2:
$(proxy) <<EOF
$(twolinerecipe)
EOF
test3:
$(proxy) <<EOF
$(threelinerecipe)
EOF
The solution I would love to avoid: transform multiline macros into single lines.
define threelinerecipe :=
echo l1;
echo l2;
echo l3
endef
test3:
$(proxy) <<< "$(strip $(threelinerecipe))"
This works (I use gmake 4.0 and bash as make's shell) but it requires changing my recipes and I have a lot.
Strip removes the newlines, from the macro, then everything is written in a single line.
My end goal is: proxy := docker run ...
Using the line .ONESHELL: somewhere in your Makefile will send all recipe lines to a single shell invocation, you should find your original Makefile works as expected.
When make sees a multi-line block in a recipe
(i.e., a block of lines all ending in \, apart from the last),
it passes that block un-modifed to the shell.
This generally works in bash,
apart from here docs.
One way around this is to strip any trailing \s,
then pass the resulting string to bash's eval.
You do this in make by playing with ${.SHELLFLAGS} and ${SHELL}.
You can use both of these in target-specific form if you only want it to kick in for a few targets.
.PHONY: heredoc
heredoc: .SHELLFLAGS = -c eval
heredoc: SHELL = bash -c 'eval "$${#//\\\\/}"'
heredoc:
#echo First
#cat <<-there \
here line1 \
here anotherline \
there
#echo Last
giving
$ make
First
here line1
here anotherline
Last
Careful with that quoting, Eugene.
Note the cheat here:
I am removing all backslashes,
not just the ones at the ends of the line.
YMMV.
With GNU make, you can combine multi-line variables with the export directive to use a multi-line command without having to turn on .ONESHELL globally:
define script
cat <<'EOF'
here document in multi-line shell snippet
called from the "$#" target
EOF
endef
export script
run:; # eval "$$script"
will give
here document in multi-line shell snippet
called from the "run" target
You can also combine it with the value function to prevent its value from being expanded by make:
define _script
cat <<EOF
SHELL var expanded by the shell to $SHELL, pid is $$
EOF
endef
export script = $(value _script)
run:; # eval "$$script"
will give
SHELL var expanded by the shell to /bin/sh, pid is 12712
Not a here doc but this might be a useful workaround.
And it doesn’t require any GNU Make’isms.
Put the lines in a subshell with parens, prepend each line with echo.
You’ll need trailing sloshes and semi-colon and slosh where appropriate.
test:
( \
echo echo I Need This ;\
echo echo To Work ;\
echo ls \
) \
| sh
I'm trying to check whether a variable is defined using ifndef/ifdef, but I keep getting a not found error from the execution. I'm using GNU Make 3.81, and here is a snippet of what I have:
all: _images
$(call clean, .)
$(call compile, .)
#$(OPENER) *.pdf &
_images:
$(call clean, "images")
$(call compile, "images")
define clean
#rm -f ${1}/*.log ${1}/*.aux ${1}/*.pdf
endef
define compile
ifdef ${1}
dir = ${1}
else
dir = .
endif
ifdef ${2}
outdir = ${2}
else
outdir = ${1}
endif
#$(COMPILER) -output-directory ${outdir} ${dir}/*.tex
endef
And the exact error:
$ make
ifdef "images"
/bin/sh: 1: ifdef: not found
make: *** [_images] Error 127
Edit:
Considering Barmar comments, here goes the conclusions:
The contents of a define are shell command lines, not make directives;
to break lines inside commands within a define block, the linebreak must be escaped -- with \;
also, each block corresponding to one-liner commands is executed separately, each in a different shell execution, which means that, defining local variables won't work if the intention is to access the variable value in the next one-liner block.
Thanks tripleee for the nice work around.
You can combine the shell's facilities with Make's to get a fairly succinct definition.
define compile
#dir="${1}"; outdir="${2}"; outdir=$${outdir:-"$dir"}; \
$(COMPILER) -output-directory "$${outdir}" "$${dir:-.}/*.tex
The double-dollar is an escape which passes a single dollar sign to the shell. The construct ${variable:-value} returns the value of $variable unless it is unset or empty, in which case it returns value. Because ${1} and ${2} are replaced by static strings before the shell evaluates this expression, we have to take the roundabout route of assigning them to variables before examining them.
This also demonstrates how to combine two "one-liners" into a single shell invocation. The semicolon is a statement terminator (basically equivalent to a newline) and the sequence of a backslash and a newline causes the next line to be merged with the current line into a single "logical line".
This is complex enough that I would recommend you omit the leading # but I left it in just to show where it belongs. If you want silent operation, once you have it properly debugged, run with make -s.