From the docs:
The 'shell' function performs the same function that backquotes ('`')
perform in most shells: it does "command expansion". This means that it
takes as an argument a shell command and evaluates to the output of the
command. The only processing 'make' does on the result is to convert
each newline (or carriage-return / newline pair) to a single space. If
there is a trailing (carriage-return and) newline it will simply be
removed.
Version 1 of makefile, is:
foo := $(shell echo 'bar'; foo)
all:
#echo 'foo is: $(foo)'
.PHONY: all
Running, we get:
/bin/sh: 1: foo: not found
bar
foo is:
Version 2 of makefile, is:
foo := $(shell echo 'bar'; false)
all:
#echo 'foo is: $(foo)'
.PHONY: all
Running, we get:
foo is: bar
Now, given the above quote from the documentation:
The 'shell' function performs the same function that backquotes ('`')
perform in most shells: it does "command expansion".
We try and compare the following:
# Equivalent to version 1 of makefile above.
$ foo=`echo 'bar'; foo`
sh: 1: foo: not found
$ echo ${foo}
bar
# Equivalent to version 2 of makefile above.
$ foo=`echo 'bar'; false`
$ echo ${foo}
bar
So, not only is the shell function not consistent, but it differs from "command expansion" of shells, referenced to, in the docs, cited above!
Is there a rationale behind all this?
Converting a comment into a substantiated answer.
The exit status from echo 'bar'; foo is 127; the exit status from echo 'bar'; false is 1. Exit status 127 means that the command failed to execute in some way.
In the comment, I said that my suspicion is that make treats the two exit statuses differently.
When I create makefile.v3 that contains:
foo := $(shell echo 'bar'; exit ${exit})
all:
#echo 'foo is: $(foo)'
.PHONY: all
and run it as shown:
$ make -f makefile.v3
foo is: bar
$ make -f makefile.v3 exit=0
foo is: bar
$ make -f makefile.v3 exit=1
foo is: bar
$ make -f makefile.v3 exit=127
bar
foo is:
$ make -f makefile.v3 exit=128
foo is: bar
$ make -f makefile.v3 exit=129
foo is: bar
$ make -f makefile.v3 exit=126
foo is: bar
$ make -f makefile.v3 exit=255
foo is: bar
$
As hypothesized in the comment, the exit status of 127 is treated differently by make.
(For the record: tested with GNU Make 3.81 on Mac OS X 10.10.4.)
Related
Shell commands sometimes take a long time to run, so you may not want to do VAR = $(shell slow-cmd) (with =, the slow-cmd will be run every time the variable is referenced). Using VAR := $(shell slow-cmd) can be useful, but if you are building a target that does not ever need the variable expanded, you will get one more invocation of the slow-cmd than is needed. In the following makefile (with gnu-make), you can get the desired behavior: the shell command to define a value for V2 is never invoked more than once, and for the target foo it is not invoked at all. But this is a heinous kludge. Is there a more reasonable way to ensure that a variable is only defined when needed, but never evaluated more than once?
V1 = $(shell echo evaluating V1 > /dev/tty; echo V1 VALUE)
all: foo bar V2
#echo $(V1) $#
#echo $(V2) $#
foo:
#echo $(V1) $#
bar: V2
#echo $(V1) $#
#echo $(V2) $#
V2:
$(eval V2 := $(shell echo evaluating V2 > /dev/tty; echo V2 VALUE))
.PHONY: all foo bar
There's no way to do it without tricks, but there's a cleaner way (maybe) than you're using. You can use:
V1 = $(eval V1 := $$(shell some-comand))$(V1)
For more details and explanation of exactly how this works see this page.
Target-specific deferred variables are an option:
host> cat Makefile
foo: VFOO = $(shell echo "VFOO" >> log.txt; echo "VFOO")
foo:
#echo '$(VFOO)' > $#
bar: VBAR = $(shell echo "VBAR" >> log.txt; echo "VBAR")
bar:
#echo '$(VBAR)' > $#
host> make foo
host> cat log.txt
VFOO
host> make foo
make: 'foo' is up to date.
host> cat log.txt
VFOO
host> make bar
host> cat log.txt
VFOO
VBAR
host> make bar
make: 'bar' is up to date.
host> cat log.txt
VFOO
VBAR
I have very strange issues using bash and exported function to give me a reliable answer to the call of the builtin caller.
Here's my setup to illustrate this issue: Bash script bar defines and exports function bar1 and bar2. bar2 calls bar1. Bash script bar then execute bash script foo which will call bar1.
The caller builtin will then break only after the call of bar1. Is this normal ? can you explain why ? can you simplify the following code that exposes the issue ?
To be perfectly clear, here's how to reproduce on your system: Build both file:
cd /tmp
cat <<"EOF" > foo
#!/bin/bash
bar1
EOF
chmod +x foo
cat <<"EOF" > bar
#!/bin/bash
bar2() {
echo "$FUNCNAME IN: $(caller 0)" >&2
}
export -f bar2
bar1() {
echo "$FUNCNAME BEFORE: $(caller 0)" >&2
bar2
echo "$FUNCNAME AFTER: $(caller 0)" >&2
}
export -f bar1
./foo
EOF
chmod +x bar
You can then fiddle and see:
$ ./bar
bar1 BEFORE: 3 main ./foo
bar2 IN:
bar1 AFTER:
I expected (with acceptable variations on the line numbers):
$ ./bar
bar1 BEFORE: 9 main ./foo
bar2 IN: 5 bar ./foo
bar1 AFTER: 9 main ./foo
Ultimately, my question would be: how could I circumvent this issue and get the caller in all cases ?
ADDITIONAL INFO:
bash version: 4.3.42(1)-release (x86_64-pc-linux-gnu) from ubuntu package 4.3-14ubuntu1.
This is a bug in bash. It was fixed in version 4.4.
In the presence of exported functions, the BASH_SOURCE variable is not properly maintained. You can check it by displaying the contents of the FUNCNAME, BASH_SOURCE, BASH_LINENO special variables:
cd /tmp
cat <<"EOF" > foo
#!/bin/bash
bar1
EOF
chmod +x foo
cat <<"EOF" > bar
#!/bin/bash
bar2() {
echo "$FUNCNAME IN: $(caller 0) [${FUNCNAME[#]}] [${BASH_SOURCE[#]}] [${BASH_LINENO[#]}]" >&2
}
export -f bar2
bar1() {
echo "$FUNCNAME BEFORE: $(caller 0) [${FUNCNAME[#]}] [${BASH_SOURCE[#]}] [${BASH_LINENO[#]}]" >&2
bar2
echo "$FUNCNAME AFTER: $(caller 0) [${FUNCNAME[#]}] [${BASH_SOURCE[#]}] [${BASH_LINENO[#]}]" >&2
}
export -f bar1
./foo
EOF
chmod +x bar
Output of ./bar:
bar1 BEFORE: 3 main ./foo [bar1 main] [./foo] [3 0]
bar2 IN: [bar2 bar1 main] [./foo] [1 3 0]
bar1 AFTER: [bar1 main] [] [3 0]
As you can see, stack frames corresponding to invocations of exported functions aren't added to BASH_SOURCE, but whenever a function returns the topmost stack frame is popped.
Note that the FUNCNAME variable is not affected by this bug. Thus, if you need only the name of the caller you can obtain it as ${FUNCNAME[1]}.
If there a way to affect the '?' variable in bash?
For a regular variable, it is possible to just use FOO=bar, thus 'echo $FOO' will output bar, but for some reason, it is not working with the '?' variable.
I have found two workaround but they are quite unsatisfactory.
First, it is possible to use true and false that will set $? to respectively 0 and 1.
#! /bin/bash
echo $?
true
echo $?
false
echo $?
This will output xxx, 0, 1. This workaround is limited, because it only allow to affect the values 0 and 1.
Then, it is possible to write some code in C (or other) that will just return the value in parameter via exit and then call this function. Example :
#! /bin/bash
rm foo.c
touch foo.c
echo "#include <stdio.h>" >> foo.c
echo "#include <stdlib.h>" >> foo.c
echo "int main(int argc, char **argv)" >> foo.c
echo "{" >> foo.c
echo " return atoi(argv[1]);" >> foo.c
echo "}" >> foo.c
gcc -o foo foo.c
./foo 42
echo $?
That will output 42. Even though it works, it is pretty nasty for doing something so simple, not to mention that this is only a simplified version without all the checkings that would have to be done in order to be sure not to overwrite anything. In addition, this require gcc to be present on the system.
There's no need to involve C at all if you just want to set the most recent return code (although I'm wondering why you're trying to do this). A simple shell function is sufficient:
ret() { return $1; }
Call it like ret 42.
Of course you know that '$?' holds the exit code of the last executed command. If the command is your own written executable, then '$?' holds its exit code.
The default value is 0 (success).
To sum up: use exit in your own scripts.
$ ls
$ echo $?
0
$ echo '#!/bin.bash' > script1.sh && chmod u+x script1.sh
$ ./script1.sh
$ echo $?
0
$ echo -e '#!/bin.bash\nexit 42' > script2.sh && chmod u+x script2.sh
$ ./script2.sh
$ echo $?
42
Also, have a look at Return value in bash script
In this example, I have a process that takes some time. Let's says 1 second. If I write the following Makefile, FOO will be expanded 3 times for make all and none for make clean.
If I want to save some execution time for all I can assign FOO using := instead of =. However this will cause FOO to be expanded for the target clean even if it doesn't use it.
FOO = $(shell echo -e "+1" >> foo && echo "Hello" && sleep 1)
all:
#echo $(FOO)
#echo $(FOO)
#echo $(FOO)
#cat foo
clean:
rm foo
The output:
$ make
Hello
Hello
Hello
+1
+1
+1
I would like to force Make to expand a variable only once only if required.
Is it possible to do it so?
The "best" solution I can come up with looks like this:
$ cat coin.mk
FOO = FOO:=$(shell echo -e "+1" >> foo && echo "Hello" && sleep 1)
defined = $(and $(filter-out undefined,$(origin $1)),$($1))
all:
echo $(eval $(FOO))$(FOO)
echo $(FOO)
echo $(FOO)
cat foo
clean:
rm foo
$ time make -f coin.mk clean
rm foo
rm: cannot remove `foo': No such file or directory
make: *** [clean] Error 1
real 0m0.003s
user 0m0.003s
sys 0m0.000s
$ time make -f coin.mk
echo Hello
Hello
echo Hello
Hello
echo Hello
Hello
cat foo
+1
real 0m1.009s
user 0m0.002s
sys 0m0.004s
$ time make -f coin.mk clean
rm foo
real 0m0.003s
user 0m0.000s
sys 0m0.003s
Which works but requires special-casing the first use of the variable in the make run ($(eval $(FOO)) run a second time will cause a make error).
I tried briefly to encapsulate the eval logic inside the value of FOO but most attempts were blocked by make complaining that *** Recursive variable 'FOO' references itself (eventually). Stop.
Try this:
FOO = $(eval FOO := $(shell echo "+1" >> foo && echo "Hello" && sleep 1))$(value FOO)
The first time make expands $(FOO) it will first expand the eval, which resets the variable FOO using :=. Then it resolves the value of the FOO variable. In subsequent expansions, due to the eval, FOO expands directly to the value.
I should also point out that if you have at least GNU make 4.0 you can use a new feature added to the POSIX standard for make recently, the != operator:
FOO != echo "+1" >> foo && echo "Hello" && sleep 1
which does exactly what you want here.
Why does the code
date
bash -c "date"
declare -x date='() { echo today; }' #aka export date='() { echo today; }'
date
bash -c "date"
print
Wed Sep 24 22:01:50 CEST 2014
Wed Sep 24 22:01:50 CEST 2014
Wed Sep 24 22:01:50 CEST 2014
today
?
Where (and why) does the evaluation
date$date
happen and getting
date() {echo today; }
Ad: #Etan Reisner
I exporting a variable - not a function. Bash makes a function from it. The
export date='someting'
is still a variable regardless of its content. So, why is
export date='() { echo something; }' #Note, it is a variable, not function.
converted to an function?
The mentioned security advisory talks about the execution of the command following the variable, for example,
x='() { echo I do nothing; }; echo vulnerable' bash -c ':'
^^^^^^^^^^^^^^^
This is executed - this vunerability is CLOSED in version 4.3.25(1).
The command after the env-definition isn't executed in the latest Bash.
But the question remains - Why does Bash convert the exported variable to a function?
It is a bug ;) Full demo, based on #chepner's answer:
#Define three variables
foo='() { echo variable foo; }' # ()crafted
qux='() { echo variable qux; }' # ()crafted
bar='variable bar' # Normal
export foo qux bar # Export
#Define the same name functions (but not qux!)
foo() { echo "function foo"; }
bar() { echo "function bar"; }
declare -fx foo bar #Export
#printouts
echo "current shell foo variable:=$foo="
echo "current shell foo function:=$(foo)="
echo "current shell bar variable:=$bar="
echo "current shell bar function:=$(bar)="
echo "current shell qux variable:=$qux="
echo "current shell qux function:=$(qux)="
#subshell
bash -c 'echo subshell foo variable:=$foo='
bash -c 'echo subshell foo command :=$(foo)='
bash -c 'echo subshell bar variable:=$bar='
bash -c 'echo subshell bar command :=$(bar)='
bash -c 'echo subshell qux variable:=$qux='
bash -c 'echo subshell qux command :=$(qux)='
prints
current shell foo variable:=() { echo variable foo; }=
current shell foo function:=function foo=
current shell bar variable:=variable bar=
current shell bar function:=function bar=
current shell qux variable:=() { echo variable qux; }=
tt: line 20: qux: command not found
current shell qux function:==
subshell foo variable:== #<-- LOST the exported foo variable
subshell foo command :=function foo=
subshell bar variable:=variable bar=
subshell bar command :=function bar=
subshell qux variable:== #<-- And the variable qux got converted to
subshell qux command :=variable qux= #<-- function qux in the subshell (!!!).
Avoiding the long comments, here is code from the Bash sources:
if (privmode == 0 && read_but_dont_execute == 0 && STREQN ("() {", string, 4))
^^^^^^^^ THE PROBLEM
{
string_length = strlen (string);
temp_string = (char *)xmalloc (3 + string_length + char_index);
strcpy (temp_string, name);
temp_string[char_index] = ' ';
strcpy (temp_string + char_index + 1, string);
if (posixly_correct == 0 || legal_identifier (name))
parse_and_execute (temp_string, name, SEVAL_NONINT|SEVAL_NOHIST);
/* Ancient backwards compatibility. Old versions of bash exported
functions like name()=() {...} */
The "ancient" (seems) was better... :)
if (name[char_index - 1] == ')' && name[char_index - 2] == '(')
name[char_index - 2] = '\0';
The key point to remember is that
foo='() { echo 5; }'
only defines a string parameter with a string that looks a lot like a function. It's still a regular string:
$ echo $foo
() { echo 5; }
And not a function:
$ foo
bash: foo: command not found
Once foo is marked for export,
$ export foo
any child Bash will see the following string in its environment:
foo=() { echo 5; }
Normally, such strings become shell variables, using the part preceding the = as the name and the part following the value. However, Bash treats such strings specially by defining a function instead:
$ echo $foo
$ foo
5
You can see that the environment itself is not changed by examining it with something other than Bash:
$ perl -e 'print $ENV{foo}\n"'
() { echo 5
}
(The parent Bash replaces the semicolon with a newline when creating the child's environment, apparently). It's only the child Bash that creates a function instead of a shell variable from such a string.
The fact that foo could be both a parameter and a function within the same shell;
$ foo=5
$ foo () { echo 9; }
$ echo $foo
5
$ foo
9
explains why -f is needed with export. export foo would cause the string foo=5 to be added to the environment of a child; export -f foo is used to add the string foo=() { echo 9; }.
You are essentially manually exporting a function with the name date. (Since that is the format that bash uses internally to export functions. Which is suggested by Barmar in his answer. This mechanism is mentioned here at the very least.)
Then when you run bash it sees that exported function and uses it when you tell it to run date.
Is the question then where is that mechanism specified? My guess is it isn't since it is an internal detail.
This should show the merging of the behaviours if that helps anything.
$ bar() { echo automatic; }; export -f bar
$ declare -x foo='() { echo manual; }'
$ declare -p foo bar
declare -x foo="() { echo manual; }"
-bash: declare: bar: not found
$ type foo bar
-bash: type: foo: not found
bar is a function
bar ()
{
echo automatic
}
$ bash -c 'type foo bar'
foo is a function
foo ()
{
echo manual
}
bar is a function
bar ()
{
echo automatic
}
The answer to your question comes directly from man bash:
The export and declare -x commands allow parameters and functions
to be added to and deleted from the environment. If the value of a
parameter in the environment is modified, the new value becomes part
of the environment, replacing the old.
Thus
declare -x date='() { echo today; }'
replaces date in the environment. The next immediate call to date gives date as it exists in the script (which is unchanged). The call to bash -c "date" creates a new shell and executes date as defined by declare -x.