Makefile: writing filtering outputs to separate files [duplicate] - shell

I found a difference of behaviour between GNU Make 4.1 and 3.81 and wonder whether my code is not POSIX compliant which 4 is enforcing more strictly, or whether something else is going on.
I distilled the failure case to this Makefile
.POSIX:
all: test-b
test-a:
cat a.txt b.txt c.txt >results.txt
test-b:
cat {a,b,c}.txt >results.txt
Assuming those files have been created with cat {a,b,c}.txt, target test-a always works, yet test-b works on Make 3.81 but fails on 4.1.
The output for 3.81:
$ make --version
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program built for i386-apple-darwin11.3.0
$ make
cat {a,b,c}.txt >results.txt
$ echo $?
0
The output for 4.1:
$ make --version
GNU Make 4.1
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
$ make
cat {a,b,c}.txt >results.txt
cat: {a,b,c}.txt: No such file or directory
Makefile:9: recipe for target 'test-b' failed
make: *** [test-b] Error 1
$ echo $?
2
It's possible the cat command is actually failing in 3.81 and it just isn't pointing it out, as later versions of GNU Make mention passing the -e flag to the shell when invoking target commands to make it more POSIX compliant, but I can't see how that command could be failing.
I assume the wildcards are handled solely by the shell, so I can't see how invoking the shell via a make target command should be any different.
Which is these behaviours are correct? If wildcards like that don't work in Makefiles, which other wildcards can I assume to work?
test-b still fails in 4.1 even if .POSIX: is removed from the file.

Recipes are sent to the shell. They are not interpreted by make. So your question is really, are curly-brace expansions supported by the shell?
That depends on which shell make uses. They are not supported by POSIX standard sh. They are supported by bash (and many other shells).
Make always invokes /bin/sh, regardless of what shell you personally use, unless you specifically set the make SHELL variable to something else. On some systems, /bin/sh is a symlink to /bin/bash so they are the same thing (bash runs in a "POSIX emulation" mode when invoked as /bin/sh but most bash features are still available). Other systems use different shells, such as dash, as /bin/sh which do not have extra bash features.
So, you can either (a) not have a portable makefile and assume /bin/sh is the same as /bin/bash, (b) set SHELL := /bin/bash in your makefile to force it to use bash always (but fail on systems that don't have bash installed), or (c) write your makefile recipes to use only POSIX sh features so it works regardless of which shell is used for /bin/sh.

Related

if statement in ONESHELL Makefile not working

I get /bin/sh: -c: line 1: syntax error: unexpected end of file for the if statement in the makefile below — what's wrong? (My prompt is /tmp >.)
/tmp > make test
set -e
if [[ -f /tmp/device ]] ; then
/bin/sh: -c: line 1: syntax error: unexpected end of file
make: *** [test] Error 2
/tmp > cat Makefile
.ONESHELL:
test:
set -e
if [[ -f /tmp/device ]] ; then
echo 'Do something'
fi
I am using GNU Make 3.81.
Using GNU Make 3.81, the .ONESHELL feature is demonstrably not supported — you showed that, and I showed that. I tested on Mac OS X 10.11.5, with the supplied /usr/bin/make as GNU Make 3.81. I demonstrated to my satisfaction by adding strategically placed echo PID=$$$$ lines before set -e, after it, and so on, and observed different PID values. GNU Make version 3.81 is from 2006 (there was also version 3.82 from 2010, before version 4.x was released, starting with 4.0 in 2013).
The current version of GNU Make is 4.2.1 (June 2016). That version does support the feature; your script works as expected when using a sufficiently recent version of GNU Make. It is a feature that has been around for a while — you probably don't have to upgrade to the latest to get the support, but why would you go with a down-version if you're going to upgrade anyway.
If you wish to use the .ONSHELL: feature, you'll have to ensure you are using a new enough version of GNU Make (newer than version 3.81). If that's not feasible, don't use the feature.
Reading the NEWS file from 4.2.1, it is clear that ONESHELL was added to GNU Make version 3.82:
Version 3.82 (28 Jul 2010)
…
* New special target: .ONESHELL instructs make to invoke a single instance
of the shell and provide it with the entire recipe, regardless of how many
lines it contains. As a special feature to allow more straightforward
conversion of makefiles to use .ONESHELL, any recipe line control
characters ('#', '+', or '-') will be removed from the second and
subsequent recipe lines. This happens _only_ if the SHELL value is deemed
to be a standard POSIX-style shell. If not, then no interior line control
characters are removed (as they may be part of the scripting language used
with the alternate SHELL).

How can I debug the Bourne Shell with gdb?

I built a toolchain script to prepare a Linux build environment. The script can be found here: https://github.com/mynameismevin/prometheus/blob/toolchain/ptool-make.sh
The script runs perfectly until after the Perl section around line 416. After Perl is done, when it goes to unzip sed, it complains with this error:
tar (child): sed-4.2.2.tar.bz2: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
However, if I run the sections by hand, it completes without errors. If I split the script into ptool-make1.sh (which ends at Perl) and ptool-make2.sh (which starts at sed) and run them sequentially then they both complete without any issues at all. At this point, I assert the issue isn't with the script, and I would like to debug the shell to see if it's an issue with the shell.
Here are some useful configurations:
user#ubuntu:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.2 LTS"
user#ubuntu:~$ bash --version
GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
user#ubuntu:~$ ls -lh $(which bash)
-rwxr-xr-x 1 root root 998K Oct 7 2014 /bin/bash
I don't think Ubuntu bash comes with debugging symbols, so I would assume I would have to recompile to include those?
How would I use gdb to debug the shell when I run a script? Or how would I have the shell log to a file while the script runs so I can open it with gdb after it's done? I know how to debug a shell script, I don't want to debug the script, I want to debug the shell.
Edit: It doesn't look like Ubuntu bash comes with debugging symbols out of the box. Reading symbols from bash...(no debugging symbols found)...done.
Edit: $PROMETHEUS is set in my root shell. At the end of Perl cd .. results in the same results a cd $PROMETHEUS/sources/.
Shells come with their own debugging tools. The most simple is running them with -x (bash -x ...script...) which will print each command after variable expansion but before it is executed. That's usually enough to determine the problem.
For more ideas, see How to debug a bash script?
You should also consider to write helper functions to reduce the size of the script to just a few lines. You could move special options to configure or post-build steps into extra files and run them from the helper function if they exist.
Looking at the code, it seems that this line is the culrit:
cd $PROMETHEUS/sources/
everywhere else, you just use cd ... If PROMETHEUS isn't defined (the script doesn't define it, that becomes cd /sources/ which should also fail but doesn't abort your script. Try
cd $PROMETHEUS/sources/ || exit 1
instead. Also #!/bin/bash -u might be useful (abort on undefined variables).
Lastly, you can use pushd and popd to navigate the folders.

Setting PATH variable from inside makefile does not work for make 3.81

Change the environment variable `PATH' in Makefile does NOT take effect with make in CLT, and it's OK with the make util which I compiled from original source.
The simple Makefile
PATH := $(PATH):/opt/bin
export PATH
all:
#cscope --version
My tests
/tmp $ echo $PATH
/usr/bin:/bin:/usr/sbin:/sbin
/tmp $ ls /opt/bin/cscope
/opt/bin/cscope
/tmp $ which make
/usr/bin/make
/tmp $ make
make: cscope: No such file or directory
make: *** [all] Error 1
/tmp $ ./_install/bin/make
cscope: version 15.7a
/tmp $ make --version
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program built for i386-apple-darwin11.3.0
/tmp $ ./_install/bin/make --version
GNU Make 3.82
Built for x86_64-apple-darwin12.0.0
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Can anyone help?
You are confusing make variables with shell variables. With your setup, the shell command that gets invoked is not influenced by the make variable PATH that you have set.
You should set the PATH variable inside your recipe, like in
all:
#PATH=$(PATH):/opt/bin; cscope --version
You have to to both on one line because every line in the recipe will be run in another shell, effectively losing the PATH setting that you just did. Or you can divide it over multiple lines by adding a backslash \ at the end of each line:
all:
#PATH=$(PATH):/opt/bin; \
cscope --version
Update
Sorry, I missed the importance of your makefile working with make 3.82. I tried with both versions and indeed, 3.81 and 3.82 behave differently in this case.
I did get this to work with 3.81 though by invoking make as follows:
make SHELL="/bin/sh -c"
or
make SHELL=/bin/bash

Makefile odd behaviour

I've got this Makefile that is presenting some odd behaviour:
$ javac -d Classes -sourcepath .. -classpath `for x in \`ls Classes/jars/*\`; do echo -n $x:; done` PCA/PCAClassifier.java
compiles the java just fine. But for somereason when I call
make PCA
I get:
Compiling PCAClassifier
javac -d Classes -sourcepath .. -classpath `for x in \`ls Classes/jars/*\`; do echo -n $x:; done` PCA/PCAClassifier.java
javac: invalid flag: Classes/jars/Jama.jar:
Usage: javac <options> <source files>
use -help for a list of possible options
make: * [Classes/RobotSuite/PCA/PCAClassifier.class] Error 2
I am so confused. Anyone have a solution?
Make version info:
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program built for i386-apple-darwin10.0
Background: Working on a group project. I'm running Ubuntu, my partner is running Mac OS X. For whatever reason, This makefile works just fine on my computer, but not on his. Even though the command works in his BASh terminal, somehow Make isn't sending it correctly.
You have given us almost no information to go on. Nonetheless, you are in luck!
On Mac OS X, bash has been built with --enable-strict-posix-default and hence the xpg_echo shell option defaults to being on in POSIX mode. POSIX mode is on when the shell has been invoked as /bin/sh, as it has been when it is invoked by Make, unless you instruct it otherwise by setting the $(SHELL) make variable (which you probably shouldn't).
This is the difference between Linux and Mac OS that is killing your makefile. When xpg_echo is on, the shell's built-in echo treats -n as just another argument to be printed (and hence also prints a newline). So the single classpath argument that you're trying to construct ends up as a bunch of separate arguments (half of which are "-n") and javac gets confused.
(This doesn't happen on the command line, even on Mac OS X, because then the shell has been invoked as /bin/bash, so it's not in POSIX mode and xpg_echo is off.)
So you have a number of options for fixing this:
Use /bin/echo -n in your shell snippet; unlike the built-in one, the real echo command these days mostly always understands -n;
Construct the classpath argument in a less roundabout way than your shell for loop; for example
... -classpath `ls Classes/jars/* | tr '\n' :` ...
or, if you are already assuming GNU Make, using make wildcards and functions instead of a shell snippet;
Have your colleague add shopt -u xpg_echo to an appropriate bash startup file on their machine (however this will just lead to future confusion when your next Mac OS-using colleague comes along and you've all long since forgotten how you fixed this this time).
Finally, a general note about debugging makefiles: when a javac ... command in a makefile recipe is giving incomprehensible error messages, change it to echo javac ... instead. Then you'll be able to see exactly how it's being invoked -- which, as seen here, may not be what you intended.

How to handle shell expansions in GNU Make under Ubuntu?

Given this very simple Makefile:
all:
#mkdir -pv test/{a,b}
I get this output on OS X 10.6.8 and CentOS 5.5:
mkdir: created directory `test'
mkdir: created directory `test/a'
mkdir: created directory `test/b'
But on Ubuntu 11.04 I get this:
mkdir: created directory `test'
mkdir: created directory `test/{a,b}'
Running the command mkdir -pv test/{a,b} manually in the shell on all platforms gives the expected result.
The version of GNU Make is the same on all platforms:
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program is built for [PLATFORM]
What's different under Ubuntu and why doesn't the shell expansion work there?
The problem is probably that Make spawns /bin/sh. It is usually a symlink to your system's default shell.
Option 1
You could make sure it points to bash (as this is a bashism). Probably, it is now /bin/dash or /bin/sh, depending on your version of Ubuntu.
Option 2
Easier option:
SHELL=/bin/bash
all:
#echo a{3,4}
#bash -c 'echo a{3,4}'
This prints the same output twice unless you comment-out the SHELL= line
Option 3
If you can't/don't want to modify the make file, you can invoke it like so:
make SHELL=/bin/bash
beware of interactions with sub-makefiles or includes. You might want to look at the make -e option and the make export keyword: http://www.gnu.org/s/hello/manual/make/Variables_002fRecursion.html
Make has a function to do this:
#mkdir -pv $(addprefix test/,a b)
gets expanded to
#mkdir -pv test/a test/b
The docs are very thorough.
It's been a long, long time since I've used Make...
There are several ways to specify a particular shell to use. The default shell for old Make was the original Bourne shell. If you wanted a different shell, you had to set it yourself.
You're using Linux and GNU, so I'll assume that you're using BASH as the default shell. Try this command in your Make:
echo "random = $RANDOM"
If this simply prints random = and doesn't include a random number, your Make is using Bourne shell as its default shell instead of BASH. (Which is weird because I didn't think there was a real Bourne shell in Linux...) To get around this:
You can add in a SHELL macro pointing to the BASH shell.
You can include the shell in your command.
Instead of:
#mkdir -pv test/{a,b}
Put this:
/bin/bash -c #mkdir -pv test/{a,b}
This specifies you want to use BASH and not the standard /bin/sh Bourne shell.
If the echo random = $RANDOM does print a random number, you're using BASH (or at least Kornshell), but the BRACE EXPANSION might not be set. Try using this in your Makefile:
set -o
And make sure braceexpand is on. It could be off when you run Make.

Resources