I'm trying to reproduce a situation which happens only in some machines. To reproduce it, I create a directory with 2000 files:
mkdir /tmp/test
cd /tmp/test
for f in $(seq 1 2000); do touch $f.txt; done
Then I use the following Makefile (simplified from the real use case):
FILES:=$(shell find . -name '*.txt')
%.done: %.txt
#echo "done $#"
toolong:
#$(foreach file,$(sort $(FILES)), \
if $(MAKE) $(file); \
then echo "did $(file)" >> $#; \
else echo "failed $(file)" >> $#; fi; )
Running make produces, unsurprisingly, an error:
make: execvp: /bin/sh: Argument list too long
This question presents a solution which does work. However, I need to understand exactly why this error does not happen on my colleagues' computers. I tried the following things:
Increasing stack limit (ulimit -s gives the same result on both machines, 8192, and increasing it does not change anything);
Checking getconf ARG_MAX (2097152 in both machines);
Checking MAX_ARG_STRLEN (131072 in both machines);
Using a different shell (zsh is being used in both machines; I also tried bash, dash and sh, via export SHELL=<shell> make, and also by replacing the symlink /bin/sh -> /bin/bash with a link to dash).
Finally, I tried recompiling Make from source, and realized that, even when I compile the same version of Make (4.1) in my Ubuntu test machine, I get the same behavior as I had in my Fedora, that is, the error "argument list too long".
make --version only shows a single difference between them:
Version from the apt package:
GNU Make 4.1
Built for x86_64-pc-linux-gnu
Version compiled from source (./configure && make):
GNU Make 4.1
Built for x86_64-unknown-linux-gnu
I even tried compiling make-dfsg, which results in an identical make --version, but the result is the same as with my other manually-compiled make.
By increasing the number of files on Ubuntu, I managed to identify that the actual limits in the size of the generated command line are:
Fedora or Arch Linux (both with Make 4.2.1), or Ubuntu with manually-compiled Make 4.1: 128 KB (~1200 files);
Debian Sid or Ubuntu, both with Make 4.1 installed from apt package: 2 MB (~19300 files).
I'd really like to understand (1) why this difference exists, and (2) how could I compile Make to obtain the higher limit, so that I can have the exact same behavior on both machines.
Your recipe expands as a one-line shell compound command (because of the line ending backslashes). And this line is probably way too long. Did you check what it looks like (the shell compound command) on all machines? I suggest that you wrap it in $(info...):
toolong:
#$(info $(foreach file,$(sort $(FILES)), \
if $(MAKE) $(file); \
then echo "did $(file)" >> $#; \
else echo "failed $(file)" >> $#; fi; ))
and see if the output is the same on your machine and on the others. If, for any reason, yours is longer, it could be the explanation. Else it must be a OS difference...
Note: you could have one rule per target, instead of one single rule with a huge recipe for all.
Note: your recipe (independently from its fantastic length) does nothing useful. As the files already exist and do not have pre-requisites, all your sub-make calls will just tell you that the files are up-to-date.
Related
I want to add a Make target that runs a program with envdir.
ENVDIR := $(shell command -v envdir)
serve: | $(ENVDIR)
$(ENVDIR) /usr/local/myservice/env python -m SimpleHTTPServer 8000
The idea is that if the user doesn't have envdir installed, it will install it for them. I need the pipe character because I don't always want it to run; I just want any version of it to be present. Unfortunately, I can't predict very well where envdir will be installed to.
UNAME := $(shell uname)
$(ENVDIR):
ifeq ($(UNAME), Darwin)
brew install daemontools
endif
ifeq ($(UNAME), Linux)
sudo apt-get install daemontools
endif
# Add other operating systems here as appropriate...
The other problem I run into is that if the user doesn't have envdir installed, the variable evaluates to the empty string. So a) My "install envdir" target doesn't actually run, and b) after it runs, $(ENVDIR) is still set to the empty string, so loading it in the serve target doesn't work.
Is there a way to a) get the install target to run, even if ENVDIR is undefined, and b) to redefine the value of the variable at the end of the "install envdir" target run?
Please don't respond with "why are you using envdir" - I run into this problem with a variety of different binaries I depend upon, this just happened to be an easily shareable example.
For the "empty" issue you can do this:
ENVDIR := $(or $(shell command -v envdir),envdir)
so that if the shell function expands to the empty string, envdir will be substituted (the or function was added in GNU make 3.81). That will probably be enough to solve all your problems since after the installation the envdir command will now appear on the PATH, so you don't need the full path.
You could do it completely in the shell as so:
ENVDIR := $( command -v envdir 2> /dev/null || ( install envdir > /dev/null && echo newEnvDirPath ) )
This is simplest, but causes envdir to be installed, even if it's not required for the current target. If you want it in a target, you could do it something like:
ENVDIR:=$(cat .ENVDIR 2> /dev/null)
$(type -v $(ENVDIR) > /dev/null || rm .ENVDIR )
.ENVDIR:
command -v envdir 2> /dev/null > $# || ( install envdir > /dev/null && echo newEnvDirPath > $#)
$(eval ENVDIR:=$$(cat $#))
shell: | .ENVDIR
If the file .ENVDIR does not exist, it will run the recipe, which will either write the installed path to .ENVDIR, or it will install and write the new path to .ENVDIR. Either way, when the recipe line finishes, .ENVDIR will contain the executable path. The target then runs $(eval) to set ENVDIR. Notice that calling eval causes the makefile to be reparsed, which is not ideal, but should only happen once on the system. (Also note the double $$ in front of the cat command, which prevents make from expanding this at read time...).
Alternatively, if .ENVDIR does exist, it will read the file to determine which envdir to try to use. A check is done on the next line to ensure that that version still exists on the system, and if not, it deletes the file, forcing it to be recreated.
While following the tutorial on the Chisel official website for installation, I came to the point where I should test if the installation was done correctly. Doing so yields this error:
set -e -o pipefail; "sbt" -Dsbt.log.noformat=true -DchiselVersion="2.+" "run Parity --genHarness --compile --test --backend c --vcd --targetDir /home/me/chisel-tutorial/generated/examples " | tee /home/me/chisel-tutorial/generated/examples/Parity.out
/bin/bash: sbt: command not found
make: *** [/home/me/chisel-tutorial/generated/examples/Parity.out] Error 127
There is another question regarding the same problem here, where the suggestion to add SHELL=/bin/bash to the Makefile is made. That did not work for me. Another suggestion is to remove set -e -o pipefail: this suggestion actually works but is it OK to remove that option? what does it do?
Edit_1:
I have installed sbt and added its path to the PATH variable.
$ which sbt
/usr/bin/sbt
But still I am getting this error when running make Parity.out
set -e -o pipefail; "sbt" -Dsbt.log.noformat=true -DchiselVersion="2.+" "run Parity --genHarness --compile --test --backend c --vcd --targetDir /home/me/chisel-tutorial/generated/examples " | tee /home/me/chisel-tutorial/generated/examples/Parity.out
/bin/sh: 1: set: Illegal option -o pipefail
make: *** [/home/me/chisel-tutorial/generated/examples/Parity.out] Error 2
If I edit this part of the file suffix.mk:
$(objdir)/%.dot: %.scala
set -e -o pipefail; "$(SBT)" $(SBT_FLAGS) "run $(notdir $(basename $<)) --backend dot --targetDir $(objdir) $(CHISEL_FLAGS)"
$(objdir)/%.out: %.scala
set -e -o pipefail; "$(SBT)" $(SBT_FLAGS) "run $(notdir $(basename $<)) --genHarness --compile --test --backend c --vcd --targetDir $(objdir) $(CHISEL_FLAGS)" | tee $#
By deleting the -o option in the set -e -o pipefail it works, I get the PASSED and [success] message after running $ make Parity.out. So what is going on?
Edit_2:
It is working fine now after I added the SHELL=/bin/bash to the Makefile, so it was first a problem of not having sbt as Nathaniel pointed out then editing the Makefile to include SHELL=/bin/bash.
set -e -o pipefail is a way of making sure that the execution of the bash script both works as expected and that if there is a failure, it halts immediately (rather than at some later stage). Removing it might work - but if there is a failure it might get swallowed and hide the fact it's broken.
But I think your problem lies here, making the other question a bit of a red herring:
/bin/bash: sbt: command not found
Do you have sbt installed on your system? Run which sbt as the user that executes the script. For instance, on my system:
$ which sbt
/opt/local/bin/sbt
If you don't have it on your system, nothing will be returned by running which.
The script clearly needs access to sbt and is failing when it doesn't find it. If you do have it on your system, then there is a mismatch between the user running the script and access to that file. You'll need to post more information about how you're executing the script: in that case it is likely you'll have to update your PATH variables to be able to find the sbt executable.
Given that, after fixing this, you still have a problem, you have to ensure that you're running in bash, and not another terminal type. The reason for this is that bash supports set -o pipefail but a lot of other terminals don't. We suspect this might be the case because of the error messages:
/bin/sh: 1: set: Illegal option -o pipefail
Here we see that /bin/sh (the shell) is being invoked by the program. Use ls -l /bin/sh to determine if your /bin/sh is pointing to a particular shell. If it is not pointed to a bash shell, then you either need to repoint it (be careful! this is probably another question in it's own right), or need to specify to your Scala program to use a specific shell.
I have a makefile including the following lines:
buildrepo:
#$(call make-repo)
define make-repo
for dir in $(C_SRCS_DIR); \
do \
mkdir -p $(OBJDIR)/$$dir; \
done
endef
On the line with the commands for dir in $(C_SRCS_DIR); \ I get the following error message:
"dir not expected at this moment"
make: *** [buildrepo] Error 255
I am using GNU make.
Can anybody tell me what is going wrong?
Actually this for ... in ... ; do ... done statement is a Unix command not a GNU make command, therefore I guess you are using a Windows machine (or any other one). You have to find the equivalent for your system.
But GNU make has a foreach function which works like this :
$(foreach dir,$(C_SRCS_DIR),mkdir -p $(OBJDIR)/$(dir);)
Also note that in your very specific case (not related to GNU make but to Windows) you can create all the dirs without a for/foreach loop, just like this :
mkdir -p $(addprefix $(OBJDIR)/,$(C_SRCS_DIR))
If I run mkdir -pv, the -p works, but I do not get verbose output; however, if I run just mkdir -v, the verbose output appears as expected. Also potentially of note, the longform of -v does not seem to work at all.
From my testing:
mkdir -p a/b/c: creates a/, a/b/, and a/b/c/, gives no output to terminal (as expected)
mkdir -v d: creates d/ and outputs mkdir: created directory 'd' (as expected)
mkdir -pv e/f/g: creates e/, e/f/, and e/f/g/, gives no output to terminal (why?)
mkdir --verbose h: gives a illegal option -- - error (why?)
Update: I filed a bug report with Apple for this issue, and received the following reply:
Their answer that "-v is not applicable" does not make much sense to me, since mkdir -v works just fine, but since there are various workarounds and I am no longer even using OSX, I don't think it's worth pursuing any further to me.
Macs use BSD-based code that is (mostly) POSIX compliant but (mostly) without GNU extensions (such as double-dash long options). The man page does document -v and -p, and -p works but does seem to suppress the -v option (which is probably when it is most useful).
One of your options is to file a bug with the Darwin or BSD teams, or with Apple. That's the way it is; it is arguably not the way it should be. (GNU mkdir supports -v and prints the directories it creates when used with -p, which makes more sense, and supports the 'it is a bug' contention.)
With thanks to SnoringFrog:
Another option is to install and use the GNU mkdir command on OSX. It is part of GNU coreutils and you could install it as explained in How to replace Mac OS X utilities with GNU core utilities at Ask Different. Then, you could alias mkdir to point to gmkdir to get the expected behavior (assuming you don't use --default-names when installing the GNU tools).
You could also do this if logging is critical (since the -v flag works in the simple case):
mkdir -v path &&
mkdir -v path/to &&
mkdir -v path/to/{destination1,destination2,etc} ;
Think Different or Think Again?!
I'd like to use $(error ...) to abort my make process if certain preconditions aren't met. The fails_to_work target should abort when failing test -d /foobar.
BAD.mk
all: this_works fails_to_work
this_works:
#echo echo works...
#test -d ~ || echo ~ is not a directory
#test -d /foobar || echo /foobar is not a directory
fails_to_work:
#echo error does not work...
#test -d ~ || $(error ~ is not a directory)
#test -d /foobar || $(error /foobar is not a directory)
$ make -f BAD.mk
echo works...
/foobar is not a directory
BAD.mk:9: *** ~ is not a directory. Stop.
As you can see, not even "error does not work..." is echoed to the screen. The recipe for fails_to_work fails before it gets started. How do I solve this? One of my use cases is#test -d $(MY_ENV_VAR), but I don't think that differs from the hard-coded paths given in the example.
UPDATE (version information)
$ make --version
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program built for x86_64-pc-linux-gnu
You're trying to get the shell stuff in a recipe to conditionally invoke makefile stuff, which doesn't work, as you've found.
I can think of two options:
Simply remove the $(error) stuff. If test fails, then it will return a non-zero exit status, and the Make process will terminate at that point.
Take the test out of the rule, and use a Make conditional (which in turn invokes shell functionality), e.g.:
ifeq ($(shell test -d /foobar; echo $$?),1)
$(error Not a directory)
endif
Shell commands for a make recipe are effectively stored as a single recursively expanded variable. At the point make decides to run the recipe, it expands the variable, and then runs each line in its own shell invocation. Any $(error ...) that gets expanded will cause make to abort even before invoking the first command.
Note though that the untaken branch of a $(if ...) or $(or ...) &c. will not be expanded. Thus, you could do
.PHONY: rule-with-assert
rule-with-assert:
$(if $(realpath ${should-be-file}/),$(error Assertion failure: ${should-be-file} is a folder!))
⋮
Note that trailing / in the realpath.
Of course macros help to tidy this up a lot.
assert-is-file = $(if $(realpath $1/),$(error Assertion failure: [$1] is a folder!))
.PHONY: rule-with-assert
rule-with-assert:
$(call assert-is-file,${should-be-file})
⋮
It's worth noting again that it doesn't matter where you put the $(call assert-is-file,…) in the recipe.
Any $(error)will be generated as the recipe is expanded,
before any shell commands are run.
Why don't you just use exit 1 shell command instead of $(error ...)? Is there any reason to use the latter?
try_this:
#test -d /foobar || { echo /foobar is not a directory; exit 1; }
or_this:
#if [ ! -d /foobar ]; then echo /foobar is not a directory; exit 1; fi
Both of these will abort the make process unless -k flag is specified.
-k
--keep-going
Continue as much as possible after an error. While the target that failed, and those that depend on it, cannot be remade, the other prerequisites of these targets can be processed all the same.