Makefile max load on remote server - makefile

we have a huge project with very time consuming sub-tasks running on Unix. The whole make process runs multiple hours. So building in parallel is essential for us. The expensive jobs are preformed on a remote server via ssh. Everything works fine. But I'm afraid that accidentally a team member could forget to specify the number for the -j flag. The --load-average flag uses the load on the local server so no matter how busy the remote server is it would generate hundreds of sub-tasks on the remote server which would slow down the entire company. Please note that I don't have admin rights on either server. We could also live with limiting the number of jobs (hardcoded).
Thanks a lot in Advance
Karl

If you have GNU make 4.2 or better, you can check that the user didn't give a raw -j by looking at MAKEFLAGS:
ifeq (-j,$(filter -j,$(MAKEFLAGS)))
$(error You cannot use -j without specifying a number of jobs)
endif
If you have an earlier version of GNU make you can do this by checking if both -j was given AND the jobserver is active:
ifneq (,$(filter -j,$(MAKEFLAGS)))
ifneq (,$(filter --jobserver-%,$(MAKEFLAGS)))
$(error You cannot use -j without specifying a number of jobs)
endif
endif
(this version will work for newer versions as well).
It's possible this won't work with versions of GNU make <4.0; I didn't test it.

For older make versions you can consider the following. That is not pure makefile way, but works.
Solution
ifneq (0,$(shell ps --pid $$PPID -o args= | grep --perl-regexp -c '^make.*-j(?!\s*[0-9]+\b)'))
$(error You cannot use -j without specifying a number of jobs)
endif
# Or oneliner:
#$(if $(shell ps --pid $$PPID -o args= | grep --perl-regexp -c '^make.*-j(?!\s*[0-9]+\b)' | grep -v '^0$$'),$(error You cannot use -j without specifying a number of jobs))
# Below part just for testing
all a 1: ; #exit 0
Testing
$ make
$ make a
$ make 1
$ make -j1
$ make -j1 a
$ make -j1 1
$ make -j 1
$ make -j 1a
Makefile:6: *** You cannot use -j without specifying a number of jobs. Stop.
$ make -j a
Makefile:6: *** You cannot use -j without specifying a number of jobs. Stop.
$ make a -j
Makefile:6: *** You cannot use -j without specifying a number of jobs. Stop.
$ make 1 -j a
Makefile:2: *** You cannot use -j without specifying a number of jobs. Stop.
How it works
It is taking information from parent process (that is make process) using ps utility, and later filter out arguments using grep (regular expression is more or less: find make command with -j parameter that is not followed by a number).

Based on to the answer from Kuchara above I needed the following enhancement, since I call some subsystems with:
$(MAKE) $(MFLAGS) ...
and this will add the jobserver and remove the number from the -j flag in the subsystem:
yy:
#echo yy mflags: $(MFLAGS)
#echo yy ps: $(shell ps --pid $$PPID -o args=)
$(MAKE) $(MFLAGS) zz
zz:
#echo zz mflags: $(MFLAGS)
#echo zz ps: $(shell ps --pid $$PPID -o args=)
Testing:
make -j 3 yy
yy mflags: - --jobserver-fds=3,4 -j
yy ps: make -j 3 yy
zz mflags: -w --jobserver-fds=3,4 - --jobserver-fds=3,4 -j
zz ps: make - --jobserver-fds=3,4 -j zz
I needed to combine the answers from MadScientist and Kuchara:
ifneq (0,$(shell ps --pid $$PPID -o args= | grep --perl-regexp -c '^make.*-j(?!\s*[0-9]+\b)'))
ifeq (0,$(shell ps --pid $$PPID -o args= | grep --perl-regexp -c '^make.*--jobserver'))
$(error You cannot use -j without specifying a number of jobs)
endif
endif
Thanks a lot again at both for your great ideas.

Related

How to specify multiple targets in makefile?

I am using make with the version below:
$ make -v
GNU Make 4.3
Built for x86_64-pc-cygwin
Copyright (C) 1988-2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
I have a makefile with multiple targets as below. I use the “& :” syntax from the manual https://www.gnu.org/software/make/manual/html_node/Multiple-Targets.html
hello.asm.mk
src_prefix = ../../src/hello/
hellomain.s sayhello.s &: $(src_prefix)hellomain.c $(src_prefix)sayhello.c
gcc -S $(src_prefix)hellomain.c -o hellomain.s
gcc -S $(src_prefix)sayhello.c -o sayhello.s
.PHONY: clean
clean :
-rm hellomain.s sayhello.s
But it seems that it does not work, as below.
$ ll
total 1
-rwxrwx---+ 1 Payne None 250 2020-08-29 17:48:54.837841300 +0800 hello.asm.mk
$ make -f hello.asm.mk
gcc -S ../../src/hello/hellomain.c -o hellomain.s
gcc -S ../../src/hello/sayhello.c -o sayhello.s
$ ll
total 3
-rwxrwx---+ 1 Payne None 250 2020-08-29 17:48:54.837841300 +0800 hello.asm.mk
-rw-rw-r--+ 1 Payne None 463 2020-08-30 18:21:22.863042900 +0800 hellomain.s
-rw-rw-r--+ 1 Payne None 449 2020-08-30 18:21:23.045053400 +0800 sayhello.s
$ rm sayhello.s
$ ll
total 2
-rwxrwx---+ 1 Payne None 250 2020-08-29 17:48:54.837841300 +0800 hello.asm.mk
-rw-rw-r--+ 1 Payne None 463 2020-08-30 18:21:22.863042900 +0800 hellomain.s
$ make -f hello.asm.mk
make: 'hellomain.s' is up to date.
Could someone give me any advice? I am new to makefile and am still learning it.
In the first place, make recognizes a default target, not a default rule. The default target is the first one appearing in the file that does not start with a . and a capital letter. If you run make without designating a target to build, then it assumes you want the default target built, and in your case, that is hellomain.s. (And only hellomain.s, even though that appears in a rule that designates more than one target.)
The behavior that make exhibits for you is thus utterly normal, notwithstanding any use of a grouped-target rule (see below). You run make without designating a target, so it interprets you to want to build the default target. That target already exists and is up to date. That another target described in the makefile is missing is irrelevant. If you want to rebuild it with your current makefile then you could tell make so explicitly:
make sayhello.s
Now a few words about grouped-target rules. Note well that support for these is
specific to rather recent versions of GNU make, and not available on most make implementations, including, for example, the versions of GNU make included in many Linux distributions as of the time of this writing.
not intended for the use to which you are putting it.
Grouped-target rules address a longstanding issue of make's design: how to describe targets that are unavoidably created together. The canonical example would probably be the outputs of yacc or bison: a C source file and corresponding header file, both generated together by one run of the same program. You probably haven't yet the experience with make to appreciate how hard it is to describe that properly to traditional makes, but it is simple in GNU make 4.3 and later.
You have created a similar situation artificially. Don't do that. If you want to build multiple targets via one rule, then the idiomatic way to do it is to give them their own rules, and to name them as prerequisites to the same rule. Moreover, it is conventional, albeit not obligatory, to name the default target "all" in such cases. Example:
src_prefix = ../../src/hello
all: hellomain.s sayhello.s
hellomain.s : $(src_prefix)/hellomain.c
gcc -S $(src_prefix)/hellomain.c -o hellomain.s
sayhello.s : $(src_prefix)/sayhello.c
gcc -S $(src_prefix)/sayhello.c -o sayhello.s
clean :
-rm -f hellomain.s sayhello.s
.PHONY : all clean
Not that that makefile really exhibits good form generally. I've kept it pretty close to your original for didactic purposes, but if I were writing it, and assuming GNU make, then I would probably go with something more like this:
src_prefix = ../../src/hello
# A variable naming the targets, so I don't need to repeat the list, or risk
# different copies getting out of sync:
ASSEMBLY_TARGETS = hellomain.s sayhello.s
# specify the assembler command as a variable, near the top, to enable it to be
# easily changed:
AS = gcc -S
# default target named "all", with all targets I want to build as its prerequisites
all: $(ASSEMBLY_TARGETS)
# One pattern rule covering both targets. Pattern rules are GNU-specific, but
# I already stipulated that I'm writing for GNU make
%.s : $(src_prefix)/%.c
$(AS) $^ -o $#
# The clean rule also relies on the ASSEMBLY_TARGETS variable, so that we don't need
# to repeat the target list explicitly here.
clean :
-rm -f $(ASSEMBLY_TARGETS)
.PHONY : all clean
You should always provide the version of GNU make you're using and the operating system you're running on.
My best guess is that you're using an older version of GNU make that doesn't support the &: syntax. You need GNU make 4.3 to be able to use this feature.
It's always best to read the version of the GNU make manual that comes with your operating system, since that will be the version associated with the version of GNU make you're using. Also you can check the NEWS file for when features were added:
http://git.savannah.gnu.org/cgit/make.git/tree/NEWS
One example using a pattern rule:
SRC_DIR := ../../src/hello
ASM_TARGETS := hellomain.s sayhello.s
.PHONY: all clean
all: $(ASM_TARGETS)
# build .s file from .c file placed in a specified source dir
%.s: $(SRC_DIR)/%.c
gcc -S $^ -o $#
clean:
rm -f $(ASM_TARGETS)

Pass a path to the "." source in a makefile

In a directory I have a config file with my db variables.
This file (db/database.ini) looks like this:
[PostgreSQL]
host=localhost
database=...
user=postgres
password=...
I have another file (db/create_stmts.sql) where I have all my raw create table statements, and i am trying to experiment the use of a Makefile to have a command like this:
make create-db from_file=db/create_stmts.sql
In order not to repeat myself, I thought of tailing the variables of db/database.ini to a file which I would then source, creating shell variables to pass to psql in the make file.
Here's my plan:
make-db:
# from_file: path to .sql file with all create statements to create the database where to insert
# how to run: make create-db from_file={insert path to sql file}
file_path=$(PWD)/file.sh
tail -n4 db/database.ini > file.sh && . $(file_path)
# -U: --user
# -d: --database
# -q: --quiet
# -f: --file
psql -U $(user) -d $(database) -q -f $(from_file) && rm file.sh
Which I run by: make create-db from_file=db/create_stmts.sql
Which gives me this message - from which i kindof understand that the sourcing just did not work.
#from_file: path to .sql file with all create statements to create the database where to insert
# how to run: make create-db from_file={insert path to sql file}
file_path=/home/gabriele/Desktop/TIUK/companies-house/file.sh
tail -n4 db/database.ini > file.sh && .
# -U: --user
# -d: --database
# -q: --quiet
# -f: --file
psql -U -d -q -f db/schema_tables.sql && rm file.sh
psql: FATAL: Peer authentication failed for user "-d"
Makefile:3: recipe for target 'create-db' failed
make: *** [create-db] Error 2
Any help?
Another solution, perhaps simpler to understand:
make-db:
file_path=$$PWD/file.sh; \
tail -n4 db/database.ini > file.sh && . $$file_path; \
psql -U $$user -d $$database -q -f $$from_file && rm file.sh
Note using ; and \ to convince make to run all commands in a single shell, and using $$ to escape the $ and use shell variable references.
The error is in the text, namely
psql -U -d -q -f db/schema_tables.sql && rm file.sh
This happens because the variables $(user) and $(database) aren't set. Every line within a target is executed in a sub shell. There is now way to use source like you would in a regular script.
You could create a file named database.mk in which you define these variables and use include database.mk at the top of your makefile to include them:
Makefile
CONFILE ?= database
include $(CONFILE).mk
test:
#echo $(user)
#echo $(database)
database.mk
user := user
database := data
If you want to parse the ini file you could do that as such
CONFILE := db/database.ini
make-db: _setup_con
echo $(user) $(database)
# your target
_setup_con:
$(eval user=$(shell grep "user=" $(CONFILE) | grep -Eo "[^=]*$$"))
$(eval database=$(shell grep "database=" $(CONFILE) | grep -Eo "[^=]*$$"))
# and so forward
I would make it more Make-way by using feature of automatic Makefile generation. Given that a configuration file is a simple properties file, its syntax is easily parseable by Make, it's sufficient to just get the lines with variables, i.e.:
include database.mk
database.mk: db/database.ini
grep -E '^\w+=\w+$$' $< > $#
.PHONY: create-db
create-db: $(from_file)
psql -U $(user) -d $(database) -q -f $<
Some additional notes:
create-db should be made .PHONY to avoid situation when nothing is done due to somebody creating (accidentally or not) a file named create-db,
by making create-db depending on from_file one can get a clean and readable error from make that a file does not exist instead of possibly cryptic error later.

How to execute a make target on file change automatically?

How do I write a make target that will watch for any file changes in specific folders and execute some other make target to compile files? I am looking for a way that can do this with minimal dependency on tools in addition to make itself to keep things simple.
For the watching you can use fswatch. (There's also a go version of this program which may be easier to install: fswatch) For example:
fswatch -ext cpp,c,h make -f Makefile
Anytime you change a cpp, c or h file it will run make again.
Make can be a bit slow for this, so I tend to use ninja instead, but that really depends on the size of your project.
Another option is tup, which has watching built-in:
tup monitor
But, sadly, only for linux.
You can use entr and adjust your Makefile similar to this one
.DEFAULT_GOAL := run
SHELL := /bin/bash
run:
clear && \
cp one.txt two.txt && \
rm -f _* *.l2m *.o2m && \
Ganlib < testgan2.x2m
watch:
while sleep 1 ; do find . -name '*.x2m' -o -name '*.c2m' \
| entr -d make -f ./Makefile ; done
.PHONY: run watch
followed by
$ make watch

Setting a variable in make

This is my makefile
file1:
uglifyjs myfile1.js -c | gzip -c -9 > myfile1.min.js
file2:
uglifyjs myfile2.js -c | gzip -c -9 > myfile2.min.js
How can I change my makefile to remove duplicate code:
file1:
FILE=myfile1.js
#How to call build target?
file2:
FILE=myfile2.js
build:
uglifyjs $(FILE).js -c | gzip -c -9 > $(FILE).min.js
I know I can use make build but is there another way to do this without invoking make recursively?
Use automatic variables:
file1 file2:
uglifyjs my$#.js -c | gzip -c -9 > my$#1.min.js
I don't know why you're using targets like file1 when the file you're actually building is myfile1.min.js. That's not a good makefile.
But, that's not the question you asked.
Use a pattern rule to run the command, and then make your targets depend on the files you want:
file1: myfile1.min.js
file2: myfile2.min.js
%.min.js: %.js
uglifyjs $< -c | gzip -c -9 >$#
The pattern rule tells make how to build a .min.js file from a .js file, and the other rules tell it to build specific files.

*** missing separator. Stop. Make file

i'm getting this error:
make:24: *** missing separator. Stop.
Although i changed all space character with tab in line 24.
Line24:arm_v5t_le-gcc $FILES $INCLUDES $LIBS -o $TARGET
Here is the code:
#DM_serial2_make
export PATH="$PATH:/opt/mv_pro_5.0/montavista/pro/devkit/arm/v5t_le/bin:/opt/mv_pro_5.0/montavista/pro/bin:/opt/mv_pro_5.0/montavista/common/bin"
TARGET="/home/cilem/Desktop/06.05.2012/DM_serial2"
INCLUDES=" -I /home/cilem/Desktop/06.05.2012/libxml2 \
-I /home/cilem/Desktop/06.05.2012/gstreamer-0.10 \
-I /home/cilem/Desktop/06.05.2012/gstreamer-0.10/gst/interfaces \
-I /home/cilem/Desktop/06.05.2012/glib-2.0 \
-I /home/cilem/Desktop/06.05.2012/glib-2.0/include"
LIBS=" -L /home/cilem/Desktop/06.05.2012/lib/ -l:libgstreamer-0.10.so.0 \
-L /home/cilem/Desktop/06.05.2012/lib/ -l:libgstinterfaces-0.10.so.0 \
-L /home/cilem/Desktop/06.05.2012/lib/ -l:libgobject-2.0.so.0 \
-L /home/cilem/Desktop/06.05.2012/lib/ -l:libgmodule-2.0.so.0 \
-L /home/cilem/Desktop/06.05.2012/lib/ -l:libxml2.so.2 \
-L /home/cilem/Desktop/06.05.2012/lib/ -l:libgthread-2.0.so.0 \
-L /home/cilem/Desktop/06.05.2012/lib/ -l:libglib-2.0.so.0"
FILES="DM_serial2.c"
arm_v5t_le-gcc $FILES $INCLUDES $LIBS -o $TARGET
That looks like a shell script. Shell scripts are not makefiles, and vice versa. You need to find a good tutorial on make, or read the GNU make manual.
For example, you should not have any quoting in your variable values.
Second, variable expansions in make require the variables to be surrounded by parens or curly braces: $(FILES) or ${FILES}.
Third, as piokuc says, that line is not a valid make rule. A make rule has the form:
<target> : <dependencies...>
<commands...>
where the indentation of the commands... must be TAB characters. This rule says "you can build target whenever it's older than any of dependencies... by running commands...". The target and dependencies must (usually) be files, so you definitely don't want to use $(INCLUDES) or $(LIBS) in that list as those are compiler flags.
You probably want something like this, although it could be improved:
$(TARGET): $(FILES)
arm_v5t_le-gcc $(FILES) $(INCLUDES) $(LIBS) -o $(TARGET)
You've got other weird things here. You don't need to provide the same directory over and over with the -L flag. Once is enough. Also I'm not familiar with the -l:libfoo.a construct; usually it's just -lfoo.
I think the last line should be replaced with something like:
$TARGET: $FILES $INCLUDES $LIBS
arm_v5t_le-gcc $FILES $INCLUDES $LIBS -o $TARGET
The above line (the one starting with arm_v5t_le-gcc) should start with a tab, not spaces.

Resources