I am writing a makefile that will download and build all programs that I need on an HPC cluster. Before doing anything else I want it to run this:
module add openmpi/1.8.4-gcc
module add slurm
These 2 commands modify some environment variables like MPICC and PATH, so that I can compile mpi programs and schedule them for running. Right now I have them in my .bashrc. How do I move them from there to my makefile?
If you run them before running make, it will work. If you want those lines to appear in the Makefile, you can use .ONESHELL:
Otherwise, make will spawn one shell process per line, making the module load lines useless.
$ cat Makefile
CC=g++
SHELL=/bin/bash
.ONESHELL:
all:
module load R
R --version
While the module is not loaded
$ R
-bash: R: command not found
running make will succeed:
$ make
module load R
R --version
R version 3.2.1 (2015-06-18) -- "World-Famous Astronaut"
Copyright (C) 2015 The R Foundation for Statistical Computing
Platform: x86_64-unknown-linux-gnu (64-bit)
[...]
But this will need every rule to include the module load commands.
Related
I have some python code which works with a C library that I compile with a Makefile. So I create some install/uninstall/installed routines to load/unload/check my library. I add the python commands to it too so everything works.
Now the problem is, not all the machines have the same python version and python might not be set to the most up to date version. I would like to execute python-{x.y} -m pip {install -e . | uninstall {libname} | show {libname}}.
My current approach was:
create a configured.dat file with the following contents:
python3.4 python3.6 python3.8
And the following Makefile routine:
CONFIGURED_FILE=configured.dat
INSTALLED_PYTHON_VERSIONS :=$(file < $(CONFIGURED_FILE))
RED := \033[0;31m
GREEN := \033[1;32m
CYAN := \033[0;36m
NO_COLOR := \033[0m
ifneq ($(VERBOSE),)
VERBOSE := ''
else
VERBOSE := '-q'
endif
EDITABLE := '-e'
.PHONY: install
install:
# make C libs
$(foreach f,$(INSTALLED_PYTHON_VERSIONS), echo -e 'installing $(CYAN)$f$(NO_COLOR)'; $f -m pip install $(VERBOSE) $(EDITABLE) .;)
On my development machine this works. On my deployment machine it doesn't. The variable INSTALLED_PYTHON_VERSIONS is empty. How? Why? The only change I found between the environments that should impact it is the make version, 3.82 on prod, and 4.3 on dev. Prod is centos 7.9.2009 and dev is ubuntu 22.04.1.
The root problem is the wanting to invoke different python versions, the superficial problem is being unable to read the file contents into a list of variables. A good answer to either would be fine by me.
You found the problem yourself: on one machine you're using GNU make 3.82 and on the other 4.3. According to the GNU make NEWS file, the $(file ...) function was added in GNU make 4.0 (released in 2013, so 9 years ago).
You can just use cat:
INSTALLED_PYTHON_VERSIONS := $(shell cat $(CONFIGURED_FILE))
I should also point out that echo -e is not portable. There are no POSIX-specified options to echo and lots of different implementations accept, or not, different options. You probably want to switch to printf which is portable and well-defined.
Within a Makefile on linux, we can excute shell/bash commands with it to move directories or excute another files. However when porting the same Makefile over to macOS, all the commands are not readible (therefore path and execution are broken). Is there a universal command or workflow that can work on both?
Example of Makefile
.ONESHELL:
COMMAND ?= none
GIT_HASH ?= githash
alpine:
#cd images/alpine
#make ${COMMAND} GIT_HASH=${GIT_HASH} ALPINE_VERSION=3.6.5 TAG=3.6
so in a linux box both #CD and #MAKE are executed but not for macOS Catalina. I would like to make it universal so that both system will respect the appropriate command that follows.
Chances are that your MacOS box uses its default GNU make version (3.81). .ONESHELL was introduced with 3.82. Upgrade with Homebrew or MacPort. Anyway, better avoid make in recipes, prefer $(MAKE), and instead of cd; make you can use GNU make's -C option: $(MAKE) -C images/alpine ...
I'm trying to install a package called fminuit http://www.fis.unipr.it/~giuseppe.allodi/Fminuit/Fminuit_building.html
on ubuntu 18.04 machine using Octave. The installation step "make -f Makefile.f2c_lnx.Octave" gives me the following error
WrapIO_Matlab.c:4:10: fatal error: mex.h: No such file or directory
Any idea how to remedy this,
cheers, Damir
The build instructions provided by FMINUIT ask you to manually adapt the Makefile to your setup. I'm guessing you did one of those steps incorrectly. I'm running Octave 6.0.0 (current development sources) and worked fine:
$ wget http://www.fis.unipr.it/~giuseppe.allodi/Fminuit/fminuit-src.tar.gz
$ tar xzf fminuit-src.tar.gz
$ cd fminuit-2011.05.31/fminuit/
# modify Makefile.f2c_lnx.Octave
$ make -f Makefile.f2c_lnx.Octave
$ make -f Makefile.f2c_lnx.Octave install
The tricky part is knowing what to modify on the Makefile. For my case, these were the lines (you need to know the exact Octave version and where you installed it):
#Octave prefix directory (typically /usr or /usr/local): modify if needed
-PREFIX=/usr
+PREFIX=/usr/local
#major version number
-OCTAVE_MAJOR=2
+OCTAVE_MAJOR=6
#minor-release version number
-OCTAVE_MINOR=9.12
+OCTAVE_MINOR=0.0
OBJS= mnintr_wrkrnd.o intrac.o WrapIO_Matlab.o doflush.o
MINUIT=Minuit_.o
INSTDIR=../bin/linux_$(ARCH)/octave$(OCTAVE_MAJOR)
The fminuit Makefile will "install" inside the fminuit source directory. You may also want to adjust its INSTDIR value. You need to adjust your Octave path to use it:
>> addpath('/wherever/you/build/fmunuit/fminuit-2011.05.31/bin/linux_x86_64/octave6')
>> fminuit # you probably can figure out how to call this function
error: fminuit: Too few input arguments
I'm trying to run Python and Julia from the Cygwin command line, but I've installed Python and Julia to two separate directories. I can run Python from Cygwin with
$ python testfile.py
because I added export PATH=/cygdrive/c/anaconda2:$PATH to bash.rc so that I can run Python. However, my Julia installation sits in a different directory. Can I add something like export PATH2=/cygdrive/c/Julia-0.5.1/bin:$PATH2 to run Julia files from command line with
$ julia testfile.jl
or what should I do?
Why not add both paths in the bash.rc?
export PATH=/cygdrive/c/anaconda2:/cygdrive/c/Julia-0.5.1/bin:$PATH
I have a binary that runs under my default shell.
The binary runs perfectly o.k. with:
./binary input.dat
However, if I put this inside a make file:
SHELL=/bin/bash
runos:
./binary input.dat
The code crashes and leaves me quite helpless.
Here is what I tested so far, everything inside my Make file and in the shell:
ulimit -a: identical.
Set the shell to bash as seen above.
diff of the environment variables in SHELL and Make with:
env | sort > vars.1
inside make
env | sort > vars.2
Then run the binary with the extra variables in Make with the following command:
env SHLVL=2 MAKELEVEL=1 MAKEFLAGS= ./binary input.dat
strace in the shell and inside make:
strace -o debug binary input.dat
The code keeps on crashing in Make, and runs in the shell. I am already thinking to dump Make for my test cases and just write shell scripts. But I am curious to know what is the difference.
The Fortran code (a mix of F77, F90 and F95) was compiled with gfortran-4.4 and the following options:
FFLAGS= -g -fbacktrace
So, the concrete question is, what can I do to make this binary run under make in Debian!?
update:
I just tested again in a CentOS machine (v5.8), The code inside Makefile does not crash (GNU Make version 3.81).
I also tested on my Debian Wheezy and openSUSE 11.4, both with GNU Make version 3.82 - It crashes!
I tested on Debian Squeeze with GNU Make version 3.81, and it does crash. So, I think it is not dependent on the GNU Make version.
error when crashing:
enter timeloop
------------------------------------------------------------------------
timestep: 1 time: 2.500E-02 days delt: 2.500E-02 days
-------------------------------------------
terminated in routine react_snia
maximum number of iterations exceeded
bye now ...
-------------------------------------------
failure in timeloop
no further time step reduction possible
try reducing min. time step, bye now ...
trying to work around 'GNU Make' using 'waf'
It has been a while since I wanted to test waf, so here is another interesting observation:
I wrote a wscript which contains a function:
import os
def run(ctx):
os.system('./binary input.dat')
And waf run runs!
If I changed the run method to:
import subprocess as sp
def run(ctx):
sp.call('./binary input.dat', shell=True)
The binary also works as expected.
So, now I am thinking GNU Make forks a new sub-shell in a way that causes may binary to fail (although, under RHEL 5.8 Make did work).
solution: compile make from sources ...
Read to find out more.
OK, so after being pretty much desperate, I did what I simply should have done before blame make for all my troubles.
I thought the problem is Debian specific. But I am guessing the version in CentOS-5.8 is a patched version, although it says it's v.3.81.
So, for those who wonder my solution was:
wget http://ftp.gnu.org/gnu/make/make-3.82.tar.gz
tar xvzf make-3.82.tar.gz
cd make-3.82
./configure
./build.sh
# copy make to the directory with the binary and input and run the local make version
./make
# everything works as expected !!!
I thought let's narrow it down -
wget http://ftp.gnu.org/gnu/make/make-3.80.tar.gz
tar xvzf make-3.80.tar.gz
cd make-3.80
./configure
./build.sh
# copy make to the directory with the binary and input and run the local make version
./make
# everything works as expected !!!
Is it the version 3.81 ?
wget http://ftp.gnu.org/gnu/make/make-3.81.tar.gz
tar xvzf make-3.81.tar.gz
cd make-3.81
./configure
./build.sh
# copy make to the directory with the binary and input and run the local make version
./make
# FAIL! Like with the make version in Debian.
Hence, I think I bumped into some very weird bug in GNU Make v.3.81.