In plain language, what is a kernel and what it is used for?
I have seen code for parallelization like:
#!/bin/bash
#SBATCH --job-name=bt-mz
#SBATCH --output=bt-mz_%j.out
#SBATCH --error=bt-mz_%j.err
#SBTACH --nodes=1
#SBATCH --ntasks=12
#SBATCH --cpus-per-task=4
#SBATCH --qos=debug
#MPI + OpenMP
export NAS_PATH=$HOME/bin
export OMP_NUM_THREADS=4
srun $NAS_PATH/bt-mz.C.12 srun $NAS_PATH/sp-mz.C.12
Where sp-mz.C.12 and bt-mz.C.12 are called kernels (BT-MZ-MPI SP). What are they?
This is a benchmark including many components, some of them being related to solving a synthetic system of nonlinear PDEs (BT, SP, LU). Solving such PDEs involves so-called kernels, that are basically functions with specific properties. Such functions are also used in probability density estimation, signal analysis, etc.
Related
make -j$(nproc)
When compiling gcc from source, I saw there is a step of using -j to match the available cores. Must the argument be the same as core numbers? Does it do any harm if it is less than actual core numbers? For example, if I have 8 cores, but I use:
make -j 4
Any consequences?
Summary: What do the -f and -m in gcc and clang compiler options stand for?
Details:
When using clang I've noticed that many compiler options start with -f and others start with -m. I assume that there is some historical reason for this and I was curious so I looked at the gcc help and saw the following:
Options starting with -g, -f, -m, -O, -W, or --param are automatically
passed on to the various sub-processes invoked by gcc. In order to
pass other options on to these processes the -W options must
be used.
If I had to guess I think that -f might stand for frontend and -m for machine. But I'd be interested to hear a more comprehensive answer, possibly including the other sub-processes that gcc invokes.
I don't have specific sources that state what 'f' and 'm' mean, but we can infer based on usage patterns found in documentation.
'f' stands for 'flag'.
Flags are on if specified via '-fFLAG' and off via '-fno-FLAG'
ex:
-fpic # flag to set position independent code
-fno-builtin # don't recognize build in functions ...
The technical definition is that 'f' defines "Control the interface conventions used in code generation".
Src:
https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html
(e.g -fpci, when this flag is set)
'm' stands for mode. One general characteristic is that it sometimes has parameters. e.g
-mabi=name #abi mode = name
-mcpu=cpu
Src:
https://gcc.gnu.org/onlinedocs/gccint/Standard-Names.html (e.g ... when this mode...)
According to gcc onlinedocs, options of the form -ffoo and -fno-foo stand for machine independent code generation conventions.
Examples: fpic, -fno-pic
-m options stand for machine dependent options.
eg: -mcpu, -march, -matomic
https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html#Code-Gen-Options or https://home.cs.colorado.edu/~main/cs1300/doc/gnu/gcc_2.html#SEC43
https://gcc.gnu.org/onlinedocs/gcc/Submodel-Options.html#Submodel-Options or https://home.cs.colorado.edu/~main/cs1300/doc/gnu/gcc_2.html#SEC16
I want to set some variables based on whether or not parallel builds are enabled, so I tried this:
jobs:
»·echo "executing jobs job"
ifneq (,$(findstring -j,$(MAKEFLAGS)))
»·$(warning "parallel!")
else
»·$(warning "not parallel!")
endif
And this is what happens:
$ make -j2
Makefile:2: "not parallel!"
echo "executing jobs job"
executing jobs job
I also tried testing $(JOBS), but no luck.
Is there a way for me to tell inside a Makefile that the --jobs parameter was used?
Additional info:
$ make --version
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program built for x86_64-pc-linux-gnu
Surprisingly, ${MAKEFLAGS} will only gain the -j when it is expanded at recipe expansion time.
Makefile:
$(warning [${MAKEFLAGS}])
.PHONY: all
all:
$(warning [${MAKEFLAGS}])
echo Now do something useful
Run:
$ make -j5
1:1: []
1:5: [ -j --jobserver-fds=3,4]
echo Now do something useful
Now do something useful
About the MAKEFLAGS expansion in #bobbogo's answer: If we look at the code I think I can explain the behavior:
Looking at the code, main function of make calls the define_makeflags function several times.
/* Define the MAKEFLAGS and MFLAGS variables to reflect the settings of the
command switches. Include options with args if ALL is nonzero.
Don't include options with the 'no_makefile' flag set if MAKEFILE. */
static struct variable *
define_makeflags (int all, int makefile)
{
......
Call locations in main:
1)
/* Set up the MAKEFLAGS and MFLAGS variables for makefiles to see.
Initialize it to be exported but allow the makefile to reset it. */
define_makeflags (0, 0)->export = v_export;
2)
/* Set up MAKEFLAGS and MFLAGS again, so they will be right. */
define_makeflags (1, 0);
3)
/* Set up 'MAKEFLAGS' specially while remaking makefiles. */
define_makeflags (1, 1);
There are other calls in sub-functions but this should be enough to explain.
The first call sets all parameter to false. The others set to true. With all set to false, the define_makeflags function only parses "simple flags" and j is not one of them. In order to understand the parsing one needs to look into this switch statement and the definition of command line params.
My SWAG is like the following:
I presume the parsing of ifneq statements happen after the first call to define_makeflags but before the subsequent calls. I can guess the reason of keeping the MAKEFLAGS simple at the start is to continue to support documented Makefile patterns like the following.
From doc1, doc2:
archive.a: ...
ifneq (,$(findstring t,$(MAKEFLAGS)))
+touch archive.a
+ranlib -t archive.a
else
ranlib archive.a
endif
If MAKEFLAGS contained long options or options that take parameters, then searching for single char flags in MAKEFLAGS would not be possible.
There is some guesstimate in my answer. Maybe someone who was involved in the design decision can also weigh in. Given this change Paul Smith may have an idea.
Summary: What do the -f and -m in gcc and clang compiler options stand for?
Details:
When using clang I've noticed that many compiler options start with -f and others start with -m. I assume that there is some historical reason for this and I was curious so I looked at the gcc help and saw the following:
Options starting with -g, -f, -m, -O, -W, or --param are automatically
passed on to the various sub-processes invoked by gcc. In order to
pass other options on to these processes the -W options must
be used.
If I had to guess I think that -f might stand for frontend and -m for machine. But I'd be interested to hear a more comprehensive answer, possibly including the other sub-processes that gcc invokes.
I don't have specific sources that state what 'f' and 'm' mean, but we can infer based on usage patterns found in documentation.
'f' stands for 'flag'.
Flags are on if specified via '-fFLAG' and off via '-fno-FLAG'
ex:
-fpic # flag to set position independent code
-fno-builtin # don't recognize build in functions ...
The technical definition is that 'f' defines "Control the interface conventions used in code generation".
Src:
https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html
(e.g -fpci, when this flag is set)
'm' stands for mode. One general characteristic is that it sometimes has parameters. e.g
-mabi=name #abi mode = name
-mcpu=cpu
Src:
https://gcc.gnu.org/onlinedocs/gccint/Standard-Names.html (e.g ... when this mode...)
According to gcc onlinedocs, options of the form -ffoo and -fno-foo stand for machine independent code generation conventions.
Examples: fpic, -fno-pic
-m options stand for machine dependent options.
eg: -mcpu, -march, -matomic
https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html#Code-Gen-Options or https://home.cs.colorado.edu/~main/cs1300/doc/gnu/gcc_2.html#SEC43
https://gcc.gnu.org/onlinedocs/gcc/Submodel-Options.html#Submodel-Options or https://home.cs.colorado.edu/~main/cs1300/doc/gnu/gcc_2.html#SEC16
Can I measure the code size with the help of an fseek() function and store it to a shell variable?
Is it possible to extract the code size, compilation time and execution time using milepost gcc or a GNU Profiler tool? If yes, how to store them into shell variables?
Since my aim is to find the best set of optimization technique upon the basis of the compilation time, execution time and code size, I will be expecting some function that can return these parameters.
MyPgm=/root/Project/Programs/test.c
gcc -Wall -o1 -fauto-inc-dec $MyPgm -o output
time -f "%e" -o Output.log ./output
while read line;
do
echo -e "$line";
Val=$line
done<Output.log
This will store the execution time to the variable Val. Similarly, I want to get the values of code size as well as compilation time.
I will prefer something that I can do to accomplish this, without using an external program!
for code size on linux, you can use size command on terminal.
$size file-name.out
it will give size of different sections. use text section for code size. you can use data and bss if you want to consider global data size as well.
You can use the size(1) command http://www.linuxmanpages.com/man1/size.1.php
Or open the ELF file, walk over section headers and sum the sizes of all the section with type SHT_PROGBITS and the SHF_EXECINSTR flag set.
On non-Linux / non-GNU-utils systems (where you may have neither GNU size nor readelf), the nm program can be used to dump symbol information (including sizes) from object files (libraries / executables). The syntax is slightly system-dependent:
OpenGroup manpage for nm (the "portable subset")
Linux/BSD manpage for nm (GNU version)
Solaris manpage for nm
AIX manpage for nm
nm usage on HP/UX (this says "PA-RISC" but the utility is present / usable on Itanium)
Windows: Doesn't have nm as such, but see: Microsoft equivalent of the nm command
Unfortunately, while the utility is available almost everywhere, its output format is not as portable as could be, so some system-specific scripting is necessary.