Bazel says that best practice is to not use binary dependencies, but to build the dependency from source using Bazel. I have a dependency - xgboost - that builds using Makefiles, and I was wondering what the canonical strategy is to convert Makefiles to Bazel - as well as if there are any standard tools that programmers use as part of the conversion.
There is no "canonical way" to do it right now. The easiest way is probably to just shell out to make in a non-hermetic generule and declare the needed output of the rule, e.g.:
genrule(
name = "xgboost",
srcs = glob(["xgboost/**"]),
cmd = "\n".join([
"DIR=$$(mktemp -d $${TMPDIR-/tmp}/tmp.XXXXXXX)",
"(cd $(execution_root)/xgboost && cp -fr . \"$${DIR}\")",
"(cd \"$${DIR}\" && make target)",
"cp $${DIR}/output $#",
]),
outs = ["xgboost-output"],
local = 1,
)
Of course, target, output and xgboost-output are to be replaced by what works for you.
Related
I am dealing with autotools and here is the situation:
By default libdir is set to '${exec_prefix}/lib' with exec_prefix set to '${prefix}' and prefix set to '/usr/local'.
Key here recursion and at first level libdir variable contain another variable and a following path.
So how to convert ${libdir} into a variable containing '/usr/local/lib' ?
Info : all 3 (libdir, exec_prefix, prefix) can change according configuration.
Consider the following files:
configure.ac
AC_PREREQ([2.59])
AC_INIT([test], [0.0.0] )
AM_INIT_AUTOMAKE()
AC_CONFIG_SRCDIR([test.cpp])
AC_LANG([C++])
AC_PROG_CXXCPP
AC_PROG_CXX
AC_CONFIG_FILES([Makefile path.conf])
AC_MSG_NOTICE([">>> Before ac_output prefix=${prefix}"])
AC_OUTPUT
AC_MSG_NOTICE([">>> after ac_output prefix=${prefix}"])
Makefile.am
bin_PROGRAMS = test
test_SOURCES = test.cpp
test.cpp
int main()
{}
path.conf.in
#libdir#
Then after invoking :
aclocal && autoconf && automake -a --foreign && ./configure
configure log show:
configure: ">>> Before ac_output prefix=NONE"
...
...
...
configure: ">>> after ac_output prefix=/usr/local"
And generated file path.conf contains
${exec_prefix}/lib
The goal is to have a variable containing the expanded version of the path to be used in a path.conf.in so autotools generate path.conf with that expanded path.
Edit: Bash only solution
Digging related topics and helped by #Aserre answer, I manage to do the following with a regex.
while expr match "${libdir}" '^.*\$.*$' 1>/dev/null;
do
echo ">${libdir}"
libdir="$(eval echo ${libdir})"
done
Which means : While $libdir contain one $ expand with eval.
But does not work in configure.ac script before AC_OUTPUT
The goal is to have a variable containing the expanded version of the path to be used in a path.conf.in so autotools generate path.conf with that expanded path.
The Autotools provide special handling for default values of the installation-directory variables, in order to enable the user to specify or override the installation prefix at make install time:
make install prefix=/my/special/prefix
What you propose to do will break that. If a user specifies a different installation prefix at the installation stage than they tell configure (or that they let configure choose by default) then you will end up with a broken installation.
The best way to do address problems like this is to build the configuration file under make's control, at make install time, instead of doing it at configuration time. If the project uses Automake, then that might mean something like this:
install-data-local:
$(SED) -e 's,[#]libdir[#],$(libdir),' path.conf.in > $(sysconfdir)/path.conf
chmod 0644 $(sysconfdir)/path.conf
chown root:root $(sysconfdir)/path.conf
uninstall-local:
rm $(sysconfdir)/path.conf
You can of course substitute more output variables than that if you like. It's pretty close to what configure does itself.
And of course, if you do it this way then you do not need to worry about performing extra expansions.
If you are 100% sure of the content of the ${exec_prefix} variable, you could use the following line to achieve what you want :
libdir="$(eval echo ${exec_prefix})"
Note that in a lot of cases, the use of eval is discouraged. Here, if the user has overriden the content of the variable exec_prefix, for instance with exec_prefix='a; rm -rf /', all the code written will be executed.
If you are in total control of your environment (i.e. you are certain of the value of the variables when you launch the script), there should be no problem, otherwise be wary of the potential side effects
Say there are a bunch of (e.g. nearly 200) modules that all depend on a core module. All are using Autotools.
The core module installs a core.m4 file which the dependents already use for various things.
All the dependents also have some lines in their install-data-local rule to generate some scripts and install them, e.g.
core_stuffdir=$(prefix)/share/core/stuff/
install-data-local:
generate-stuff stuff.xml
test -d $(DESTDIR)$(core_stuffdir) || mkdir $(DESTDIR)$(core_stuffdir)
stuff=`xmllint --xpath '//stuff[#install="yes"]/#name' stuff.xml`; \
$(INSTALL_DATA) $$stuff $(DESTDIR)$(core_stuffdir); \
rm $$stuff
…
$(INSTALL_DATA) other-module-specific-stuff …
…
I would like to remove those five lines which are currently redundantly duplicated over ~200 files, and instead define the lines in the core.m4. The dependents should be able to say something like SOMEVAR: somevalue (or something similarly short, at worst a one-line thing in install-data-local) and have those lines executed during make install.
Is there a nice way to do define these lines in core.m4 and make them available to Makefile.am? I can't find any similar examples on the net.
The only solution I can think of now is that the m4 spits out a shell script that I can call from install-data-local, but I'm not sure that's the best (or most autotoolish) way.
Alternatively, is there a simple way to distribute an automake fragment.am file from my core module? (This seems to be the way e.g. Python compilation rules are defined.)
After some digging, I found
http://thread.gmane.org/gmane.comp.sysutils.automake.general/5704/focus=5708
which has a solution to a similar problem (I couldn't get the
AC_CONFIG_FILES solution mentioned there to work).
So core.m4 now defines CORE_MKINCLUDE:
AC_DEFUN([CORE_MKINCLUDE],
[
AC_SUBST_FILE(core_include)
core_include=$srcdir/core_include.am
cat >$srcdir/core_include.am <<EOF
core_stuffdir=\$(prefix)/share/core/stuff/
install-stuff:
generate-stuff stuff.xml
test -d \$(DESTDIR)\$(core_stuffdir) || mkdir \$(DESTDIR)\$(core_stuffdir)
stuff=\`xmllint --xpath '//stuff#<:##install="yes"#:>#/#name' stuff.xml`; \\
\$(INSTALL_DATA) \$\$stuff \$(DESTDIR)\$(core_stuffdir); \\
rm \$\$stuff
EOF
])
Ie. the goal is now printed to the file core_include.am, and has its
own name. Each module's configure.ac calls CORE_MKINCLUDE, and each
Makefile.am just has
#core_include#
install-data-local: install-stuff
An improvement at least.
On an IBM i system, using PASE (AIX emulator), i try to compile RPG sources using a makefile.
I have RPG sources and try to build a PGM program.
Every single source will be compile in a distinct PGM.
Here is the syntax i tried first
VPATH=.
SYSTEM=system -iv
CRTRPGMOD=CRTRPGMOD
BIN_LIB=MR_CH
CURRENT_PATH=/currentPath #${PWD} doesn't work
#With this line active, it works
SRC_RPGLE = src1.rpgle src2.rpgle
#With this line active, it doesn't work
#SRC_RPGLE = $(shell echo *.rpgle) #Should list every sources i want to compile
TARGETS = $(SRC_RPGLE:.rpgle=.rpgleMod) #Should list every program i want to build
.SUFFIXES: .rpgle .rpgleMod
.rpgle.rpgleMod:
$(SYSTEM) "$(CRTRPGMOD) MODULE($(BIN_LIB)/$(*F)) SRCSTMF('$(CURRENT_PATH)/$<')" > $(*F)_1_crtrpgmod.log
ln -fs $(*F).rpgMod
all: $(TARGETS)
I tried to apply GNU shell syntax using AIX make command
Any suggestions ?
I'm not familiar with the AIX implementation of make but assuming that the linked man page is all there is to it, then it looks like a bare-bones implementation of POSIX make (for an older POSIX spec).
Therefore, the only way to do what you want (expand a list of files) is to use recursion, so that you get access to the shell, like this:
SYSTEM=system -iv
CRTRPGMOD=CRTRPGMOD
BIN_LIB=MR_CH
CURRENT_PATH=/currentPath
CURRENT_PATH=/home/CHARLES/Projets/MRSRC/tmp
recurse:
$(MAKE) all SRC_RPGLE="`echo *.rpgle`"
TARGETS = $(SRC_RPGLE:.rpgle=.rpgleMod)
.SUFFIXES: .rpgle .rpgleMod
.rpgle.rpgleMod:
$(SYSTEM) "$(CRTRPGMOD) MODULE($(BIN_LIB)/$(*F)) SRCSTMF('$(CURRENT_PATH)/$<')" > $(*F)_1_crtrpgmod.log
ln -fs $(*F).rpgMod
all: $(TARGETS)
The recurse rule MUST be the first target in the makefile. Also this won't help if you want to run other targets like make foobar; it will only help you run all properly.
Alternatively you can obtain GNU make and build it for your system, and use that. In the end that might be a more straightforward approach.
I'm wondering if there is a way to remove dependency checks from a configure script when make dist is run. The purpose is that the version of the package in the repository uses emacs to execute lisp files and generate .c files, whereas those generated source files are included in the tarball when make dist is run. Automake supports including built files in the output tarball simply by setting EXTRA_DIST in the Makefile. I would like to be able to check for emacs only when someone tries to compile from the repository, but not when someone compiles from the tarball.
More concretely, in configure.ac I have:
AM_PATH_LISPDIR
AS_IF([test "$EMACS" = no], [AC_MSG_ERROR([cannot find emacs])])
but I would like that check not to occur after make dist is run. Any way to do this? If not, how is this case normally handled?
The best method I've come up with is to check for the presence of the Bootstrap file. Bootstrap is removed from the tarball after make dist is run.
AC_CHECK_FILE([Bootstrap])
if test "$ac_cv_file_Bootstrap" == yes ; then
<check for emacs>
fi
I'd normally set a conditional if I find the tool in configure.ac:
AM_PATH_LISPDIR
AM_CONDITIONAL([HAVE_EMACS_IN_REPOSITORY], [test "x$EMACS" != xno -a -d ".git"])
AS_IF([test "$EMACS" = no -a -d ".git"], [AC_MSG_ERROR([cannot find emacs])])
Then in Makefile.am
if HAVE_EMACS_IN_REPOSITORY
Execute lisp files...
Generate .c files...
endif
Usually configure ends up in the same location as .git. If not, the path can be adjusted.
Is it possible to tell make to only build a target if it's sources md5 checksum has changed (instead of the edit time)?
I'm using make to compile my .tex files and I need to prevent it from building everything twice all the time.
I've tried using Scons, but I feel this isn't adaptable to other needs I have.
No, this is not supported by Make — as you've found out, support for this feature is one of the reasons why tools like Scons exist.
I found a manual recipe for GNU make, though. Maybe you can use that as a work around.
I found 3 ways: From simply
Using temporary md5-files over an
elegant but unreadable hack to
patching GNU Make yourself
(from patchy but simple to clean but hard to realize)
Maybe my scons latex and org-mode recipe helps you:
## I need a couple of special builders for my projects
# the $SOURCE replacement only uses the first source file. $SOURCES gives all.
# specifying all source files makes it possible to rerun the build if a single source file changed.
orgexportpdf = 'emacs --batch --visit "$SOURCE" --funcall org-export-as-pdf'
pyxplot = 'pyxplot $SOURCE'
# pdflatex is quite dirty. I directly clean up after it with rm.
pdflatex = 'pdflatex $SOURCE -o $TARGET; rm -f *_flymake* flymake* *.log *.out *.toc *.aux *.snm *.nav *.vrb'
# build the PhD thesis from emacs org-mode.
Command("doktorarbeit.pdf", "doktorarbeit.org",
orgexportpdf)
# create plots
Command(["images/comp-t3-s07-tem-boas.png",
"images/comp-t3-s07-tem-bona.png"],
["nee-comp.pyx",
"nee-comp.txt"],
pyxplot)
# build my sink.pdf
Command("sink.pdf",
["sink.tex",
"images/comp-t3-s07-tem-boas.png",
"images/comp-t3-s07-tem-bona.png",
"images/bona-marble.png",
"images/boas-marble.png"],
pdflatex)
# My editors leave tempfiles around. I want them gone after a build clean. This is not yet supported!
tempfiles = Glob('*~') + Glob('#*#') + Glob('*.bak')
# using this here would run the cleaning on every run.
#Command("clean", [], Delete(tempfiles))
It is the counterpart for my Makefile:
all: doktorarbeit.pdf sink.pdf
sink.pdf : sink.tex images/comp-t3-s07-tem-boas.png images/comp-t3-s07-tem-bona.png images/bona-marble.png images/boas-marble.png
pdflatex sink.tex
rm -f *_flymake* flymake* *.log *.out *.toc *.aux *.snm *.nav *.vrb # kill litter
comp-t3-s07-tem-boas.png comp-t3-s07-tem-bona.png : nee-comp.pyx nee-comp.txt
pyxplot nee-comp.pyx
# http://www.reddit.com/r/emacs/comments/dy9yt/anyone_know_of_a_script_that_i_can_feed_an/
# http://blog.nguyenvq.com/2010/10/30/bash-batch-script-to-convert-org-mode-file-to-html/comment-page-1/#comment-27013
doktorarbeit.pdf : doktorarbeit.org
emacs --batch --visit "doktorarbeit.org" --funcall org-export-as-pdf
# this is not what clean is intended to do, but I do it anyway.
clean :
rm -f \#* *~ *.bak # kill editor backups
# alternatively run scons. Should I switch to SCons, just put this in all :)
sconsrun : scons
python scons/bootstrap.py -Q
scons :
hg clone https://bitbucket.org/ArneBab/scons