Include directory off-by one in libxml2 for nix - gcc

I'm bundling a .cpp program that depends on libxml, but g++ cannot find it. I've done the appropriate steps to make libxml available, however the directory structure is slightly different than the source expects. I want to know about nix-based solutions to this problem.
The compiler (from a nix-shell session) says that libxml is not in my system path:
fatal error: libxml/encoding.h: No such file or directory
#include <libxml/encoding.h>
However, it should be. In my .nix file I made sure to included libxml2
buildInputs = [ ... libxml2 ...];
and my nix-shell environment confirms that the library is present:
NIX_CFLAGS_COMPILE= [...] -isystem /nix/store/zf1nyqyx2zd6y944ln2rxnhd5m4265n4-libxml2-2.9.9-dev/include [...]
If I look in that directory, I find the search path is off. The path to the file is actually (relative to NIX_CFLAGS_COMPILE path):
libxml2/libxml/encoding.h
I found that I can compensate for this off-by-one path error by adding the following option to my compile command:
-isystem /nix/store/zf1nyqyx2zd6y944ln2rxnhd5m4265n4-libxml2-2.9.9-dev/include/libxml2
(The same path as the NIX_CFLAGS_COMPILE, but starting at the libxml2 directory.)
Given that information, I am wondering about a nix-based solution to this problem. The source is not mine, so changing them is the last thing I want to do. I see two other options.
First, I could add the path in the makefile that is triggered during buildPhase. However, I'm not exactly sure how to simply grab that path. For instance, inheriting libxml2 in my derivation makes the libxm2-2.9.9-bin directory available, when I need the libxml2-2.9.9-dev. I suppose I could grep or sed on the NIX_CFLAGS_COMPILE path, but I'd rather not.
My preferred solution would be to make a simple libxml derivation based off the libxml2 and just add that to my buildInputs, but that seems like it might not be simple either.

Try running pkg-config --cflags libxml-2.0 to get the appropriate compiler options.
If everything is configured correctly, it will find the appropriate /nix/store/*-libxml2-2.9.9-dev/lib/pkgconfig/libxml-2.0.pc file on your computer, and extract the needed compiler options from that.
You could invoke GCC using a one-liner like this:
g++ program.cpp $(pkg-config --libs --cflags libxml-2.0) -o program
This is a general solution that doesn't just apply to Nix, but you should be able to get it to work in Nix by ensuring you have the pkg-config utility on your PATH and ensuring the proper environment variables are set so that it can find libxml2. Nix probably has a bunch of shell scripts or something to help you set up your environment in that way, because this is a common need.

I solved this by defining a libxml attribute in my derivation and adding that to my buildInputs. The new library is just a link to the old, but with path corrected for. Here's the relevant parts of the derivation:
{ stdenv, libxml2, ... } : stdenv.mkDerivation
rec
{ buildInputs = [ libxml ];
libxml = stdenv.mkDerivation {
name = "libxml" ;
system = builtins.currentSystem;
outputs = [ "bin" "dev" "out" ];
phases = ["buildPhase"];
buildPhase =
''echo "my command out = $out"
echo ${libxml2.dev}
mkdir -p $out
mkdir -p $dev
mkdir -p $bin
ln -s ${libxml2.dev}/include/libxml2 $dev/include
ln -s ${libxml2.dev}/lib $dev/lib
''; };
}

Related

How to recursively substitute variable for a path in bash variable in configure.ac?

I am dealing with autotools and here is the situation:
By default libdir is set to '${exec_prefix}/lib' with exec_prefix set to '${prefix}' and prefix set to '/usr/local'.
Key here recursion and at first level libdir variable contain another variable and a following path.
So how to convert ${libdir} into a variable containing '/usr/local/lib' ?
Info : all 3 (libdir, exec_prefix, prefix) can change according configuration.
Consider the following files:
configure.ac
AC_PREREQ([2.59])
AC_INIT([test], [0.0.0] )
AM_INIT_AUTOMAKE()
AC_CONFIG_SRCDIR([test.cpp])
AC_LANG([C++])
AC_PROG_CXXCPP
AC_PROG_CXX
AC_CONFIG_FILES([Makefile path.conf])
AC_MSG_NOTICE([">>> Before ac_output prefix=${prefix}"])
AC_OUTPUT
AC_MSG_NOTICE([">>> after ac_output prefix=${prefix}"])
Makefile.am
bin_PROGRAMS = test
test_SOURCES = test.cpp
test.cpp
int main()
{}
path.conf.in
#libdir#
Then after invoking :
aclocal && autoconf && automake -a --foreign && ./configure
configure log show:
configure: ">>> Before ac_output prefix=NONE"
...
...
...
configure: ">>> after ac_output prefix=/usr/local"
And generated file path.conf contains
${exec_prefix}/lib
The goal is to have a variable containing the expanded version of the path to be used in a path.conf.in so autotools generate path.conf with that expanded path.
Edit: Bash only solution
Digging related topics and helped by #Aserre answer, I manage to do the following with a regex.
while expr match "${libdir}" '^.*\$.*$' 1>/dev/null;
do
echo ">${libdir}"
libdir="$(eval echo ${libdir})"
done
Which means : While $libdir contain one $ expand with eval.
But does not work in configure.ac script before AC_OUTPUT
The goal is to have a variable containing the expanded version of the path to be used in a path.conf.in so autotools generate path.conf with that expanded path.
The Autotools provide special handling for default values of the installation-directory variables, in order to enable the user to specify or override the installation prefix at make install time:
make install prefix=/my/special/prefix
What you propose to do will break that. If a user specifies a different installation prefix at the installation stage than they tell configure (or that they let configure choose by default) then you will end up with a broken installation.
The best way to do address problems like this is to build the configuration file under make's control, at make install time, instead of doing it at configuration time. If the project uses Automake, then that might mean something like this:
install-data-local:
$(SED) -e 's,[#]libdir[#],$(libdir),' path.conf.in > $(sysconfdir)/path.conf
chmod 0644 $(sysconfdir)/path.conf
chown root:root $(sysconfdir)/path.conf
uninstall-local:
rm $(sysconfdir)/path.conf
You can of course substitute more output variables than that if you like. It's pretty close to what configure does itself.
And of course, if you do it this way then you do not need to worry about performing extra expansions.
If you are 100% sure of the content of the ${exec_prefix} variable, you could use the following line to achieve what you want :
libdir="$(eval echo ${exec_prefix})"
Note that in a lot of cases, the use of eval is discouraged. Here, if the user has overriden the content of the variable exec_prefix, for instance with exec_prefix='a; rm -rf /', all the code written will be executed.
If you are in total control of your environment (i.e. you are certain of the value of the variables when you launch the script), there should be no problem, otherwise be wary of the potential side effects

Automake: Remove configure script checks with make dist

I'm wondering if there is a way to remove dependency checks from a configure script when make dist is run. The purpose is that the version of the package in the repository uses emacs to execute lisp files and generate .c files, whereas those generated source files are included in the tarball when make dist is run. Automake supports including built files in the output tarball simply by setting EXTRA_DIST in the Makefile. I would like to be able to check for emacs only when someone tries to compile from the repository, but not when someone compiles from the tarball.
More concretely, in configure.ac I have:
AM_PATH_LISPDIR
AS_IF([test "$EMACS" = no], [AC_MSG_ERROR([cannot find emacs])])
but I would like that check not to occur after make dist is run. Any way to do this? If not, how is this case normally handled?
The best method I've come up with is to check for the presence of the Bootstrap file. Bootstrap is removed from the tarball after make dist is run.
AC_CHECK_FILE([Bootstrap])
if test "$ac_cv_file_Bootstrap" == yes ; then
<check for emacs>
fi
I'd normally set a conditional if I find the tool in configure.ac:
AM_PATH_LISPDIR
AM_CONDITIONAL([HAVE_EMACS_IN_REPOSITORY], [test "x$EMACS" != xno -a -d ".git"])
AS_IF([test "$EMACS" = no -a -d ".git"], [AC_MSG_ERROR([cannot find emacs])])
Then in Makefile.am
if HAVE_EMACS_IN_REPOSITORY
Execute lisp files...
Generate .c files...
endif
Usually configure ends up in the same location as .git. If not, the path can be adjusted.

How to customize the meaning of the "-l" flag for GCC?

I have problems with GCC and I would like to use the -l flags in a customized way.
I would like to specify the search path for the correspondant libfoo specified by -lfoo , I also would like to override any internal search path in GCC, i don't want GCC to use any random lib that can solve the symbols, I only want GCC to compile with a really specific lib when -l is specified.
I know that there are utils such as pkg-config but my problem is more gcc-centric because i'm focusing on having more control on the compilation steps.
There is an undocumented syntax for specifying an absolute lib path to gcc:
$ gcc -o test test.c -l:/usr/lib/libfoo.so #(note the colon)
See here: https://code.ros.org/lurker/message/20130119.001059.fad11362.de.html
A more standard way to do this would simply be:
$ gcc -o test test.c /usr/lib/libfoo.so
Really, the only reason to use the -l: syntax is if you have a conflicting library of the same name in your search path and you can't change the search path.

check header file of iconv.h

I am using FreeBSD and have installed iconv package form port collection.I wrote
AC_CHECK_HEADER([iconv.h], ,[AC_MSG_ERROR([can not find iconv.h])])
AC_CHECK_LIB([iconv], [iconv_open], ,[AC_MSG_ERROR([can not find iconv_open])])
in the configure.ac.But when I run ./configure, it gave me the following message
checking iconv.h usability... no
checking iconv.h presence... no
checking for iconv.h... no
configure: error: can not find iconv.h
I am sure there is iconv.h, libiconv.la, libiconv.so in the directory of /usr/local/include and /usr/local/lib.So how should I write the correct statement to check the header file and library.Thanks advance!!
You should not modify your configure.ac in any way. The user is responsible for telling the tool chain where to look for libraries. For the user (which is who you are when you run configure), an easy thing to do is to set up a config.site so that the appropriate flags are set. For example, in your .bashrc: export CONFIG_SITE=$HOME/CONFIG_SITE, and then in $HOME/CONFIG_SITE, something like:
test "$prefix" = NONE && pfx=/usr/local || pfx=$prefix
: ${CPPFLAGS=-I$pfx/include}
: ${LDFLAGS=-L$pfx/lib}
This will insert appropriate flags to the compiler and linker to always look in some common locations.
Most likely the paths /usr/local/include and /usr/local/lib are not searched by default by the compiler. You can do it be modifying the CFLAGS and LDFLAGS variables in your configure.ac script:
CFLAGS="$CFLAGS -I/usr/local/include"
LDFLAGS="$LDFLAGS -L/usr/local/lib"
Add those two lines before you check for the header and library.
Possibly you might have to set CPPFLAGS (for the pre-processor) and CXXFLAGS (if you are building C++ code) as well, to add the include path.
Edit: To set these flags in configure.ac is, as noted by William Pursell, not really recommended. Instead you set the these as environment variables when invoking the generated configure script:
$ CPPFLAGS=-I/usr/local/include LDFLAGS=-L/usr/local/lib ./configure
There's always the AM_ICONV automake macro (iconv.m4) which is part of GNU gettext. You need not use gettext.m4.

How to compile OpenSSL with relative rpath

I have been trying to compile openssl 1.0.0g with the following rpath:
$ORIGIN/../lib64
Everytime I readelf -d apps/openssl, I am getting results like the following depending on what escaping variation I tried:
\RIGIN/../lib64
RIGIN/../lib64
ORIGIN/../lib64
I want to setup my rpath without using external tools like chrpath. Is it at all possible? I will basically accept anything that does not involve using external tools like chrpath (though I would already be done with that).
Ideally, I would like to do it by passing options on the command line (any form of -Wl,-rpath,$ORIGIN/../lib64).
I don't mind editing the generated Makefile, which is what I have been trying last. If only I could get it to print a stupid dollar sign!!! I tried modifying LIBRPATH under the BUILDENV= block with no luck. My best results so far:
LIBRPATH=$$'ORIGIN/../lib64 # result: /../lib64
LIBRPATH=$$$$'ORIGIN/../lib64 # result: 12345<pid>/../lib64
I have read various rpath related questions and tried various escaping and quoting tricks but nothing worked so far!
In your makefile try:
-Wl,-rpath,${ORIGIN}/../lib64
I am assuming that the ORIGIN is a shell variable.
EDIT
I have just found an answer to your question (better late then never):
You need to prevent make from interpolating variables, to do that you need to use $$ (double dolar sign):
-Wl,-rpath,'$$ORIGIN/../lib64'
I know that it works because I have tested it with my own application, enjoy :)
I went the chrpath way.
http://enchildfone.wordpress.com/2010/03/23/a-description-of-rpath-origin-ld_library_path-and-portable-linux-binaries/
It is quite complicated to counter shell expansion of `$$ORIGIN`` in openssl. Sooner or later, it gets expanded because of the dollar sign. If you really want to go this way, you can do it. I have found the following to work with openssl 1.0.1g on Linux. In Makefile.shared, look for this line:
DO_GNU_APP=LDFLAGS="$(CFLAGS) -Wl,-rpath,$(LIBRPATH)"
Replace it with the following. This quoting-fu neutralize the expansion of $. The double $$ is the way to get a single dollar sign in makefiles.
DO_GNU_APP=LDFLAGS="$(CFLAGS) -Wl,-rpath,'"'$$'"ORIGIN/../lib64'"
After compiling:
readelf -d apps/openssl | grep RPATH
0x000000000000000f (RPATH) Library rpath: ['$ORIGIN/../lib64']
OK I spent several hours fighting with this same issue and trying all manner of crazy escaping, at one point I was up to eight $ signs, at which point I decided that there must be another way.
In fact, it appears that there is, at least with GNU ld.
Instead of -Wl,-rpath,\\$$$\$\$\$$$\$\\\\$ or some other elder god invoking monstrosity, just do this:
echo '-rpath=$ORIGIN/../lib64' > rpathorigin
./config -Wl,#$(pwd)/rpathorigin ...
I don't see that ld.gold documents the # flag, and I have no idea about, say, lld. But if you are using GCC and it is invoking BFD ld, the above may just work for you.
Of course, the actual path used with origin should be customized as needed, and I have no opinion on ./config vs ./Configure. But using the response file trick seems to entirely sidestep the shell/make escaping nightmare.
I don't mind editing the generated Makefile, which is what I have been trying last...
I'm not sure you can set it with a shell variable and relative path. I don't think ldd expands the $ORIGIN in $ORIGIN/../lib64. In this case, I think you need to use ldconfig to add $ORIGIN/../lib64 to the library search paths. See finding ldd search path on Server Fault for more details.
Since I'm not sure, I'll provide the instructions anyway. You don't need to change the Makefiles. As a matter of fact, I did not have any luck doing so in the past because things get overwritten, and other things like CFLAGS and LDFLAGS get ignored.
Also see Build OpenSSL with RPATH? Your question and the cited question are different question that converge on similar answers (no duplicates between them). But it provides the OpenSSL dev's position on RPATHs. It was a private email, so I shared the relevant details rather than the whole message.
If you manage to embed $ORIGIN/../lib64 in the ELF section and it works, then please report back. Below, I am using /usr/local/ssl/lib for my RPATH. You should substitute $ORIGIN/../lib64 for /usr/local/ssl/lib.
OpenSSL supports RPATH's out of the box for BSD targets (but not others). From Configure:
# Unlike other OSes (like Solaris, Linux, Tru64, IRIX) BSD run-time
# linkers (tested OpenBSD, NetBSD and FreeBSD) "demand" RPATH set on
# .so objects. Apparently application RPATH is not global and does
# not apply to .so linked with other .so. Problem manifests itself
# when libssl.so fails to load libcrypto.so. One can argue that we
# should engrave this into Makefile.shared rules or into BSD-* config
# lines above. Meanwhile let's try to be cautious and pass -rpath to
# linker only when --prefix is not /usr.
if ($target =~ /^BSD\-/)
{
$shared_ldflag.=" -Wl,-rpath,\$(LIBRPATH)" if ($prefix !~ m|^/usr[/]*$|);
}
The easiest way to do it for OpenSSL 1.0.2 appears to be add it to linker flags during configuration
./config -Wl,-rpath=/usr/local/ssl/lib
You can also edit Configure line and hard code the rpath. For example, I am working on Debian x86_64. So I opened the file Configure in an editor, copied linux-x86_64, named it linux-x86_64-rpath, and made the following change to add the -rpath option:
"linux-x86_64-rpath", "gcc:-m64 -DL_ENDIAN -O3 -Wall -Wl,-rpath=/usr/local/ssl/lib::
-D_REENTRANT::-Wl,-rpath=/usr/local/ssl/lib -ldl:SIXTY_FOUR_BIT_LONG RC4_CHUNK DES_INT DES_UNROLL:
${x86_64_asm}:elf:dlfcn:linux-shared:-fPIC:-m64:.so.\$(SHLIB_MAJOR).\$(SHLIB_MINOR):::64",
Above, fields 2 and 6 were changed. They correspond to $cflag and $ldflag in OpenSSL's builds system.
Then, Configure with the new configuration:
$ ./Configure linux-x86_64-rpath shared no-ssl2 no-ssl3 no-comp \
--openssldir=/usr/local/ssl enable-ec_nistp_64_gcc_128
Finally, after make, verify the settings stuck:
$ readelf -d ./libssl.so | grep -i rpath
0x000000000000000f (RPATH) Library rpath: [/usr/local/ssl/lib]
$ readelf -d ./libcrypto.so | grep -i rpath
0x000000000000000f (RPATH) Library rpath: [/usr/local/ssl/lib]
$ readelf -d ./apps/openssl | grep -i rpath
0x000000000000000f (RPATH) Library rpath: [/usr/local/ssl/lib]
Once you perform make install, then ldd will produce expected results:
$ ldd /usr/local/ssl/lib/libssl.so
linux-vdso.so.1 => (0x00007ffceff6c000)
libcrypto.so.1.0.0 => /usr/local/ssl/lib/libcrypto.so.1.0.0 (0x00007ff5eff96000)
...
$ ldd /usr/local/ssl/bin/openssl
linux-vdso.so.1 => (0x00007ffc30d3a000)
libssl.so.1.0.0 => /usr/local/ssl/lib/libssl.so.1.0.0 (0x00007f9e8372e000)
libcrypto.so.1.0.0 => /usr/local/ssl/lib/libcrypto.so.1.0.0 (0x00007f9e832c0000)
...
Don't ask me why but this worked for me in OpenSSL 1.1.1i in getting around the $ sign issue:
\$\$\$$ORIGIN
Example:
./Configure linux-x86_64 '-Wl,-rpath,\$\$\$$ORIGIN'
Alternatively, if this command line hack isn't congruent with you, you can always use chrpath after building as others have suggested:
./Configure linux-x86_64 '-Wl,-rpath,XORIGIN'
make depend
make all
chrpath -r "\$ORIGIN" libssl.so

Resources