Phase 1: I have installed Hadoop 2.6.0 on Ubuntu 14.04 64bit, I have started it in pseudo-distributed mode and I ran the sample program they provide . Everything ok here.
Phase 2: I have installed Hadoop 2.5.1 on 3 computers that run Ubuntu 10.04 32bit. I have started the HDFS and YARN, I can see them in the web GUI. But when try to compile the same test program I get this:
gcc test.c -I$HADOOP_HDFS_HOME/include -L/usr/lib/java/jdk1.7.0_71/jre/lib/amd64/server/ -L$HADOOP_HDFS_HOME/lib/native -lhdfs -ljvm -o test
/usr/bin/ld: skipping incompatible /usr/local/hadoop/lib/native/libhdfs.so when searching for -lhdfs
/usr/bin/ld: skipping incompatible /usr/local/hadoop/lib/native/libhdfs.a when searching for -lhdfs
/usr/bin/ld: cannot find -lhdfs
collect2: ld returned 1 exit status
Now, when I do file libhdfs.so.0.0.0 , I get :
libhdfs.so.0.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
So, the native libraries are 64bit and my OS is 32bit.
Question:
Is it necessary to recompile entire Hadoop from source or can I only recompile native libraries and replace them in the existing installation?
What is the shortest path ?
Please provide info on recompilation.
Edit:
I have compiled for 32bit on my 64bit machine, I've exported CLFAGS=-m32 , but I had this error :
[exec] /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so: error adding symbols: File in wrong format
[exec] collect2: error: ld returned 1 exit status
[exec] make[2]: *** [target/usr/local/lib/libhdfs.so.0.0.0] Error 1
[exec] make[1]: *** [CMakeFiles/hdfs.dir/all] Error 2
What I did next was to manually install and configure update-alternatives to point java8 JDK 32bit.
I only had a problem with libfuse-dev, which I have uninstalled.
Since it was not mandatory.
After that I ran : mvn package -Pnative -DskipTests -Dtar
and found the libs here : hadoop-hdfs-project/hadoop-hdfs/target/native/target/usr/local/lib :
libhdfs.a libhdfs.so libhdfs.so.0.0.0
The recent hadoop tarballs only seem to ship with 64 bit libraries. You can compile them yourself, altough i haven't tested this myself.
You will have to download the source tarball of hadoop and install the following packages:
GCC (or another C compiler)
GNU Autotools (autoconf, automake, libtool)
zlib dev package
openssl dev package
maven
Once this is installed you can use the standard hadoop pom.xml file contained in the source tarball to compile the native libraries:
$ mvn package -Pdist,native -DskipTests -Dtar
The libraries can be found in:
$ hadoop-dist/target/hadoop-2.6.0/lib/native
The bin/hadoop script ensures the native hadoop library is on the classpath via the system property -Djava.library.path. You can modify the bin/hadoop script to point to your newly compiled libraries. For more information be sure to check the documentation that is available here.
Related
I cross-compiled ARMNN for my ARM Cortex-A9 (Debian 9) device on my host system which is x86_64 (Ubuntu 18.04).
I have successfully built ARMNN and all it's dependencies without any errors, but when I try to run the sample mnist on Cortex-A9, it gives the following error:
duu#34d456-45433g:~/ML-examples/armnn-mnist$ make test
arm-linux-gnueabihf-g++ -O3 -std=c++14 -I/home/duu/armnn/armnn/include mnist_caffe.cpp -o mnist_caffe -L/home/duu/build -larmnn -larmnnCaffeParser
/home/admin/build/libarmnn.so: undefined reference to `exp2f#GLIBC_2.27'
/home/admin/build/libarmnn.so: undefined reference to `logf#GLIBC_2.27'
/home/admin/build/libarmnn.so: undefined reference to `log2f#GLIBC_2.27'
/home/admin/build/libarmnn.so: undefined reference to `powf#GLIBC_2.27'
/home/admin/build/libarmnn.so: undefined reference to `expf#GLIBC_2.27'
collect2: error: ld returned 1 exit status
Makefile:12: recipe for target 'mnist_caffe' failed
make: *** [mnist_caffe] Error 1
I figured later that the device should have the same compiler and same GLIBC package, both with same versions as the host on which the library was compiled.
I compiled ARMNN using gcc-6.4.0 and glibc-2.27 on the host and I have gcc-6.3.0 and glibc-2.24 on Cortex-A9 device.
I managed to match the GCC versions and solve the compiler error but I am not able to find matching versions of GLIBC on Ubuntu 18.04 and debian 9.
Is there any way to install a GLIBC version on either of the systems which is common for both of them?
P.S. : Any other method to solve this issue is highly appreciated.
Here is a list of the possible options I can think of right now:
run your application while having LD_LIBRARY_PATH pointing to a directory containing the arm glibc dynamic libraries(v2.27) copied from your x86_64 system - see this post.
Re-compile your application on your target system, if possible, if your target system can use NFS for example,
cross-compile a static version of your library/application, linking it using -static -static-libgcc -static-libstdc++ - see this post.
use arm-linux-musleabihf-cross or one of his friends for cross-compiling a static version of your library/application if this did not work with gcc/g++ and glibc.
run your application in a chrooted environment containing the arm dynamic libraries you linked your application with on the x86_64 system - see this post for more details,
install docker on your Cortex-A9 system, and build a minimal debian/ubuntu docker image that contain a version of debian/ubuntu using glibc 2.27, along with your library and application, and execute the application in a container.
I'm working with some legacy hardware, and I need to compile a customised version of opensslv0.9.8e onto a 32-bit system.
My Fedora 17 VM comes with gcc-4.7.2. Unfortunately, compiling the openssl version with this doesn't work, I get illegal instruction errors. I found posts here and here that suggest that this version of openssl doesn't compile with gcc 4.2.0 or higher
when compiling openssl 0.9.8e with gcc 4.2.0, some errors show up (see
below). The test suite then fails with "instruction error". (The resulting
programs also fail.)
I built and checked gcc-4.1.2 on the system where the reported error arose.
I then rebuilt openssl with gcc-4.1.2 and could successfully run the test
suite.
My current plan is to build 4.1.2 onto my system. Unfortunately, when doing so, I run into the following error:
WARNING: 'makeinfo' is missing on your system. You should only need it if you modified a '.texi'
or a '.texinfo' file, or any other file indirectly affecting the aspect of the manual. The
spurious call might also be the consequence of using a buggy 'make' (AIX, DU, IRIX). You might
want to install the 'Texinfo' package or the 'GNU make' package. Grab either from any GNU archive site
Then,
make[3]: ***[fastjar.info] Error 1
make[3]: Leaving directory '/home/charles/build/make/fastjar'
. . .
make: *** [all] Error 2
I've installed texinfo, but I can't find gmake on the yum package manager.
What are the next steps I should take? I'm stumped at the moment ):
Installed Vala for Windows64 from the MSYS2 project. Started running various examples from https://wiki.gnome.org/Projects/Vala/Examples. Some examples produce errors like
... the "Clutter Samples:Animated Actors"
D:\Projects\Vala\Examples>valac.exe --version
Vala 0.36.1-dirty
D:\Projects\Vala\Examples>valac.exe --pkg clutter-1.0 clutter.vala -o clutter.exe
D:/Projects/Vala/Examples/clutter.vala.c:7:29: fatal error: clutter/clutter.h: No such file or direc
tory
#include <clutter/clutter.h>
^
compilation terminated.
error: cc exited with status 1
Compilation failed: 1 error(s), 0 warning(s)
... and the "Vala GTK+ 3.x Examples: Basic Sample"
D:\Projects\Vala\Examples>valac.exe --version
Vala 0.36.1-dirty
D:\Projects\Vala\Examples>valac.exe --pkg gtk+-3.0 gtk+3-hello.vala -o gtk+3-hello.exe
D:/Projects/Vala/Examples/gtk+3-hello.vala.c:17:21: fatal error: gtk/gtk.h: No such file or director
y
^
compilation terminated.
error: cc exited with status 1
Compilation failed: 1 error(s), 0 warning(s)
...and the Vala Collections: libgee sample
D:\Projects\Vala\Examples>valac.exe --version
Vala 0.36.1-dirty
D:\Projects\Vala\Examples>valac.exe --pkg=gee-0.8 libgee-collections.vala -o libgee-collections.exe
error: Package `gee-0.8' not found in specified Vala API directories or GObject-Introspection GIR di
rectories
Compilation failed: 1 error(s), 0 warning(s)
I realize pkg:gee-0.8 is not installed, but I did find 0.20.0-1 using pacman. I dont know how to reference this library?
$ pacman -Ss gee -v
Root : /
Conf File : /etc/pacman.conf
DB Path : /var/lib/pacman/
Cache Dirs: /var/cache/pacman/pkg/
Hook Dirs : /usr/share/libalpm/hooks/ /etc/pacman.d/hooks/
Lock File : /var/lib/pacman/db.lck
Log File : /var/log/pacman.log
GPG Dir : /etc/pacman.d/gnupg/
Targets : gee
mingw32/mingw-w64-i686-libgee 0.20.0-1
A collection library providing GObject-based interfaces and classes for
commonly used data structures (mingw-w64)
mingw64/mingw-w64-x86_64-libgee 0.20.0-1
A collection library providing GObject-based interfaces and classes for
commonly used data structures (mingw-w64)
Obviously my installation is incorrect/incomplete. What do I need to change or install to fix these type of issues?
Thanks for your help!
Running valac.exe directly can be a problem.
Usually when using msys2 to compile something you first enter the msys2 shell with the MSYSTEM environment variable set up correctly. (there are scripts and in newer versions executables in the msys2 root folder that can do that for you).
For example when you set MSYSTEM=MINGW64 (caveat: MINGW64 must be upper case!) and then run mingtty.exe or bash.exe --login you can then check your PKG_CONFIG_PATH var:
$ echo $PKG_CONFIG_PATH
/mingw64/lib/pkgconfig:/mingw64/share/pkgconfig
Be sure to understand the difference of the three MSYSTEM settings:
MSYS - Only used to bootstrap the core msys2 system and for packaging purposes
MINGW32 / MINGW64 - The "normal" environment (32-Bit / 64-Bit) you work in when using msys2.
The pacman package names for the different MSYSTEM environment have prefixes. I.e. the packages that start with mingw-w64-x86_64 are used in the MSYSTEM=MINGW64 environment.
vala uses pkg-config (be sure to have mingw64/mingw-w64-x86_64-pkg-config installed) when you specify something with --pkg and pkg-config then searches for .pc files (like gee-0.8.pc).
The mingw-w64-x86_64-libgee package installs its pc file as /mingw64/lib/pkgconfig/gee-0.8.pc.
With gee-0.8 the 0.8 is the interface version number, not the release version number. So install release 0.20.0 and the relevant interface files should become available. That also applies to Clutter and GTK+3.
I'm trying to build a c++ project on Ubuntu 12.04 64-bit.
the project is 32-bit.
At the terminal output I receive such lines:
ibcrypto.a(bio_b64.o)' is incompatible with i386 output
/usr/bin/ld: i386:x86-64 architecture of input file `/usr/lib/x86_64-linux-
gnu/libcrypto.a(bio_asn1.o)' is incompatible with i386 output
collect2: ld returned 1 exit status
make[2]: * [linux/release_32/cesmagent] Error 1
I thought it had something to do with libssl0.9.8, but so I did "sudo apt-get install libssl0.9.8:i386" but there already was the newest version of it.
Can any one please help me here?
The linker is trying to link against the 64-bit version of libcrypto.a because it can't find a 32-bit version.
You say you have this:
$ locate libcrypto.a
/usr/lib/x86_64-linux-gnu/libcrypto.a
Whereas, you need this:
$ locate libcrypto.a
/usr/lib/i386-linux-gnu/libcrypto.a
(I thought you could have both, but it turns out, on 12.04, it's impossible to have both because the packages conflict.)
You can install the i386 library like this:
sudo apt-get install libssl-dev:i386
Basically, you probably had the run-time libraries installed correctly, but you did not have the developer libraries.
The workaround which solved my problem:
I installed 32-bit version library
sudo apt-get install libssl-dev:i386
the 64-bit library was removed by that.
in the directory /usr/lib/i386-linux-gnu I found 32-bit version libcrypto.a and put it in /usr/lib32
Then I installed 64-bit library
sudo apt-get install libssl-dev
in the directory /usr/lib/x86_64-linux-gnu I found 64-bit version libcrypto.a and put it in /usr/lib64
in the Makefile of a project I adjusted paths /usr/lib32/libcrypto.a and /usr/lib64/libcrypto.a to be dependent on the corresponding command argument.
the project builds!
I want to build "gcc cross-compiler" to compile "c/c++" applications on "Linux" environment but for "Windows" target.
I have made this so far:
Installed the necessary tools and packages for building GCC listed on "Prerequisites for GCC" page.
Downloaded required sources:
"gcc-core-4.4.1", "gcc-g++-4.4.1", "binutils-2.19.1", "w32api-3.13-mingw32", "mingwrt-3.16-mingw32"
Created this directory hierarchy:
"${HOME}/gcc/" - for final cross-compiler
"${HOME}/src/" - for sources
"${HOME}/src/build-binutils/i386-mingw32/" - for building binutils to "i386-mingw32" target
"${HOME}/src/build-gcc/i386-mingw32/" - for building gcc to "i386-mingw32" target
Builded binutils package:
cd "${HOME}/src/build-binutils/i386-mingw32/"
../../binutils-2.19.1/configure --prefix="${HOME}/gcc" --target=i386-mingw32 --disable-nls
make
make install
Copied "w32api" and "mingwrt" headers to the install directory:
cp -R "${HOME}/src/w32api-3.13-mingw32/include" "${HOME}/gcc/i386-mingw32"
cp -R "${HOME}/src/mingwrt-3.16-mingw32/include" "${HOME}/gcc/i386-mingw32"
And now when I am trying to build the "c (only) cross-compiler":
cd "${HOME}/src/build-gcc/i386-mingw32/"
../../gcc-4.4.1/configure --prefix="${HOME}/gcc" --target=i386-mingw32 --enable-languages=c --with-headers="${HOME}/gcc/i386-mingw32/include" --disable-nls
make<br>
it was building something about 4 minutes and then gives me these errors:
${HOME}/gcc/i386-mingw32/bin/ld: dllcrt2.o: No such file: No such file or directory
collect2: ld returned 1 exit status
make[2]: *** [libgcc_s.dll] Error 1
make[2]: Leaving directory `${HOME}/src/build-gcc/i386-mingw32/i386-mingw32/libgcc'
make[1]: *** [all-target-libgcc] Error 2
make[1]: Leaving directory `${HOME}/src/build-gcc/i386-mingw32'
make: *** [all] Error 2
From that error message I really don't know what to do now :-((( .
Does anybody know where is the problem?
Thanks.
That's actually OK: the way things go, you need to
build binutils
install headers
build the a partial C compiler: enough to create object files, but not enough to link
build the win32api and mingw runtime (which includes your missing dllcrt2.o)
build a complete C compiler (and other front-ends, such as C++, Fortran, Ada, whatever, if you want them)
You have successful performed step 3 above; it fails building libgcc (which is a GCC support library), but that means the C compiler core is functionnal (although it won't be able to link, it can still create valid object files). You can check that by looking at the gcc/xgcc file in your GCC build directory.
So, you need to go to the next step, not worrying about your current failure.
(To actuall install the partial C compiler, you should run make with the -k option, to have it do it best, even in the face of errors. For example, use make -k install.)
There are precompiled cross-compilers of MinGW-w64 available.
This allows to compile native 32- and 64-bit Windows binaries from Linux, a two minute tutorial is available at http://www.blogcompiler.com/2010/07/11/compile-for-windows-on-linux/
Just in case you don't want to spend a lot of time trying to build it yourself.
I grepped through the MinGW sources, and found that dllcrt2.o is something built off the mingwrt package. I assume you have to compile and install that, not just copy the headers?