Workarounds for compilations on remote machine - compilation

Recently I have been working on a remote cluster (Scientific Linux 6 operating system), where I don't have root permissions to install the libraries I need to compile my C codes.
I ran into several issues:
1) If you can't run yum to install what you need, what can you do?
2) If you can't make install, what can you do?
3) How do I run my executables (assuming I could compile them) when they might need to call on libraries not installed on the machine?
I have posted this question so I can share what I have found to work around this problem. I hope it is useful...

Here we go!
1) If you can't install via "yum", you can try
1.1) Find out what ".rpm" repository has the libraries you need (google for it).
1.2) Download the relevant ".rpm" onto your home machine.
1.3) Then scp the "rpm" repository to your remote machine - or
scp repository.rpm user#remotemachine.com:/home/user
1.4) Then
ssh user#remote.com
There do the following (to "install" the rpm file without root permissions):
export ins=repository.rpm
rpm2cpio $ins | cpio -idv
This will unpack your rpm file, usually to a folder /home/user/usr/lib64.
1.5) You can then
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/user/usr/lib64
2) Instead of "make install":
2.1) Run
make install --prefix=/home/user/file
to "install" your libraries into /home/user/file
2.2) Then add /home/user/file to your LD library path as above.
3) Compiling when you have the libraries in "funny" directories:
3.1) Copy all your necessary libraries into a new directory /home/user/libs
3.2) Put this directory on LD library path i.e.
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/user/libs
3.3) To your Makefile for your executables, add
$LIBDIR = /home/user/libs
LIBS = -L$(LIBDIR) -llib1 -llib2 ... -llibn
(where lib1...libn are the designators for your libraries (like -llapack for the LAPACK libraries for example).
Hope that helps! All the best with your remote computing and good luck!

Related

Why can this dynamic library file include all dependencies? [duplicate]

Program is part of the Xenomai test suite, cross-compiled from Linux PC into Linux+Xenomai ARM toolchain.
# echo $LD_LIBRARY_PATH
/lib
# ls /lib
ld-2.3.3.so libdl-2.3.3.so libpthread-0.10.so
ld-linux.so.2 libdl.so.2 libpthread.so.0
libc-2.3.3.so libgcc_s.so libpthread_rt.so
libc.so.6 libgcc_s.so.1 libstdc++.so.6
libcrypt-2.3.3.so libm-2.3.3.so libstdc++.so.6.0.9
libcrypt.so.1 libm.so.6
# ./clocktest
./clocktest: error while loading shared libraries: libpthread_rt.so.1: cannot open shared object file: No such file or directory
Is the .1 at the end part of the filename? What does that mean anyway?
Your library is a dynamic library.
You need to tell the operating system where it can locate it at runtime.
To do so,
we will need to do those easy steps:
Find where the library is placed if you don't know it.
sudo find / -name the_name_of_the_file.so
Check for the existence of the dynamic library path environment variable(LD_LIBRARY_PATH)
echo $LD_LIBRARY_PATH
If there is nothing to be displayed, add a default path value (or not if you wish to)
LD_LIBRARY_PATH=/usr/local/lib
We add the desired path, export it and try the application.
Note that the path should be the directory where the path.so.something is. So if path.so.something is in /my_library/path.so.something, it should be:
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/my_library/
export LD_LIBRARY_PATH
./my_app
Reference to source
Here are a few solutions you can try:
ldconfig
As AbiusX pointed out: If you have just now installed the library, you may simply need to run ldconfig.
sudo ldconfig
ldconfig creates the necessary links and cache to the most recent
shared libraries found in the directories specified on the command
line, in the file /etc/ld.so.conf, and in the trusted directories
(/lib and /usr/lib).
Usually your package manager will take care of this when you install a new library, but not always, and it won't hurt to run ldconfig even if that is not your issue.
Dev package or wrong version
If that doesn't work, I would also check out Paul's suggestion and look for a "-dev" version of the library. Many libraries are split into dev and non-dev packages. You can use this command to look for it:
apt-cache search <libraryname>
This can also help if you simply have the wrong version of the library installed. Some libraries are published in different versions simultaneously, for example, Python.
Library location
If you are sure that the right package is installed, and ldconfig didn't find it, it may just be in a nonstandard directory. By default, ldconfig looks in /lib, /usr/lib, and directories listed in /etc/ld.so.conf and $LD_LIBRARY_PATH. If your library is somewhere else, you can either add the directory on its own line in /etc/ld.so.conf, append the library's path to $LD_LIBRARY_PATH, or move the library into /usr/lib. Then run ldconfig.
To find out where the library is, try this:
sudo find / -iname *libraryname*.so*
(Replace libraryname with the name of your library)
If you go the $LD_LIBRARY_PATH route, you'll want to put that into your ~/.bashrc file so it will run every time you log in:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/library
Update
While what I write below is true as a general answer about shared libraries, I think the most frequent cause of these sorts of message is because you've installed a package, but not installed the -dev version of that package.
Well, it's not lying - there is no libpthread_rt.so.1 in that listing. You probably need to re-configure and re-build it so that it depends on the library you have, or install whatever provides libpthread_rt.so.1.
Generally, the numbers after the .so are version numbers, and you'll often find that they are symlinks to each other, so if you have version 1.1 of libfoo.so, you'll have a real file libfoo.so.1.0, and symlinks foo.so and foo.so.1 pointing to the libfoo.so.1.0. And if you install version 1.1 without removing the other one, you'll have a libfoo.so.1.1, and libfoo.so.1 and libfoo.so will now point to the new one, but any code that requires that exact version can use the libfoo.so.1.0 file. Code that just relies on the version 1 API, but doesn't care if it's 1.0 or 1.1 will specify libfoo.so.1. As orip pointed out in the comments, this is explained well at here.
In your case, you might get away with symlinking libpthread_rt.so.1 to libpthread_rt.so. No guarantees that it won't break your code and eat your TV dinners, though.
You need to ensure that you specify the library path during
linking when you compile your .c file:
gcc -I/usr/local/include xxx.c -o xxx -L/usr/local/lib -Wl,-R/usr/local/lib
The -Wl,-R part tells the resulting binary to also look for the library
in /usr/local/lib at runtime before trying to use the one in /usr/lib/.
Try adding LD_LIBRARY_PATH, which indicates search paths, to your ~/.bashrc file
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path_to_your_library
It works!
The linux.org reference page explains the mechanics, but doesn't explain any of the motivation behind it :-(
For that, see Sun Linker and Libraries Guide
In addition, note that "external versioning" is largely obsolete on Linux, because symbol versioning (a GNU extension) allows you to have multiple incompatible versions of the same function to be present in a single library. This extension allowed glibc to have the same external version: libc.so.6 for the last 10 years.
cd /home/<user_name>/
sudo vi .bash_profile
add these lines at the end
LD_LIBRARY_PATH=/usr/local/lib:<any other paths you want>
export LD_LIBRARY_PATH
Another possible solution depending on your situation.
If you know that libpthread_rt.so.1 is the same as libpthread_rt.so then you can create a symlink by:
ln -s /lib/libpthread_rt.so /lib/libpthread_rt.so.1
Then ls -l /lib should now show the symlink and what it points to.
I had a similar error and it didn't fix with giving LD_LIBRARY_PATH in ~/.bashrc .
What solved my issue is by adding .conf file and loading it.
Go to terminal an be in su.
gedit /etc/ld.so.conf.d/myapp.conf
Add your library path in this file and save.(eg: /usr/local/lib).
You must run the following command to activate path:
ldconfig
Verify Your New Library Path:
ldconfig -v | less
If this shows your library files, then you are good to go.
running:
sudo ldconfig
was enough to fix my issue.
I had this error when running my application with Eclipse CDT on Linux x86.
To fix this:
In Eclipse:
Run as -> Run configurations -> Environment
Set the path
LD_LIBRARY_PATH=/my_lib_directory_path
Wanted to add, if your libraries are in a non standard path, run ldconfig followed by the path.
For instance I had to run:
sudo ldconfig /opt/intel/oneapi/mkl/2021.2.0/lib/intel64
to make R compile against Intel MKL
All I had to do was run:
sudo apt-get install libfontconfig1
I was in the folder located at /usr/lib/x86_64-linux-gnu and it worked perfectly.
Try to install lib32z1:
sudo apt-get install lib32z1
If you are running your application on Microsoft Windows, the path to dynamic libraries (.dll) need to be defined in the PATH environment variable.
If you are running your application on UNIX, the path to your dynamic libraries (.so) need to be defined in the LD_LIBRARY_PATH environment variable.
The error occurs as the system cannot refer to the library file mentioned. Take the following steps:
Running locate libpthread_rt.so.1 will list the path of all the files with that name. Let's suppose a path is /home/user/loc.
Copy the path and run cd home/USERNAME. Replace USERNAME with the name of the current active user with which you want to run the file.
Run vi .bash_profile and at the end of the LD_LIBRARY_PATH parameter, just before ., add the line /lib://home/usr/loc:.. Save the file.
Close terminal and restart the application. It should run.
I got this error and I think its the same reason of yours
error while loading shared libraries: libnw.so: cannot open shared
object file: No such file or directory
Try this. Fix permissions on files:
cd /opt/Popcorn (or wherever it is)
chmod -R 555 * (755 if not ok)
I use Ubuntu 18.04
Installing the corresponding -dev package worked for me,
sudo apt install libgconf2-dev
Before installing the above package, I was getting the below error:
turtl: error while loading shared libraries: libgconf-2.so.4: cannot open shared object file: No such file or directory
I got this error and I think its the same reason of yours
error while loading shared libraries: libnw.so: cannot open shared object
file: No such file or directory
Try this. Fix permissions on files:
sudo su
cd /opt/Popcorn (or wherever it is)
chmod -R 555 * (755 if not ok)
chown -R root:root *
A similar problem can be found here.
I've tried the mentioned solution and it actually works.
The solutions in the previous questions may work. But the following is an easy way to fix it.
It works by reinstalling the package libwbclient
in fedora:
dnf reinstall libwbclient
You can read about libraries here:
https://domiyanyue.medium.com/c-development-tutorial-4-static-and-dynamic-libraries-7b537656163e

How to install wxHaskell?

I have downloaded the latest Haskell Platform(8.0.1 version) for Windows 7, wxWidgets-3.0.2, set all the needed environment variables. I have successfully installed "wxdirect" using "cabal install", but when I try to install "wxc" it fails. Install logs:
C:\wxWidgets-3.0.2\lib\gcc_dll\mswud\..: getDirectoryContents: does not exist
(The system cannot find the path specified.)
cabal: Leaving directory '.'
Where do I need to get that "gcc_dll\mswud\" folder?
P.S Neither Abriline nor Achelanne works for me for some reason.
P.P.S I tried to install this library following Haskell-wiki instructions, but it didn't work complaining about a ton of dependencies.
UPDATE
I tried installing the wxHaskell using Haskell Platform 2014.2.0.0 with Abriline installer. It stopped on building wxc. Logs:
Building wxc
setup.exe: Maybe.fromJust: Nothing
1] Compiling Main ( wxc-0.92.2.0\dist\setup\setup.hs, wxc-0.92.2.0\dist\setup\Main.o )
Linking .\wxc-0.92.2.0\dist\setup\setup.exe ...
The bitness is correct
Configuring wxc-0.92.2.0...
Configuring wxc to build against wxWidgets 3.0.2
cabal: Leaving directory '.\wxc-0.92.2.0'
UPDATE
Extended logs with "cabal install wxc -v3":
http://rgho.st/7BJcHzJTF
From the log it seems that maybe pkg-config is missing. From the platform's built in msys2 run pacman -S pkg-config and install it, see if it works after that.
I've installed it as follows.
Download wxInstall-Achelanne-64-0.1.zip, unzip, and set this folder as the current directory in a terminal
Type
Set WXWIN=%CD%\wxWidgets
Set WXCFG=gcc_dll\mswu
Set PATH=%CD%;%PATH%
cabal unpack wxdirect
cabal install ./wxdirect-xxx
(where xxx is e.g. 0.92.3).
Find the file wxdirect.exe and put it in the current directory.
Rename the file wxWdigets/include/wx/setup_redirect.h to wxWdigets/include/wx/setup.h
Type
cabal unpack wxc
cabal install ./wxc-xxx
Find the file wxc.dll and put it in the DLLs folder.
Put the folders DLLs and wxWidgets\lib\gcc_dll in your search path.
Now you can install wxcore and wx.

Installing liblapack without root permissions? (Trying to get Armadillo/Lapack/Blas working)

I'm trying to install Armadillo (and thus also BLAS and LAPACK) on a linux server for which I do not have root permission. I have jumped through a few of the first hurdles, but I am getting an error:
/usr/bin/ld: cannot find -llapack
This question addresses the same problem, with the solution being to install
liblapack-dev
liblapack3
libopenblas-base
libopenblas-dev
The problem is the only way I've found to do this online is by doing something like sudo-aptget install or yum install. Both are not allowed on the server I use. I can download binaries and install them locally - that is it.
My question is: how can I install these packages without the above permissions and get on my way to using Armadillo?
I'm using Centos linux, if it helps.
Since you do not have root permissions, the best way is to download and build LAPACK and BLAS. You can download the source code from netlib.
Description of installation instructions can be found here and here.
The basic steps are:
Unzip and tar the file.
Copy and edit the file LAPACK/make.inc.example to LAPACK/make.inc.
Edit the file LAPACK/Makefile
type make.
Similarly you can download and build BLAS.

Golang zmq binding, ZMQ4, returns package error not finding file zmq.h

I am trying to include ZMQ sockets in a Go app but both zmq4 and gozmq (the referred ZMQ binding libraries for Go) are giving me problems. I would like to understand why zmq4 specifically isn't importable on my system.
I am running a Windows 8 system and I used the windows installer from the ZMQ website for version 4.0.3. I am primarily concerned about getting zmq4 set up and here is the result of my "go get" query on the github library's location:
> go get github.com/pebbe/zmq4
# github.com/pebbe/zmq4
polling.go:4:17: fatal error: zmq.h: No such file or directory
compilation terminated.
This issue is not alleviated by cloning the Github repository - the error remains the same.
I know the issue has to do with the C library zmq.h that is located in the "include" folder of my ZMQ installation, but whether the dependency is held up by a pathing issue or an external tool issue is a mystery to me.
A similar error has come up in regards to node.js and is the solution I see others referred to, outside of node scripting, but it was unsuccessful in my case.
I've so far included the path to the "include" folder in my PATH environment variable and previously placed zmq.h inside of the zmq4 top-level folder. I don't have much of an arsenal otherwise to understand this problem because I am new to C and C-importing packages in Go
I wanted to do the same thing, but on Windows 7, and here is what I had to do.
Since the Go bindings are using cgo to integrate with zeromq, you need zeromq built with gcc. There are no pre-built binaries, so you'll have to build them yourself, with mingw or similar, but this process is easier than it may sound, and nicely described on the zeromq site.
As #photoionized pointed out, C_INCLUDE_PATH and LIBRARY_PATH need to be set when building the Go bindings.
(In my case, I ran into a problem when compiling libzmq with IN6_ADDR not being defined. The only solution I found was, inspired by this issue, to manually add the line #include <in6addr.h> to the windows.hpp file.)
The Windows installer version of ZeroMQ won't work with zmq4, you need to compile from source with gcc, I recommend using MSYS2.
Install and update MSYS2 following the instructions from
http://sourceforge.net/p/msys2/wiki/MSYS2%20installation/
Start the mingw32_shell.bat or mingw64_shell.bat based on Go arch (32bit or 64bit)
pacman -S mingw-w64-(x86_64|i686)-toolchain make (x86_64 for 64bit, i686 for 32bit)
cd into zeromq src folder (C:\ path starts with /c/ inside the shell)
./configure
make
make install
CGO_CFLAGS=-I/usr/local/include CGO_LDFLAGS=-L/usr/local/lib go get github.com/pebbe/zmq4
Copy the following dlls and put them next to your go program (.exe):
/usr/local/bin/libzmq.dll
/mingw(32|64)/bin/libgcc*.dll
/mingw(32|64)/bin/libwinpthread*.dll
/mingw(32|64)/bin/libstdc++*.dll
Here's updated steps for #user2172816's MSYS2 solution:
Install and update MSYS2 following the instructions from http://sourceforge.net/p/msys2/wiki/MSYS2%20installation/
Start the mingw32_shell.bat or mingw64_shell.bat based on Go arch (32bit or 64bit)
pacman -S mingw-w64-(x86_64|i686)-toolchain make (x86_64 for 64bit, i686 for 32bit)
Add C:\msys64\mingw64\bin to your Path (pkg-config is there)
Restart the msys2 shell to get the new Path
Download and unzip libsodium source: https://github.com/jedisct1/libsodium/releases
cd into libsodium folder (C:\ path starts with /c/ inside the shell)
./configure --build=(x86_64|i686)-w64-mingw32
make
make install
Add /usr/local/lib to PKG_CONFIG_PATH (export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig)
cd into zeromq src folder
./configure --build=(x86_64|i686)-w64-mingw32
Add
#ifdef ZMQ_HAVE_MINGW32
#include <winsock2.h>
#include <windows.h>
#include "netioapi.h"
#endif
To the top of src/tcpaddress.cpp
make
make install
CGO_CFLAGS=-I/usr/local/include CGO_LDFLAGS=-L/usr/local/lib go get github.com/pebbe/zmq4
CGO_CFLAGS=-I/usr/local/include CGO_LDFLAGS=-L/usr/local/lib go build in your project directory
Copy the following dlls and put them next to your go program (.exe):
/usr/local/bin/libzmq.dll
/mingw(32|64)/bin/libgcc*.dll
/mingw(32|64)/bin/libwinpthread-*.dll
/mingw(32|64)/bin/libstdc++*.dll
/usr/local/bin/libsodium-*.dll
maybe? /usr/local/bin/libsodium-*.def
An updated answer using MSYS2.
Install MSYS2 MSYS2 installation guide.
Make sure to choose the correct installation 32bit or 64bit.
Open the appropriate shell MSYS2 MinGW 64-bit or MSYS2 MinGW 32-bit. All further steps assume you are using this shell.
Update packages following instructions at the installation guide.
Install libtool pacman -Sy libtool.
Download zmq source code to a location of your choice.
Navigate to the zmq source folder.
To generate the configure file, run the autogen tool by running ./autogen.sh.
In the probable case that step 8 fails:
Find the file at fault (probably version.sh).
Replace line endings by (replace file by the actual filename).
cp file file.bak
tr -d '\r' <file.bak> file
If this fails you'll have to dive in the code and find the problem.
Run the configure tool ./configure.
In the probable case of failure. Comment out empty else clauses in the configure file.
Add Go to Path: PATH=${PATH}:<go bin directory>.
Install Go Package: CGO_CFLAGS=-I/usr/local/include CGO_LDFLAGS=-L/usr/local/lib go get github.com/pebbe/zmq4
To install ZMQ in windows: Problem in Installing Golang ZMQ for windows - fatal error: czmq.h: No such file or directory
First of all, install the msys64. Download the software from https://www.msys2.org/ and install it on C:\msys64.
Then add C:\msys64\mingw64\bin to PATH environment variable of the windows.
Then run the following commands (in CMD) one by one.
pacman -Su
pacman -S --needed base-devel mingw-w64-x86_64-toolchain
pacman -S base-devel gcc vim cmake
pacman -S mingw-w64-x86_64-libsodium
pacman -S mingw-w64-x86_64-zeromq
Finally, run the Go install command:
go get github.com/pebbe/zmq4
Finished.

Build shared libraries in ATLAS

I've read the entire ATLAS installation guide, and it says all you need to build shared (.so) libraries is to pass the --shared flag to the configure script. However, when I build, the only .so files that appear in my lib folder are libsatlas.so and libtatlas.so, though the guide says that there should be six others:
libatlas.so, libcblas.so, libf77blas.so, liblapack.so, libptcblas.so, libptf77blas.so
After installation some of the tests fail because these libraries are missing. Furthermore, FFPACK wants these libraries during installation.
Has anyone encountered this? What am I doing incorrectly?
In my experience, it's a lot more complex than that, see our EasyBuild implementation of the ATLAS build procedure at https://github.com/hpcugent/easybuild-easyblocks/blob/master/easybuild/easyblocks/a/atlas.py .
We needed to:
enable the -fPIC compiler option
run 'make shared cshared ptshared cptshared' in the 'lib' directory
We're not even using --shared for configure, probably because it doesn't do much.
If you want to build ATLAS (and whatever you will be linking it with) without headaches, look into EasyBuild.
(disclaimer: I'm a developer for EasyBuild)
First if you have incorrectly specified the --force-tids flag for configure then the parallel libs won't build. To check this you can run make ptcheck. I have question regarding the specification of this flag here
Then if I examine my resulting ATLAS Makefile it says " ... only when atlas is built to one lib" and indeed only two "fat" libs are constructed: libsatlas.so and libtatlas.so.
I quess you can either link FFPACK against those libs or change the resulting ATLAS Makefile to contain the targets you need (Which won't be too hard since the static libs are available).
I had to manually create links to the .so.3 files.
So the versioned library files existed, but not the files the cmake was looking for.
Running
sudo ln -s libatlas.so.3 libatlas.so
sudo ln -s libcblas.so.3 libcblas.so
sudo ln -s liblapack_atlas.so.3
(I didn't build the cblas, atlas or lapack but installed them with apt-get. Wondering why the links were not automatically created).

Resources