I have the need of running a Rust executable (made with cargo build --release) inside a chroot. Normally, I just copy the files reported by ldd
$ldd hello_world_rust
linux-vdso.so.1 (0x00007ffef48c6000)
libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f3224c3e000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f3224a21000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007f3224819000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f3224603000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f3224261000)
/lib64/ld-linux-x86-64.so.2 (0x00007f3224e42000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f3223f5d000)
But my Rust programs crashes when run inside the jail
thread '<unnamed>' panicked at 'assertion failed: `(left == right)` (left: `2`, right: `0`)', /build/rust/src/rustc-1.1.0/src/libstd/sys/unix/thread.rs:204
fatal runtime error: Could not unwind stack, error = 5
Illegal instruction (core dumped)
When checking things with strace (inside the jail) I noticed the following
strace -e file hello_world_rust
.... 14 lines of loading dynlibs cut
readlink("/etc/je_malloc.conf", 0x7fff7c2ed380, 4096) = -1 ENOENT (No such file or directory)
open("/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/proc/stat", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/proc/cpuinfo", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/proc/self/maps", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
I believe the crash occurs because there are no /proc and /sys inside the chroot.
Is my belief correct? If so, why are they necessary? Is there a way I can compile my rust program so that it does not need /sys and /proc?
I believe this is a known issue. According to Alex Crichton:
it looks like the failure in question is our detection of the stack start of the main thread in setting up the first guard page. I forget how reliable it is that linux sets up a guard page for us, and it would be tough to remove for now at least.
Related
Just out of curiosity I would like to examine what is inside these commands. Like mkdir or adb command or other things inside /bin or /sbin folder. I know they are binaries so, i searched some answers like below. I found xxd -b <filename> or `strings . But i want more clearer view, is it possible. And is there a way we can know in which language they are written? For example for mac commands, what language they are written?
How to view files in binary from bash?
Practical answer: most binaries are written in 'C', some are written in 'C++'. See below on logic to tell which one. Note that I'm not macos expert - I believe Linux tools will work the same for macos
Most of the files in /bin and /usr/bin are stripped binaries (the minority are symbolic links which create "aliases", or small scripts that implement existing commands, usually using other binaries).
The fact the the binary is stripped means that a debugger (gdb, or other tools like ltrace, etc.) to see function names, variable names, etc. (do man strip to see what is removed). One of the items that can NOT be 'stripped' is the list of shared libraries.
Given this information, possible to get some clues from the list of shared libraries. C++ programs will usually link with the 'standard C++' shared library (libstdc++.so.N, where N is the version, currently at 6). Most other languages have their own run-time shared library that will be linked. Most 'C' programs will link with the standard C runtime (/lib.c.so.N).
Bottom line: consider the following process:
Do file -L /bin/sh (or any other binary):
If output contain 'script' in the output, the program is implemetned a s script
If output indicates a binary (ELF shared object, or similar):
Run 'ldd /bin/sh`
If output contain 'libstdc++.so' - code contain SOME c++ - most like C++ source
If output contain 'libc.so.' - program contain SOME c - most like C program
owner#vm1:~/Projects/SO/ff1$ file /bin/bash
/bin/bash: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/l, for GNU/Linux 3.2.0, BuildID[sha1]=452da38d6212b692cd80eb0dd1c99cf853da31ae, stripped
owner#vm1:~/Projects/SO/ff1$ ldd /bin/bash
linux-vdso.so.1 (0x00007ffcb87fe000)
libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f5e7cd8e000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5e7cb8a000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5e7c799000)
/lib64/ld-linux-x86-64.so.2 (0x00007f5e7d2d2000)
wner#vm1:~/Projects/SO/ff1$ file /usr/bin/zlib-flate
/usr/bin/zlib-flate: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/l, for GNU/Linux 3.2.0, BuildID[sha1]=bbc1bc3fbbb71c7ae3cf9ef9f2876098df3fd752, stripped
owner#vm1:~/Projects/SO/ff1$ ldd !$
ldd /usr/bin/zlib-flate
linux-vdso.so.1 (0x00007fffeb3be000)
libqpdf.so.21 => /usr/lib/x86_64-linux-gnu/libqpdf.so.21 (0x00007f2f8ee45000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f2f8eabc000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f2f8e8a4000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2f8e4b3000)
libjpeg.so.8 => /usr/lib/x86_64-linux-gnu/libjpeg.so.8 (0x00007f2f8e24b000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f2f8e02e000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2f8dc90000)
/lib64/ld-linux-x86-64.so.2 (0x00007f2f8f313000)
Note:
* Technically, possible to "trick" this logic with static linking, but most (all) binaries will use dynamic linking.
I've been trying to link a shared library into my program and I want its path to be relative to my RPATH.
However, when I ran ldd, I noticed that the absolute path of the shared library was linked. Any ideas why?
Edit:
/home/projects/my_files/winter_fresh.so
libgcc_s.so.1 => /home/tomo/anaconda3/lib/libgcc_s.so.1 (0x00007f0a3bf64000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0a3bd47000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0a3b97d000)
/lib64/ld-linux-x86-64.so.2 (0x00007f0a3e369000)
libstdc++.so.6 => /home/tomo/anaconda3/lib/libstdc++.so.6 (0x00007f0a3b643000)
The issue is the first file. I don't want the library for winter_fresh to be an absolute path, since I have an RPATH that contains it.
The issue is the first file. I don't want the library for winter_fresh to be an absolute path
This most usually happens when you link against your library like so:
gcc ... /home/projects/my_files/winter_fresh.so ...
and your library does not have SONAME (you didn't use -soname linker option when building it).
To fix this, either add SONAME to winter_fresh.so (a good practice in general), or link against it like so:
gcc ... -L /home/projects/my_files -l:winter_fresh.so
An even better approach might be to rename winter_fresh.so to libwinter_fresh.so, and then link against it like so:
gcc ... -L /home/projects/my_files -lwinter_fresh
My guess is, you compiled your program using winter_fresh.so as source file, not by linking against it.
If you encoded the path to shared library/executable as /home/projects/my_files/winter_fresh.so, you can put your shared library inside RPATH directory, like this:
$ mkdir some_dir
$ mkdir -p some_dir/home/projects/my_files
$ cp /home/projects/my_files/winter_fresh.so some_dir/home/projects/my_files
$ RPATH=$(pwd)/some_dir ./executable
The linker searches for library named /home/projects/my_files/winter_fresh.so under RPATH.
Now a simple test:
// main.c
int main() {
int external_function(void);
return external_function();
}
// exlib.c
#include <stdio.h>
int external_function(void) {
return printf("%s\n", __func__);
}
Now, let's create bad.out compiled with exlib.so shared library used as source:
$ gcc -shared -fPIC -o exlib.so exlib.c
$ gcc /tmp/exlib.so main.c -o bad.out
$ ldd bad.out
linux-vdso.so.1 (0x00007ffd921db000)
/tmp/exlib.so (0x00007fe4470f7000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007fe446d3b000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fe4474fb000)
As you see the string /tmp/exlib.so points to the shared library. I can run the program, using RPATH to point the linker to exlib.so location. I need to create subtree /tmp/exlib.so inside RPATH, like this:
$ mkdir -p lib/tmp
$ mv exlib.so lib/tmp
$ RPATH=$(pwd)/lib ./bad.out
external_function
When running bad.out, linker searches for a file named /tmp/exlib.so inside RPATH.
Linux uses a convention for naming shared libraries. Now let's link against good.out:
$ gcc -shared -fPIC -o libexlib.so exlib.c
$ gcc -I /tmp -lexlib main.c -o good.out
$ ldd good.out
linux-vdso.so.1 (0x00007ffcb01bf000)
libexlib.so => not found
libc.so.6 => /usr/lib/libc.so.6 (0x00007fc1230ef000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fc1236ad000)
Now you see that good.out is linked against libexlib.so. gcc searched for alibrary named libexlib.so inside /tmp directory when linking. I can run good.out by specyfing LD_LIBRARY_PATH to the path libexlib.so resides:
$ LD_LIBRARY_PATH=/tmp ldd ./good.out
external_function
Make seems to compare a file modification time with some sort of internal cache of modification times. Does anyone know where is this cache located or how to access it?
Use a utility such as strace to see what it's doing. Here we can follow all file actions (-e trace=file) that the make program performs.
Let's say that we have foo.c and a foo which is built from it using a simple Makefile that looks like:
$ cat Makefile
foo: foo.c
Let's run make:
$ strace -e trace=file make
execve("/usr/bin/make", ["make"], [/* 20 vars */]) = 0
...
open("Makefile", O_RDONLY) = 3
stat("Makefile", {st_mode=S_IFREG|0644, st_size=11, ...}) = 0
...
stat("foo", 0x7ffd4373c8c0) = -1 ENOENT (No such file or directory)
stat("foo.c", {st_mode=S_IFREG|0644, st_size=36, ...}) = 0
cc foo.c -o foo
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=27717, si_status=0, si_utime=0, si_stime=0} ---
stat("foo", {st_mode=S_IFREG|0755, st_size=8549, ...}) = 0
+++ exited with 0 +++
Here you see that it checks for the existence of foo and fails to find it, it checks for the existence of foo.c, and it then calls the rule to compile it.
Now you can call make again, and you get slightly different results:
$ strace -e trace=file make
...
stat("foo", {st_mode=S_IFREG|0755, st_size=8549, ...}) = 0
stat("foo.c", {st_mode=S_IFREG|0644, st_size=36, ...}) = 0
make: `foo' is up to date.
+++ exited with 0 +++
If there was some kind of a cache file being used, we'd expect to see an open() and a read() against it. If this cache was being used between runs, then we'd expect to see a write() to it as well.
The cache as such is your file system:
The make program reads the makefile,
determines the list of targets and dependencies,
observes the timestamps for these targets and dependencies,
works backward from the target you specify (or the default target, i.e., the first one in the makefile), to find a target whose dependencies are newer.
it runs the rule for the out-of-date target, and repeats until everything is up to date.
As it performs rules, the target it builds is expected to be updated (newer), but it moves on to the next out-of-date target from its initial inspection of the filesystem even if the target is not actually updated.
Re-executing make forces it to start over with the analysis, using the current state of your file system.
I've installed pbs-drmaa library, but I get an error while running a C program.
Here's the compilation, I've used -L and -I to indicate the proper folders for pbs-drmaa files. As it shows, there's no compilation error:
$ gcc teste_drmaa1.c -L /usr/lib/pbs-drmaa/lib -I /usr/lib/pbs-drmaa/include -ldrmaa -o teste_drmaa1
But when I try to run, I get an error:
$./teste_drmaa1
./teste_drmaa1: error while loading shared libraries: libdrmaa.so.1: cannot open shared object file: No such file or directory
But the file exists in the path given for -L:
ls /usr/lib/pbs-drmaa/lib
libdrmaa.so# libdrmaa.so.1# libdrmaa.so.1.0.10
What am I doing wrong?
Thanks in advance.
Solved running ldconfig.
Now compilation is:
gcc teste_drmaa1.c -ldrmaa -o teste_drmaa1
Is their location hardcoded into gcc code or does gcc just call as and we must have as location in our PATH variable?
And in the latter case, how could we create two completely separate gcc toolchains? I mean, how can we make gcc-A invoke as-A and gcc-B invoke as-B if as-A and as-B are both called as?
Some of the paths (e.g., to cc1) are compiled in. Others (e.g., as) use normal lookup in $PATH. This can vary depending on the options GCC is configured with.
You can tell fairly easily by running with strace, and grepping for exec|stat.
$ strace -f gcc foo.c -o foo |& grep exec
⋮
[pid 24943] execve("/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.6.1/cc1", …
That is a call to cc1 by a compiled-in path, as you can see from the
lack of looking for it. Its also not in $PATH.
[pid 24944] execve("/home/anthony/bin/as", ["as", "--64", "-o", "/tmp/ccCIrcGi.o", "/tmp/ccbw3PkL.s"], [/* 51 vars */]) = -1 ENOENT (No such file or directory)
[pid 24944] execve("/usr/local/bin/as", ["as", "--64", "-o", "/tmp/ccCIrcGi.o", "/tmp/ccbw3PkL.s"], [/* 51 vars */]) = -1 ENOENT (No such file or directory)
[pid 24944] execve("/usr/bin/as", ["as", "--64", "-o", "/tmp/ccCIrcGi.o", "/tmp/ccbw3PkL.s"], [/* 51 vars */]) = 0
That is looking for as in $PATH. You can tell because its trying each
location in $PATH in order.
I've omitted a lot of strace output—even with just stat and exec, its
several pages long.
Running gcc -v will show you some of the compiled-in paths (as part of the configure line).
How could we create two completely separate gcc toolchains?
Compile GCC from source twice, detailed instructions at: Multiple glibc libraries on a single host
Everything is hardcoded and highly coupled as far as I can see, I don't think there is any other decent solution.
Query the GCC search path
You can also query the GCC search path with:
gcc -print-search-dirs | grep -E '^programs' | tr ':' '\n'
sample output:
programs
=/usr/lib/gcc/x86_64-linux-gnu/6/
/usr/lib/gcc/x86_64-linux-gnu/6/
/usr/lib/gcc/x86_64-linux-gnu/
/usr/lib/gcc/x86_64-linux-gnu/6/
/usr/lib/gcc/x86_64-linux-gnu/
/usr/lib/gcc/x86_64-linux-gnu/6/../../../../x86_64-linux-gnu/bin/x86_64-linux-gnu/6/
/usr/lib/gcc/x86_64-linux-gnu/6/../../../../x86_64-linux-gnu/bin/x86_64-linux-gnu/
/usr/lib/gcc/x86_64-linux-gnu/6/../../../../x86_64-linux-gnu/bin/
and a specific program with:
gcc -print-prog-name=cc1
sample output:
/usr/lib/gcc/x86_64-linux-gnu/6/cc1
GCC spec files
It is wort mentioning that what actually determines the final cpp, ld, as are the "spec" files in the GCC source code, see also: What are GCC's passes and invoked programs?
There's an ad-hoc option for that: -B*prefix*, quoting gcc docs:
For each subprogram to be run, the compiler driver first tries the -B prefix, if any. If
that name is not found, or if -B was not specified, the driver tries two standard prefixes,
which are /usr/lib/gcc/ and /usr/local/lib/gcc/. [...]