I am experimenting to print the value pid from the task_strcut using bpf inside the kernel with the following program.
from __future__ import print_function
from bcc import BPF
prog = """
#include <linux/sched.h>
int trace(void *ctx) {
int pid = current->pid;
bpf_trace_printk("val (%d)", pid);
return 0;
}
"""
b = BPF(text=prog)
b.attach_kprobe(event="<a kernel function>", fn_name="trace")
print("PID MESSAGE")
try:
b.trace_print(fmt="{1} {5}")
except KeyboardInterrupt:
exit()
Following is the error:
#define __HAVE_BUILTIN_BSWAP16__
^
<command line>:3:9: note: previous definition is here
#define __HAVE_BUILTIN_BSWAP16__ 1
^
3 warnings generated.
error: invalid operand in inline asm: 'movq %gs:${1:P}, $0' at line 2149017352
This error was fixed upstream in bcc with commit https://github.com/iovisor/bcc/commit/d089013e8c6ee0b82d012c1814f822b00695691f. You'll need bcc version v0.20.0 or newer to have this fix.
In short, the issue was in the order of definitions. bcc would have its fallback macro definition before the kernel's, hence compilation failed because the macro is already defined. Moving the bcc fallback definition last solves this.
Related
I am getting the following error
rudimentary_calc.c: In function ‘main’:
rudimentary_calc.c:9:6: error: conflicting types for ‘getline’
9 | int getline(char line[], int max) ;
| ^~~~~~~
In file included from rudimentary_calc.c:1:
/usr/include/stdio.h:616:18: note: previous declaration of ‘getline’ was here
616 | extern __ssize_t getline (char **__restrict __lineptr,
| ^~~~~~~
when I ran the following code
#include <stdio.h>
#define maxline 100
int main()
{
double sum, atof(char[]);
char line[maxline];
int getline(char line[], int max) ;
sum = 0;
while (getline(line, maxline) > 0)
printf("\t %g \n", sum += atof(line));
return 0;
}
What am I doing wrong? I am very new to C, so I don't know what went wrong.
Generally, you should not have to declare "built-in" functions as long as you #include the appropriate header files (in this case stdio.h). The compiler is complaining that your declaration is not exactly the same as the one in stdio.h.
The venerable K&R book defines a function named getline. The GNU C library also defines a non-standard function named getline. It is not compatible with the function defined in K&R. It is declared in the standard <stdio.h> header. So there is a name conflict (something that every C programmer has do deal with).
You can instruct GCC to ignore non-standard names found in standard headers. You need to supply a compilation flag such as -std=c99 or -std=c11 or any other std=c<year> flag that yout compiler supports.
Live demo
Always use one of these flags, plus at least -Wall, to compile any C code, including code from K&R. You may encounter some compiler warnings or even errors. This is good. Thy will tell you that there are some code constructs that were good in the days of K&R, but are considered problematic now. You want to know about those. The book is rather old and the best practices and the C language itself have evolved since.
Is it possible in Pybind11 to use mpi4py on the Python side and then to hand over the communicator to the C++ side?
If so, how would it work?
If not, is it possible for example with Boost? And if so, how would it be done?
I searched the web literally for hours but didn't find anything.
Passing an mpi4py communicator to C++
using pybind11 can be done using the
mpi4py C-API. The corresponding header files can be located using the
following Python code:
import mpi4py
print(mpi4py.get_include())
To convenietly pass communicators between Python and C++, a custom pybind11
type caster
can be implemented. For this purpose, we start with the typical preamble.
// native.cpp
#include <pybind11/pybind11.h>
#include <mpi.h>
#include <mpi4py/mpi4py.h>
namespace py = pybind11;
In order for pybind11 to automatically convert a Python type to a C++ type,
we need a distinct type that the C++ compiler can recognise. Unfortunately,
the MPI standard does not specify the type for MPI_comm. Worse, in
common MPI implementations MPI_comm can be defined as int or void*
which the C++ compiler cannot distinguish from regular use of these types.
To create a distinct type, we define a wrapper class for MPI_Comm which
implicitly converts to and from MPI_Comm.
struct mpi4py_comm {
mpi4py_comm() = default;
mpi4py_comm(MPI_Comm value) : value(value) {}
operator MPI_Comm () { return value; }
MPI_Comm value;
};
The type caster is then implemented as follows:
namespace pybind11 { namespace detail {
template <> struct type_caster<mpi4py_comm> {
public:
PYBIND11_TYPE_CASTER(mpi4py_comm, _("mpi4py_comm"));
// Python -> C++
bool load(handle src, bool) {
PyObject *py_src = src.ptr();
// Check that we have been passed an mpi4py communicator
if (PyObject_TypeCheck(py_src, &PyMPIComm_Type)) {
// Convert to regular MPI communicator
value.value = *PyMPIComm_Get(py_src);
} else {
return false;
}
return !PyErr_Occurred();
}
// C++ -> Python
static handle cast(mpi4py_comm src,
return_value_policy /* policy */,
handle /* parent */)
{
// Create an mpi4py handle
return PyMPIComm_New(src.value);
}
};
}} // namespace pybind11::detail
Below is the code of an example module which uses the type caster. Note, that
we use mpi4py_comm instead of MPI_Comm in the function definitions
exposed to pybind11. However, due to the implicit conversion, we can use
these variables as regular MPI_Comm variables. Especially, they can be
passed to any function expecting an argument of type MPI_Comm.
// recieve a communicator and check if it equals MPI_COMM_WORLD
void print_comm(mpi4py_comm comm)
{
if (comm == MPI_COMM_WORLD) {
std::cout << "Received the world." << std::endl;
} else {
std::cout << "Received something else." << std::endl;
}
}
mpi4py_comm get_comm()
{
return MPI_COMM_WORLD; // Just return MPI_COMM_WORLD for demonstration
}
PYBIND11_MODULE(native, m)
{
// import the mpi4py API
if (import_mpi4py() < 0) {
throw std::runtime_error("Could not load mpi4py API.");
}
// register the test functions
m.def("print_comm", &print_comm, "Do something with the mpi4py communicator.");
m.def("get_comm", &get_comm, "Return some communicator.");
}
The module can be compiled, e.g., using
mpicxx -O3 -Wall -shared -std=c++14 -fPIC \
$(python3 -m pybind11 --includes) \
-I$(python3 -c 'import mpi4py; print(mpi4py.get_include())') \
native.cpp -o native$(python3-config --extension-suffix)
and tested using
import native
from mpi4py import MPI
import math
native.print_comm(MPI.COMM_WORLD)
# Create a cart communicator for testing
# (MPI_COMM_WORLD.size has to be a square number)
d = math.sqrt(MPI.COMM_WORLD.size)
cart_comm = MPI.COMM_WORLD.Create_cart([d,d], [1,1], False)
native.print_comm(cart_comm)
print(f'native.get_comm() == MPI.COMM_WORLD '
f'-> {native.get_comm() == MPI.COMM_WORLD}')
The output should be:
Received the world.
Received something else.
native.get_comm() == MPI.COMM_WORLD -> True
This is indeed possible. As was pointed out in the comments by John Zwinck, MPI_COMM_WORLD will automatically point to the correct communicator, so nothing has to be passed from python to the C++ side.
Example
First we have a simple pybind11 module that does expose a single function which simple prints some MPI information (taken from one of the many online tutorials). To compile the module see here pybind11 cmake example.
#include <pybind11/pybind11.h>
#include <mpi.h>
#include <stdio.h>
void say_hi()
{
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
char processor_name[MPI_MAX_PROCESSOR_NAME];
int name_len;
MPI_Get_processor_name(processor_name, &name_len);
printf("Hello world from processor %s, rank %d out of %d processors\n",
processor_name,
world_rank,
world_size);
}
PYBIND11_MODULE(mpi_lib, pybind_module)
{
constexpr auto MODULE_DESCRIPTION = "Just testing out mpi with python.";
pybind_module.doc() = MODULE_DESCRIPTION;
pybind_module.def("say_hi", &say_hi, "Each process is allowed to say hi");
}
Next the python side. Here I reuse the example from this post: Hiding MPI in Python and simply put in the pybind11 library. So first the python script that will call the MPI python script:
import sys
import numpy as np
from mpi4py import MPI
def parallel_fun():
comm = MPI.COMM_SELF.Spawn(
sys.executable,
args = ['child.py'],
maxprocs=4)
N = np.array(0, dtype='i')
comm.Reduce(None, [N, MPI.INT], op=MPI.SUM, root=MPI.ROOT)
print(f'We got the magic number {N}')
And the child process file. Here we simply call the library function and it just works.
from mpi4py import MPI
import numpy as np
from mpi_lib import say_hi
comm = MPI.Comm.Get_parent()
N = np.array(comm.Get_rank(), dtype='i')
say_hi()
comm.Reduce([N, MPI.INT], None, op=MPI.SUM, root=0)
The end result is:
from prog import parallel_fun
parallel_fun()
# Hello world from processor arch_zero, rank 1 out of 4 processors
# Hello world from processor arch_zero, rank 2 out of 4 processors
# Hello world from processor arch_zero, rank 0 out of 4 processors
# Hello world from processor arch_zero, rank 3 out of 4 processors
# We got the magic number 6
I am learning KLEE now and I wrote a simple code:
#include "klee/klee.h"
#include <stdio.h>
#include <stdlib.h>
int test(int *p)
{
int *q = (int *) malloc(sizeof(int));
if ((*p) == (*q)) {
printf("reading uninitialized heap memory");
}
return 0;
}
int main()
{
int *p = (int *) malloc(sizeof(int));
test(p);
return 0;
}
First, I generate LLVM bitcode, and then I execute KLEE to the bitcode.
Following is all output:
KLEE: output directory is "/Users/yjy/WorkSpace/Test/klee-out-13"
Using STP solver backend
KLEE: WARNING: undefined reference to function: printf
KLEE: WARNING ONCE: calling external: printf(140351601907424)
reading uninitialized heap memory
KLEE: done: total instructions = 61
KLEE: done: completed paths = 4
KLEE: done: generated tests = 4
I suppose that KLEE should give me an error that the q pointer is not initialized, but it doesn't. Why KLEE does not give me an error or warning about this? KLEE can not detect this error? Thanks in advance!
TLTR: KLEE has not implemented this feature. Clang can check this directly.
KLEE currently support add/sub/mul/div overflow checking. To use this feature, you have to compile the source code with clang -fsanitize=signed-integer-overflow or clang -fsanitize=unsigned-integer-overflow .
The idea is that a function call is inserted into the bytecode (e.g. __ubsan_handle_add_overflow) when you use the clang sanitizer. Then KLEE will handle the overflow checking once it meets the function call.
Clang support
MemorySanitizer, AddressSanitizer UndefinedBehaviorSanitizer. They are defined in projects/compiler-rt/lib directory. MemorySanitizer is the one you are looking for, which is a detector of uninitialized reads.
You can remove the KLEE function call and check with clang directly.
➜ ~ clang -g -fsanitize=memory st.cpp
➜ ~ ./a.out
==16031==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x490954 (/home/hailin/a.out+0x490954)
#1 0x7f21b72f382f (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
#2 0x41a1d8 (/home/hailin/a.out+0x41a1d8)
SUMMARY: MemorySanitizer: use-of-uninitialized-value (/home/hailin/a.out+0x490954)
Exiting
I am trying to run sample rsa/dsa code using libtomcrypt.
I have installed LibTomMath first as make install, as a result following files are created.
/usr/lib/libtommath.a
/usr/include/tommath.h
After that I installed libtomcrypt with LibTomMath as external library
CFLAGS="-DLTM_DESC -DUSE_LTM -I/usr/include" EXTRALIBS="/usr/lib/libtommath.a " make install
As a result following file is created
/usr/lib/libtomcrypt.a
I am not getting any error while running following command
CFLAGS="-DLTM_DESC -DUSE_LTM -I/usr/include" EXTRALIBS="/usr/lib/libtommath.a " make test
I have gone through this document libtomcrypt_installation and libtomcrypt_resolved to successfully compile using
gcc -DLTM_DESC rsa_make_key_example.c -o rsa -ltomcrypt
or
gcc rsa_make_key_example.c -o rsa -ltomcrypt
no compile error. However when I try to run, I got following error.
./rsa
LTC_ARGCHK 'ltc_mp.name != NULL' failure on line 34 of file src/pk/rsa/rsa_make_key.c
Aborted
Here is my sample rsa code
#include <tomcrypt.h>
#include <stdio.h>
int main(void) {
# ifdef USE_LTM
ltc_mp = ltm_desc;
# elif defined (USE_TFM)
ltc_mp = tfm_desc;
# endif
rsa_key key;
int err;
register_prng(&sprng_desc);
if ((err = rsa_make_key(NULL, find_prng("sprng"), 1024/8, 65537,&key)) != CRYPT_OK) {
printf("make_key error: %s\n", error_to_string(err));
return -1;
}
/* use the key ... */
return 0;
}
Here is my sample dsa code
#include <tomcrypt.h>
#include <stdio.h>
int main(void) {
# ifdef USE_LTM
ltc_mp = ltm_desc;
# elif defined (USE_TFM)
ltc_mp = tfm_desc;
# endif
int err;
register_prng(&sprng_desc);
dsa_key key;
if ((err = dsa_make_key(NULL, find_prng("sprng"), 20, 128,&key)) != CRYPT_OK) {
printf("make_key error: %s\n", error_to_string(err));
return -1;
}
/* use the key ... */
return 0;
}
Here is how I have compiled it successfully,
gcc dsa_make_key_example.c -o dsa -ltomcrypt
When I try to run the code , I am getting following error .
./dsa
segmentation fault
EDIT 1:
I investigated further and found the reason for segmentation fault
#ifdef LTC_MPI
#include <stdarg.h>
int ltc_init_multi(void **a, ...)
{
...
...
if (mp_init(cur) != CRYPT_OK) ---> This line causes segmentation fault
Where am I making mistakes ? How to resolve this problem to run these programs successfully?
I am using linux , gcc. Any help/link will be highly appreciated. Thanks in advance.
It's been a year or so since this was asked, but I have some component of an answer, and a workaround.
The reason mp_init fails is that the "math_descriptor" is uninitialized. mp_init is a defined as
#define mp_init(a) ltc_mp.init(a)
where ltc_mp is a global struct (of type ltc_math_descriptor) that holds pointers to the math routines.
There are several implementations of the math routines available, and a user can choose which they want. For whatever reason, there does not seem to be a default math implementation chosen for certain builds of libtomcrypt. Thus, the init member of ltc_mp is null, and we get the SIGSEGV.
Here is a manual workaround:
You can make your desired ltc_math_descriptor struct available to your main() routine by #defineing one of
LTM_DESC -- built-in math lib
TFM_DESC -- an external fast math package
GMP_DESC -- presumably a GNU MultiPrecision implementation?
Before #include <tomcrypt.h> (or by using -D on the command-line).
Whichever you choose, a corresponding object will be declared:
extern const ltc_math_descriptor ltm_desc;
extern const ltc_math_descriptor tfm_desc;
extern const ltc_math_descriptor gmp_desc;
To use it, manually copy it to the global math descriptor:
E.g., in my case, for the local math imlpementation,
ltc_mp = ltm_desc;
Now libtomcrypt works.
I encountered "SYS#0" at the top of a stack and cannot find any documentation as to what that means.
Compiler: g++
OS: Solaris 9
Arch: SPARC
Memory Manager libhoard_32.so from Hoard 3.5.1
We used "gcore" to generate a core file. Looking at the output of running the "pstack" command against the core file, the only thread that was doing anything interesting had the following at the very top of its call stack:
ff309858 SYS#0 ()
ff309848 void MyHashMap<const void*,unsigned,AlignedMmapInstance<65536U>::SourceHeap>::set(const void*,unsigned) (ff31eed4, 9bf20000, 10000, 40, 9bf1fff0, ff31e738) + 134
...
pflags for that LWP shows:
/8: flags = PR_STOPPED|PR_ISTOP|PR_ASLEEP
why = PR_REQUESTED
sigmask = 0xfffffeff,0x00003fff
I could not find any mention of this syntax in the Sun documentation.
Edit: The process appears to have hung sometime prior to doing the gcore. Is "SYS#0" somehow interrelated with process hangs?
Edit: Added next stack frame and link to Hoard, pflags output
Edit: The accepted answer is correct. In addition, at least on SPARC, the g1 register should contain the system call number, but this did not appear to be the case in our core file.
The topic "what is an indirect system call?" is probably good material for another question.
Try this:
$ cat foo.c
#include <stdio.h>
int main(int argc, char *argv[]) {
char buf[1024];
proc_sysname(0, buf, 1024);
printf("%s\n", buf);
}
$ gcc -ofoo -lproc foo.c
$ ./foo
SYS#0
$
SYS#0 is therefore the string that represents system call zero. If you look in <sys/syscall.h> (the system call table) you will find the following:
/* syscall enumeration MUST begin with 1 */
/*
* SunOS/SPARC uses 0 for the indirect system call SYS_syscall
* but this doesn't count because it is just another way
* to specify the real system call number.
*/
#define SYS_syscall 0
The indirect system call syscall(SYS_syscall, foo, bar, ...) is equivalent to the direct call syscall(foo, bar, ...).