Trying to load pyd with LoadLibraryEx and got failed - winapi

I have a pyd pytest.pyd, where declared two functions: say_hello and add_numbers. So I want to load this pyd dynamically in C++ with LoadLibraryEx. But, when I try to call initpytest func, it fails.
const char* funcname = "initpytest";
HINSTANCE hDLL = LoadLibraryEx(L"D:\\msp\\myproj\\Test\\pytest.pyd", NULL, LOAD_WITH_ALTERED_SEARCH_PATH);
FARPROC p = GetProcAddress(hDLL, funcname);
(*p)(); // fail
In output error: Fatal Python error: PyThreadState_Get: no current thread
Unhandled exception at 0x00007FFCD0CC286E (ucrtbase.dll) in Test.exe: Fatal program exit requested.
Here code of the extension before generating to pyd:
#include "Python.h"
static PyObject* say_hello(PyObject* self, PyObject* args)
{
const char* msg;
if(!PyArg_ParseTuple(args, "s", &msg))
{
return NULL;
}
printf("This is C world\nYour message is %s\n", msg);
return Py_BuildValue("i", 25);
}
static PyObject* add_numbers(PyObject* self, PyObject* args)
{
double a, b;
if (!PyArg_ParseTuple(args, "dd", &a, &b))
{
return NULL;
}
double res = a + b;
return Py_BuildValue("d", res);
}
static PyMethodDef pytest_methods[] = {
{"say_hello", say_hello, METH_VARARGS, "Say Hello!"},
{"add_numbers", add_numbers, METH_VARARGS, "Adding two numbers"},
{NULL, NULL, 0, NULL}
};
PyMODINIT_FUNC initpytest(void)
{
Py_InitModule("pytest", pytest_methods);
}

In the absence of a proper minimal, reproducible example it's impossible to be certain. However, it's probably because you haven't initialized the interpreter: (see this question for example). You need to call Py_Initialize before using any Python functions.
Can I suggest that you use the normal Python C-API tools for running modules (rather than doing it yourself with LoadLibraryEx!) until you fully understand what how embedding Python works. You might consider PyImport_AppendInittab (before initializing) to set up your function directly and avoid the Python search path.

Related

What is the stack used for in CPython, if anything?

As far as I understand:
The OS kernel (e.g. Linux) always allocates a stack for each system-level thread when a thread is created.
CPython is known for using a private heap for its objects, including presumably the call stack for Python subroutines.
If so, what is the stack used for in CPython, if anything?
CPython is an ordinary C program. There is no magic in running Python script / module / REPL / whatever: every piece of code must be read, parsed, interpreted — in a loop, until it's done. There is whole bunch of processor instructions behind every Python expression and statement.
Every "simple" top-level thing (parsing and production of bytecode, GIL management, attribute lookup, console I/O, etc) is very complex under the hood. If consists of functions, calling other functions, calling other functions... which means there is stack involved. Seriously, check it yourself: some of the source files span few thousand lines of code.
Just reaching the main loop of the interpreter is an adventure on it's own. Here is the gist, sewed from pieces from all around the code base:
#ifdef MS_WINDOWS
int wmain(int argc, wchar_t **argv)
{
return Py_Main(argc, argv);
}
#else
// standard C entry point
#endif
int Py_Main(int argc, wchar_t **argv)
{
_PyArgv args = /* ... */;
return pymain_main(&args);
}
static int pymain_main(_PyArgv *args)
{
// ... calling some initialization routines and checking for errors ...
return Py_RunMain();
}
int Py_RunMain(void)
{
int exitcode = 0;
pymain_run_python(&exitcode);
// ... clean-up ...
return exitcode;
}
static void pymain_run_python(int *exitcode)
{
// ... initializing interpreter state and startup config ...
// ... determining main import path ...
if (config->run_command) {
*exitcode = pymain_run_command(config->run_command, &cf);
}
else if (config->run_module) {
*exitcode = pymain_run_module(config->run_module, 1);
}
else if (main_importer_path != NULL) {
*exitcode = pymain_run_module(L"__main__", 0);
}
else if (config->run_filename != NULL) {
*exitcode = pymain_run_file(config, &cf);
}
else {
*exitcode = pymain_run_stdin(config, &cf);
}
// ... clean-up
}
int PyRun_AnyFileExFlags(FILE *fp, const char *filename, int closeit, PyCompilerFlags *flags)
{
// ... even more routing ...
int err = PyRun_InteractiveLoopFlags(fp, filename, flags);
// ...
}
int PyRun_InteractiveLoopFlags(FILE *fp, const char *filename_str, PyCompilerFlags *flags)
{
// ... more initializing ...
do {
ret = PyRun_InteractiveOneObjectEx(fp, filename, flags);
// ... error handling ...
} while (ret != E_EOF);
// ...
}

std::thread wrapper with run method

I am trying to write a wrapper for std::thread with a run method , which will allow thread to execute only once run is called.
class ThreadRAII_WITHRUN {
public:
enum class DtorAction { join, detach };
template< class Function, class... Args >
ThreadRAII_WITHRUN(DtorAction a,Function&& f, Args&&... args)
: action(a)
, t(std::thread([&](){pro.get_future().wait(); std::forward<Function>(f)(std::forward<Args>(args)...);}))
{
}
void run()
{
pro.set_value();
}
~ThreadRAII_WITHRUN()
{
if (t.joinable()) { // joinability test
if (action == DtorAction::join) {
t.join();
} else {
t.detach();
}
}
}
ThreadRAII_WITHRUN(ThreadRAII_WITHRUN&&) = default; // support
ThreadRAII_WITHRUN& operator=(ThreadRAII_WITHRUN&&) = default; // moving
std::thread& get() { return t; }
private: // as before
DtorAction action;
std::promise<void> pro;
std::thread t;
};
This code compiles with gcc6.1.0 but not with gcc4.8.5
with gcc4.8.5 with -std=c++11 flag I get below error.
g++ -std=c++11 thread.cpp -pthread
thread.cpp: In lambda function:
thread.cpp:74:94: error: parameter packs not expanded with â...â:
, t(std::thread([&](){pro.get_future().wait(); std::forward<Function>(f)(std::forward<Args>(args)...);}))
^
thread.cpp:74:94: note: âargsâ
thread.cpp: In instantiation of âstruct ThreadRAII_WITHRUN::ThreadRAII_WITHRUN(ThreadRAII_WITHRUN::DtorAction, Function&&, Args&& ...) [with Function = threadwithraii_run()::__lambda10; Args = {int}]::__lambda9â:
thread.cpp:74:106: required from âThreadRAII_WITHRUN::ThreadRAII_WITHRUN(ThreadRAII_WITHRUN::DtorAction, Function&&, Args&& ...) [with Function = threadwithraii_run()::__lambda10; Args = {int}]â
thread.cpp:104:117: required from here
thread.cpp:74:94: error: using invalid field âThreadRAII_WITHRUN::ThreadRAII_WITHRUN(ThreadRAII_WITHRUN::DtorAction, Function&&, Args&& ...)::__lambda9::__argsâ
does anyboday know the reason for the same?
A bit off-topic, that lambda captures constructor arguments by references. By the time run is called some or all of these references may have become invalid. Capturing arguments by value may be safer, e.g.:
std::thread([=](){ ... }))
^
|
capture by value

std::list::remove_if goes crazy if combined with a generic lambda

I found a problem that I guess is due to a bug in GCC.
Anyway, before opening an issue, I would like to be sure.
Consider the code below:
#include<algorithm>
#include<list>
template<typename U>
struct S {
using FT = void(*)();
struct T { FT func; };
template<typename>
static void f() { }
std::list<T> l{ { &f<int> }, { &f<char> } };
void run() {
l.remove_if([](const T &t) { return t.func == &f<int>; }); // (1)
l.remove_if([](const auto &t) { return t.func == &f<int>; }); // (2)
}
};
int main() {
S<void> s;
s.run();
}
clang v3.9 compiles both (1) and (2) as expected.
GCC v6.2 compiles (1), but it doesn't compile (2).
The returned error is:
error: 'f' was not declared in this scope
Moreover, note that GCC compiles (2) if it is modified as it follows:
l.remove_if([](const auto &t) { return t.func == &S<U>::f<int>; }); // (2)
As far as I know, using an const auto & instead of const T & should not alter the behavior in this case.
Is it a bug of GCC?
Per [expr.prim.lambda]:
8 - [...] [For] purposes of name lookup (3.4) [...] the compound-statement is considered in the context of the lambda-expression. [...]
MCVE:
template<int>
struct S {
template<int> static void f();
S() { void(*g)(char) = [](auto) { f<0>; }; }
};
S<0> s;
Hoisting the compound-statement to the context of the lambda-expression gives a clearly valid program:
template<int>
struct S {
template<int> static void f();
S() { f<0>; }
};
S<0> s;
So yes, this is a bug in gcc.

performing static cast from a non template base class to a templated derived class which has variadic template parameter (c++)

I wanted to store a vector of function pointers, each taking different no. of arguments in a class "Store". So, wrote a templated class "Func" that would store the function as a std::function and its arguments in a tuple.
I derived this "Func" class from a non-template base class "IFunc", so that i can store a vector of pointers to this base class in the class "Store".
template<typename... Args>
class Func : public IFunc
{
public:
std::function<void (Args...)> f;
std::tuple<Args...> args;
template <typename F,typename... Ar>
Func(F&& func,Ar&&... arg): f(std::forward<F>(func)),args(std::make_tuple(std::forward<Ar>(arg)...))
{
}
virtual ~NonMemfun()
{
}
//other methods to unpack the tuple and call the function
};
The IFunc class:
class IFunc
{
public:
Ifunc(){}
virtual ~Ifunc(){}
};
The Store class:
class Store
{
std::vector<Ifunc*> funcs;
public:
template<typename... Args,typename... Args2>
void registerfunc(std::string name,int runs,void(*f)(Args...),Args2&&... arg)
{
Func<Args2...>* sample = new Func<Args2...>(f,arg...);
Ifunc* fp = sample;
funcs.push_back(fp);
}
};
I want to iterate through the vector and call each function. To do that i need to do a static cast like this:
Func<>* der = static_cast<Func<>*>(funcs[0]);
When i try to do this, the cast doesn't happen properly. I cannot specify the template paramenters(variadics) since this class(Store) is not aware of them.
I am totally stuck at this place. Something is wrong with my design i guess. Can someone please suggest me a better way to do this. Thank you.
Rather than trying to do a cast from IFunc to Func<>, you should make a pure virtual function, Apply() in IFunc, which Func<> defines as apply(f, args...);. As you iterate over the vector of IFunc pointers, simply call IFunc->Apply(), which will dispatch to the Func<>::Apply() and do the actual apply.
I'm not much of a C++ programmer, but I think you may find this useful.
I'm sure you know that templates are a compile time thing in C++ so your functions need to be known at build time.
With that said, if you do know your functions and you just want to map them to say a string command and then dynamically bind arguments from something like a stream then this code should help you. It is actually able to use a dynamic_cast to retrieve the command from the map.
this snippet is from a school project I did a while back that had a similar goal:
#include <map>
#include <string>
#include <sstream>
#include <tuple>
using namespace std;
class Shell {
class Command {
public:
virtual ~Command() {};
virtual void executeWithArgStream(Shell*, istream& s)=0;
};
template <typename... ArgTypes>
class ShellCommand : public Command {
private:
// FIXME: its probably more apropriate for FuncType to return an int for exit code...
typedef function<void(Shell*, ArgTypes...)> FuncType;
FuncType _f;
tuple<ArgTypes...> args;
template<int... Is>
struct seq { };
template<int N, int... Is>
struct gen_seq : gen_seq<N - 1, N - 1, Is...> { };
template<int... Is>
struct gen_seq<0, Is...> : seq<Is...> { typedef seq<Is...> type; };
template<size_t I = 0, class ...P>
typename std::enable_if<I == sizeof...(P)>::type
// template for functions with no arguments
parseArgs(istream& is, std::tuple<P...> &) {}
template<size_t I = 0, class ...P>
typename std::enable_if<I < sizeof...(P)>::type
parseArgs(istream& is, std::tuple<P...> & parts) {
// this is the magic bit that takes a tuple of pointers (representing the command arguments)
// created at compile time and creates new instances of each argument type and populates it from
// the given input stream :D
auto& part = std::get<I>(args);
// hmmm should we delete or recycle...
delete part;
part = new typeof(*part);
is >> *part;
parseArgs<I + 1>(is, parts);
}
template<int ...S>
void callFunc(Shell* shell, seq<S...>) {
_f(shell, get<S>(args) ...);
}
public:
static constexpr size_t numArgs = sizeof...(ArgTypes);
ShellCommand(FuncType f) : _f(f) {};
void operator()(Shell* shell, ArgTypes... args) {
_f(shell, args...);
};
void executeWithArgStream(Shell* shell, istream& s)
{
parseArgs(s, args);
callFunc(shell, typename gen_seq<sizeof...(ArgTypes)>::type());
};
};
private:
typedef shared_ptr<Command> CommandPtr;
typedef map<string, CommandPtr> FMap;
FMap _cmdMap;
ostream& _out;
istream& _in;
public:
Shell(istream& is = cin, ostream& os = cout)
: _out(os), _in(is)
{
// populate
_cmdMap.insert(pair<string, CommandPtr>("chdir", make_shared<ShellCommand<string*>>(&Shell::chdir)));
_cmdMap.insert(pair<string, CommandPtr>("list", make_shared<ShellCommand<>>(&Shell::list)));
_cmdMap.insert(pair<string, CommandPtr>("count", make_shared<ShellCommand<>>(&Shell::count)));
};
int run();
// FIXME: its probably more apropriate for execute to return an int for exit code...
template <typename... ArgTypes>
void execute(string& command, ArgTypes... args);
void executeWithArgStream(string& command, istream& istr);
// shell commands:
// any command parameters must be done as a pointer!
// the magic that parses string arguments into real types depends on it!
void list() {
list command
};
void chdir(string* dir) {
// chdir command
};
void count() {
// count command
};
};
template <typename... ArgTypes>
void Shell::execute(string& command, ArgTypes... args)
{
typedef ShellCommand<ArgTypes...> CommandType;
CommandType* c = dynamic_cast<CommandType*>(_cmdMap[command].get());
// TODO: neeed to diferentiate between invalid commands and some kind of dynamic_cast failure
if (c) {
(*c)(this, args...);
} else {
// dynamic cast failure
throw runtime_error("Broken Implementation for:" + command);
}
}
void Shell::executeWithArgStream(string& command, istream& istr)
{
Command* c = _cmdMap[command].get();
if (c) {
c->executeWithArgStream(this, istr);
} else {
throw runtime_error("Invalid Shell Command: " + command);
}
}
int Shell::run()
{
do {
string cmd, argString;
_out << _currentDir->name() << "> ";
_in.clear();
_in >> cmd;
if (cmd == "q") {
return 0;
}
if (_in.peek() == ' ')
_in.ignore(1, ' ');
getline(cin, argString);
if (_cmdMap[cmd]) {
try {
if (argString.length()) {
istringstream s(argString);
executeWithArgStream(cmd, s);
} else {
execute(cmd);
}
} catch (runtime_error& e) {
_out << e.what() << endl;
}
} else {
_out << "unrecognized command: " << cmd << endl;
}
} while (true);
}
int main(int argc, const char * argv[])
{
// start the interactive "shell"
Shell shell();
return shell.run();
}

Is there a way to create an elegant class-member window-function?

The Window-Procedure in the Win32 API must be static \ global function since it cannot take a class-object (the this) parameter. One can of-course use workarounds like a hWnd->object dictionary and such.
I wonder if D has a way to elegantly solve it, like create a tiny member function copy for each object (to call the object's real handler) or anonymous function that I can assign to WNDCLASS.lpfnWndProc (I know there are anonymous functions, but I cannot use the extern(Windows) property on them) ?
Can I do something like this :
class Window {
extern (Windows)
LRESULT delegate (HWND hWnd, UINT msg, WPARAM w, LPARAM l) MyWinProcDelegate;
this() {
MyWinProcDelegate = &Events;
}
extern (Windows)
LRESULT Events (HWND hWnd, UINT msg, WPARAM w, LPARAM l) {
MessageBoxA(null , "Success!!!" , null ,0);
return DefWindowProcA(hWnd, message, wParam, lParam);
}
}
(Omitting the registration\creation\msg-loop...)
The Events() doesn't seem to fire... am I missing something ?
Here I made this for you (based on BCS' answer):
version (Windows)
{
import std.c.windows.windows;
void makeExecutable(ubyte[] code)
{
DWORD old;
VirtualProtect(code.ptr, code.length, PAGE_EXECUTE_READWRITE, &old);
}
}
else
version (linux)
{
import core.sys.posix.sys.mman;
import core.sys.posix.unistd;
static if (!is(typeof(&mprotect)))
extern(C) int mprotect(void*, size_t, int);
void makeExecutable(ubyte[] code)
{
auto pageSize = sysconf(_SC_PAGE_SIZE);
auto address = ((cast(size_t)code.ptr) & ~(pageSize-1));
int pageCount =
(address/pageSize == (address+code.length)/pageSize) ? 1 : 2;
mprotect(cast(void*)address, pageSize * pageCount,
PROT_READ | PROT_WRITE | PROT_EXEC);
}
}
else
static assert(0, "TODO");
R function(A) delegate2function(R, A...)(R delegate(A) d)
{
enum size_t TEMPLATE1 = cast(size_t)0x01234567_01234567;
enum size_t TEMPLATE2 = cast(size_t)0x89ABCDEF_89ABCDEF;
static R functionTemplate(A args)
{
R delegate(A) d;
d.ptr = cast(typeof(d.ptr ))TEMPLATE1;
d.funcptr = cast(typeof(d.funcptr))TEMPLATE2;
return d(args);
}
static void functionTemplateEnd() {}
static void replaceWord(ubyte[] a, size_t from, size_t to)
{
foreach (i; 0..a.length - size_t.sizeof + 1)
{
auto p = cast(size_t*)(a.ptr + i);
if (*p == from)
{
*p = to;
return;
}
}
assert(0);
}
auto templateStart = cast(ubyte*)&functionTemplate;
auto templateEnd = cast(ubyte*)&functionTemplateEnd;
auto templateBytes = templateStart[0 .. templateEnd - templateStart];
// must allocate type with pointers, otherwise GC won't scan it
auto functionWords = new void*[(templateBytes.length / (void*).sizeof) + 3];
// store context in word-aligned boundary, so the GC can find it
functionWords[0] = d.ptr;
functionWords[1] = d.funcptr;
functionWords = functionWords[2..$];
auto functionBytes = (cast(ubyte[])functionWords)[0..templateBytes.length];
functionBytes[] = templateBytes[];
replaceWord(functionBytes, TEMPLATE1, cast(size_t)d.ptr );
replaceWord(functionBytes, TEMPLATE2, cast(size_t)d.funcptr);
makeExecutable(functionBytes);
return cast(typeof(return)) functionBytes.ptr;
}
void main()
{
import std.stdio;
auto context = 42;
void del(string s)
{
writeln(s);
writeln(context);
}
auto f = delegate2function(&del);
f("I am a pretty function");
}
Tested on Windows 32-bit and Linux 64-bit.
How about storing this in the window itself, with SetWindowLong?
One very un-portable solution would be to dynamically create a function that wraps the call. I would do this by writing a function that looks like this:
extern(C) RetType TestFn(Arg arg /* and any others */) {
Class c = cast(Class)(0xDEAD_BEEF);
return c.Method(arg);
}
You can then compile this function as un-optimized PIC, de-compile it, and find a byte sequence that can be mashed into what you need. The end result would be a type (likely a struct) that has a methoud returning a function pointer and that, when constructed, populates an internal void array with the bytes you found from the above step and pokes the object in question into the appropriate places.
A slightly more advanced solution would populate a delegate with both the object and the method pointer so both can be provided to the constructor. An even more advanced solution would template the type and take advantage of knowledge of the C and D calling conventions to dynamically generate the argument forwarding code.

Resources