I started to use C++11 std::thread (mingw 4.8) so far so good. I ran into a situation with overlapped I/O where sleepEx was used to put the thread in an alertable wait state. This worked quite well, until QueueUserAPC had to be used, which returned an "invalid handle error".
After some searching found out that std::thread uses the pthread library under Windows.
Is there any way to use windows API calls which expect a thread handle with std::thread ?
Or do I need to stick with Windows threads for overlapped I/O ?
To solve your issue, MinGW-w64 winpthreads (the pthreads implementation you are using), just like pthreads-win32, allows you to get the native Win32 thread handle for a pthread:
void * pthread_gethandle (pthread_t t);
Note that this is currently an undocumented function.
The corresponding function in pthreads-win32 is:
HANDLE pthread_getw32threadhandle_np(pthread_t thread);
I'd bet this will make your intermixing of the two work, or at least bring to light some bugs in winpthreads which can be fixed. In the latter case, please report them to MinGW-w64.
If the above returns an invalid handle, your best bet is to ask on the MinGW-w64-public mailing list (subscribe first, otherwise you'll have to wait for manual moderation which is silly).
Is there any way to use windows API calls which expect a thread handle with std::thread ?
No, because the std::thread in your MinGW build isn't implemented in terms of thread handles. Edit: it is, but indirectly, see rubenvb's answer for how to get the native thread handle from a pthread_t, and you should be able to use std::thread::native_handle() to get the pthread_t.
Noone has implemented the necessary support in GCC for the C++11 thread library to use native Windows threads directly.
I had some ideas for a new thead model that would be implemented in terms of native mutexes and condition variables. That would allow you to call std::thread::native_handle() to get the underlying thread handle to use with the Windows API.
I got as far as rebuilding GCC with my changes applied, but couldn't test them. There was almost no interest in my suggestions and no offers to help from any MinGW contributors, so as I'm not a Windows user, and working on Windows and building MinGW was so painful and frustrating, I gave up. I should put my changes online somewhere, so that someone with more patience than me can finish the work one day.
There is already a native win32 implementation of std::thread and sync primitives, see:
https://github.com/meganz/mingw-std-threads
This is a header-only library and works with any version of MinGW that has proper language support for C++11
Related
In C/C++, Windows executables are linked against static libraries that import DLL files containing Windows API procedures.
But how do we access those procedures from Forth code (e.g. GForth)? Is it possible at all?
I'm aware that there's Win32Forth capable of doing Win32 stuff, but I'm interested how (and if) this could be done in Forth implementations that lack this functionality from the box (yet do run on target OS and are potentially able to interact with it on a certain level).
What currently comes up to my mind is loading the DLL files in question and somehow locating the address of a procedure to execute - but then, execute how? (All I know is that Windows API uses the stdcall
convention). And how do we locate a procedure without a C header? (I'm very new to Forth and just a bit less new to C++. Please bear with me if my musings are nonsense).
In general case, to implement foreign functions interface (FFI) for dynamically loaded libraries in some Forth system as extension (i.e., without changing source code and recompilation), we need the dlopen and dlsym functions, Forth assembler, and intimate knowledge of the Forth-system organization and ABI.
Sometimes it could be done even without assembler. For example, though SP-Forth has FFI, foreign calls were also implemented in pure Forth as a result of native code generation and union of the return stack with the native hardware stack.
Regarding Gforth, it seems that in the version 0.7.9 (see releases) it doesn't have FFI for stdcall calling convention out of the box (it supports cdecl only), although it has dlopen and dlsym, and an assembler. So, it should be feasible to implement FFI for stdcall.
Yes, you could do this in Gforth according to its documentation. The biggest problem will be dealing with call backs, which the Windows API relies on rather heavily. There is an unsupported package to deal with this, see 5.25.6 Callbacks. I have not attempted this myself in Gforth, but the documentation looks adequate.
You might also want to check MPE's VFXForth. From their website:
Windows API Access
VFX Forth can access all the standard Windows API calls, as well as functions in any other DLLs. The function interface allows API calls to be defined by cut and paste from other language reference manuals, for example:
EXTERN: int PASCAL CreateDialogIndirectParam( HINSTANCE, void *,HWND, WNDPROC, LPARAM );
EXTERN: int PASCAL SetWindowText( HANDLE, LPSTR );
EXTERN: HANDLE PASCAL GetDlgItem( HANDLE, int );
This is down the page a bit at VFX Forth for Windows.
As I do my Forth on Mac and Linux, I can't work through the Windows for Gforth to provide more detail, sorry.
Gforth 0.7.9 provides Windows API calls generated by Swig from the Windows header files. The C interface uses a wrapper library, which is compiled by the C compiler, to pass parameters from the Forth stack to the system functions; as the C compiler understands stdcall, and the header files declare Windows API as stdcall, this "just works".
As all pre-generated C bindings live in the directory "unix" (for historical reasons), include unix/win32.fs gives you the win32 part of the Windows API.
Callbacks in the event loop are still a problem, as Gforth is a Cygwin program, and Cygwin has its special event loop task... but I hope that problem can be fixed.
In the MinGW-W64 online installer there are several fields you can select. However I cannot find any documentation on this, and the guesses I've made don't give me the behaviour I want.
Clearly a lot of work has gone into this project so it seems a pity that uptake is being held back by lack of basic documentation.
The "Version" and "Architecture" fields are self-explanatory but the other fields I have trouble with are (values shown as of current installer):
Threads, options posix and win32
Exception, options dwarf and sjlj
Build revision, options 0, 1, 2.
The values I chose on my previous install were win32, seh and 1 (clearly the options have changed since then but I am none the wiser as to what's what).
What are the pros and cons of each option, especially the threading model and exception handling, and which version is "best"?
The specific problems I have encountered using x86_64-win32-seh-rev1 are:
std::thread and std::condition_variable are not supported
When debugging (using Code::Blocks as IDE), if an exception is thrown it does not jump to the exception handler; selecting Next Line does nothing 3 times and then aborts the run.
I can cope with the debugging problem but it would be really nice to have working C++11 threads.
Exceptions
Please see this answer for all three models (dwarf, sjlj and seh).
Threads
You can decide what kind of threads you want to use: POSIX threads or Windows API threads. The posix threads have the advantage of portability; you can use your code on other posix platforms (eg. linux) without modifications. The win32 threading api is windows only. If you are 100% on windows and like it's api that's no problem though.
If you use new C++ features like std::thread the impact is less visible since you already have a standard api for threading. I'm not sure if there's really a big difference if you don't use posix- / win32 thread api directly (maybe std::thread native handles?)
See also: mingw-w64 threads: posix vs win32
Build revision
I guess that's just another version number since Mingw(-w64) follows GCC versions (4.8.x, 4.9.x etc.). If you don't need an specific build, you should use the latest version.
Threading issue
If the exception thrown is:
terminate called after throwing an instance of 'std::system_error'
what(): Enable multithreading to use std::thread: Operation not permitted
then just link pthreads - and the problem is solved.
Recommendation
If you don't have reasons to use a specific option; my personal recommendation:
posix - dwarf - 2
Posix enable C++11 <thread>, <mutex> and <future>
dwarf is faster
2 because it's the latest release
I need to hijack all operating system calls of my own process. I cannot rewrite code as it is partly not my code (plug-ins). I need to be able to decide within my implementation of a specific system call, if I want to call the original implementation or not.
Operating systems will be at first windows xp and higher versions. Later os x 10.5 and higher will follow. Starting on windows with 32 bit versions, later for all operating systems also 64 bit versions.
I found a lot of documentation and tools about hooking other processes but I would hope my job is much simpler and I would hope for some source code.
Thanks a lot in advance, Bernd.
There are many hooking libraries that will let you do this, for example Detours or madCodeHook on Windows. No doubt there are similar libraries on OSX, I just don't know them!
It's very easy to hook a routine and replace it with your own implementation. It's less easy to retain the option of running the original routine in some circumstances, and that's where using a hooking library will take the pain away for you.
On Mac OS X, you can override functions with the DYLD_INTERPOSE macro (and DYLD_INSERT_LIBRARIES, if needed). This answer has an example: Ansi C patch using dlsym compiles OK under linux but fails on Mac Os X
For Windows, there is the open source alternative to Microsoft Detours called EasyHook:
CodePlex: EasyHook
Code Project: EasyHook - The reinvention of Windows API hooking
I am using pthread win32 library to implement mqueue.
But when it runs into following code, it throw #40 error should be ENOSYS, means system not supported.
pthread_mutexattr_setpshared(&mattr, PTHREAD_PROCESS_SHARED);
i = pthread_mutex_init(&mqhdr->mqh_lock, &mattr);
pthread_mutexattr_destroy(&mattr); /* be sure to destroy */
i is 40 after it goes wrong. Any body has idea about this? or do you have some other alternative solution, like use what kind of WIN32 thread function to replace it.
Note: If anyone successfully implement a mqueue in win32?
Thanks
You will want to read up on Windows interprocess synchronization functions.
For an inter-process mutex in Windows, your choices are to implement your own using shared memory and InterlockedCompareExchange (spin then sleep or watch for Event).
Or easier to program but not as performant is to use the OS provided named Mutex object. These perform about 10 times worse than using CriticalSection within threads of a process.
In my own production code I was porting from Linux pthreads, I played with the first solution, but ended up releasing the code using the Mutex solution. It was more reliable and I was sure it would work in all cases.
I recognize the code you are using ...just comment the 2 lines in the code
pthread_mutexattr_setpshared(&mattr, PTHREAD_PROCESS_SHARED);
pthread_condattr_setpshared(&cattr, PTHREAD_PROCESS_SHARED);
...it works fine as a intra-process message queue ...unless you need it across processes.
I don't know if you feel comfortable hacking inside the Win32 PThread library, but, while the full PTHREAD_PROCESS_SHARED behavior cannot be attained, it IS possible to duplicate handles to kernel objects into other processes using the DuplicateHandle API - so it should be possible to add some windows specific extensions (that would compile out in unix builds) that allow a mutex to be shared between processes.
•A child process created by the CreateProcess function can inherit a handle to a mutex object if the lpMutexAttributes parameter of CreateMutex enabled inheritance. This mechanism works for both named and unnamed mutexes.
•A process can specify the handle to a mutex object in a call to the DuplicateHandle function to create a duplicate handle that can be used by another process. This mechanism works for both named and unnamed mutexes.
•A process can specify a named mutex in a call to the OpenMutex or CreateMutex function to retrieve a handle to the mutex object.
I believe that is Aurelio Medina's code from 2000.
Unfortunately, his test code was a single process, so it didn't care if the PTHREAD_PROCESS_SHARED flag was set or not, since pthreads-32 has never supported it. When he built it in 2000, I bet that pthreads did't even throw an error, so his test code run fine.
Unfortunately for all of us, it seems he died in 2013, so he's not going to finish his opus.
I've taken up the torch and rewrote the mutex/signal handling to use native windows mutex and events. Please look here for the code:
https://github.com/marklakata/mqueue-w32
I would like a low-overhead method of monitoring the I/O of a Windows process.
I got several useful answers to Monitoring certain system calls done by a process in Windows. The most promising was about using Windows Performance Toolkit to get a kernel event trace. All necessary information can indeed be pulled from there, but the WPT is a massive overkill for what I need and subsequently has a prohibitive overhead.
My idea was to implement an alternative approach to detecting C/C++ dependency graphs. Usually this is done by passing an option to the compiler (-M, for example). This works fine for compilers and tools which have such an option, but not all of them do, and those who do often implement them differently. So, I implemented an alternative way of doing this on Linux using strace to detect which files are opened. Running gcc (for example) in this way has a 50% overhead (ballpark figure), and I was hoping to figure out a way to do this on windows with a similarish overhead.
The xperf set of tools have two issues which prevents me from using them in this case:
There is no way to monitor file-I/O events for a single process; I have to use the kernel event trace which traces every single process and thus generates huge amounts of data (15Mb for the time it takes to run gcc, YMMV).
As a result of having to use the kernel event trace, I have to run as administrator.
I really don't need events at the kernel level; I suppose I could manage just as well if I could just monitor, say, the Win32 API call CreateFile(), and possibly CreateProcess() if I want to catch forked processes.
Any clever ideas?
Use API hooking. Hooking NtCreateFile and a few other calls in ntdll should be enough. I've had good experience using easyhook as a framework to do the hooking itself - free and open source. Even supports managed hooking (c# etc) if you wanted to do that. It's quite easy to set up.
It's at located at http://easyhook.codeplex.com
Edit: btw detours does not allow 64 bit hooking (unless you buy a license for a nominal price of 10,000USD)
EasyHook does not allow native hooks across a WOW64 boundary. It allows managed hooking across WOW64 boundaries though.
I used Microsoft's Detours in the past to track memory allocations by intercepting particular API calls. You could use it to track CreateFile and CreateProcess.
It seems like Dr. Memory's System Call Tracer for Windows is exactly what I was looking for. It is basically a strace implementation for Windows.