I run a Visual C++ console test program inside the daily build. Every now and then the test would call some function that was changed by other developers improperly, descend into an infinite loop and hang thus blocking the build.
I need a watchdog solution as simple as possible. Here's what I came up with. In the test program entry point I start a separate thread that loops continuosly and checks elapsed time. If some predefined period is exceeded it calls TerminateProcess(). Pseudocode:
DWORD WatchDog( LPVOID)
{
DWORD start = GetTickCount();
while( true ) {
Sleep( ReasonablePeriod );
if( GetTickCount() - start > MaxAllowed ) {
TerminateProcess( GetCurrentProcess(), 0 );
}
}
return 0;
}
Is this solution any worse than a watchdog implemented as a separate master program?
I think it's preferable to implement the watchdog as a separate process. It's easier to re-use it, it's easier to detect if your app crashed and to get its return code.
Related
I thought, if I didn't call the ev_loop_fork in the child, then the watcher in child wouldn't be triggered.
This is my code, I build the ev_loop with EVBACKEND_EPOLL and EVFLAG_NOENV flags.
So there is no EVFLAG_FORKCHECK flag.
Then I comment the ev_loop_fork call in the child.
If everything goes well, I thought the child will not trigger the timeout callback function.
But actually, the output is something like this:
$ 4980 fork 4981
$ time out at 4980
$ time out at 4981
it seemed that the watchers still has been triggered in the child, it behaved the same as call ev_loop_fork .
So what's the problem, thank you.
#include<ev.h>
#include<stdio.h>
#include<unistd.h>
void timeout_cb(EV_P_ ev_timer *w,int revents)
{
printf("time out at %d\n", getpid());
ev_break(EV_A_ EVBREAK_ONE);
}
int main()
{
int ret;
ev_timer timeout_watcher;
struct ev_loop *loop = ev_default_loop(EVBACKEND_EPOLL | EVFLAG_NOENV);
ev_timer_init(&timeout_watcher,timeout_cb,5.5,0.);
ev_timer_start(loop,&timeout_watcher);
ret = fork();
if(ret>0) printf("%d fork %d\n",getpid(),ret);
else if(ret==0)
{
//ev_loop_fork(EV_DEFAULT);
}
else return -1;
ev_run(loop,0);
return 0;
}
The libev manual does not say that after a fork an event loop will be stopped. All it says is that to be sure that the event loop will properly work in the child, you need to call ev_loop_fork(). What's actually happening depends on the backend.
And technically, timers will even be more resilient against forks in most backends: select(), poll(), epoll(), kqueue all allow for specification of a timeout value after which these functions return in case of no event. libev uses this feature to be able to trigger timeouts when they are supposed to be triggered. So there's no need to re-register any file descriptors for timeouts to work.
I am using a Win32 multimedia timer to put a delay between the dispatch of large numbers of UDP packets, but i am finding that the resulting delay is substantially longer than it should be. Delays of ~40ms are sometimes nearer 1000ms, even when using Windows Miltimedia timers and upping the timer resolution. Below is a simplifed version of the code i used:
if( timeGetDevCaps(&tc,sizeof(TIMECAPS)) == TIMERR_NOERROR)
{
timeRes = min( max(tc.wPeriodMin,1), tc.wPeriodMax);
timeBeginPeriod(timeRes);
printf("Timer Res: %u\n", timeRes);
}
/* ... */
while( ptrHead )
{
NALU_t *ptrLink = ptrHead;
unsigned long tsNALU = ptrLink->timestamp - tsFirst;
printf("Timestamp: %umsec\n", ptrLink->timestamp / 90 );
int idxPort;
for(idxPort=0;idxPort<12;idxPort++)
{
ip4Addr.sin_port = htons( 60000 + idxPort );
struct sockaddr *saAddr = (struct sockaddr*)&ip4Addr;
sendto(fdSocket,(char*)ptrLink->ptrData,ptrLink->lenData,
0,saAddr,lenAddr);
}
if( 1 )
{
unsigned long millis = (tsNALU - tsPrev) / 90;
valTime.QuadPart = 10000;
valTime.QuadPart *= millis;
valTime.QuadPart *= -1;
if(SetWaitableTimer(hdlTimer,&valTime,0,NULL,NULL,TIME_ONESHOT))
WaitForSingleObject(hdlTimer,INFINITE);
}
tsPrev = tsNALU;
ptrHead = ptrLink->next;
free( ptrLink );
}
I suspect the problem is that Windows7 no longer guarantees the resolution of timers when signalled by events as opposed to call-backs, but i am loathed to use the latter. Anyone know why even supposedly high-resolution timers in single-threaded test cases are so wildly inaccurate?
If timing is critical it's best to run in a busy loop (you can give up a timeslice every iteration using Sleep(0) if you want), using the QueryPerformanceCounter() API to measure elapsed time.
From subsequent experiments, my best guess is that Windows moving threads between CPU cores (possibly for load-balancing reasons - this is on a Quad-core i7) is disruptive to timing functions. I used SetThreadAffinityMask() to lock my timing-critical thread to one CPU (and my non-timing threads to all other cores), and that has sorted out the problems.
I'm trying and failing to cancel a call to WNetAddConnection2 with CancelSynchronousIo.
The call to CancelSynchronousIo succeeds but nothing is actually cancelled.
I'm using a 32-bit console app running on Windows 7 x64.
Has anyone done this successfully? Am I doing something dumb? Here's a sample console app (which needs to be linked with mpr.lib):
DWORD WINAPI ConnectThread(LPVOID param)
{
NETRESOURCE nr;
memset(&nr, 0, sizeof(nr));
nr.dwType = RESOURCETYPE_ANY;
nr.lpRemoteName = L"\\\\8.8.8.8\\bog";
// result is ERROR_BAD_NETPATH (i.e. the call isn't cancelled)
DWORD result = WNetAddConnection2(&nr, L"pass", L"user", CONNECT_TEMPORARY);
return 0;
}
int _tmain(int argc, _TCHAR* argv[])
{
// Create a new thread to run WNetAddConnection2
HANDLE hThread = CreateThread(0, 0, ConnectThread, 0, 0, 0);
if (!hThread)
return 1;
// Retry the cancel until it fails; keep track of how often
int count = 0;
BOOL ok;
do
{
// Sleep to give the thread a chance to start
Sleep(1000);
ok = CancelSynchronousIo(hThread);
++count;
}
while (ok);
// count will equal two here (i.e. one successful cancellation and
// one failed cancellation)
// err is ERROR_NOT_FOUND (i.e. nothing to cancel) which makes
// sense for the second call
DWORD err = GetLastError();
// Wait for the thread to finish; this takes ages (i.e. the
// WNetAddConnection2 call is not cancelled)
WaitForSingleObject(hThread, INFINITE);
return 0;
}
According to Larry Osterman (I hope he doesn't mind me quoting him): "The question was answered in the comments: wnetaddconnection2 isn’t a simple IOCTL call." So the answer (unfortunately) is no.
First, WNetAddConnection2 is system-wide, not per-process. This is important, as calling WNetAddConnection2 many times can wreck system stability - particularly with explorer.
I use WNetGetResourceInformation first to check if the connection already exists before even thinking of calling it - my process may have previously run and then shutdown. The connection may still exist. When my Windows service(s) needs to add such a connection I use a nasty little trick in order to prevent these totally non-abortable API's from stalling my own service shutdown.
The trick is to run these calls in a separate process: they are system-wide, after all. You can normally wait for the process to complete as if you called the functions yourself but you can terminate the process and give up waiting if you need to abort in order to shutdown.
Sadly, however, certain Windows resources, such as named pipe handles and handles to files open on remote computers, can take about 16 seconds to close following failure or shutdown of a remote machine. CancelSynchronousIo does not seem to even help with those but will likely add additional long delay.
I am doing a performance evaluation between Windows CE and Linux on an arm imx27 board. The code has already been written for CE and measures the time it takes to do different kernel calls like using OS primitives like mutex and semaphores, opening and closing files and networking.
During my porting of this application to Linux (pthreads) I stumbled upon a problem which I cannot explain. Almost all tests showed a performance increase from 5 to 10 times but not my version of win32 events (SetEvent and WaitForSingleObject), CE actually "won" this test.
To emulate the behaviour I was using pthreads condition variables (I know that my implementation doesn't fully emulate the CE version but it's enough for the evaluation).
The test code uses two threads that "ping-pong" each other using events.
Windows code:
Thread 1: (the thread I measure)
HANDLE hEvt1, hEvt2;
hEvt1 = CreateEvent(NULL, FALSE, FALSE, TEXT("MyLocEvt1"));
hEvt2 = CreateEvent(NULL, FALSE, FALSE, TEXT("MyLocEvt2"));
ResetEvent(hEvt1);
ResetEvent(hEvt2);
for (i = 0; i < 10000; i++)
{
SetEvent (hEvt1);
WaitForSingleObject(hEvt2, INFINITE);
}
Thread 2: (just "responding")
while (1)
{
WaitForSingleObject(hEvt1, INFINITE);
SetEvent(hEvt2);
}
Linux code:
Thread 1: (the thread I measure)
struct event_flag *event1, *event2;
event1 = eventflag_create();
event2 = eventflag_create();
for (i = 0; i < 10000; i++)
{
eventflag_set(event1);
eventflag_wait(event2);
}
Thread 2: (just "responding")
while (1)
{
eventflag_wait(event1);
eventflag_set(event2);
}
My implementation of eventflag_*:
struct event_flag* eventflag_create()
{
struct event_flag* ev;
ev = (struct event_flag*) malloc(sizeof(struct event_flag));
pthread_mutex_init(&ev->mutex, NULL);
pthread_cond_init(&ev->condition, NULL);
ev->flag = 0;
return ev;
}
void eventflag_wait(struct event_flag* ev)
{
pthread_mutex_lock(&ev->mutex);
while (!ev->flag)
pthread_cond_wait(&ev->condition, &ev->mutex);
ev->flag = 0;
pthread_mutex_unlock(&ev->mutex);
}
void eventflag_set(struct event_flag* ev)
{
pthread_mutex_lock(&ev->mutex);
ev->flag = 1;
pthread_cond_signal(&ev->condition);
pthread_mutex_unlock(&ev->mutex);
}
And the struct:
struct event_flag
{
pthread_mutex_t mutex;
pthread_cond_t condition;
unsigned int flag;
};
Questions:
Why doesn't I see the performance boost here?
What can be done to improve performance (e.g are there faster ways to implement CEs behaviour)?
I'm not used to coding pthreads, are there bugs in my implementation maybe resulting in performance loss?
Are there any alternative libraries for this?
Note that you don't need to be holding the mutex when calling pthread_cond_signal(), so you might be able to increase the performance of your condition variable 'event' implementation by releasing the mutex before signaling the condition:
void eventflag_set(struct event_flag* ev)
{
pthread_mutex_lock(&ev->mutex);
ev->flag = 1;
pthread_mutex_unlock(&ev->mutex);
pthread_cond_signal(&ev->condition);
}
This might prevent the awakened thread from immediately blocking on the mutex.
This type of implementation only works if you can afford to miss an event. I just tested it and ran into many deadlocks. The main reason for this is that the condition variables only wake up a thread that is already waiting. Signals issued before are lost.
No counter is associated with a condition that allows a waiting thread to simply continue if the condition has already been signalled. Windows Events support this type of use.
I can think of no better solution than taking a semaphore (the POSIX version is very easy to use) that is initialized to zero, using sem_post() for set() and sem_wait() for wait(). You can surely think of a way to have the semaphore count to a maximum of 1 using sem_getvalue()
That said I have no idea whether the POSIX semaphores are just a neat interface to the Linux semaphores or what the performance penalties are.
I know that I can use condition variable to synchronize work between the threads, but is there any class like this (condition variable) to synchronize work between the processes, thanks in advance
Use a pair of named Semaphore objects, one to signal and one as a lock. Named sync objects on Windows are automatically inter-process, which takes care of that part of the job for you.
A class like this would do the trick.
class InterprocessCondVar {
private:
HANDLE mSem; // Used to signal waiters
HANDLE mLock; // Semaphore used as inter-process lock
int mWaiters; // # current waiters
protected:
public:
InterprocessCondVar(std::string name)
: mWaiters(0), mLock(NULL), mSem(NULL)
{
// NOTE: You'll need a real "security attributes" pointer
// for child processes to see the semaphore!
// "CreateSemaphore" will do nothing but give you the handle if
// the semaphore already exists.
mSem = CreateSemaphore( NULL, 0, std::numeric_limits<LONG>::max(), name.c_str());
std::string lockName = name + "_Lock";
mLock = CreateSemaphore( NULL, 0, 1, lockName.c_str());
if(!mSem || !mLock) {
throw std::runtime_exception("Semaphore create failed");
}
}
virtual ~InterprocessCondVar() {
CloseHandle( mSem);
CloseHandle( mLock);
}
bool Signal();
bool Broadcast();
bool Wait(unsigned int waitTimeMs = INFINITE);
}
A genuine condition variable offers 3 calls:
1) "Signal()": Wake up ONE waiting thread
bool InterprocessCondVar::Signal() {
WaitForSingleObject( mLock, INFINITE); // Lock
mWaiters--; // Lower wait count
bool result = ReleaseSemaphore( mSem, 1, NULL); // Signal 1 waiter
ReleaseSemaphore( mLock, 1, NULL); // Unlock
return result;
}
2) "Broadcast()": Wake up ALL threads
bool InterprocessCondVar::Broadcast() {
WaitForSingleObject( mLock, INFINITE); // Lock
bool result = ReleaseSemaphore( mSem, nWaiters, NULL); // Signal all
mWaiters = 0; // All waiters clear;
ReleaseSemaphore( mLock, 1, NULL); // Unlock
return result;
}
3) "Wait()": Wait for the signal
bool InterprocessCondVar::Wait(unsigned int waitTimeMs) {
WaitForSingleObject( mLock, INFINITE); // Lock
mWaiters++; // Add to wait count
ReleaseSemaphore( mLock, 1, NULL); // Unlock
// This must be outside the lock
return (WaitForSingleObject( mSem, waitTimeMs) == WAIT_OBJECT_0);
}
This should ensure that Broadcast() ONLY wakes up threads & processes that are already waiting, not all future ones too. This is also a VERY heavyweight object. For CondVars that don't need to exist across processes I would create a different class w/ the same API, and use unnamed objects.
You could use named semaphore or named mutex. You could also share memory between processes by shared memory.
For a project I'm working on I needed a condition variable and mutex implementation which can handle dead processes and won't cause other processes to end up in a deadlock in such a case. I implemented the mutex with the native named mutexes provided by the WIN32 api because they can indicate whether a dead process owns the lock by returning WAIT_ABANDONED. The next issue was that I also needed a condition variable I could use across processes together with these mutexes. I started of with the suggestion from user3726672 but soon discovered that there are several issues in which the state of the counter variable and the state of the semaphore ends up being invalid.
After doing some research, I found a paper by Microsoft Research which explains exactly this scenario: Implementing Condition Variables with Semaphores . It uses a separate semaphore for every single thread to solve the mentioned issues.
My final implementation uses a portion of shared memory in which I store a ringbuffer of thread-ids (the id's of the waiting threads). The processes then create their own handle for every named semaphore/thread-id which they have not encountered yet and cache it. The signal/broadcast/wait functions are then quite straight forward and follow the idea of the proposed solution in the paper. Just remember to remove your thread-id from the ringbuffer if your wait operation fails or results in a timeout.
For the Win32 implementation I recommend reading the following documents:
Semaphore Objects and Using Mutex Objects as those describe the functions you'll need for the implementation.
Alternatives: boost::interprocess has some robust mutex emulation support but it is based on spin locks and caused a very high cpu load on our embedded system which was the final reason why we were looking into our own implementation.
#user3726672: Could you update your post to point to this post or to the referenced paper?
Best Regards,
Michael
Update:
I also had a look at an implementation for linux/posix. Turns out pthread already provides everything you'll need. Just put pthread_cond_t and pthread_mutex_t in some shared memory to share it with the other process and initialize both with PTHREAD_PROCESS_SHARED. Also set PTHREAD_MUTEX_ROBUST on the mutex.
Yes. You can use a (named) Mutex for that. Use CreateMutex to create one. You then wait for it (with functions like WaitForSingleObject), and release it when you're done with ReleaseMutex.
For reference, Boost.Interprocess (documentation for version 1.59) has condition variables and much more. Please note, however, that as of this writing, that "Win32 synchronization is too basic".