I wanted to know how I can I make the io do something like a thread.join() wait for all tasks to finish.
io_type->post( strand->wrap(boost::bind &somemethod,ptr,parameter)));
In the above code if 4 threads were initially launched this would give work to the next available thread. However I want to know how I could actually wait for all the threads to finish work. Like we do with threads.join().
If this really needs to be done, then you could setup a mutex or critical section to stop your io handlers from processing messages off of the socket. This would need to be activated from another thread. But, more importantly...
Perhaps you should rethink your design. The problem with having the io wait for other threads to finish is that the io would then be unresponsive. In general, not a good idea. I suspect that most developers working on networking software would not even consider it. If you are receiving messages that are not ready to be processed yet due to other processing that is going on, then consider storing them in a queue and process them on a different thread when the other threads have signaled that they have completed their work.
Related
I'm trying to pull some data from HDFS. I'm running the listHDFS and fetchHDFS processor for this.
When I stopped the fetchHDFS processor, there were a number of active threads even after stopping the processor. To kill these threads, I used the "terminate" option.
Just wanted to know the working of the terminate option.
Does it gracefully shut all the connections with the FS?
Since all the threads are killed, do I lose out on the data that was consumed by these threads?
Is it advised the terminate option only when the threads are stuck or the flow enters a frozen state?
When you stop a processor it tells the NiFi framework to no longer schedule/execute the processor, but there may already be threads executing which need to finish what they were doing. Usually these threads should complete and you will see the active threads go away, but sometimes a thread is blocked (typically when trying to make a network connection somewhere without having proper timeouts set) and this thread may never complete, and therefore needs to be terminated.
The terminate option will issue an interrupt to the thread and then quarantine it, which takes it out of the pool for further execution. The thread may then complete in the background, or if it did not respond to the interrupt and is blocked then it may stay stuck in the background until the next restart of NiFi.
In the FetchHDFS case, assuming it was successfully fetching data, it was most likely in the middle of reading a file from HDFS and just needs a few minutes to complete and shouldn't need to use terminate. If it was never fetching data and was stuck connecting to HDFS then you would use terminate.
I am in the process of designing a crash handler solution for one of our applications that creates a crash dump file using the MiniDumpWriteDump() function. While reading up on the topic I have seen the recommendations to invoke MiniDumpWriteDump() from an external process to maximize the chance that the dump file contains the correct information. The common solution seems to be to run a watchdog process in parallel to the application process. When the application crashes it somehow contacts the watchdog process, providing it with the information that is required to create the crash dump. Then the application goes to sleep until it is terminated by the watchdog process.
I can imagine such a watchdog process being run continually as a background service. This has many implications, starting with "who creates the service?", but also "which user does the service run as?", and "how does the application contact the service?" etc. It seems a pretty heavy-weight solution which I don't feel is appropriate for the scope of my task.
A simpler approach is suggested by this SO answer: Launch a guard process on application startup that is tightly coupled to the application process. This is pretty good, but it still leaves me with the tasks of 1) keeping the information somewhere in the application how I can contact the guard process in case of a crash; and 2) making sure to terminate the guard process if the application process shuts down normally.
The simplest solution of all would be to launch the crash dump handler process at the time the crash occurs, passing all the information that is required to create the crash dump as arguments to the process. This information consists of
The process ID of the application process that crashed
The thread ID of the thread that crashed
The adress of the EXCEPTION_POINTERS structure that describes the exception that caused the crash
This "fire and forget" approach is compelling because it does not require any state retention, nor any complicated over-time process management. In fact, the approach seems so overwhelmingly simple that I cannot help but feel that I am overlooking something.
What are the arguments against such an approach?
The main argument against the "fire and forget" approach, as I called it, is that it is not safe to launch a new process at a time when the application is already in a state where it is about to crash.
Because of that I went for the "guard process" approach. It brings a number of challenges with it, for which Hans Passant has outlined a solution.
I also added a bit of code in this answer that should help with deep-copying the all-important EXCEPTION_POINTERS data structure.
Using WER, as proposed in the comments, also looks like a good alternative to writing your own guard process. I must admit I have not investigated this any further, though.
What happens if a block is asynchronously executing on a background queue when the user quits the app? Will it be terminated? Is it down to me to handle in –applicationWillTerminate:?
I'm not talking about long running tasks.. just wondering if there is an inherent danger in doing anything that could lead to inconsistent state asynchronously?
All threads (GCD created or not) are terminated when an app exits, so yes, the block will not be run (ever) if it hasn't already. If there's work being done that needs to complete before the app can safely exit then you can wait for that in -applicationWillTerminate, but it's really not a good idea to set things up that way since your app could also be force quit by the user and -applicationWillTerminate may never run.
I'm running an eventmachine process on heroku, and it seems to be hitting their memory limit of 512MB after an hour or so. I start seeing messages like this:
Error R14 (Memory quota exceeded)
Process running mem=531M(103.8%)
I'm running a lot of events through the reactor, so I'm thinking maybe the reactor is getting backed up (I'm imagining it as a big queue)? But there could be some other reason, I'm still fairly new to eventmachine.
Are there any good ways to profile eventmachine and and get some stats on it. As a simple example, I was hoping to see how many events were scheduled in the queue to see if it was getting backed up and keeping those all in memory. But if anyone has other suggestions I'd really appreciate it.
Thanks!
I use eventmachine extensively and never ran into any memory leak inside the reactor so your bet is that the ruby code is but without knowing more about your application it is hard to give you a real answer.
The only queue I can think of right now is the thread pool, each time you use the defer method the block is either given to a free thread or queued waiting for a free thread, I suppose if all your threads are blocking waiting for something the queue could grow and use all the memory available.
The leak turned out to be in Mongoid's identity_map (nothing to do with EventMachine). Setting Mongoid.identity_map_enabled = false at the beginning of the event machine process resolved it.
I am spawning few threads inside ioctl call to my driver. I am also assigning kernel affinity to my driver. I want to ensure one of the thread does not get scheduled out till a particular event is flagged by the other thread. Is there any way to not allow windows scheduler to context out my thread. Using _disable() may hang the system as event may take couple of seconds.
Environment is windows 7,64bit
Thanks,
What you are probably after is a spin lock. However this is probably a bad idea unless you can guarantee that your driver/application is always run on a multi-processor system, even then it is still very bad practice. On a single processor system if a thread spin locks then the other thread signalling the spin locked thread will never be scheduled and so can't signal your event. Spin locks are meant to be used sparingly and only when the lock is for a very short time, never a couple of seconds.
It sounds like you need to use an event or other signally mechanism to synchronise your threads and let the windows scheduler do its job. If you need to respond to an event very quickly then interrupts or a Deferred Procedure Call (DPC) could be used instead.