i've got problem with tuxedo tmshutdown command. One of processes still runs (with huge cpu usage) though tmshutdown stops it succesfull. There is also one opened IPC shared memory which i can close when I kill existing process. There are other servers but only this one is problematic. Is it possible that the problem is in code (tpsvrdone is exiting without errors)?
Tmshudown normally sends a SIGTERM signal to tuxedo serves unless you use -k KILL (which sends a SIGKILL)
If the source code of the Tuxedo server implements a handler of the signal, you could get the behavior you explained.
http://www.thegeekstuff.com/2012/03/catch-signals-sample-c-code/
Also, if it is not possible to shutdown a server, or remove a service advertisement, a diagnostic is written on the ULOG.
Related
I'm trying to pull some data from HDFS. I'm running the listHDFS and fetchHDFS processor for this.
When I stopped the fetchHDFS processor, there were a number of active threads even after stopping the processor. To kill these threads, I used the "terminate" option.
Just wanted to know the working of the terminate option.
Does it gracefully shut all the connections with the FS?
Since all the threads are killed, do I lose out on the data that was consumed by these threads?
Is it advised the terminate option only when the threads are stuck or the flow enters a frozen state?
When you stop a processor it tells the NiFi framework to no longer schedule/execute the processor, but there may already be threads executing which need to finish what they were doing. Usually these threads should complete and you will see the active threads go away, but sometimes a thread is blocked (typically when trying to make a network connection somewhere without having proper timeouts set) and this thread may never complete, and therefore needs to be terminated.
The terminate option will issue an interrupt to the thread and then quarantine it, which takes it out of the pool for further execution. The thread may then complete in the background, or if it did not respond to the interrupt and is blocked then it may stay stuck in the background until the next restart of NiFi.
In the FetchHDFS case, assuming it was successfully fetching data, it was most likely in the middle of reading a file from HDFS and just needs a few minutes to complete and shouldn't need to use terminate. If it was never fetching data and was stuck connecting to HDFS then you would use terminate.
Our group has long running processes which run daily. The processes are typically started at 9pm on any given day and run until 7pm the next day. Thus they typically run 22hrs/day. They are started by scheduled tasks on servers under a particular generic user ID, and they start and run regardless of whether or not that user ID is logged on. Thus, they are windowless console executables.
The tasks orchestrate computations running on a large server farm. Generally these controlling tasks run uninterrupted for the full 22hrs/day. However, we often have a need to stop and restart these processes. Because they control a multitude of tasks running on our server farm, it is important that they be shut down cleanly, so that they can stop and shut down all the server farm processes. Which brings me to our problem.
The controlling process has been programmed to respond to ctrl-C and ctrl-break signals. This works fine when the process is manually started in a console where we have access to the console and can "type" ctrl-c or ctrl-break in the console window. However, as mentioned, the processes typically run as windowless scheduled tasks. Hence we cannot "type" anything into a non-existent console window. Because they are console processes that execute without a logon process, the also must be able to execute in a completely windowless environment. So, how do we set up the process to listen for a shut-down signal?
While the process does indeed listen for a ctrl-C and ctrl-break signal, I can see no way to send that signal to a process. This seems to be a fundamental problem in Windows, or am I wrong? I am aware of SendSignal.exe, but so far have been unable to get it to work. It fails as follows:
>SendSignal 26320
Sending signal to process 26320...
CreateRemoteThread failed with 0x00000005.
StartRemoteThread failed with 0x00000005.
0x00000005 == Access is denied.
Trying "taskkill" without -F results in:
>taskkill /PID 24840
ERROR: The process with PID 24840 could not be terminated.
Reason: This process can only be terminated forcefully (with /F option).
All other "kill" functions kill the process immediately rather than sending a signal.
One possible solution would be a file-watch based solution: create a watch for some modification of a specific file. But this is a hack and we would prefer to do it with appropriate signaling. Has anyone solved this issue? It seems to be so very basic a functionality, and it is certainly trivial to do it in a Unix environment. Surely Microsoft has provided SOME mechanism to allow clean shut down of a windowless executable?
I am aware of the thread below, whose question is virtually identical (save for the specification of why the answer is necessary, i.e. why one needs to be able to do this for a windowless, console-less process), but there is no answer there excpet for "use SendSignal", which, as I said, does not work for us:
Can I send a ctrl-C (SIGINT) to an application on Windows?
There are other similar questions, but no answers as yet.
Any help appreciated.
[Upgrading #Anon's comment to an answer for visibility]
windows-kill worked perfectly and managed to resolve access denial issues faced with SendSignal. A privileged user would have to run it as well of course.
windows-kill also supports both ctrl-c and ctrl-break signals.
I have a named pipe server similar to the MSDN sample at http://msdn.microsoft.com/en-us/library/windows/desktop/aa365588(v=vs.85).aspx and would like to allow clients to send an "exit" message which causes the server to gracefully stop.
So in the "InstanceThread()", if a special message is received, I would like to make the server stop.
I tried to stop the call to ConnectNamedPipe() in the main thread from the separate thread for "InstanceThread()" by closing the pipe handle, but this does not work.
I already tried various things, among others closing the overall pipe, exiting directly from the InstanceThread, ... but none of them causes the call to ConnectNamedPipe() to stop.
I played with SetNamedPipeHandleState(), but it complicates the implementation hugely, also using overlapped I/O seems overkill for this simple requirement.
So is there an easier way to get ConnectNamedPipe() to return when the server process should be stopped and not wait endlessly for client connections?
If you don't need to support Windows XP, you could try using CancelSynchronousIo.
If the process is exiting, you don't need to do anything; the thread will be terminated when Windows tears down the process.
Alternatively, you could make the call to ConnectNamedPipe exit simply by connecting to the named pipe yourself.
how can I catch windows taskkill command in order to prevent it?
You cannot prevent your process being killed. The only way to keep a program alive is to have a second Watchdog application that is constantly pooling that process to ensure it's running.
So, you have two Applications; AppA (Main) and AppB (Watchdog).
AppA checks if AppB is running. If
not, AppA starts AppB.
AppB checks if AppA is running. If
not, AppB starts AppA.
But this is usually a design for a malicious process, so I hope you're not doing anything untoward.
You can see the following events in the message queue when End process is clicked:
WM_CLOSE // UI app
CTRL_CLOSE_EVENT //
Console app
But if your application does not close in a timely manner, then the kill command will be sent, which cannot be captured.
I'm pretty sure it is not possible, otherwise a single application could stop windows from ever shutting down. Even when the process is hung, killing it in task manager eventually works. If it was possible, malicous software, etc would probably have a hey-day using it - not only could they take control of various parts of your PC, they'd make sure you could never get it back again.
I'm very interested in the answer to another question regarding watchdog timers for Windows services (see here). That answer stated:
I have also used an internal watchdog system running in another thread. That thread looks at the main thread for activity like log output or a toggling event. If the activity is not seen then the service is considered hung and I shutdown the service.
In this case you can configure windows to auto-restart a stopped service and that might clear the problem (as long as it's not an internal logic bug).
Also services I work with have text logs that are written to a log. In addition for services that are about to "sleep for a bit", I log the time for the next wake up. I use MTAIL to watch a log for output."
Could anyone give some sample code how to use an internal watchdog running in another thread, since I currently have a task to develop a windows service which will be able to self restart in case it failed, hung up, etc.
I really appreciate your help.
I'm not a big fan of running a watchdog as a thread in the process you're watching. That means if the whole process hangs for some reason, the watchdog won't work.
Watchdogs are an idea lifted from the hardware world and they had it right. Use an external circuit as simple as possible (so it can be provably correct). Typical watchdogs simply ran an timer and, if the process hadn't done something before the timer expired (like access a memory location the watchdog was watching), the whole thing was reset. When the watchdog was "kicked", it would restart the timer.
The act of the process kicking the watchdog protected that process from summary termination.
My advice would be to write a very simple stand-alone program which just monitored an event (such as file update time being modified). If that event didn't occur within the required time, kill the process being watched (and let Windows restart it).
Then have your watched program periodically rewrite that file.
Other approaches you might want to consider besides regularly modifying the lastwritetime of a file would be to create a proper performance counter or even a WMI object. We do the later in our build infrastructure, the 'trick' is to find a meaningful work unit in the service being monitored and pulse your 'heartbeat' each time a unit is finished.
The advantage of WMI or Perf Counters over a the file approach is that you then become visible to a whole bunch of professional MIS / management tools. This can add a lot of value.
You can configure from service properties to self restart in case of failure
Services -> right-click your service -> Properties -> First failure : restart the service -> Second failure : restart the service -> Subsequent failure : restart