In lttng live streaming and looking at it through babeltrace; when the node restarts, it continues to send the traces but babeltrace has already done a shutdown because the flow has ended.
So in this stage if babeltrace is restarted manually, it resumes its functionality and works fine.
But how we can make the babeltrace survive the restart and resume after restart? We need this as we post-process babeltrace output and the tests are run during the night.
Babeltrace maintainer here!
There is currently no "retry" option in Babeltrace's live mode. If I understand correctly, you would like for Babeltrace to retry to connect to the remote relay daemon periodically after a connection is dropped? (Ungracefully, I assume)
I have no immediate recommendation except perhaps writing a shell script which re-launches Babeltrace if it exits.
Feel free to ask more questions on the lttng-dev#lists.lttng.org mailing list.
Related
I have configured a queue worker as a daemon on Forge, then used the recommended deployment script command (php artisan queue:restart).
How do I manually stop and restart the queue worker? If I stop it, supervisor will just restart it. Do I need kill the active worker in Forge first?
This may be required on an ad-hoc basis. For example, if I want to clear a log file that the queue has open.
I've been pretty vocal in deployment discussions, and I always tell people to stop their worker processes with the supervisorctl command.
supervisorctl stop <name of task>
Using the queue:restart command doesn't actually restart anything. It sets an entry in the cache which the worker processes check, and shutdown. As you noticed, supervisor will then restart the process.
This means that queue:restart has one huge problem, ignoring the naming and the fact that it doesn't restart; it will cause all worker processes on all servers that uses the same cache to restart. I think this is wrong, I think a deployment should only affect the current server currently being deployed to.
If you're using a per-server cache, like the file cache driver, then this has another problem; what happens if your deployment entirely removes the website folder? The cache would change, the queues would start again, and the worker process may have a mix of old and new code. Fun things to debug...
Supervisor will signal the process when it is shutting down, and wait for it to shut down cleanly, and if it doesn't, forcefully kill it. These timeouts can be configured in the supervisor configuration file. This means that using supervisorctl to stop the queue process will not terminate any jobs "half-way through", they will all complete (assuming they run for a short enough time, or you increase the timeouts).
Is it possible to create a script that is always running on my VPS server? And what need i to do to run it the hole time? (I haven't yet a VPS server, but if this is possible i wants to buy one!
Yes you can, there are many methods to get your expected result.
Supervisord
Supervisord is a process control system that keeps any process running. It automatically start or restart your process whenever necessary.
When to use it: Use it when you need a process that run continuously, eg.:
A queue worker that reads a database continuously waiting for a job to run.
A node application that acts like a daemon
Cron
Cron allow you running processes regularly, in time intervals. You can for example run a process every 1 minute, or every 30 minutes, or any time interval you need.
When to use it: Use it when your process is not long running, it do a task and end, and you do not need it beign restarted automatically like on Supervisord, eg.:
A task that collects logs everyday and send it on a gzip by email
A backup routine.
Whatever you choose, there are many tutorials on the internet on how configuring both, so I'll not go into this details.
In terms of a forced system reboot, from (updates, nightly patch push from HQ, etc), what type of efficacy might a shutdown or logoff script have in overcoming that?
Whether it's just running shutdown -a , or whether it's a vbs with a do loop that runs until 3=4 ... What I'm getting at is that in the past I have used shutdown scripts to do stuff prior to my PC getting automatically/forced shutdown overnight from HQ.
Since the script always worked, i"m wondering if a person could almost conclude from that, that the machine will 'ALWAYS' wait until the script is finished running? Get my drift?
A machine will not necessarily wait for scripts to finish execution before shutting down. A forced shutdown will always happen immediately. A properly timed shutdown -a should still abort it, but it's really unpredictable.
Our group has long running processes which run daily. The processes are typically started at 9pm on any given day and run until 7pm the next day. Thus they typically run 22hrs/day. They are started by scheduled tasks on servers under a particular generic user ID, and they start and run regardless of whether or not that user ID is logged on. Thus, they are windowless console executables.
The tasks orchestrate computations running on a large server farm. Generally these controlling tasks run uninterrupted for the full 22hrs/day. However, we often have a need to stop and restart these processes. Because they control a multitude of tasks running on our server farm, it is important that they be shut down cleanly, so that they can stop and shut down all the server farm processes. Which brings me to our problem.
The controlling process has been programmed to respond to ctrl-C and ctrl-break signals. This works fine when the process is manually started in a console where we have access to the console and can "type" ctrl-c or ctrl-break in the console window. However, as mentioned, the processes typically run as windowless scheduled tasks. Hence we cannot "type" anything into a non-existent console window. Because they are console processes that execute without a logon process, the also must be able to execute in a completely windowless environment. So, how do we set up the process to listen for a shut-down signal?
While the process does indeed listen for a ctrl-C and ctrl-break signal, I can see no way to send that signal to a process. This seems to be a fundamental problem in Windows, or am I wrong? I am aware of SendSignal.exe, but so far have been unable to get it to work. It fails as follows:
>SendSignal 26320
Sending signal to process 26320...
CreateRemoteThread failed with 0x00000005.
StartRemoteThread failed with 0x00000005.
0x00000005 == Access is denied.
Trying "taskkill" without -F results in:
>taskkill /PID 24840
ERROR: The process with PID 24840 could not be terminated.
Reason: This process can only be terminated forcefully (with /F option).
All other "kill" functions kill the process immediately rather than sending a signal.
One possible solution would be a file-watch based solution: create a watch for some modification of a specific file. But this is a hack and we would prefer to do it with appropriate signaling. Has anyone solved this issue? It seems to be so very basic a functionality, and it is certainly trivial to do it in a Unix environment. Surely Microsoft has provided SOME mechanism to allow clean shut down of a windowless executable?
I am aware of the thread below, whose question is virtually identical (save for the specification of why the answer is necessary, i.e. why one needs to be able to do this for a windowless, console-less process), but there is no answer there excpet for "use SendSignal", which, as I said, does not work for us:
Can I send a ctrl-C (SIGINT) to an application on Windows?
There are other similar questions, but no answers as yet.
Any help appreciated.
[Upgrading #Anon's comment to an answer for visibility]
windows-kill worked perfectly and managed to resolve access denial issues faced with SendSignal. A privileged user would have to run it as well of course.
windows-kill also supports both ctrl-c and ctrl-break signals.
I've got a Launch Daemon that runs as root outside of the user context. This Daemon is always running and monitoring a server. On occasion, my Daemon will get a command from the server telling it to display something to the user. Obviously I can't do this from a Launch Daemon since it's not running in a "WindowsServer" environment - so the other option is to spawn a separate process for each user that has access to be able to display a GUI.
One option I have is to make a Launch Agent and have the OS start it up for me. The launch agent could then communicate with the Launch Daemon to determine when something needs to be displayed.
The problem is this though: the server will sometimes provide updates of the actual binary files. So the server would send the Launch Daemon a series of files which it is expected to copy out on top of the existing files. If I have a Launch Agent running, how can I shut it down so that the agent itself can be updated? Likewise, how could I start the Agent back up after the update is done?
Off the cuff, I would say that you should arrange a way for your daemon to request the launch agent to exit on demand, the same way you would arrange for it to launch on demand. Once you know the agent has exited, your daemon can proceed with the update.
Note that your agent probably should not live forever, and it should definitely not be polling the daemon for work. Instead, you should rig it up so that, when the daemon tries to connect to the agent, launchd launches the agent. You could do this by having the daemon connect to a Unix domain socket registered on behalf of the agent by launchd, for example.
You might do better to look at how Google's UpdateEngine handles software updates. I believe it too has a launch daemon perform the app updates. Since this is already working on many computers, its solution to software update problems is a proven one. (Sparkle's solution is thoroughly proven, as well, but it relies on user interaction, which you seem to want to avoid.)