I have a architecture with 2 modules (one in C++ and one in Python) that communicates via ZeroMQ with PUSH/PULL sockets.The PUSHER is the C++ module and the PULLER is the Python module. A third module communicate in REQ/REP with these 2 modules to request for specific command (e.g start, stop, kill, ...).
My Python module cand be started in 3 different modes and can switch between them when requested via REQ/REP. When switching between modes, both C++ en python modules are stopped (going into IDLE STATE) and then started again. the C++ has only one mode but it goes into IDLE_STATE because orthewise it works for nothing and continue filling its sending queue for nothing.
But the problem is that the python module first receives stop request and then the C++ module. So, C++ module has time to continue filling its queue before stopping. Thus, when starting again in another mode, the python module will first get messages from the previous mode before getting messages from the actual mode.
So, is there a way to avoid this behavior? I was thinking of making a "clearQueue" function in python module so that it receives the remaining messages (and throw them to trash) before starting the C++ module.
I did more research to try to find a solution, but it seems that there is no simple solution to it. The Hight Water mark (i.e the qeue size) can be set to a specific value, so I was thinking of setting it to one, but it can still end up with one message remaining in the queue. No one to help ?
EDIT: I actually added the following on the python side, wich i found quit awfull. But cannot get another solution.
try:
while True:
self.bus.receiveData(zmq.NOBLOCK)
except:
pass
Related
I am using Streamlink to help some of my technically challenged older friends watch streams from selected sites that webcast LIVE two or thee times a week and are ingested into Youtube. In between webcasts, it would nice to show the User when the next one will begin via the apps Status page.
The platform is Raspberry Pi 3 B+. I have modified the Youtube plugin to allow/prohibit non-live streams. If '--youtube-live-required' is in the command line, then only LIVE streams will play. This prevents the LIVE webcast from re-starting after it has ended, and also prevents videos that Youtube randomly selects, from playing. I have also applied a 'soon to be released' patch that fixes a breaking-change that Youtube made recently. I mention these so you know that I have at least a minimal understanding of the Streamlink code, and am not looking for a totally free ride. But for some reason, I cannot get my head around how to add a feature to get the 'scheduledStartTime' value from the Youtube.py plugin. I am hoping someone with a deep understanding of the Streamlink code can toss me a clue or two.
Once the 'scheduledStartTime' value is obtained (it is in epoch notation), a custom module will send that value to the onboard Python server, via socketio, which can then massage the data and push it to the Status page of connected clients.
Within an infinite loop, Popen starts Streamlink. The output of Popen is PIPEd and observed in order to learn what is happening, and then sends that info to the Server, again using socketio. It is within this loop that the 'scheduledStartTime' data would be gleaned (I think).
How do I solve the problem?
I have a solution to this problem that I am not very proud of, but it solves the problem and I can close this project, finally. Also, it turns out that this solution did not have to utilize the streamlink youtube.py plugin, but since it fetches the contents of the URL of interest anyways, I decided to hack the plugin and keep all of this business in one place.
In a nutshell, a simple regex gets the value of scheduledStartTime IF it is present in the fetched URL contents. The Hack: That value is printed out as a string 'SCHEDULE START TIME:epoch time value', which surfaces through streamlink via Popen PIPE which is polled for such information, in a custom module. Socket.io then sends the info to the on-board server, that sends a massaged version of the info to the app's Status Page (Ionic framework, typescript, etc). Works. Simple. Ugly. Done.
I am wondering what would be the best practice for deploying updates to a (MVC) Go web application. Imagine the following scenario :
1) Code and test some changes for my Go Web Application
2) Deploy update without anyone currently using the previous version getting interrupted.
I don't know how to make sure point 2) can be covered - when somebody is sending a request to the server and I rebuild/restart it just in this moment, he gets an error - even if the request just uses a part of the code I did not touch or that is backwards-compatible, or if I just added a new Request-handler.
Maybe I'm missing something trivial or a well-known pattern as I am just in the process of learning go and my previous web applications were ASP.NET- or php-applications where this was no issue as I did not need to restart the webserver on code changes.
It's not just an issue with Go, but in general we can divide the problem into two separate ones:
Making sure current requests do not get terminated and affect user experience.
Making sure there is no down-time in which new requests cannot be handled.
The first one is easier to tackle: You just don't violently kill your server, but tell it to exit, causing a "Drain phase", in which it does not accept new requests and only finishes the currently running requests, and exits. This can be done by listening on signals for example, and entering the app into a special state.
It's not trivial with Go as the default http server doesn't support shutting it down, but you can start a server with a net.Listener, and then keep a reference to it an close it when the time is due.
Now, doing only approach one and then starting the service again will cause new requests not to be accepted while this is going on, and we all know this can take a number of seconds in extreme cases.
So what we need is another instance of the server already running with the new code, the instant the old one is not responding to new requests, right? That can be done in several ways:
Having more than one server, and a load-balancer on top of them, allowing one (or more) server to take the load while we restart another. That's the simplest way, and the way most people do it. If you need N servers to take the load of your users, just keep N+1 and restart one at a time.
Using socket sharing tricks. In Newer Linux kernels, Many processes can listen and accept on the same port. What you do is simply start the new instance and then tell the old one to finish and exit. This way there is no pause. This is done by setting SO_REUSEPORT on the listening socket.
The above can be automated with ready to ship solutions, like Einhorn, that deals with all the details for you, see https://github.com/stripe/einhorn
Another approach is documented in this blog post: http://blog.nella.org/?p=879
This a follow-up based on MQ (Websphere 7) persist message to file system.
How do you set up an .exe program from an MQ support pack (such as the Q utility in MA01) to execute each time a message is received? Can this be setup in MQ Explorer? (We are using 7.0 on Windows 2008/R2).
There are a few ways to do this.
Set the application up under Windows control (since you mentioned the QMgr is on Windows.) The app would run as a service, with recovery options to restart it if the service fails. The application would be programmed (or configured in the case of Q) to process the queue using GETs with a VERY long wait period. Some additional error handling is required to do something sensible when the QMgr is not available. This works great for copying messages to multiple queues but isn't appropriate for queue-to-file since the file would never close.
Run the application as a WebSphere MQ service. Defining the SERVICE object using CONTRIL(QMGR) causes MQ to start the service when the QMgr boots and stop it when the QMgr is shut down. Since the service is a child process of the QMgr, no need to worry about how to handle errors when the QMgr isn't available.
Trigger the program based on non-zero queue depth. Define a process object describing the command to be executed, enable triggering on the queue with TRIGTYPE(FIRST) and run a trigger monitor. Whenever the queue has depth > 0 and no open input handles, the process object fires and executes the command. The thing you start must know how to parse the command line so the easiest thing to do if you have someone else's executable is use a script to start it. The script parses the trigger message and fires off the executable. Or perhaps the script ignores the trigger message and just runs the exe. I generally use Korn Shell or Perl and both are available on Windows.
I wrote an article showing how to keep event queues from filling using a triggered version of Q. The article assumes you want the queues to remain mostly full so uses triggering on depth of about 80%. The same techniques could be used (in a much simpler implementation, by the way) to fire off the Q program whenever the queue depth became non-zero.
UPDATE
Probably should have linked the article. ;-)
Mission:Messaging: Easing administration and debugging with circular queues
I am trying to find out if any http requests are made during installation of an msi package. It appears to me that the process under which the http request is made shares no lineage with the process under which the installer executes.
For example, I install an app that makes http calls during installation. Using SysInternals process monitor, I see the process created when the install kicks off. Using MS NetworkMonitor I can see the process used to generate the http request. Filtering in Process Monitor after the fact shows that there is no relationship between the http process, and the install process.
I am thinking that somehow the OS says to use a new process whenever an http request is made. My most important requirement is that I be able to relate one to the other, in order to definitively say "This app installation called these http resources during install". So I don't have to have a perfect understanding of how it all works under the covers, but, I am at a standstill right now. I've concluded that there is no way to relate the two. Am I wrong?
Okay, let's assume that msiexec.exe invokes a helper and that helper invokes whatever is causing the HTTP traffic. Right after the first helper spawns the child it kills itself. This process is too short-lived to normally see the relationships here.
Enter the "Process Tree" feature of Process Monitor. Keep Process Monitor running without any filters on process events. After you are done you can then press Ctrl+T to see the Process Tree (see below).
The grayed icons tell you the process is not active anymore. Furthermore the last column is the end time of the process. But best of all you can see which process created which other process from this, even for very very short-lived processes.
Mark Russinovich, author of Process Monitor and its predecessors, demonstrated this at TechEd about a month ago.
Although this may not answer the question entirely, it should get you going in the right direction. After all Process Monitor also includes network activity filtering (albeit crude, compared to Network Monitor and Wireshark :)).
btw: the green bar in the above screenshot is the "timeline" where you can see the runtime of the process in relation to other processes. Very nifty.
Are you using a custom action for making web requests? Windows Installer Service Process runs custom actions in a separate instance of MSIEXEC than your main set-up instance.
More information here:
http://blogs.msdn.com/b/astebner/archive/2005/03/02/384088.aspx
I'm working on a consumer web app that needs to do a long running background process that is tied to each customer request. By long running, I mean anywhere between 1 and 3 minutes.
Here is an example flow. The object/widget doesn't really matter.
Customer comes to the site and specifies object/widget they are looking for.
We search/clean/filter for widgets matching some initial criteria. <-- long running process
Customer further configures more detail about the widget they are looking for.
When the long running process is complete the customer is able to complete the last few steps before conversion.
Steps 3 and 4 aren't really important. I just mention them because we can buy some time while we are doing the long running process.
The environment we are working in is a LAMP stack-- currently using PHP. It doesn't seem like a good design to have the long running process take up an apache thread in mod_php (or fastcgi process). The apache layer of our app should be focused on serving up content and not data processing IMO.
A few questions:
Is our thinking right in that we should separate this "long running" part out of the apache/web app layer?
Is there a standard/typical way to break this out under Linux/Apache/MySQL/PHP (we're open to using a different language for the processing if appropriate)?
Any suggestions on how to go about breaking it out? E.g. do we create a deamon that churns through a FIFO queue?
Edit: Just to clarify, only about 1/4 of the long running process is database centric. We're working on optimizing that part. There is some work that we could potentially do, but we are limited in the amount we can do right now.
Thanks!
Consider providing the search results via AJAX from a web service instead of your application. Presumably you could offload this to another server and let you web application deal with the content as you desire.
Just curious: 1-3 minutes seems like a long time for a lookup query. Have you looked at indexes on the columns you are querying to improve the speed? Or do you need to do some algorithmic process -- perhaps you could perform some of this offline and prepopulate some common searches with hints?
As Jonnii suggested, you can start a child process to carry out background processing. However, this needs to be done with some care:
Make sure that any parameters passed through are escaped correctly
Ensure that more than one copy of the process does not run at once
If several copies of the process run, there's nothing stopping a (not even malicious, just impatient) user from hitting reload on the page which kicks it off, eventually starting so many copies that the machine runs out of ram and grinds to a halt.
So you can use a subprocess, but do it carefully, in a controlled manner, and test it properly.
Another option is to have a daemon permanently running waiting for requests, which processes them and then records the results somewhere (perhaps in a database)
This is the poor man's solution:
exec ("/usr/bin/php long_running_process.php > /dev/null &");
Alternatively you could:
Insert a row into your database with details of the background request, which a daemon can then read and process.
Write a message to a message queue which a daemon then read and processed.
Here's some discussion on the Java version of this problem.
See java: what are the best techniques for communicating with a batch server
Two important things you might do:
Switch to Java and use JMS.
Read up on JMS but use another queue manager. Unix named pipes, for instance, might be an acceptable implementation.
Java servlets can do background processing. You could do something similar to this technology in a web technology with threading support. I don't know about PHP though.
Not a complete answer but I would think using AJAX and passing the 2nd step to something thats faster then PHP (C, C++, C#) then a PHP function pick the results off of some stack most likely just a database.