I probably have found a similar question (and answer) but I wanted to know if any better alternative is available.
Link to similar question:
http://mqseries.net/phpBB/viewtopic.php?t=72601&sid=f62d9730d61ee2ee2a59986dd79defd1
I want to schedule a particular message flow every 5 seconds (or so). I'm using IIB 10 and it's not associated with MQ. So, Timer nodes are non functional.
I've read about scheduling it with cronjob but again it's getting dependent on the OS which is not my preference. Is there any alternative to the timeout notification node?
Can we use java.util.TimerTask or something similar to to it? Any helping hands please?
I don't know of any solution that does not require a cron job or other external scheduler.
Many organisations use a distributed scheduler like Ctrl-M for a wide range of tasks, and adding a couple of jobs to support the integration layer is not seen as a problem.
You can write you own timer flow using an infinite WHILE loop with SLEEP and PROPAGATE TO TERMINAL functions and sending HTTP requests or configure a "CallableFlow".
Related
I need a better way to use my alerting code.Right now I have a code that check for space free on aws ecs and sends a simple notification to slack if space is less than 5gb using slack api.I used this code in jenkins and setup a periodic schedule to run every 15 min.But once the notification is triggered I wanted it to stop the check for 4 hours so, it won't fill the slack channel with messages .So, i used sleep 14400 after condition is triggered.But this leaves an executor of jenkins waiting.Is there a better way to do this?
If you really want a better way, you should use better tools. there are many tools (some free) out there, that can monitor something in a stateful manner (for example, using a daemon).
Writing to log (or slack channel) in this context of using Jenkins is sort of stateless, for example you cannot check whether an alarm is currently triggered or not.
Since you cannot check if an alarm is already triggered - using jenkins with the logic you requested in your question ('snooze feature') can be very ugly.
In general I would recommend using Conditional BuildStep to trigger a step if a condition is met (i.e. if alarm not already triggered), but since there is no way for you to poll this information, or achieve this with Jenkins without the solution being 'hackish' like creating a file to indicate alert is on, and deleting it from another job if it was created more than 4 hrs ago - I would suggest looking at tools more suitable for the job.
I'm building a monitoring service similar to pingdom but monitoring different aspects of a system and using sidekiq to queue the tasks which is working well. What I need to do is to schedule sending out pings every minute, rather than using a cron based system which would require spinning up a new ruby instance every minute I have gone down the route of using sidetiq (notice the different spelling with a "t") which uses sidekiq's own queue to schedule future tasks. This feels like a neat solution, however I am concerned this may not be the most reliable way of scheduling tasks? If there are issues with the system (as there inevitable will be at some point) will this method of scheduling tasks be less reliable than using a cron based method and why?
Thanks
You are giving too short description of your system needs but I'll try to guess how it could be:
In the first place using sidekiq means that you'll also need an instance of redis and also means that you'll need a way to monitor the sidekiq process and restart it in case of failure and possibly redis server.
A method based on cron tasks will have fewer requirements therefore much less possibilities of failing.
cron has been around for a long time and it's battle tested and it's very very reliable, but has it's drawbacks too.
Said that, you can build a system with separate instances of redis in a master/slave configuration and you can also use Redis sentinel to implement a failover in case of the master failure, implement a monitoring/alerting system on this setup (you can use something super simple like this http://contribsys.com/inspeqtor/ from the sidekiq author) and you can also start several instances of sidekiq in different machines.
With all of that, you can have a quite reliable system for running sidekiq with sidetiq.
Hope it helps
I'm currently working on a Spring Batch POC and have got a pretty good handle on most of the actual Spring Batch features. I've currently got a program that uses Spring Integration to receive an HttpRequest and use message channels to eventually send the job executions to the job launcher in a queue. What we'd really like to do is implement some kind of "scheduler/load balancer" (not quite sure what to call it) before the job launcher that will look at the currently running worker nodes and the size of the input file and make a decision on how many worker nodes the job should be allowed. We would probably also want to be able to change the amount of worker nodes a job has while it is running to allow more jobs to run.
The idea is that we'd have a server running that could accept many job requests at any time, and a large cluster of machines that jobs will be partitioned onto. We'd like to be able to scale horizontally, so whenever the server isn't busy it can make full use of the hardware, as well as being able to make sure that small jobs don't get constantly blocked by larger jobs.
From my research it seems like we'd have to implement another framework to do this (do GridGain and Hadoop allow this?), but I figured I'd ask to see what people recommended to do something like this, and if there's a way to do it without implementing another large framework.
Sorry if anything is unclear or confusing, I'm just a lowly intern who started learning Spring and Spring Batch last month and I'm far from completely understanding everything, especially this scaling stuff. Just ask and I'll try to clear things up.
Thanks for any help!
Take a look at the 'spring-batch-integration' project under the spring-batch-admin umbrella project https://github.com/SpringSource/spring-batch-admin
It has a number of examples of using spring-integration to distribute work to other nodes. IN particular see the chunk and partition packages. Just swap out the spring integration channels with jms channel adapters. By distributing work partitions via JMS, you can scale out the number of worker nodes as needed.
There are a number of threads on this subject in the spring integration forum; search for 'PartitionHandler'.
Hope that helps.
We Want to Create an online game like this.I think that ,this type of games have a scheduling software on web server. For Example : Player Click to create a resource And resource creation will be take a moment like 20 Minutes.(Every resource creation time will be different). This message will send to web server application but this message will not processed at same time for example must be processed after 20 Minutes. The web server application after getting the message must be put the order in the Queue.
We have Some big problems :
1- The Jobs must be complete by the web server application Even the player Exit the Game. I think that we must create something like Windows service on Web Server. Can we do it? or Is there a better way?
2- The Second problem depended on problem 1 .Because we will have many Jobs (every player can create 20,30 Jobs in every Loggin and we will have thousands of users) , So Our Scheduling System Must be Work On time . it's possible that , there is 100 , 1000 jobs in a same second , if application Can't Done Jobs in him Second will be use the next second Time and the next second jobs will shift to next second and etc. How We can do for this problem ?
Platform : .Net 3.5 On Windows 2003 Web server
Cheers
Asad Safari
Agile Coach , Scrum Master
On unix, use cron to schedule a script to run every minute - the script then handles all the jobs.
http://en.wikipedia.org/wiki/Cron
Well the specific answer would depend on which technology/language/platform you're using. In java, check the scheduling services provided by Quartz to schedule jobs; and check JMS/MDBs to implement the asynchronous event processing you mention in question 2.
I think my answer won't exactly fit your question but it will solve your problem. Both points at once.
First at all, why do you want to automatize the execution of this events? As far as I understand, you need to give the player access to his resource if and only if he created the resource an especific time ago.
Well, this is my solution:
Insert a field in your resource model class named usable_since or something like that and, when the player create his resource, just set it the value of this field to the current time plus the desired waiting time.
In this way you beat the two problems:
the resource will be usable even if the player gets offline and
you just won't need to set up so big job handler, even with a million users.
I hope it helps. I wish you the best luck in developing the game. Make it fun!
I had a somewhat similar question, as I'm also developing an application which relies on server-side scheduling: Job-Scheduling in Play! Framework. I use the Play Framework (Java) and it works like a charm; except I have no experience in how many jobs can be scheduled concurrently without bringing the server to its knees.
I'm working on a consumer web app that needs to do a long running background process that is tied to each customer request. By long running, I mean anywhere between 1 and 3 minutes.
Here is an example flow. The object/widget doesn't really matter.
Customer comes to the site and specifies object/widget they are looking for.
We search/clean/filter for widgets matching some initial criteria. <-- long running process
Customer further configures more detail about the widget they are looking for.
When the long running process is complete the customer is able to complete the last few steps before conversion.
Steps 3 and 4 aren't really important. I just mention them because we can buy some time while we are doing the long running process.
The environment we are working in is a LAMP stack-- currently using PHP. It doesn't seem like a good design to have the long running process take up an apache thread in mod_php (or fastcgi process). The apache layer of our app should be focused on serving up content and not data processing IMO.
A few questions:
Is our thinking right in that we should separate this "long running" part out of the apache/web app layer?
Is there a standard/typical way to break this out under Linux/Apache/MySQL/PHP (we're open to using a different language for the processing if appropriate)?
Any suggestions on how to go about breaking it out? E.g. do we create a deamon that churns through a FIFO queue?
Edit: Just to clarify, only about 1/4 of the long running process is database centric. We're working on optimizing that part. There is some work that we could potentially do, but we are limited in the amount we can do right now.
Thanks!
Consider providing the search results via AJAX from a web service instead of your application. Presumably you could offload this to another server and let you web application deal with the content as you desire.
Just curious: 1-3 minutes seems like a long time for a lookup query. Have you looked at indexes on the columns you are querying to improve the speed? Or do you need to do some algorithmic process -- perhaps you could perform some of this offline and prepopulate some common searches with hints?
As Jonnii suggested, you can start a child process to carry out background processing. However, this needs to be done with some care:
Make sure that any parameters passed through are escaped correctly
Ensure that more than one copy of the process does not run at once
If several copies of the process run, there's nothing stopping a (not even malicious, just impatient) user from hitting reload on the page which kicks it off, eventually starting so many copies that the machine runs out of ram and grinds to a halt.
So you can use a subprocess, but do it carefully, in a controlled manner, and test it properly.
Another option is to have a daemon permanently running waiting for requests, which processes them and then records the results somewhere (perhaps in a database)
This is the poor man's solution:
exec ("/usr/bin/php long_running_process.php > /dev/null &");
Alternatively you could:
Insert a row into your database with details of the background request, which a daemon can then read and process.
Write a message to a message queue which a daemon then read and processed.
Here's some discussion on the Java version of this problem.
See java: what are the best techniques for communicating with a batch server
Two important things you might do:
Switch to Java and use JMS.
Read up on JMS but use another queue manager. Unix named pipes, for instance, might be an acceptable implementation.
Java servlets can do background processing. You could do something similar to this technology in a web technology with threading support. I don't know about PHP though.
Not a complete answer but I would think using AJAX and passing the 2nd step to something thats faster then PHP (C, C++, C#) then a PHP function pick the results off of some stack most likely just a database.