I am working on a ruby application that creates todos and meetings.
There will be reminders that are sent out with respect to each meeting or todo as you would imagine.
We are already using sidekiq and it would be nice to use sidekiq to create the scheduled jobs in x number of days/hours etc.
My concern is that we will lose the jobs if redis restarts.
Am I write in assuming that if redis restarts, we lose the jobs and if so, is there anything that can be done about it?
If not sidekiq, what else could I use?
There are several ways of doing that, just go through the link http://redis.io/topics/persistence. Snapshotting is a technique to snapshots of the dataset on disk.
Related
Rails app which handle and activation of a license using an external service, the external service sometime delays the handling of rails request to over 30s, which will then return an error to front end (I'm running heroku, so max is 30s).
I tried using ActiveJobs and the default rails async adapter (Rails 5), and I can see that is working in Heroku out of the box. I keep reading that I should be using another web process and for example redis, but if the background job should just be performed straight after the request is done and if is just hitting another API outside which may be slower, is it so bad to use the default async?
I can see that this is handle in an in-process thread but I don't see a reason for such small job to be having another web process.
I use the async adapter in production for sending emails. This is a very small job. An email could take up to 3 seconds to send.
The doc said it's a poor fit for production because it will drop pending jobs on restart. If I remember correctly, Heroku restarts dynos once a day.
If your job is pending during the restart, the job will be lost. For my case, a pending email during the restart is pretty slim. So far so good.
But if you have jobs taking 30 seconds, I'll use Resque or DelayedJob.
If for small background job in production, which does not require 100% persistence in case of failure/server restart, whose duration is relatively short and thus separate process would be an overkill, I'd recommend using Sucker Punch.
Sucker Punch gem is designed to handle exactly such case. It prepares execution thread pool for each Job you create, using the concurrent-ruby gem, which is (probably) the most robust concurrency library in Ruby. It also hooks on_exit to finish all the pending tasks, so I guess you can expect this gem to be more reliable than the AsyncJob.
One thing to note is that although Sucker Punch is supported on Active Job, the adapter is not well written. Or, at least, when you use Sucker Punch adapter, it's behavior would be just like that of async adapter. So, I'd recommend using bare Sucker Punch if you wanted something just a little more useful/robust than AsyncJob.
I'm building a monitoring service similar to pingdom but monitoring different aspects of a system and using sidekiq to queue the tasks which is working well. What I need to do is to schedule sending out pings every minute, rather than using a cron based system which would require spinning up a new ruby instance every minute I have gone down the route of using sidetiq (notice the different spelling with a "t") which uses sidekiq's own queue to schedule future tasks. This feels like a neat solution, however I am concerned this may not be the most reliable way of scheduling tasks? If there are issues with the system (as there inevitable will be at some point) will this method of scheduling tasks be less reliable than using a cron based method and why?
Thanks
You are giving too short description of your system needs but I'll try to guess how it could be:
In the first place using sidekiq means that you'll also need an instance of redis and also means that you'll need a way to monitor the sidekiq process and restart it in case of failure and possibly redis server.
A method based on cron tasks will have fewer requirements therefore much less possibilities of failing.
cron has been around for a long time and it's battle tested and it's very very reliable, but has it's drawbacks too.
Said that, you can build a system with separate instances of redis in a master/slave configuration and you can also use Redis sentinel to implement a failover in case of the master failure, implement a monitoring/alerting system on this setup (you can use something super simple like this http://contribsys.com/inspeqtor/ from the sidekiq author) and you can also start several instances of sidekiq in different machines.
With all of that, you can have a quite reliable system for running sidekiq with sidetiq.
Hope it helps
I have sidekiq running on heroku that does a lot of syncing with user's emails etc.
Every so often, we were getting the following error:
Error R14 (Memory quota exceeded)
To counter this, I created a rake task that is executed by the heroku scheduler.
The rake task restarts all the dynos and flushes all the sidekiq jobs from redis with this code:
Sidekiq.redis { |r| r.flushall }
I have a new requirement whereby users want to schedule certain jobs to run in the future like this:
DeliverEmail.delay_until(email.send_time).perform_async(email.id)
Am I right in saying that flushall from the above code sample would flush any scheduled jobs that are created?
If that is the case, is there anything I can do to avoid this?
When you send a redis FLUSHALL command, it's truncating the entire redis datastore. This is a dangerous thing to do and probably not what you want.
It sounds like what you want to do is clear some types of enqueued work, while preserving others. You will need to flush each queue you are using, most likely just the default queue unless you've setup others:
Sidekiq::Queue.new('default').clear
This will remove the queue within redis, but preserve your scheduled jobs, statistics, and other data within redis.
I want to schedule jobs to happen at a specific time and date but I'm getting confused by the wide range of options for doing so.
My requirements:
These are not recurring jobs, they only need to happen once at a specified date and time
I'm the only user of the app so don't need to deal with heavy traffic
I would like to minimise the cost of running this on Heroku, i.e. not paying idle dynos
Any tips on which combinations of gems etc. to use?
Have you looked into using https://github.com/bvandenbos/resque-scheduler? You'll need the Redis To Go addon on Heroku. This will cost you $36 a month because you'll need a scheduler process running alongside your web process. However, I've done this for free. See the README here: https://github.com/austinthecoder/pinger.
Good luck!
Due to Heroku Scheduler (default Heroku add-on) doesn't allow you to schedule your job as specific time. It is best to rely on a clock process to do the job. Gem such has Clockwork could be set up please see Heroku's Clockwork guide. You need to combine Clock with a background queuer such as resque or sidekiq. I highly recommend you go for sidekiq. Please bear in mind that both resque and sidekiq requires redis which is offered by redistogo add-on and it will cost you money to run it.
I am new at heroku and resque.
I have a queue in resque and i should hire and release workers according to current amount of jobs in my queue automatically. I tried hirefireapp but it just hire workers while queue expands and not release any worker unless there are no job waiting in queue. So i make some research and find out that there is no way to say a worker not take a new job after finished current one and shutdown yourself. Resque developers and users also have pointed out this issue in this link https://github.com/defunkt/resque/issues/319 and create a new branch which is keepalive to resque ( https://github.com/hone/resque/tree/keepalive ). It seems it is the solution for my issue. However, since i am new in using resque i couldn't find out that how to fire a worker via resque safely.
If anyone who have more experienced in resque and heroku help me, i will be really glad.
Thanks.
You'll want to run a separate process to control the scaling of workers.
resque-heroku-scaler is one option.
A single additional scaler process helps you manage workers efficiently.
This isn't really what Resque is designed for as it's meant to be sitting there working a queue, not deciding whether or not to start up/shut down.
Personally, unless the money required to run the worker 24/7 is that hard to come by I would just leave it running.