I am running sidekiq in a worker on Heroku as follows:
bundle exec sidekiq -t 25 -e $RAILS_ENV -c 3
One of the operations uses more memory (>500mb) than the worker allows. After the job has completed, the memory still hasn't been released and I get these errors in the heroku rails log files:
2018-11-13T00:56:05.642142+00:00 heroku[sidekiq_worker.1]: Process running mem=646M(126.4%)
2018-11-13T00:56:05.642650+00:00 heroku[sidekiq_worker.1]: Error R14 (Memory quota exceeded)
Is there a way to automatically restart Sidekiq when the memory usage exceeds a certain amount?
Thanks!
have you tried to reduce memory fragmentations? here how you can do it in Heroku.
if that wasn't good enough you can use Heroku platform gem and periodically restart the sidekiq
Related
I'm lost in the weeds. I'm trying to optimize my Sidekiq / Heroku setup with Puma. But I'm either getting a massive amount of jobs backed up in the queue or a huge amount failing and going in the retry queue.
I understand the concurrency optimization math to be:
Puma workers x RAILS_MAX_THREADS = which is 10 in my case
Or is it:
Puma workers x MIN/MAX PUMA THREADS = would be between 2 and 32 in my case
My working is running on 1 Pm dyno.
My puma config is this:
workers 2
threads 1, 16
My database.yml is this:
default: &default
adapter: postgresql
pool: 100
timeout: 5000
And RAILS_MAX_THREADS is 5
I also am using REDIS with a connection limit of 256, but this doesn't seem to be a factor??
And I'm booting sidekiq on Heroku with this:
worker: bundle exec sidekiq -c 15
When I was running sidekiq with 5 threads it was backing up, and 25 threads it was filling up the retry queue.
Is 10 sidekiq threads the best answer here?
EDIT:
I'm trying 10 now to see how it goes.
We are using Amazon AWS Ubuntu machine for our server.
Rails 4.2.7 and Ruby 2.3
I have installed redis with these instructions (Redis)
and installed sidekiq with Sidekiq
I started redis, then starting sidekiq with just bundle exec sidekiq it give me error as running the jobs
FATAL: password authentication failed for user "ubuntu"
Not sure why this error. Any suggestions ?
Closing this thread as it has been fixed.
Changed the permissions for Redis from root level to user level.
And running sidekiq in production mode.
I have a rake task I'm going to use only once, to import a massive CSV file.
heroku run:detached bundle exec rake owners:import
The rake task works ok locally. It runs for > 8 hours on Heroku, but eventually crashes. The task is not completed. Checking processes:
heroku ps
=== web (Standard-1X): bin/rails server -p $PORT -e $RAILS_ENV (1)
web.1: up 2018/01/09 22:52:22 +0000 (~ 10h ago)
=== worker (Standard-1X): bundle exec rake jobs:work (1)
worker.1: crashed 2018/01/10 09:19:05 +0000 (~ 28m ago)
The log does not give any information at all
> heroku logs --app mydemoapp --dyno run.8945
>
I have a 1x standard worker process, which I thought this would use automatically. Is that wrong? Do I set it to use the worker process myself somewhere?
Pardon my ignorance, but is there to increase the number of processes per dyno for Resque workers? And if so, how?
I'm currently using Unicorn to add concurrency to the web dynos, which has been working great so far. I would like to extend this to the Resque workers. I followed Heroku's guide to set up the concurrency.
Update: The solution below works, but is not recommended. For resque concurrency on heroku use the resque-pool gem.
It is possible if you use the COUNT=* option. Your procfile will look something like:
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
resque: env TERM_CHILD=1 COUNT=2 RESQUE_TERM_TIMEOUT=6 QUEUE=* bundle exec rake resque:workers
It is impotant to note that the rake task in the Procfile is resque:workers and not resque:work.
Update Explanation
There are major problems with the COUNT=* option and the rake resque:workers invocation in production on heroku. Because of the way resque starts up the multiple workers using threads, all of the SIGTERM, SIGKILL, etc. handling that allows workers to stop the current job, re-enqueue job, and shut down properly (including de-registering) will ever happen. This is because the the signals are handled by the main process not trapped by the threads. This can cause phantom workers to remain in the worker list long after they've been killed. This is probably why there is a comment in the resque code that warns that the resque:workers should only be used in development mode.
I'm using HireFireApp to autoscale my web and worker dynos on Heroku. However, when I navigate to the Resque app on my application it says
"0 of 46 Workers Working"
Does this mean that I'm using 46 worker dynos???
Update:
Running heroku ps shows:
web.1 up for 21m bundle exec thin start -p $PORT
worker.1 starting for 1s bundle exec rake resque:work QUEUE..
From the command line in your heroku app have a look at the output of
heroku ps
that will show you how many workers you are running.