We are using Celery 4.1.0 with Django 1.33.1 on Heroku. There are two dynos configured in the Procfile, one for web, one for both Celery worker and beat:
web: gunicorn dms.wsgi --log-file=-
worker: celery -A dms.tasks worker -B --scheduler django_celery_beat.schedulers:DatabaseScheduler --without-gossip --without-mingle --without-heartbeat
There are several scheduled tasks configured which execute just fine. However, when trying to send tasks to the broker it says ConnectionRefusedError: [Errno 111] Connection refused, seemingly regardless of the type of broker (Actually the process just times out and only shows the error when sending a task manually but we suspect this is related to a different bug)
I tried RabbitMQ and Redis and got the same error message on both. It is a staging environment with a single user and I double-checked we didn't hit any queue/connection limits.
dms/settings.py:
CELERY_BROKER = os.environ.get('CELERY_BROKER')
CELERY_TASK_IGNORE_RESULT = True
BROKER_POOL_LIMIT = 1
CELERY_IMPORTS = (
'dms.tasks',
'core.tasks',
)
dms/tasks.py:
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'dms.settings')
app = Celery('dms', broker=settings.CELERY_BROKER)
app.config_from_object('django.conf:settings')
app.autodiscover_tasks()
We also contacted Heroku and CloudAMQP support.
Related
I'm trying to connect from an external client (my laptop) to a broker in a Kafka cluster that I have running on ec2 machines. When I try and connect from my local machine I get the following error:
$ ./kafka-console-producer --broker-list AWS.PRIV.ATE.IP:9092 --topic test
>hi
>[2018-09-20 13:28:53,952] ERROR Error when sending message to topic test with key: null, value: 2 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for test-0: 1519 ms has passed since batch creation plus linger time
The topic exists because if I run (from local machine)
$ ./kafka-topics --list --zookeeper AWS.PRIV.ATE.IP:2181
__confluent.support.metrics
__consumer_offsets
_schemas
connect-configs
connect-offsets
connect-status
test
The cluster configuration is from Confluent's AWS quickstart template: https://github.com/aws-quickstart/quickstart-confluent-kafka/blob/master/templates/confluent-kafka.template and I'm running the open source version.
The three broker ec2 instances are visible to my local machine, which I verified by stopping the Kafka broker, starting a simple HTTP server on port 9092, and successfully curling that server using the internal IP address of the ec2 instance.
If I ssh into one of the broker instances I can successfully produce and consume messages across the cluster. The only update I've made to the out-of-the-box configuration provided by the template is changing listeners=PLAINTEXT://ec2-AWS-PUB-LIC-IP.compute-1.amazonaws.com:9092 in server.properties on each machine and then restarted the kafka server.
I can provide more configuration or debugging info if necessary. Believe the issue is something regarding IP address discoverability/visibility but I'm not entirely sure what.
You need to set advertised.listeners too.
See https://rmoff.net/2018/08/02/kafka-listeners-explained/ for details.
I am using CloudAMPQ addon for Heroku. As RabbitMQ needs a unique node name for each of its process, I run into warning when I scale my worker dynos from 1 to 2 or more:
/app/.heroku/python/lib/python3.6/site-packages/kombu/pidbox.py:71: UserWarning: A node named coworker#fstrk.io is already using this process mailbox!
Maybe you forgot to shutdown the other node or did not do so properly?
Or if you meant to start multiple nodes on the same host please make sure
you give each node a unique node name!
My Procfile line looks like this
coworker: celery -l info -A getmybot worker -Q slack -c ${COWORKER_PROCESSES:-4} --hostname coworker#fstrk.io --without-gossip --without-mingle --without-heartbeat
how do I go about it?
Try change --hostname coworker#fstrk.io to --hostname coworker#%%h
More details in official docs:
http://docs.celeryproject.org/en/latest/reference/celery.bin.worker.html
Trying to run the 1 Master 2 Slave with 3 Sentinel setup on localhost (Windows), using redis-3.0.5 64 bit, as described here :
https://github.com/ServiceStack/redis-config.git
All instances come up fine and seems to be communication till I test the failover using the command :
redis-cli -p 6380 DEBUG sleep 30
this commands returns after 30 seconds during which the Master becomes unavailable and my Java app, using Spring-Data-Redis with Jedis client, goes into a connection retry mode.
There are no log messages on the console or log file for any of the Redis or Sentinel instances. No indcation of any of the 2 slaves being promoted to master.
The command "SENTINEL get-master-addr-by-name mymaster" shows the same Master IP and Port all through the 30 seconds of sleep and after.
Am I missing something ?
Previous night I was tinkering with Elixir running code on my both machines at home, but when I woke up, I asked myself Can I actually do the same using heroku run command?
I think theoretically it should be entirely possible if setup properly. Obviously heroku run iex --sname name executes and gives me access to shell (without functioning backspace which is irritating) but i haven't accessed my app yet.
Each time I executed the command it gave me different machine. I guess it's how Heroku achieve sandbox. I also was trying to find a way to determine address of my app's machine but haven't got any luck yet.
Can I actually connect with the dyno running the code to evaluate expressions on it like you would do iex -S mix phoenix.server locally ?
Unfortunately it's not possible.
To interconnect Erlang VM nodes you'd need EPMD port (4369) to be open.
Heroku doesn't allow opening custom ports so it's not possible.
In case You'd want to establish a connection between your Phoenix server and Elixir node You'd have to:
Two nodes on the same machine:
Start Phoenix using iex --name phoenix#127.0.0.1 -S mix phoenix.server
Start iex --name other_node#127.0.0.1
Establish a connection using Node.ping from other_node:
iex(other_node#127.0.0.1)1> Node.ping(:'phoenix#127.0.0.1')
(should return :pong not :pang)
Two nodes on different machines
Start Phoenix using some external address
iex --name phoenix#195.20.2.2 --cookie someword -S mix phoenix.server
Start second node
iex --name other_node#195.20.2.10 --cookie someword
Establish a connection using Node.ping from other_node:
iex(other_node#195.20.2.10)1> Node.ping(:'phoenix#195.20.2.2')
(should return :pong not :pang)
Both nodes should contact each other on the addresses they usually see each other on the network. (Full external IP when different networks, 192.168.X.X when in the same local network, 127.0.0.1 when on the same machine)
If they're on different machines they also must have set the same cookie value, because by default it takes automatically generated cookie in your home directory. You can check it out by running:
cat ~/.erlang.cookie
What's last you've got to make sure that your EPMD port 4369 is open, because Erlang VM uses it for internode data exchange.
As a sidenote if you will leave it open make sure to make your cookie as private as possible, because if someone knows it, he can have absolute power over your machine.
When you execute heroku run it will start a new one-off dyno which is a temporary instance that is deprovisioned when you finish the heroku run session. This dyno is not a web dyno and cannot receive inbound HTTP requests through Heroku's routing layer.
From the docs:
One-off dynos can never receive HTTP traffic, since the routers only route traffic to dynos named web.N.
https://devcenter.heroku.com/articles/one-off-dynos#formation-dynos-vs-one-off-dynos
If you want your phoenix application to receive HTTP requests you will have to set it up to run on a web dyno.
It has been a while since you've asked the question, but someone might find this answer valuable, though.
As of 2021 Heroku allows forwarding multiple ports, which allows to remsh into a running ErlangVM node. It depends on how you deploy your application, but in general, you will need to:
Give your node a name and a cookie (i.e. --name "myapp#127.0.0.1" --cookie "secret")
Tell exactly which port a node should bind to, so you know which pot to forward (i.e. --erl "-kernel inet_dist_listen_min 9000 -kernel inet_dist_listen_max 9000")
Forward EPMD and Node ports by running heroku ps:forward 9001:4369,9000
Remsh into your node: ERL_EPMD_PORT=9001 iex --cookie "secret" --name console#127.0.0.1 --remsh "myapp#127.0.0.1"
Eventually you should start your server with something like this (if you are still using Mix tool): MIX_ENV=prod elixir --name "myapp#127.0.0.1" --cookie "secret" --erl "-kernel inet_dist_listen_min 9000 -kernel inet_dist_listen_max 9000" -S mix phx.server --no-halt
If you are using Releases, most of the setup has already been done for you by the Elixir team.
To verify that EPMD port has been forwarded correctly, try running epmd -port 9001 -names. The output should be:
epmd: up and running on port 4369 with data:
name myapp#127.0.0.1 at port 9000
You may follow my notes on how I do it for Dockerized releases (there is a bit more hustle): https://paveltyk.medium.com/elixir-remote-shell-to-a-dockerized-release-on-heroku-cc6b1196c6ad
Every time I launch my app it cannot get past the 60 second point without:
2012-05-06T22:41:11+00:00 heroku[web.1]: Stopping process with SIGKILL
2012-05-06T22:41:11+00:00 heroku[web.1]: Error R10 (Boot timeout) -> Web process failed to bind to $PORT within 60 seconds of launch
2012-05-06T22:41:11+00:00 heroku[web.1]: Process exited with status 137
2012-05-06T22:41:12+00:00 heroku[web.1]: State changed from starting to crashed
Here is my Procfile:
web: bundle exec thin start -p $PORT
Any responses will be thoroughly appreciated.
If your app does take longer than 60 seconds for "good" reasons, you can work around the 60s boot time limit with https://github.com/dblock/heroku-forward.
The solution was that I had forgotten to include the -p $PORT in my Procfile line.
in Procfile change:
web: bundle exec thin start
to
web: bundle exec thin start -p $PORT
That fixed it for me.
Heroku's boot timeout bit me too. I read several blog posts about how to get around it and ended up automating some of the solutions into a gem.
To reduce the startup time on deploy, you can trim the gems loaded at boot time (this doesn't mean you have to trim them from the app, just boot time).
gem_bench evaluates which gems are likely to not be needed at boot time.
I have an app with about 250 gems and was able to add :require => false to about 60 of them, with dramatic effects.
https://github.com/acquaintable/gem_bench
Disclaimer: I am the author of this open source ruby gem. I wrote the gem to aid myself in solving this exact problem: the 60 second timeout on Heroku.
Hi i was facing the same issue.I have resolved this issue by increase the timeout in /config/unicorn.rb
change timeout 15 to timeout 20 in /config/unicorn.rb
In my case using nodejs I solved this adding a Procfile file with content:
worker: node index.js and push it to heroku.
After that make sure to disable the check "web npm start" and turn on the check "worker node index.js" just like the image attached below
herokuResourcesConfig
I was having the same error when deploying my Node app on Heroku.
I got it solved by adding a Procfile.
web: node app.js
It tells Heroku how to start the application.
The error is because of Heroku is not able to configure on which PORT to run the application.
It can be solved by specifying the PORT for Heroku, ie: in app.js
const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
console.log(`App is running on port ${ PORT }`);
});
Error R10 (Boot timeout)
is this hidden section of heroku allows you to increase the deployment time.
https://tools.heroku.support/limits/boot_timeout
I got this error because Heroku didn't have access to the Mongo Atlas database. You need to change this in the database settings
have the same issue, solved by creating file with proxy server
https://www.npmjs.com/package/http-proxy#setup-a-basic-stand-alone-proxy-server
proxy.js:
httpProxy.createProxyServer({
target, // target that can't cant be exposed, e.g. localhost:4000
changeOrigin: true,
}).listen(process.env.PORT); // port from heroku runtime
then
node server/proxy.js