I am new to shopify api, what I have done is create node app with the following details:
shopify api V5.0.1
using redis as SessionStorage
deployed to heroku
shopify cli V2.22
Heroku Data for Redis
and here is the log from heroku using this command heroku logs -t --app<app_name_in_heroku>`
See log
2022-08-22T02:53:05.000000+00:00 app[heroku-redis]: source=REDIS addon=redis-closed-31837 sample#active-connections=2 sample#load-avg-1m=0.395 sample#load-avg-5m=0.29 sample#load-avg-15m=0.235 sample#read-iops=0 sample#write-iops=0 sample#memory-total=16085852kB sample#memory-free=10200488kB sample#memory-cached=2958216kB sample#memory-redis=355704bytes sample#hit-rate=1 sample#evicted-keys=0
2022-08-22T02:57:43.000000+00:00 app[heroku-redis]: source=REDIS addon=redis-closed-31837 sample#active-connections=2 sample#load-avg-1m=0.315 sample#load-avg-5m=0.375 sample#load-avg-15m=0.28 sample#read-iops=0 sample#write-iops=0 sample#memory-total=16085852kB sample#memory-free=10199724kB sample#memory-cached=2958228kB sample#memory-redis=355704bytes sample#hit-rate=1 sample#evicted-keys=0
2022-08-22T02:57:43.000000+00:00 app[heroku-redis]: source=REDIS addon=redis-closed-31837 sample#active-connections=2 sample#load-avg-1m=0.155 sample#load-avg-5m=0.285 sample#load-avg-15m=0.26 sample#read-iops=0 sample#write-iops=0 sample#memory-total=16085852kB sample#memory-free=10200236kB sample#memory-cached=2958236kB sample#memory-redis=355672bytes sample#hit-rate=1 sample#evicted-keys=0
2022-08-22T02:58:36.080444+00:00 app[web.1]: node:events:491
2022-08-22T02:58:36.080453+00:00 app[web.1]: throw er; // Unhandled 'error' event
2022-08-22T02:58:36.080454+00:00 app[web.1]: ^
2022-08-22T02:58:36.080454+00:00 app[web.1]:
2022-08-22T02:58:36.080455+00:00 app[web.1]: SocketClosedUnexpectedlyError: Socket closed unexpectedly
2022-08-22T02:58:36.080456+00:00 app[web.1]: at Socket.<anonymous> (/app/node_modules/#redis/client/dist/lib/client/socket.js:156:118)
2022-08-22T02:58:36.080456+00:00 app[web.1]: at Object.onceWrapper (node:events:628:26)
2022-08-22T02:58:36.080457+00:00 app[web.1]: at Socket.emit (node:events:513:28)
2022-08-22T02:58:36.080457+00:00 app[web.1]: at TCP.<anonymous> (node:net:757:14)
2022-08-22T02:58:36.080457+00:00 app[web.1]: Emitted 'error' event on Commander instance at:
2022-08-22T02:58:36.080458+00:00 app[web.1]: at RedisSocket.<anonymous> (/app/node_modules/#redis/client/dist/lib/client/index.js:344:14)
2022-08-22T02:58:36.080458+00:00 app[web.1]: at RedisSocket.emit (node:events:513:28)
2022-08-22T02:58:36.080459+00:00 app[web.1]: at RedisSocket._RedisSocket_onSocketError (/app/node_modules/#redis/client/dist/lib/client/socket.js:179:10)
2022-08-22T02:58:36.080460+00:00 app[web.1]: at Socket.<anonymous> (/app/node_modules/#redis/client/dist/lib/client/socket.js:156:107)
2022-08-22T02:58:36.080460+00:00 app[web.1]: at Object.onceWrapper (node:events:628:26)
2022-08-22T02:58:36.080460+00:00 app[web.1]: at Socket.emit (node:events:513:28)
2022-08-22T02:58:36.080461+00:00 app[web.1]: at TCP.<anonymous> (node:net:757:14)
2022-08-22T02:58:36.080466+00:00 app[web.1]:
2022-08-22T02:58:36.080467+00:00 app[web.1]: Node.js v18.7.0
2022-08-22T02:58:36.223443+00:00 heroku[web.1]: Process exited with status 1
2022-08-22T02:58:36.431160+00:00 heroku[web.1]: State changed from up to crashed
2022-08-22T02:58:36.434346+00:00 heroku[web.1]: State changed from crashed to starting
2022-08-22T02:58:40.417402+00:00 heroku[web.1]: Starting process with command `npm start`
2022-08-22T02:58:42.045346+00:00 app[web.1]:
2022-08-22T02:58:42.045359+00:00 app[web.1]: > start
2022-08-22T02:58:42.045359+00:00 app[web.1]: > npm run serve
2022-08-22T02:58:42.045360+00:00 app[web.1]:
2022-08-22T02:58:42.360327+00:00 app[web.1]:
2022-08-22T02:58:42.360333+00:00 app[web.1]: > preserve
2022-08-22T02:58:42.360333+00:00 app[web.1]: > npm run build
2022-08-22T02:58:42.360334+00:00 app[web.1]:
2022-08-22T02:58:42.663352+00:00 app[web.1]:
2022-08-22T02:58:42.663364+00:00 app[web.1]: > build
2022-08-22T02:58:42.663366+00:00 app[web.1]: > npm run build:client
2022-08-22T02:58:42.663366+00:00 app[web.1]:
2022-08-22T02:58:42.969492+00:00 app[web.1]:
2022-08-22T02:58:42.969499+00:00 app[web.1]: > build:client
2022-08-22T02:58:42.969500+00:00 app[web.1]: > vite build --outDir dist/client
2022-08-22T02:58:42.969500+00:00 app[web.1]:
2022-08-22T02:58:43.361517+00:00 app[web.1]: vite v2.9.8 building for production...
2022-08-22T02:58:43.418520+00:00 app[web.1]: transforming...
2022-08-22T02:58:51.309037+00:00 app[web.1]: ✓ 1478 modules transformed.
2022-08-22T02:58:52.060605+00:00 app[web.1]: rendering chunks...
2022-08-22T02:58:52.068929+00:00 app[web.1]: dist/client/assets/eromman_logo.6b39893a.png 8.82 KiB
2022-08-22T02:58:52.071195+00:00 app[web.1]: dist/client/index.html 0.29 KiB
2022-08-22T02:58:52.079983+00:00 app[web.1]: dist/client/assets/index.9ac88508.css 330.02 KiB / gzip: 38.73 KiB
2022-08-22T02:58:52.102140+00:00 app[web.1]: dist/client/assets/index.f69ec157.js 522.69 KiB / gzip: 144.56 KiB
2022-08-22T02:58:52.102350+00:00 app[web.1]:
2022-08-22T02:58:52.102350+00:00 app[web.1]: (!) Some chunks are larger than 500 KiB after minification. Consider:
2022-08-22T02:58:52.102351+00:00 app[web.1]: - Using dynamic import() to code-split the application
2022-08-22T02:58:52.102353+00:00 app[web.1]: - Use build.rollupOptions.output.manualChunks to improve chunking: https://rollupjs.org/guide/en/#outputmanualchunks
2022-08-22T02:58:52.102353+00:00 app[web.1]: - Adjust chunk size limit for this warning via build.chunkSizeWarningLimit.
2022-08-22T02:58:52.145873+00:00 app[web.1]:
2022-08-22T02:58:52.145874+00:00 app[web.1]: > serve
2022-08-22T02:58:52.145875+00:00 app[web.1]: > cross-env NODE_ENV=production node server/index.js
2022-08-22T02:58:52.145876+00:00 app[web.1]:
2022-08-22T02:58:53.208466+00:00 heroku[web.1]: State changed from starting to up
2022-08-22T03:02:17.000000+00:00 app[heroku-redis]: source=REDIS addon=redis-closed-31837 sample#active-connections=2 sample#load-avg-1m=0.155 sample#load-avg-5m=0.26 sample#load-avg-15m=0.255 sample#read-iops=0 sample#write-iops=0 sample#memory-total=16085852kB sample#memory-free=10198040kB sample#memory-cached=2958376kB sample#memory-redis=355672bytes sample#hit-rate=1 sample#evicted-keys=0
2022-08-22T03:02:17.000000+00:00 app[heroku-redis]: source=REDIS addon=redis-closed-31837 sample#active-connections=2 sample#load-avg-1m=0.31 sample#load-avg-5m=0.29 sample#load-avg-15m=0.27 sample#read-iops=0 sample#write-iops=0 sample#memory-total=16085852kB sample#memory-free=10199328kB sample#memory-cached=2958376kB sample#memory-redis=355672bytes sample#hit-rate=1 sample#evicted-keys=0
2022-08-22T03:07:14.000000+00:00 app[heroku-redis]: source=REDIS addon=redis-closed-31837 sample#active-connections=2 sample#load-avg-1m=0.175 sample#load-avg-5m=0.25 sample#load-avg-15m=0.26 sample#read-iops=0 sample#write-iops=0 sample#memory-total=16085852kB sample#memory-free=10200780kB sample#memory-cached=2958388kB sample#memory-redis=355672bytes sample#hit-rate=1 sample#evicted-keys=0
2022-08-22T03:07:14.000000+00:00 app[heroku-redis]: source=REDIS addon=redis-closed-31837 sample#active-connections=2 sample#load-avg-1m=0.1 sample#load-avg-5m=0.185 sample#load-avg-15m=0.23 sample#read-iops=0 sample#write-iops=0 sample#memory-total=16085852kB sample#memory-free=10199220kB sample#memory-cached=2958524kB sample#memory-redis=355672bytes sample#hit-rate=1 sample#evicted-keys=0
2022-08-22T03:11:57.000000+00:00 app[heroku-redis]: source=REDIS addon=redis-closed-31837 sample#active-connections=2 sample#load-avg-1m=0.085 sample#load-avg-5m=0.15 sample#load-avg-15m=0.21 sample#read-iops=0 sample#write-iops=0 sample#memory-total=16085852kB sample#memory-free=10199936kB sample#memory-cached=2958536kB sample#memory-redis=355672bytes sample#hit-rate=1 sample#evicted-keys=0
2022-08-22T03:13:53.045639+00:00 app[web.1]: node:events:491
2022-08-22T03:13:53.045648+00:00 app[web.1]: throw er; // Unhandled 'error' event
2022-08-22T03:13:53.045648+00:00 app[web.1]: ^
2022-08-22T03:13:53.045648+00:00 app[web.1]:
2022-08-22T03:13:53.045649+00:00 app[web.1]: SocketClosedUnexpectedlyError: Socket closed unexpectedly
2022-08-22T03:13:53.045650+00:00 app[web.1]: at Socket.<anonymous> (/app/node_modules/#redis/client/dist/lib/client/socket.js:156:118)
2022-08-22T03:13:53.045650+00:00 app[web.1]: at Object.onceWrapper (node:events:628:26)
2022-08-22T03:13:53.045651+00:00 app[web.1]: at Socket.emit (node:events:513:28)
2022-08-22T03:13:53.045651+00:00 app[web.1]: at TCP.<anonymous> (node:net:757:14)
2022-08-22T03:13:53.045651+00:00 app[web.1]: Emitted 'error' event on Commander instance at:
2022-08-22T03:13:53.045652+00:00 app[web.1]: at RedisSocket.<anonymous> (/app/node_modules/#redis/client/dist/lib/client/index.js:344:14)
2022-08-22T03:13:53.045652+00:00 app[web.1]: at RedisSocket.emit (node:events:513:28)
2022-08-22T03:13:53.045653+00:00 app[web.1]: at RedisSocket._RedisSocket_onSocketError (/app/node_modules/#redis/client/dist/lib/client/socket.js:179:10)
2022-08-22T03:13:53.045653+00:00 app[web.1]: at Socket.<anonymous> (/app/node_modules/#redis/client/dist/lib/client/socket.js:156:107)
2022-08-22T03:13:53.045653+00:00 app[web.1]: at Object.onceWrapper (node:events:628:26)
2022-08-22T03:13:53.045654+00:00 app[web.1]: at Socket.emit (node:events:513:28)
2022-08-22T03:13:53.045654+00:00 app[web.1]: at TCP.<anonymous> (node:net:757:14)
2022-08-22T03:13:53.045659+00:00 app[web.1]:
2022-08-22T03:13:53.045659+00:00 app[web.1]: Node.js v18.7.0
2022-08-22T03:13:53.674235+00:00 heroku[web.1]: State changed from up to crashed
2022-08-22T03:13:53.677381+00:00 heroku[web.1]: State changed from crashed to starting
2022-08-22T03:13:53.486062+00:00 heroku[web.1]: Process exited with status 1
2022-08-22T03:13:59.017503+00:00 heroku[web.1]: Starting process with command `npm start`
2022-08-22T03:14:00.918467+00:00 app[web.1]:
2022-08-22T03:14:00.918479+00:00 app[web.1]: > start
2022-08-22T03:14:00.918479+00:00 app[web.1]: > npm run serve
2022-08-22T03:14:00.918479+00:00 app[web.1]:
2022-08-22T03:14:02.061437+00:00 app[web.1]:
2022-08-22T03:14:02.061449+00:00 app[web.1]: > build:client
2022-08-22T03:14:02.061450+00:00 app[web.1]: > vite build --outDir dist/client
As you can see the error is
node:events:491
throw er; // Unhandled 'error' event
SocketClosedUnexpectedlyError: Socket closed unexpectedly
and the weired thing is heroku immediatly rebuild the app?!!! I think it should only restart. am i right?
My question is how to know the cause of this error??
how to resolve this error?
Because my app got rejected in final review because of this error since shopify app send me below screenshot while the are reviewing my app
Yes heroku is displaying this error but after a while(after rebuild) the app again because accessible and works fine
and after successfull installation by 1 min the error occur and heroku rebuild the app again.
Any help is appreciated and thanks in advance
I have found the cause of this issue:
my node version in package.json was >=16.13.0 after making it 16.13.0 and deploy to heroku the app crash and rebuild after 15 minutes of the first deploy and then crash then rebuild and again after 15 minutes apps crash again. After it was crashing within the 1st minute.
Again to resolve the new issue explained above I have similar issue in redis node repo, saying that i have to extend client-output-buffer-limit and maxmemory while in heroku free plan for redis you can't access redis CONFIG command at all. So, I have found another alternative which is using free plan of RedisLabs and the issue of crashing is resolved and my app is working fine now.
This is the result when I open the app in heroku after doing an update.
Application error
An error occurred in the application and your page could not be served. If you are the application owner, check your logs for details. You can do this from the Heroku CLI with the command
heroku logs --tail
This was the procedure I did when doing the updates:
heroku login
heroku info
heroku git:remote --app tcwhais
git add .
git commit -m "first update"
git heroku push master
heroku open
Because the web did not open, I did the following:
I installed the Heroku CLI (64-bit)
heroku login
heroku git:clone -a tcwhais
cd tcwhais
git add .
git commit -am"to correct error"
git push heroku master
heroku open
This is the logs of the updates:
PS C:\Users\User\Desktop\mysite\INO_TCWHAIS> heroku open
PS C:\Users\User\Desktop\mysite\INO_TCWHAIS> heroku logs --tail
2019-11-12T03:06:57.573596+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
2019-11-12T03:06:57.573598+00:00 app[web.1]: self.load_wsgi()
2019-11-12T03:06:57.573600+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
2019-11-12T03:06:57.573602+00:00 app[web.1]: self.wsgi = self.app.wsgi()
2019-11-12T03:06:57.573605+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
2019-11-12T03:06:57.573607+00:00 app[web.1]: self.callable = self.load()
2019-11-12T03:06:57.573609+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
2019-11-12T03:06:57.573610+00:00 app[web.1]: return self.load_wsgiapp()
2019-11-12T03:06:57.573612+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
2019-11-12T03:06:57.573614+00:00 app[web.1]: return util.import_app(self.app_uri)
2019-11-12T03:06:57.573616+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
2019-11-12T03:06:57.573618+00:00 app[web.1]: __import__(module)
2019-11-12T03:06:57.573620+00:00 app[web.1]: File "/app/app.py", line 7, in <module>
2019-11-12T03:06:57.573622+00:00 app[web.1]: import script1
2019-11-12T03:06:57.573624+00:00 app[web.1]: File "/app/script1.py", line 145, in <module>
2019-11-12T03:06:57.573625+00:00 app[web.1]: import requests
2019-11-12T03:06:57.573627+00:00 app[web.1]: ModuleNotFoundError: No module named 'requests'
2019-11-12T03:06:57.574274+00:00 app[web.1]: [2019-11-12 11:06:57 +0800] [10] [INFO] Worker exiting (pid: 10)
2019-11-12T03:06:57.582439+00:00 app[web.1]: [2019-11-12 11:06:57 +0800] [11] [ERROR] Exception in worker process
2019-11-12T03:06:57.582460+00:00 app[web.1]: Traceback (most recent call last):
2019-11-12T03:06:57.582463+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
2019-11-12T03:06:57.582466+00:00 app[web.1]: worker.init_process()
2019-11-12T03:06:57.582470+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
2019-11-12T03:06:57.582472+00:00 app[web.1]: self.load_wsgi()
2019-11-12T03:06:57.582474+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
2019-11-12T03:06:57.582476+00:00 app[web.1]: self.wsgi = self.app.wsgi()
2019-11-12T03:06:57.582478+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
2019-11-12T03:06:57.582480+00:00 app[web.1]: self.callable = self.load()
2019-11-12T03:06:57.582482+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
2019-11-12T03:06:57.582484+00:00 app[web.1]: return self.load_wsgiapp()
2019-11-12T03:06:57.582486+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
2019-11-12T03:06:57.582488+00:00 app[web.1]: return util.import_app(self.app_uri)
2019-11-12T03:06:57.582489+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
2019-11-12T03:06:57.582491+00:00 app[web.1]: __import__(module)
2019-11-12T03:06:57.582493+00:00 app[web.1]: File "/app/app.py", line 7, in <module>
2019-11-12T03:06:57.582495+00:00 app[web.1]: import script1
2019-11-12T03:06:57.582497+00:00 app[web.1]: File "/app/script1.py", line 145, in <module>
2019-11-12T03:06:57.582499+00:00 app[web.1]: import requests
2019-11-12T03:06:57.582506+00:00 app[web.1]: ModuleNotFoundError: No module named 'requests'
2019-11-12T03:06:57.583535+00:00 app[web.1]: [2019-11-12 11:06:57 +0800] [11] [INFO] Worker exiting (pid: 11)
2019-11-12T03:06:57.817914+00:00 app[web.1]: [2019-11-12 11:06:57 +0800] [4] [INFO] Shutting down: Master
2019-11-12T03:06:57.818030+00:00 app[web.1]: [2019-11-12 11:06:57 +0800] [4] [INFO] Reason: Worker failed to boot.
2019-11-12T03:06:57.911942+00:00 heroku[web.1]: State changed from up to crashed
2019-11-12T03:06:57.915850+00:00 heroku[web.1]: State changed from crashed to starting
2019-11-12T03:06:57.888899+00:00 heroku[web.1]: Process exited with status 3
2019-11-12T03:07:06.422412+00:00 heroku[web.1]: Starting process with command `gunicorn app:app`
2019-11-12T03:07:09.884688+00:00 heroku[web.1]: State changed from starting to up
2019-11-12T03:07:09.434491+00:00 app[web.1]: [2019-11-12 11:07:09 +0800] [4] [INFO] Starting gunicorn 19.9.0
2019-11-12T03:07:09.435332+00:00 app[web.1]: [2019-11-12 11:07:09 +0800] [4] [INFO] Listening at: http://0.0.0.0:13556 (4)
2019-11-12T03:07:09.435513+00:00 app[web.1]: [2019-11-12 11:07:09 +0800] [4] [INFO] Using worker: sync
2019-11-12T03:07:09.440985+00:00 app[web.1]: [2019-11-12 11:07:09 +0800] [10] [INFO] Booting worker with pid: 10
2019-11-12T03:07:09.470380+00:00 app[web.1]: [2019-11-12 11:07:09 +0800] [11] [INFO] Booting worker with pid: 11
2019-11-12T03:07:16.486361+00:00 app[web.1]: [2019-11-12 11:07:16 +0800] [11] [ERROR] Exception in worker process
2019-11-12T03:07:16.486392+00:00 app[web.1]: Traceback (most recent call last):
2019-11-12T03:07:16.486396+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
2019-11-12T03:07:16.486399+00:00 app[web.1]: worker.init_process()
2019-11-12T03:07:16.486401+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
2019-11-12T03:07:16.486403+00:00 app[web.1]: self.load_wsgi()
2019-11-12T03:07:16.486450+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
2019-11-12T03:07:16.486452+00:00 app[web.1]: self.wsgi = self.app.wsg
2019-11-12T03:07:16.486450+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
2019-11-12T03:07:16.486452+00:00 app[web.1]: self.wsgi = self.app.wsgi()
2019-11-12T03:07:16.486456+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
2019-11-12T03:07:16.486458+00:00 app[web.1]: self.callable = self.load()
2019-11-12T03:07:16.486460+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
2019-11-12T03:07:16.486462+00:00 app[web.1]: return self.load_wsgiapp()
2019-11-12T03:07:16.486464+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
2019-11-12T03:07:16.486467+00:00 app[web.1]: return util.import_app(self.app_uri)
2019-11-12T03:07:16.486469+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
2019-11-12T03:07:16.486471+00:00 app[web.1]: __import__(module)
2019-11-12T03:07:16.486473+00:00 app[web.1]: File "/app/app.py", line 7, in <module>
2019-11-12T03:07:16.486475+00:00 app[web.1]: import script1
2019-11-12T03:07:16.486477+00:00 app[web.1]: File "/app/script1.py", line 145, in <module>
2019-11-12T03:07:16.486479+00:00 app[web.1]: import requests
2019-11-12T03:07:16.486491+00:00 app[web.1]: ModuleNotFoundError: No module named 'requests'
2019-11-12T03:07:16.487443+00:00 app[web.1]: [2019-11-12 11:07:16 +0800] [11] [INFO] Worker exiting (pid: 11)
2019-11-12T03:07:16.512372+00:00 app[web.1]: [2019-11-12 11:07:16 +0800] [10] [ERROR] Exception in worker process
2019-11-12T03:07:16.512383+00:00 app[web.1]: Traceback (most recent call last):
2019-11-12T03:07:16.512385+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
2019-11-12T03:07:16.512388+00:00 app[web.1]: worker.init_process()
2019-11-12T03:07:16.512390+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
2019-11-12T03:07:16.512392+00:00 app[web.1]: self.load_wsgi()
2019-11-12T03:07:16.512394+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
2019-11-12T03:07:16.512396+00:00 app[web.1]: self.wsgi = self.app.wsgi()
2019-11-12T03:07:16.512399+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
2019-11-12T03:07:16.512401+00:00 app[web.1]: self.callable = self.load()
2019-11-12T03:07:16.512404+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
2019-11-12T03:07:16.512406+00:00 app[web.1]: return self.load_wsgiapp()
2019-11-12T03:07:16.512408+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
2019-11-12T03:07:16.512410+00:00 app[web.1]: return util.import_app(self.app_uri)
2019-11-12T03:07:16.512412+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
2019-11-12T03:07:16.512414+00:00 app[web.1]: __import__(module)
2019-11-12T03:07:16.512416+00:00 app[web.1]: File "/app/app.py", line 7, in <module>
2019-11-12T03:07:16.512418+00:00 app[web.1]: import script1
2019-11-12T03:07:16.512420+00:00 app[web.1]: File "/app/script1.py", line 145, in <module>
2019-11-12T03:07:16.512423+00:00 app[web.1]: import requests
2019-11-12T03:07:16.512434+00:00 app[web.1]: ModuleNotFoundError: No module named 'requests'
2019-11-12T03:07:16.513594+00:00 app[web.1]: [2019-11-12 11:07:16 +0800] [10] [INFO] Worker exiting (pid: 10)
2019-11-12T03:07:16.819875+00:00 app[web.1]: [2019-11-12 11:07:16 +0800] [4] [INFO] Shutting down: Master
2019-11-12T03:07:16.820089+00:00 app[web.1]: [2019-11-12 11:07:16 +0800] [4] [INFO] Reason: Worker failed to boot.
2019-11-12T03:07:16.959707+00:00 heroku[web.1]: State changed from up to crashed
2019-11-12T03:07:16.939371+00:00 heroku[web.1]: Process exited with status 3
How can I solve this problem?
I am trying to deploy my jhipster application to Heroku with yo jhipster:heroku command.But application doesn't start.
'heroku logs --tail' commands shows:
2017-06-22T13:06:43.498695+00:00 app[web.1]: 2017-06-22 13:06:43.498 ERROR 4 --- [ main] o.s.boot.SpringApplication : Application startup failed
2017-06-22T13:06:43.487869+00:00 app[web.1]: 2017-06-22 13:06:43.482 WARN 4 --- [ main] o.s.boot.SpringApplication : Error handling failed (ApplicationEventMulticaster not initialized - call 'refresh' before multicasting events via the context: org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext#1458ed9c: startup date [Thu Jan 01 00:00:00 UTC 1970]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext#10e41621)
2017-06-22T13:06:43.498697+00:00 app[web.1]:
2017-06-22T13:06:43.498699+00:00 app[web.1]: org.springframework.beans.FatalBeanException: Could not copy property 'password' from source to target; nested exception is java.lang.reflect.InvocationTargetException
2017-06-22T13:06:43.498713+00:00 app[web.1]: Caused by: java.lang.reflect.InvocationTargetException: null
2017-06-22T13:06:43.498714+00:00 app[web.1]: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2017-06-22T13:06:43.498715+00:00 app[web.1]: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2017-06-22T13:06:43.498714+00:00 app[web.1]: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2017-06-22T13:06:43.498716+00:00 app[web.1]: at java.lang.reflect.Method.invoke(Method.java:498)
2017-06-22T13:06:43.498716+00:00 app[web.1]: at org.springframework.beans.BeanUtils.copyProperties(BeanUtils.java:617)
2017-06-22T13:06:43.498717+00:00 app[web.1]: ... 16 common frames omitted
2017-06-22T13:06:43.498718+00:00 app[web.1]: Caused by: java.lang.IllegalStateException: Invalid URL: ${JHIPSTER_REGISTRY_URL}/config
2017-06-22T13:06:43.498719+00:00 app[web.1]: at org.springframework.cloud.config.client.ConfigClientProperties.extractCredentials(ConfigClientProperties.java:258)
2017-06-22T13:06:43.498719+00:00 app[web.1]: at org.springframework.cloud.config.client.ConfigClientProperties.getPassword(ConfigClientProperties.java:170)
2017-06-22T13:06:43.498720+00:00 app[web.1]: ... 21 common frames omitted
2017-06-22T13:06:43.498721+00:00 app[web.1]:
2017-06-22T13:06:48.579401+00:00 heroku[web.1]: Starting process with command java $JAVA_OPTS -Xmx256m -jar target/*.war --spring.profiles.active=prod,heroku --server.port=46450
2017-06-22T13:06:49.784412+00:00 app[web.1]: Setting JAVA_TOOL_OPTIONS defaults based on dyno size. Custom settings will override them.
2017-06-22T13:06:49.787364+00:00 app[web.1]: Picked up JAVA_TOOL_OPTIONS: -Xmx350m -Xss512k -Dfile.encoding=UTF-8
2017-06-22T13:06:55.512505+00:00 heroku[web.1]: State changed from starting to crashed
2017-06-22T13:06:55.502911+00:00 heroku[web.1]: Process exited with status 1
2017-06-22T13:06:55.402089+00:00 app[web.1]: 2017-06-22 13:06:55.397 WARN 4 --- [ main] o.s.boot.SpringApplication : Error handling failed (ApplicationEventMulticaster not initialized - call 'refresh' before multicasting events via the context: org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext#10a9d961: startup date [Thu Jan 01 00:00:00 UTC 1970]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext#10e41621)
2017-06-22T13:06:55.412301+00:00 app[web.1]:
2017-06-22T13:06:55.362819+00:00 app[web.1]: :: JHipster 🤓 :: Running Spring Boot 1.5.3.RELEASE ::
2017-06-22T13:06:55.412303+00:00 app[web.1]: at org.springframework.beans.BeanUtils.copyProperties(BeanUtils.java:625)
2017-06-22T13:06:55.412298+00:00 app[web.1]: 2017-06-22 13:06:55.411 ERROR 4 --- [ main] o.s.boot.SpringApplication : Application startup failed
2017-06-22T13:06:55.412303+00:00 app[web.1]: at org.springframework.beans.BeanUtils.copyProperties(BeanUtils.java:537)
2017-06-22T13:06:55.412304+00:00 app[web.1]: at org.springframework.cloud.config.client.ConfigClientProperties.override(ConfigClientProperties.java:317)
2017-06-22T13:06:55.412306+00:00 app[web.1]: at org.springframework.cloud.bootstrap.config.PropertySourceBootstrapConfiguration.initialize(PropertySourceBootstrapConfiguration.java:93)
2017-06-22T13:06:55.412302+00:00 app[web.1]: org.springframework.beans.FatalBeanException: Could not copy property 'password' from source to target; nested exception is java.lang.reflect.InvocationTargetException
2017-06-22T13:06:55.412307+00:00 app[web.1]: at org.springframework.boot.SpringApplication.applyInitializers(SpringApplication.java:611)
I am trying to deploy a rails app running on puma and jruby. the Procfile is as follows
web: bundle exec puma -C config/puma.rb -p $PORT -e $RACK_ENV
configuration for Puma placed in
config/puma.rb
if ENV['RACK_ENV'] != 'production' || ENV['RAILS_ENV'] != 'production'
workers Integer(ENV['PUMA_WORKERS'] || 4)
end
threads Integer(ENV['MIN_THREADS'] || 1), Integer(ENV['MAX_THREADS'] || 4)
rackup DefaultRackup
port ENV['PORT'] || 3000
environment ENV['RACK_ENV'] || 'development'
preload_app!
on_worker_boot do
# worker specific setup
ActiveSupport.on_load(:active_record) do
config = ActiveRecord::Base.configurations[Rails.env] ||
Rails.application.config.database_configuration[Rails.env]
config['pool'] = ENV['MAX_THREADS'] || 16
ActiveRecord::Base.establish_connection(config)
end
end
the reason I have a conditional initialisation for workers is because Heroku complains (or may be Puma) worker mode is not available for JRuby and Windows; anyway,
The strange behaviour I notice on heroku is that (unlike) my local environment a clustered Puma instance starts without any problems, and picks up two different ports to bind i.e. 3000 and 5000.
However, on production i.e. Heroku this causes a crash
2014-06-23T08:50:54.545724+00:00 heroku[web.1]: State changed from crashed to starting
2014-06-23T08:51:02.510184+00:00 app[web.1]: Picked up JAVA_TOOL_OPTIONS: -Djava.rmi.server.useCodebaseOnly=true
2014-06-23T08:51:01.333763+00:00 heroku[web.1]: Starting process with command `bundle exec puma -C config/puma.rb -p 49576 -e production`
2014-06-23T08:51:21.658663+00:00 app[web.1]: Picked up JAVA_TOOL_OPTIONS: -Djava.rmi.server.useCodebaseOnly=true
2014-06-23T08:51:44.964380+00:00 app[web.1]: * Version 2.8.2 (jruby 1.9.3), codename: Sir Edmund Percival Hillary
2014-06-23T08:51:44.962724+00:00 app[web.1]: The signal USR1 is in use by the JVM and will not work correctly on this platform
2014-06-23T08:51:44.964563+00:00 app[web.1]: * Min threads: 1, max threads: 4
2014-06-23T08:51:44.964738+00:00 app[web.1]: * Environment: production
2014-06-23T08:51:44.963925+00:00 app[web.1]: Puma starting in single mode...
2014-06-23T08:51:57.852471+00:00 app[web.1]: Errno::EADDRINUSE: Address already in use - bind - Address already in use
2014-06-23T08:51:57.830580+00:00 app[web.1]: * Listening on tcp://0.0.0.0:49576
2014-06-23T08:51:57.852474+00:00 app[web.1]: initialize at org/jruby/ext/socket/RubyTCPServer.java:118
2014-06-23T08:51:57.852478+00:00 app[web.1]: add_tcp_listener at /app/vendor/bundle/jruby/1.9/gems/puma-2.8.2-java/lib/puma/binder.rb:195
2014-06-23T08:51:57.852476+00:00 app[web.1]: new at org/jruby/RubyIO.java:852
2014-06-23T08:51:57.850256+00:00 app[web.1]: * Listening on tcp://0.0.0.0:49576
2014-06-23T08:51:57.852479+00:00 app[web.1]: parse at /app/vendor/bundle/jruby/1.9/gems/puma-2.8.2-java/lib/puma/binder.rb:96
2014-06-23T08:51:57.852481+00:00 app[web.1]: each at org/jruby/RubyArray.java:1613
2014-06-23T08:51:57.852482+00:00 app[web.1]: parse at /app/vendor/bundle/jruby/1.9/gems/puma-2.8.2-java/lib/puma/binder.rb:82
2014-06-23T08:51:57.852484+00:00 app[web.1]: load_and_bind at /app/vendor/bundle/jruby/1.9/gems/puma-2.8.2-java/lib/puma/runner.rb:119
2014-06-23T08:51:57.852486+00:00 app[web.1]: run at /app/vendor/bundle/jruby/1.9/gems/puma-2.8.2-java/lib/puma/single.rb:73
2014-06-23T08:51:57.852487+00:00 app[web.1]: run at /app/vendor/bundle/jruby/1.9/gems/puma-2.8.2-java/lib/puma/cli.rb:499
2014-06-23T08:51:57.852489+00:00 app[web.1]: (root) at /app/vendor/bundle/jruby/1.9/gems/puma-2.8.2-java/bin/puma:10
2014-06-23T08:51:57.852491+00:00 app[web.1]: load at org/jruby/RubyKernel.java:1099
2014-06-23T08:51:57.852492+00:00 app[web.1]: (root) at /app/vendor/bundle/jruby/1.9/bin/puma:23
2014-06-23T08:51:59.347012+00:00 heroku[web.1]: State changed from starting to crashed
2014-06-23T08:51:59.333790+00:00 heroku[web.1]: Process exited with status 1
This strange startup behaviour is not seen when I run it on my laptop.
The only way I have got puma working on jruby in Heroku is through this setting
if ENV['RACK_ENV'] != 'production' || ENV['RAILS_ENV'] != 'production'
workers Integer(ENV['PUMA_WORKERS'] || 4)
port ENV['PORT'] || 3000
environment ENV['RACK_ENV'] || 'development'
end
threads Integer(ENV['MIN_THREADS'] || 1), Integer(ENV['MAX_THREADS'] || 4)
rackup DefaultRackup
preload_app!
on_worker_boot do
# worker specific setup
ActiveSupport.on_load(:active_record) do
config = ActiveRecord::Base.configurations[Rails.env] ||
Rails.application.config.database_configuration[Rails.env]
config['pool'] = ENV['MAX_THREADS'] || 16
ActiveRecord::Base.establish_connection(config)
end
end
i.e. if the port and environment directives are out side the if block Heroku somehow tried to run two instances on the same port, which will obviously cause a crash.
Am I missing any configuration here or basically running in clustered mode (I even tried running on Ruby 2.0.0 instead of JRuby) is just not possible in Heroku
* Update *
Procfile
web: bundle exec puma -C config/puma.rb
config/puma.rb
if ENV['RACK_ENV'] != 'production' || ENV['RAILS_ENV'] != 'production'
workers Integer(ENV['PUMA_WORKERS'] || 4)
end
threads Integer(ENV['MIN_THREADS'] || 1), Integer(ENV['MAX_THREADS'] || 16)
rackup DefaultRackup
port ENV['PORT'] || 3000
environment ENV['RACK_ENV'] || 'development'
preload_app!
on_worker_boot do
# worker specific setup
ActiveSupport.on_load(:active_record) do
config = ActiveRecord::Base.configurations[Rails.env] ||
Rails.application.config.database_configuration[Rails.env]
config['pool'] = ENV['MAX_THREADS'] || 16
ActiveRecord::Base.establish_connection(config)
end
end
The reason for conditional block (quoted from initial problem description
the reason I have a conditional initialisation for workers is because
Heroku complains (or may be Puma) worker mode is not available for
JRuby and Windows;
And error logs (even if I remove JRUBY, the conditional worker block) and try to run on clustered mode
606 2014-06-23T07:13:42.058507+00:00 heroku[api]: Release v35 created by foo#bar.com
607 2014-06-23T07:13:47.674708+00:00 app[web.1]: [2] Puma starting in cluster mode...
608 2014-06-23T07:13:47.674777+00:00 app[web.1]: [2] * Version 2.8.2 (ruby 2.1.1-p76), codename: Sir Edmund Percival Hillary
609 2014-06-23T07:13:47.674816+00:00 app[web.1]: [2] * Min threads: 1, max threads: 16
610 2014-06-23T07:13:47.674939+00:00 app[web.1]: [2] * Preloading application
611 2014-06-23T07:13:47.674858+00:00 app[web.1]: [2] * Environment: production
612 2014-06-23T07:13:47.674900+00:00 app[web.1]: [2] * Process workers: 4
613 2014-06-23T07:13:49.078330+00:00 app[web.1]: [2] * Listening on tcp://0.0.0.0:45245
614 2014-06-23T07:13:49.078766+00:00 app[web.1]: /app/vendor/bundle/ruby/2.1.0/gems/puma-2.8.2/lib/puma/binder.rb:195:in `initialize': Address already in use - bind(2) for "0.0.0.0 " port 45245 (Errno::EADDRINUSE)
615 2014-06-23T07:13:49.078775+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.1.0/gems/puma-2.8.2/lib/puma/binder.rb:195:in `add_tcp_listener'
616 2014-06-23T07:13:49.078771+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.1.0/gems/puma-2.8.2/lib/puma/binder.rb:195:in `new'
617 2014-06-23T07:13:49.078780+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.1.0/gems/puma-2.8.2/lib/puma/binder.rb:82:in `each'
618 2014-06-23T07:13:49.078785+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.1.0/gems/puma-2.8.2/lib/puma/runner.rb:119:in `load_and_bind'
619 2014-06-23T07:13:49.078778+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.1.0/gems/puma-2.8.2/lib/puma/binder.rb:96:in `block in parse'
620 2014-06-23T07:13:49.078784+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.1.0/gems/puma-2.8.2/lib/puma/binder.rb:82:in `parse'
621 2014-06-23T07:13:49.078793+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.1.0/gems/puma-2.8.2/lib/puma/cli.rb:499:in `run'
622 2014-06-23T07:13:49.078789+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.1.0/gems/puma-2.8.2/lib/puma/cluster.rb:271:in `run'
623 2014-06-23T07:13:49.078794+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.1.0/gems/puma-2.8.2/bin/puma:10:in `<top (required)>'
624 2014-06-23T07:13:49.078798+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.1.0/bin/puma:23:in `load'
625 2014-06-23T07:13:49.078801+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.1.0/bin/puma:23:in `<main>'
626 2014-06-23T07:13:49.078565+00:00 app[web.1]: [2] * Listening on tcp://0.0.0.0:45245
627 2014-06-23T07:13:50.359244+00:00 heroku[web.1]: Process exited with status 1
628 2014-06-23T07:13:45.838792+00:00 heroku[web.1]: Starting process with command `bundle exec puma -C config/puma.rb -p 45245 -e production`
629 2014-06-23T07:13:50.385427+00:00 heroku[web.1]: State changed from starting to crashed
630 2014-06-23T07:13:50.385913+00:00 heroku[web.1]: State changed from crashed to starting
631 2014-06-23T07:13:57.445391+00:00 app[web.1]: [2] Puma starting in cluster mode...
632 2014-06-23T07:13:57.445417+00:00 app[web.1]: [2] * Environment: production
633 2014-06-23T07:13:57.445413+00:00 app[web.1]: [2] * Version 2.8.2 (ruby 2.1.1-p76), codename: Sir Edmund Percival Hillary
634 2014-06-23T07:13:57.445415+00:00 app[web.1]: [2] * Min threads: 1, max threads: 16
635 2014-06-23T07:13:57.445419+00:00 app[web.1]: [2] * Process workers: 4
636 2014-06-23T07:13:57.445423+00:00 app[web.1]: [2] * Preloading application
637 2014-06-23T07:13:54.371058+00:00 heroku[web.1]: Starting process with command `bundle exec puma -C config/puma.rb -p 33243 -e production`
638 2014-06-23T07:13:59.883490+00:00 app[web.1]: /app/vendor/bundle/ruby/2.1.0/gems/puma-2.8.2/lib/puma/binder.rb:195:in `initialize': Address already in use - bind(2) for "0.0.0.0 " port 33243 (Errno::EADDRINUSE)
639 2014-06-23T07:13:59.882892+00:00 app[web.1]: [2] * Listening on tcp://0.0.0.0:33243
640 2014-06-23T07:13:59.883501+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.1.0/gems/puma-2.8.2/lib/puma/binder.rb:96:in `block in parse'
641 2014-06-23T07:13:59.883494+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.1.0/gems/puma-2.8.2/lib/puma/binder.rb:195:in `new'
642 2014-06-23T07:13:59.883499+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.1.0/gems/puma-2.8.2/lib/puma/binder.rb:195:in `add_tcp_listener'
643 2014-06-23T07:13:59.883506+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.1.0/gems/puma-2.8.2/lib/puma/binder.rb:82:in `parse'
644 2014-06-23T07:13:59.883510+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.1.0/gems/puma-2.8.2/lib/puma/runner.rb:119:in `load_and_bind'
645 2014-06-23T07:13:59.883503+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.1.0/gems/puma-2.8.2/lib/puma/binder.rb:82:in `each'
646 2014-06-23T07:13:59.883512+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.1.0/gems/puma-2.8.2/lib/puma/cluster.rb:271:in `run'
647 2014-06-23T07:13:59.883519+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.1.0/bin/puma:23:in `load'
648 2014-06-23T07:13:59.883517+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.1.0/gems/puma-2.8.2/bin/puma:10:in `<top (required)>'
649 2014-06-23T07:13:59.883514+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.1.0/gems/puma-2.8.2/lib/puma/cli.rb:499:in `run'
650 2014-06-23T07:13:59.883525+00:00 app[web.1]: from /app/vendor/bundle/ruby/2.1.0/bin/puma:23:in `<main>'
651 2014-06-23T07:13:59.883273+00:00 app[web.1]: [2] * Listening on tcp://0.0.0.0:33243
652 2014-06-23T07:14:01.451472+00:00 heroku[api]: Set DEV-2-PASS config vars by foo#bar.com
653 2014-06-23T07:14:01.451540+00:00 heroku[api]: Release v36 created by foo#bar.com
654 2014-06-23T07:14:01.624035+00:00 heroku[web.1]: State changed from starting to crashed
655 2014-06-23T07:14:01.963688+00:00 heroku[web.1]: State changed from crashed to starting
656 2014-06-23T07:14:01.615855+00:00 heroku[web.1]: Process exited with status 1
I just had right the same issue as you.
It seems like its related to the duplicate definition of the port
First in the Procfile and later in the config/puma.rb file
I removed the port from the Procfile (as well as the environment), as this is already defined in config/puma.rb.
web: bundle exec puma -C config/puma.rb
I have a problem with a process in an app of mine hosted on Heroku (2xweb + 1x worder dynos). The process ends with quite a heavy email being created and sent via SendGrid. This took some time, causing web worker timeouts and bad usability, so it was refactored into a worker, which I thought would solve the problem, but I'm getting situations like this:
Apr 10 17:12:48 wc heroku/web.2: Processing by DealsController#show as */*
[request is processed]
Apr 10 17:12:50 wc app/worker.1: [worker sending emails]
[a lot of lines with debug data cut]
Apr 10 17:12:53 wc heroku/worker.1: source=worker.1 dyno=heroku.16079351.858a3455-0b9f-4f75-9052-b419d4653703 sample#load_avg_1m=0.29 sample#load_avg_5m=0.07 sample#load_avg_15m=0.02
Apr 10 17:12:53 wc heroku/worker.1: source=worker.1 dyno=heroku.16079351.858a3455-0b9f-4f75-9052-b419d4653703 sample#memory_total=240.45MB sample#memory_rss=240.34MB sample#memory_cache=0.11MB sample#memory_swap=0.00MB sample#memory_pgpgin=85990pages sample#memory_pgpgout=24436pages
Apr 10 17:12:53 wc heroku/web.2: source=web.2 dyno=heroku.16079351.879182ef-13f0-4908-bf35-c487ccab6153 sample#load_avg_1m=0.00 sample#load_avg_5m=0.01 sample#load_avg_15m=0.01
Apr 10 17:12:54 wc heroku/web.2: source=web.2 dyno=heroku.16079351.879182ef-13f0-4908-bf35-c487ccab6153 sample#memory_total=844.16MB sample#memory_rss=511.82MB sample#memory_cache=0.00MB sample#memory_swap=332.34MB sample#memory_pgpgin=223581pages sample#memory_pgpgout=92554pages
Apr 10 17:12:54 wc heroku/web.2: Process running mem=844M(164.9%)
Apr 10 17:12:54 wc heroku/web.2: Error R14 (Memory quota exceeded)
Apr 10 17:12:54 wc app/web.2: ** [NewRelic][04/10/14 15:12:54 +0000 879182ef-13f0-4908-bf35-c487ccab6153 (468)] INFO : Starting Agent shutdown
Apr 10 17:12:55 wc app/worker.1: ** [Bugsnag] Bugsnag exception handler 1.6.2 ready, api_key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
[8 new relic lines cut]
Apr 10 17:12:56 wc app/heroku-postgres: source=HEROKU_POSTGRESQL_WHITE sample#current_transaction=144542 sample#db_size=189720760bytes sample#tables=92 sample#active-connections=12 sample#waiting-connections=0 sample#index-cache-hit-rate=0.99981 sample#table-cache-hit-rate=0.99868 sample#load-avg-1m=0.36 sample#load-avg-5m=0.3 sample#load-avg-15m=0.285 sample#read-iops=38.367 sample#write-iops=13.221 sample#memory-total=7629464kB sample#memory-free=187884kB sample#memory-cached=6599816kB sample#memory-postgres=689216kB
Apr 10 17:12:56 wc app/worker.1: ** [NewRelic][04/10/14 15:12:55 +0000 858a3455-0b9f-4f75-9052-b419d4653703 (99)] INFO : Installing DelayedJob instrumentation hooks
[11 new relic lines cut]
Apr 10 17:12:59 wc app/worker.1: Delayed::Backend::ActiveRecord::Job Load (1.6ms) UPDATE "delayed_jobs" SET locked_at = '2014-04-10 15:12:59.482734', locked_by = 'host:858a3455-0b9f-4f75-9052-b419d4653703 pid:2' WHERE id IN (SELECT id FROM "delayed_jobs" WHERE ((run_at <= '2014-04-10 15:12:59.481969' AND (locked_at IS NULL OR locked_at < '2014-04-10 11:12:59.482002') OR locked_by = 'host:858a3455-0b9f-4f75-9052-b419d4653703 pid:2') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *
Apr 10 17:12:59 wc app/worker.1: ** [NewRelic][04/10/14 15:12:59 +0000 858a3455-0b9f-4f75-9052-b419d4653703 (99)] INFO : Starting Agent shutdown
Apr 10 17:13:09 wc app/worker.1: Delayed::Backend::ActiveRecord::Job Load (1.6ms) UPDATE "delayed_jobs" SET locked_at = '2014-04-10 15:13:09.486147', locked_by = 'host:858a3455-0b9f-4f75-9052-b419d4653703 pid:2' WHERE id IN (SELECT id FROM "delayed_jobs" WHERE ((run_at <= '2014-04-10 15:13:09.485453' AND (locked_at IS NULL OR locked_at < '2014-04-10 11:13:09.485486') OR locked_by = 'host:858a3455-0b9f-4f75-9052-b419d4653703 pid:2') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *
Apr 10 17:13:13 wc heroku/web.2: source=web.2 dyno=heroku.16079351.879182ef-13f0-4908-bf35-c487ccab6153 sample#load_avg_1m=0.00 sample#load_avg_5m=0.01 sample#load_avg_15m=0.01
Apr 10 17:13:14 wc heroku/web.2: source=web.2 dyno=heroku.16079351.879182ef-13f0-4908-bf35-c487ccab6153 sample#memory_total=463.75MB sample#memory_rss=153.29MB sample#memory_cache=0.00MB sample#memory_swap=310.46MB sample#memory_pgpgin=245188pages sample#memory_pgpgout=205946pages
Apr 10 17:13:14 wc app/web.2: Started GET "/user" for [IP.IP.IP.IP] at 2014-04-10 15:13:13 +0000
Apr 10 17:13:14 wc heroku/worker.1: source=worker.1 dyno=heroku.16079351.858a3455-0b9f-4f75-9052-b419d4653703 sample#load_avg_1m=0.21 sample#load_avg_5m=0.06 sample#load_avg_15m=0.02
Apr 10 17:13:14 wc heroku/worker.1: source=worker.1 dyno=heroku.16079351.858a3455-0b9f-4f75-9052-b419d4653703 sample#memory_total=157.25MB sample#memory_rss=157.14MB sample#memory_cache=0.11MB sample#memory_swap=0.00MB sample#memory_pgpgin=103179pages sample#memory_pgpgout=62923pages
Apr 10 17:13:16 wc app/web.2: Started GET "/user" for [IP.IP.IP.IP] at 2014-04-10 15:13:16 +0000
Apr 10 17:13:17 wc heroku/router: at=error code=H12 desc="Request timeout" method=GET path=/pages/planning host=www.cool-app.com request_id=c62a7ee5-11d8-4286-846a-a55861cc6a0e fwd="[IP.IP.IP.IP]" dyno=web.2 connect=2ms service=30000ms status=503 bytes=0
Apr 10 17:13:19 wc app/web.2: E, [2014-04-10T15:13:18.948990 #2] ERROR -- : worker=1 PID:12 timeout (31s > 30s), killing
Apr 10 17:13:19 wc app/worker.1: Delayed::Backend::ActiveRecord::Job Load (1.5ms) UPDATE "delayed_jobs" SET locked_at = '2014-04-10 15:13:19.489494', locked_by = 'host:858a3455-0b9f-4f75-9052-b419d4653703 pid:2' WHERE id IN (SELECT id FROM "delayed_jobs" WHERE ((run_at <= '2014-04-10 15:13:19.488845' AND (locked_at IS NULL OR locked_at < '2014-04-10 11:13:19.488874') OR locked_by = 'host:858a3455-0b9f-4f75-9052-b419d4653703 pid:2') AND failed_at IS NULL) ORDER BY priority ASC, run_at ASC LIMIT 1 FOR UPDATE) RETURNING *
Apr 10 17:13:20 wc app/web.2: E, [2014-04-10T15:13:19.689336 #2] ERROR -- : reaped #<Process::Status: pid 12 SIGKILL (signal 9)> worker=1
Apr 10 17:13:21 wc app/web.2: Disconnected from ActiveRecord
Apr 10 17:13:27 wc app/web.2: Processing by UsersController#show as HTML
[web worker 2 works properly from now on]
The web.2 worker still crashes resulting in a ~20 second hang for the user and and "Application Error" screen being shown. What's strange is that this happens on different pages and seems to be linked to the worker in the background.
The line that especially confuses me (and the one probably symptomatic of the crash) is:
Apr 10 17:13:19 wc app/web.2: E, [2014-04-10T15:13:18.948990 #2] ERROR -- : worker=1 PID:12 timeout (31s > 30s), killing
What does it mean? It seems to me that the web.2 dyno is being killed because the worker=1 had a timout, which seems a bit crazy.
The configuration for the dynos is:
Dynos:
1x - 2 - web bundle exec unicorn -p $PORT -c ./config/unicorn.rb
1x - 1 - worker bundle exec rake jobs:work
Any ideas?
Solution: switch from delayed_job to sidekiq
To whom it may concern - despite several hours of debugging, I was never really able to get to the bottom of this. Decided to try switching this one job over to sidekiq. This solved the issue. Probably just a bug with the particular delayed_job -> heroku interaction. Maybe this would be working today...