Docker : Worker keep exiting and respawn and 99% CPU - laravel

Supervisord can't keep redis worker on, my worker still exited with satus code 12 and respawn. My redis container is on and on the the docker network of app (where supervisor be)
I follow the doc of laravel : https://laravel.com/docs/5.8/queues#supervisor-configuration
I tried to daemonize the command and some code update
I updated my Debian to Stretch and Docker as well
I tested all of it in local, everything works fine...
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/artisan queue:work redis --sleep=3 --tries=3
autostart=true
autorestart=true
user=www-data
numprocs=8
priority=10
redirect_stderr=true
stdout_logfile=/var/log/worker.log
api:
image: gitlab.ladechetterieduweb.com:5555/lddw/backend:latest
container_name: backend-lddw-develop
restart: always
working_dir: /var/www
volumes:
- ./config/api:/var/env
- ./app/storage:/var/www/storage
- ./logs/laravel:/var/www/storage/logs
- ./logs/supervisord:/var/log
depends_on:
- db
command: /bin/bash -c "cp /var/env/.env /var/www/.env && supervisord -c /etc/supervisord.conf --nodaemon"
networks:
- app-network
redis:
image: redis:5.0.3-stretch
restart: always
ports:
- 6379:6379
container_name: redis-lddw-develop
volumes:
- redis_data:/data
- ./config/redis:/usr/local/etc/redis
command: /bin/bash -c "cp /usr/local/etc/redis/rc.local /etc/rc.local && redis-server --appendonly yes"
networks:
- app-network
2019-06-25 21:50:15,199 CRIT Set uid to user 0
2019-06-25 21:50:15,199 WARN No file matches via include "/etc/supervisor/conf.d/*.conf"
2019-06-25 21:50:15,295 INFO RPC interface 'supervisor' initialized
2019-06-25 21:50:15,295 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2019-06-25 21:50:15,457 INFO RPC interface 'supervisor' initialized
2019-06-25 21:50:15,457 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2019-06-25 21:50:15,457 INFO supervisord started with pid 1
2019-06-25 21:50:16,460 INFO spawned: 'php-fpm' with pid 10
2019-06-25 21:50:16,461 INFO spawned: 'laravel-worker_00' with pid 11
2019-06-25 21:50:16,463 INFO spawned: 'laravel-worker_01' with pid 12
2019-06-25 21:50:16,464 INFO spawned: 'laravel-worker_02' with pid 13
2019-06-25 21:50:16,467 INFO spawned: 'laravel-worker_03' with pid 14
2019-06-25 21:50:16,469 INFO spawned: 'laravel-worker_04' with pid 15
2019-06-25 21:50:16,472 INFO spawned: 'laravel-worker_05' with pid 16
2019-06-25 21:50:16,474 INFO spawned: 'laravel-worker_06' with pid 17
2019-06-25 21:50:16,476 INFO spawned: 'laravel-worker_07' with pid 18
2019-06-25 21:50:17,667 INFO success: php-fpm entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-06-25 21:50:17,667 INFO success: laravel-worker_00 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-06-25 21:50:17,667 INFO success: laravel-worker_01 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-06-25 21:50:17,667 INFO success: laravel-worker_02 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-06-25 21:50:17,667 INFO success: laravel-worker_03 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-06-25 21:50:17,668 INFO success: laravel-worker_04 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-06-25 21:50:17,668 INFO success: laravel-worker_05 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-06-25 21:50:17,668 INFO success: laravel-worker_06 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-06-25 21:50:17,668 INFO success: laravel-worker_07 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-06-25 21:50:26,205 INFO exited: laravel-worker_00 (exit status 12; not expected)
2019-06-25 21:50:26,205 INFO exited: laravel-worker_02 (exit status 12; not expected)
2019-06-25 21:50:26,205 INFO exited: laravel-worker_03 (exit status 12; not expected)
2019-06-25 21:50:26,205 INFO exited: laravel-worker_05 (exit status 12; not expected)
2019-06-25 21:50:26,205 INFO exited: laravel-worker_06 (exit status 12; not expected)
2019-06-25 21:50:26,205 INFO exited: laravel-worker_07 (exit status 12; not expected)

Your workers are exiting with status 12 :
2019-06-25 21:50:26,205 INFO exited: laravel-worker_00 (exit status 12; not expected)
2019-06-25 21:50:26,205 INFO exited: laravel-worker_02 (exit status 12; not expected)
2019-06-25 21:50:26,205 INFO exited: laravel-worker_03 (exit status 12; not expected)
2019-06-25 21:50:26,205 INFO exited: laravel-worker_05 (exit status 12; not expected)
2019-06-25 21:50:26,205 INFO exited: laravel-worker_06 (exit status 12; not expected)
2019-06-25 21:50:26,205 INFO exited: laravel-worker_07 (exit status 12; not expected)
This exit code is triggered when your worker is consuming too much memory, see :
https://github.com/laravel/framework/blob/5.8/src/Illuminate/Queue/Worker.php#L204
$this->memoryExceeded($options->memory) returns true and exit.
You have 2 options, reducing the memory footprint of your worker or increasing memory allowed to it. As the default is pretty low (128MB), you can try to add some memory.
To change memory allowed to your workers, you have to edit your supervisord configuration :
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/artisan queue:work redis --sleep=3 --tries=3 --memory=1024
autostart=true
autorestart=true
user=www-data
numprocs=8
priority=10
redirect_stderr=true
stdout_logfile=/var/log/worker.log
See the --memory I added to your conf in the command section
Regards

Related

Run cron with supervisor and docker

I'm stuck on an issue with a legacy Laravel project. It uses supervisor and cron to run the scheduled tasks, but it seems that the cronjobs won't run (and have never run apparently).
This is the Dockerfile:
FROM 704666026001.dkr.ecr.eu-central-1.amazonaws.com/laravel-prod
# Copy project
COPY . /var/www/html/
# Copy cronjob setup fro laravel scheduler
COPY docker/cron/cron.txt /etc/docker/cron/cron.txt
# Copy laravel queue worker supervisor conf
COPY docker/supervisor /etc/docker/supervisor/conf
RUN mkdir -p /var/www/html/storage/framework/cache/data \
&& /usr/bin/crontab -u www-data /etc/docker/cron/cron.txt \
&& chown -R www-data:www-data /var/www/html/
In the docker/supervisor folder, there a two files:
One named queue-worker.conf with:
[group:laravel]
programs=laravel-worker
priority=30
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/artisan queue:work --sleep=3 --tries=3
user=www-data
numprocs=1
startsecs=10
autostart=true
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
And cron.conf with:
[group:cron]
programs=crond
priority=40
[program:crond]
process_name=%(program_name)s
command=crond -f
user=www-data
autostart=true
autorestart=true
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
And the file docker/cron/cron.txt has one line:
* * * * * php /var/www/html/artisan schedule:run >> /dev/null 2>&1
The docker image does build without any errors. When i run it locally, this is the output:
2020-06-16 10:21:05,045 INFO Included extra file "/etc/docker/supervisor/conf/cron.conf" during parsing
2020-06-16 10:21:05,045 INFO Included extra file "/etc/docker/supervisor/conf/nginx.conf" during parsing
2020-06-16 10:21:05,045 INFO Included extra file "/etc/docker/supervisor/conf/php-fpm.conf" during parsing
2020-06-16 10:21:05,045 INFO Included extra file "/etc/docker/supervisor/conf/queue-worker.conf" during parsing
2020-06-16 10:21:05,062 INFO RPC interface 'supervisor' initialized
2020-06-16 10:21:05,063 INFO supervisord started with pid 1
2020-06-16 10:21:06,073 INFO spawned: 'nginxd' with pid 9
2020-06-16 10:21:06,078 INFO spawned: 'php-fpmd' with pid 10
2020-06-16 10:21:06,084 INFO spawned: 'laravel-worker_00' with pid 11
2020-06-16 10:21:06,088 INFO spawned: 'crond' with pid 12
2020/06/16 10:21:06 [notice] 9#9: using the "epoll" event method
2020/06/16 10:21:06 [notice] 9#9: nginx/1.16.1
2020/06/16 10:21:06 [notice] 9#9: OS: Linux 4.19.76-linuxkit
2020/06/16 10:21:06 [notice] 9#9: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2020/06/16 10:21:06 [notice] 9#9: start worker processes
2020-06-16 10:21:06,121 INFO success: nginxd entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2020-06-16 10:21:06,121 INFO success: php-fpmd entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2020/06/16 10:21:06 [notice] 9#9: start worker process 13
2020/06/16 10:21:06 [notice] 9#9: start worker process 14
2020/06/16 10:21:06 [notice] 9#9: start worker process 15
2020/06/16 10:21:06 [notice] 9#9: start worker process 16
2020/06/16 10:21:06 [notice] 9#9: start cache manager process 17
2020/06/16 10:21:06 [notice] 9#9: start cache loader process 18
[16-Jun-2020 10:21:06] NOTICE: fpm is running, pid 10
[16-Jun-2020 10:21:06] NOTICE: ready to handle connections
2020-06-16 10:21:07,259 INFO success: crond entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-06-16 10:21:16,253 INFO success: laravel-worker_00 entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
It does show 'crond entered RUNNING state', but the cronjob isn't run in any way.
Does anyone have an idea why? Is this setup even valid?
Thanks in advance for the help!
Supervisor stops runs if they are not doing anything for certain amount of time.
So with cron you have task only running intermittently. So it gets shut down.

Running Kafka-Manager inside Docker container on Windows

I am following this tutorial to run Kafka inside a Docker container on windows.
When I try to launch Kafka-Manager by opening http://localhost:9000 in the browser as described there, I get ERR_CONNECTION_REFUSED.
Something I think might be related is that at the first time I ran docker-compose up, PowerShell showed an error saying I needed to run some command first, to open a virtual machine or something like that.
Then I ran the command that PowerShell had told me and then I managed to run docker-compose up successfully. However the tutorial didn't mention anything about it, and since then every time I tried to run docker-compose up I managed to to it without running another command first, even if I closed and reopened PowerShell.
I suspect PowerShell remembers I'm connected to a virtual machine so docker-compose up runs Kafka inside a virtual machine, and therefore I can't reach Kafka-Manager in the browser, although I see shows the following message:
kafkamanager | [info] p.c.s.NettyServer - Listening for HTTP on
/0.0.0.0:9000
Edit:
docker logs for kafka container:
/usr/lib/python2.7/dist-packages/supervisor/options.py:296: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
'Supervisord is running as root and it is searching '
2020-02-28 08:37:37,274 CRIT Supervisor running as root (no user in config file)
2020-02-28 08:37:37,274 WARN Included extra file "/etc/supervisor/conf.d/zookeeper.conf" during parsing
2020-02-28 08:37:37,274 WARN Included extra file "/etc/supervisor/conf.d/kafka.conf" during parsing
2020-02-28 08:37:37,303 INFO RPC interface 'supervisor' initialized
2020-02-28 08:37:37,303 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2020-02-28 08:37:37,303 INFO supervisord started with pid 1
2020-02-28 08:37:38,306 INFO spawned: 'zookeeper' with pid 8
2020-02-28 08:37:38,308 INFO spawned: 'kafka' with pid 9
2020-02-28 08:37:39,372 INFO success: zookeeper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-28 08:37:39,372 INFO success: kafka entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-28 21:16:01,095 WARN received SIGTERM indicating exit request
2020-02-28 21:16:01,095 INFO waiting for zookeeper, kafka to die
2020-02-28 21:16:02,102 INFO stopped: kafka (terminated by SIGTERM)
2020-02-28 21:16:02,442 INFO stopped: zookeeper (exit status 143)
/usr/lib/python2.7/dist-packages/supervisor/options.py:296: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
'Supervisord is running as root and it is searching '
2020-02-28 21:17:50,843 CRIT Supervisor running as root (no user in config file)
2020-02-28 21:17:50,843 WARN Included extra file "/etc/supervisor/conf.d/zookeeper.conf" during parsing
2020-02-28 21:17:50,843 WARN Included extra file "/etc/supervisor/conf.d/kafka.conf" during parsing
2020-02-28 21:17:50,858 INFO RPC interface 'supervisor' initialized
2020-02-28 21:17:50,858 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2020-02-28 21:17:50,859 INFO supervisord started with pid 1
2020-02-28 21:17:51,862 INFO spawned: 'zookeeper' with pid 8
2020-02-28 21:17:51,864 INFO spawned: 'kafka' with pid 9
2020-02-28 21:17:52,926 INFO success: zookeeper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-28 21:17:52,927 INFO success: kafka entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-28 21:17:59,672 INFO exited: kafka (exit status 1; not expected)
2020-02-28 21:18:00,675 INFO spawned: 'kafka' with pid 297
2020-02-28 21:18:01,694 INFO success: kafka entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-29 19:42:18,487 WARN received SIGTERM indicating exit request
2020-02-29 19:42:18,487 INFO waiting for zookeeper, kafka to die
2020-02-29 19:42:18,488 INFO stopped: kafka (terminated by SIGTERM)
2020-02-29 19:42:18,821 INFO stopped: zookeeper (exit status 143)
/usr/lib/python2.7/dist-packages/supervisor/options.py:296: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
'Supervisord is running as root and it is searching '
2020-02-29 19:42:26,841 CRIT Supervisor running as root (no user in config file)
2020-02-29 19:42:26,841 WARN Included extra file "/etc/supervisor/conf.d/zookeeper.conf" during parsing
2020-02-29 19:42:26,842 WARN Included extra file "/etc/supervisor/conf.d/kafka.conf" during parsing
2020-02-29 19:42:26,854 INFO RPC interface 'supervisor' initialized
2020-02-29 19:42:26,854 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2020-02-29 19:42:26,855 INFO supervisord started with pid 1
2020-02-29 19:42:27,857 INFO spawned: 'zookeeper' with pid 8
2020-02-29 19:42:27,859 INFO spawned: 'kafka' with pid 9
2020-02-29 19:42:28,903 INFO success: zookeeper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-29 19:42:28,903 INFO success: kafka entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-29 19:42:34,985 INFO exited: kafka (exit status 1; not expected)
2020-02-29 19:42:35,988 INFO spawned: 'kafka' with pid 297
2020-02-29 19:42:37,014 INFO success: kafka entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-29 19:43:20,590 WARN received SIGTERM indicating exit request
2020-02-29 19:43:20,590 INFO waiting for zookeeper, kafka to die
2020-02-29 19:43:20,590 INFO stopped: kafka (terminated by SIGTERM)
2020-02-29 19:43:20,784 INFO stopped: zookeeper (exit status 143)
/usr/lib/python2.7/dist-packages/supervisor/options.py:296: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
'Supervisord is running as root and it is searching '
2020-02-29 19:45:38,600 CRIT Supervisor running as root (no user in config file)
2020-02-29 19:45:38,600 WARN Included extra file "/etc/supervisor/conf.d/zookeeper.conf" during parsing
2020-02-29 19:45:38,600 WARN Included extra file "/etc/supervisor/conf.d/kafka.conf" during parsing
2020-02-29 19:45:38,619 INFO RPC interface 'supervisor' initialized
2020-02-29 19:45:38,629 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2020-02-29 19:45:38,630 INFO supervisord started with pid 1
2020-02-29 19:45:39,632 INFO spawned: 'zookeeper' with pid 8
2020-02-29 19:45:39,634 INFO spawned: 'kafka' with pid 9
2020-02-29 19:45:40,687 INFO success: zookeeper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-29 19:45:40,689 INFO success: kafka entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-29 19:45:47,740 INFO exited: kafka (exit status 1; not expected)
2020-02-29 19:45:48,743 INFO spawned: 'kafka' with pid 297
2020-02-29 19:45:49,763 INFO success: kafka entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-02-29 19:46:20,659 WARN received SIGTERM indicating exit request
2020-02-29 19:46:20,659 INFO waiting for zookeeper, kafka to die
2020-02-29 19:46:20,660 INFO stopped: kafka (terminated by SIGTERM)
2020-02-29 19:46:20,991 INFO stopped: zookeeper (exit status 143)
/usr/lib/python2.7/dist-packages/supervisor/options.py:296: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
'Supervisord is running as root and it is searching '
2020-03-13 22:16:26,128 CRIT Supervisor running as root (no user in config file)
2020-03-13 22:16:26,128 WARN Included extra file "/etc/supervisor/conf.d/zookeeper.conf" during parsing
2020-03-13 22:16:26,128 WARN Included extra file "/etc/supervisor/conf.d/kafka.conf" during parsing
2020-03-13 22:16:26,157 INFO RPC interface 'supervisor' initialized
2020-03-13 22:16:26,162 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2020-03-13 22:16:26,162 INFO supervisord started with pid 1
2020-03-13 22:16:27,164 INFO spawned: 'zookeeper' with pid 8
2020-03-13 22:16:27,167 INFO spawned: 'kafka' with pid 9
2020-03-13 22:16:28,226 INFO success: zookeeper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-03-13 22:16:28,227 INFO success: kafka entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-03-13 22:16:36,496 INFO exited: kafka (exit status 1; not expected)
2020-03-13 22:16:37,499 INFO spawned: 'kafka' with pid 298
2020-03-13 22:16:38,511 INFO success: kafka entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-03-13 22:17:20,939 WARN received SIGTERM indicating exit request
2020-03-13 22:17:20,940 INFO waiting for zookeeper, kafka to die
2020-03-13 22:17:20,940 INFO stopped: kafka (terminated by SIGTERM)
2020-03-13 22:17:21,268 INFO stopped: zookeeper (exit status 143)
/usr/lib/python2.7/dist-packages/supervisor/options.py:296: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
'Supervisord is running as root and it is searching '
2020-03-27 21:25:59,495 CRIT Supervisor running as root (no user in config file)
2020-03-27 21:25:59,496 WARN Included extra file "/etc/supervisor/conf.d/zookeeper.conf" during parsing
2020-03-27 21:25:59,497 WARN Included extra file "/etc/supervisor/conf.d/kafka.conf" during parsing
2020-03-27 21:25:59,520 INFO RPC interface 'supervisor' initialized
2020-03-27 21:25:59,522 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2020-03-27 21:25:59,523 INFO supervisord started with pid 1
2020-03-27 21:26:00,530 INFO spawned: 'zookeeper' with pid 8
2020-03-27 21:26:00,532 INFO spawned: 'kafka' with pid 9
2020-03-27 21:26:01,620 INFO success: zookeeper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-03-27 21:26:01,620 INFO success: kafka entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
docker logs for kafka manager container seems fine:
[info] o.a.z.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
[info] o.a.z.ZooKeeper - Client environment:java.io.tmpdir=/tmp
[info] o.a.z.ZooKeeper - Client environment:java.compiler=<NA>
[info] o.a.z.ZooKeeper - Client environment:os.name=Linux
[info] o.a.z.ZooKeeper - Client environment:os.arch=amd64
[info] o.a.z.ZooKeeper - Client environment:os.version=4.9.93-boot2docker
[info] o.a.z.ZooKeeper - Client environment:user.name=root
[info] o.a.z.ZooKeeper - Client environment:user.home=/root
[info] o.a.z.ZooKeeper - Client environment:user.dir=/kafka-manager-1.3.3.4
[info] o.a.z.ZooKeeper - Initiating client connection, connectString=kafkaserver:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState#7a27a9b4
[info] o.a.z.ClientCnxn - Opening socket connection to server kafka.kafka_kafkanet/172.18.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
[info] k.m.a.KafkaManagerActor - zk=kafkaserver:2181
[info] k.m.a.KafkaManagerActor - baseZkPath=/kafka-manager
[info] o.a.z.ClientCnxn - Socket connection established to kafka.kafka_kafkanet/172.18.0.2:2181, initiating session
[info] o.a.z.ClientCnxn - Session establishment complete on server kafka.kafka_kafkanet/172.18.0.2:2181, sessionid = 0x1711de33be70001, negotiated timeout = 40000
[info] k.m.a.KafkaManagerActor - Started actor akka://kafka-manager-system/user/kafka-manager
[info] k.m.a.KafkaManagerActor - Starting delete clusters path cache...
[info] k.m.a.DeleteClusterActor - Started actor akka://kafka-manager-system/user/kafka-manager/delete-cluster
[info] k.m.a.DeleteClusterActor - Starting delete clusters path cache...
[info] k.m.a.DeleteClusterActor - Adding kafka manager path cache listener...
[info] k.m.a.DeleteClusterActor - Scheduling updater for 10 seconds
[info] k.m.a.KafkaManagerActor - Starting kafka manager path cache...
[info] k.m.a.KafkaManagerActor - Adding kafka manager path cache listener...
[info] play.api.Play - Application started (Prod)
[info] p.c.s.NettyServer - Listening for HTTP on /0.0.0.0:9000
[info] k.m.a.KafkaManagerActor - Updating internal state...
[info] k.m.a.KafkaManagerActor - Updating internal state...
[info] k.m.a.KafkaManagerActor - Updating internal state...
[info] k.m.a.KafkaManagerActor - Updating internal state...
This log is a lot longer so I've ommited the beginning but it seems fine.
Yes, there's a hypervisor, not a full VM. You can open the hyperV manager to look at it
You compose file needs a port forward
ports:
- '9000:9000'
If you are using docker toolbox on windows you can try to access kafka-manager with this address: http://192.168.99.100:9000
Note: 192.168.99.100 is the default ip address of VM which docker running on.
docker-compose.yaml is totally fine which is given in the tutorial. Can you do docker-compose down and then again bring up the docker-compose up?
Then try to browse http://localhost:9000 and you should be able to see it.
Possible errors:-
Port forwarding (already done in the docker-compose)
Instead of HTTP, you are opening HTTPS in the browser.

docker installation of openproject: Phusion passenger fails to start after installation

I am trying to install openproject using docker on centos7.6 but Phusion passenger fails to start after installation. Error is suggesting it failed to parse response.
The preloader process sent an unparseable response:. I don't know how to fix this issue.
stdout:
-----> Database setup finished.
On first installation, the default admin credentials are login: admin, password: admin
-----> Launching supervisord...
2019-05-08 08:14:46,313 CRIT Supervisor running as root (no user in config file)
2019-05-08 08:14:46,318 INFO supervisord started with pid 1
2019-05-08 08:14:47,321 INFO spawned: 'postgres' with pid 155
2019-05-08 08:14:47,325 INFO spawned: 'apache2' with pid 156
2019-05-08 08:14:47,328 INFO spawned: 'web' with pid 157
2019-05-08 08:14:47,331 INFO spawned: 'worker' with pid 158
2019-05-08 08:14:47,351 INFO spawned: 'postfix' with pid 159
2019-05-08 08:14:47,360 INFO spawned: 'memcached' with pid 160
2019-05-08 08:14:47.634 UTC [172] LOG: database system was shut down at 2019-05-08 08:14:44 UTC
2019-05-08 08:14:47,634 INFO success: postfix entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2019-05-08 08:14:47.649 UTC [172] LOG: MultiXact member wraparound protections are now enabled
2019-05-08 08:14:47.653 UTC [155] LOG: database system is ready to accept connections
2019-05-08 08:14:47.663 UTC [177] LOG: autovacuum launcher started
2019-05-08 08:14:48,670 INFO success: postgres entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-05-08 08:14:48,670 INFO success: apache2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-05-08 08:14:48,670 INFO success: web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-05-08 08:14:48,670 INFO success: worker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-05-08 08:14:48,670 INFO success: memcached entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message
2019-05-08 08:14:50,198 INFO exited: postfix (exit status 0; expected)
--> Downloading a Phusion Passenger agent binary for your platform
--> Installing Nginx 1.15.8 engine
--------------------------
[passenger_native_support.so] trying to compile for the current user (app) and Ruby interpreter...
(set PASSENGER_COMPILE_NATIVE_SUPPORT_BINARY=0 to disable)
Compilation successful. The logs are here:
/tmp/passenger_native_support-15tsfhk.log
[passenger_native_support.so] successfully loaded.
=============== Phusion Passenger Standalone web server started ===============
PID file: /app/tmp/pids/passenger.8080.pid
Log file: /app/log/passenger.8080.log
Environment: production
Accessible via: http://0.0.0.0:8080/
You can stop Phusion Passenger Standalone by pressing Ctrl-C.
Problems? Check https://www.phusionpassenger.com/library/admin/standalone/troubleshooting/
===============================================================================
[ N 2019-05-08 08:15:01.7338 404/Tb age/Cor/SecurityUpdateChecker.h:519 ]: Security update check: no update found (next check in 24 hours)
Forcefully loading the application. Use :environment to avoid eager loading.
[auth_saml] Missing settings from '/app/config/plugins/auth_saml/settings.yml', skipping omniauth registration.
hook registered
App 439 output: [auth_saml] Missing settings from '/app/config/plugins/auth_saml/settings.yml', skipping omniauth registration.
App 439 output: hook registered
Creating scope :order_by_name. Overwriting existing method Sprint.order_by_name.
App 439 output: Creating scope :order_by_name. Overwriting existing method Sprint.order_by_name.
[Worker(host:d0b3748f627a pid:158)] Starting job worker
2019-05-08T08:15:45+0000: [Worker(host:d0b3748f627a pid:158)] Starting job worker
App 439 output: /app/vendor/bundle/ruby/2.6.0/gems/passenger-6.0.1/src/ruby_supportlib/phusion_passenger/preloader_shared_helpers.rb:108:in `fork': Cannot allocate memory - fork(2) (Errno::ENOMEM)
App 439 output: from /app/vendor/bundle/ruby/2.6.0/gems/passenger-6.0.1/src/ruby_supportlib/phusion_passenger/preloader_shared_helpers.rb:108:in `handle_spawn_command'
App 439 output: from /app/vendor/bundle/ruby/2.6.0/gems/passenger-6.0.1/src/ruby_supportlib/phusion_passenger/preloader_shared_helpers.rb:78:in `accept_and_process_next_client'
App 439 output: from /app/vendor/bundle/ruby/2.6.0/gems/passenger-6.0.1/src/ruby_supportlib/phusion_passenger/preloader_shared_helpers.rb:167:in `run_main_loop'
App 439 output: from /app/vendor/bundle/ruby/2.6.0/gems/passenger-6.0.1/src/helper-scripts/rack-preloader.rb:207:in `<module:App>'
App 439 output: from /app/vendor/bundle/ruby/2.6.0/gems/passenger-6.0.1/src/helper-scripts/rack-preloader.rb:30:in `<module:PhusionPassenger>'
App 439 output: from /app/vendor/bundle/ruby/2.6.0/gems/passenger-6.0.1/src/helper-scripts/rack-preloader.rb:29:in `<main>'
[ E 2019-05-08 08:15:46.6971 404/Tc age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /app: The preloader process sent an unparseable response:
Error ID: d7825364
Error details saved to: /tmp/passenger-error-wjSTKF.html
[ E 2019-05-08 08:15:46.7028 404/T8 age/Cor/Con/CheckoutSession.cpp:276 ]: [Client 1-1] Cannot checkout session because a spawning error occurred. The identifier of the error is d7825364. Please see earlier logs for details about the error.
[ W 2019-05-08 08:34:24.7967 404/Tk age/Cor/Spa/SmartSpawner.h:572 ]: An error occurred while spawning an application process: Cannot connect to Unix socket '/tmp/passenger.PKROzbY/apps.s/preloader.hyl9g8': No such file or directory (errno=2)
[ W 2019-05-08 08:34:24.7968 404/Tk age/Cor/Spa/SmartSpawner.h:574 ]: The application preloader seems to have crashed, restarting it and trying again...
App 543 output: [auth_saml] Missing settings from '/app/config/plugins/auth_saml/settings.yml', skipping omniauth registration.
App 543 output: hook registered
App 543 output: Creating scope :order_by_name. Overwriting existing method Sprint.order_by_name.
App 543 output: /app/vendor/bundle/ruby/2.6.0/gems/passenger-6.0.1/src/ruby_supportlib/phusion_passenger/preloader_shared_helpers.rb:108:in `fork': Cannot allocate memory - fork(2) (Errno::ENOMEM)
App 543 output: from /app/vendor/bundle/ruby/2.6.0/gems/passenger-6.0.1/src/ruby_supportlib/phusion_passenger/preloader_shared_helpers.rb:108:in `handle_spawn_command'
App 543 output: from /app/vendor/bundle/ruby/2.6.0/gems/passenger-6.0.1/src/ruby_supportlib/phusion_passenger/preloader_shared_helpers.rb:78:in `accept_and_process_next_client'
App 543 output: from /app/vendor/bundle/ruby/2.6.0/gems/passenger-6.0.1/src/ruby_supportlib/phusion_passenger/preloader_shared_helpers.rb:167:in `run_main_loop'
App 543 output: from /app/vendor/bundle/ruby/2.6.0/gems/passenger-6.0.1/src/helper-scripts/rack-preloader.rb:207:in `<module:App>'
App 543 output: from /app/vendor/bundle/ruby/2.6.0/gems/passenger-6.0.1/src/helper-scripts/rack-preloader.rb:30:in `<module:PhusionPassenger>'
App 543 output: from /app/vendor/bundle/ruby/2.6.0/gems/passenger-6.0.1/src/helper-scripts/rack-preloader.rb:29:in `<main>'
[ E 2019-05-08 08:34:52.2521 404/Tk age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /app: The preloader process sent an unparseable response:
Error ID: c2ce0823
Error details saved to: /tmp/passenger-error-bpsfAC.html
[ E 2019-05-08 08:34:52.2570 404/T8 age/Cor/Con/CheckoutSession.cpp:276 ]: [Client 1-2] Cannot checkout session because a spawning error occurred. The identifier of the error is c2ce0823. Please see earlier logs for details about the error.
Thanks.
The import line in the log is this one:
App 439 output: /app/vendor/bundle/ruby/2.6.0/gems/passenger-6.0.1/src/ruby_supportlib/phusion_passenger/preloader_shared_helpers.rb:108:in `fork': Cannot allocate memory - fork(2) (Errno::ENOMEM)
This means your container is unable to allocate necessary memory. It could be that your system is in a OOM state and things are being killed or due to some other restriction on the daemon that prevents it from allocating additional memory
For reference:
https://success.docker.com/article/docker-daemon-error-cannot-allocate-memory

Supervisor stops/terminates before timeout

I know this might be a hard question as it concerns supervisord queueing but I hope someone would be able to try and think with me to solve an application-breaking problem.
The scenario I'm in at the moment is that I have an application that runs huge imports, each on its own supervisor thread/proc and even though the timeout of the queue is set to either 0 or 3600, the supervisor queue keeps terminating before the 3600 mark, generally after about 30-40 minutes.
I've tracked the RAM- & CPU-usage for an import but this could not be the problem as I am using max. 2GB / 8GB ram and the CPU is using only 1 thread with around 20% usage.
My supervisor queue looks as follows:
[program:laravel_queue]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/application/artisan queue:work --sleep=3 --tries=1 --queue=application --timeout=0
autostart=true
autorestart=true
user=administrator
numprocs=4
redirect_stderr=true
stdout_logfile=/var/www/application/storage/logs/queue/laravel_queue.out.log
stderr_logfile=/var/www/application/storage/logs/queue/laravel_queue.err.log
I've tried the same variant with --timeout=3600, but the queue still terminates at about 40 minutes.
I'm running the same setup in a Homestead/Laravel VM, and it runs perfectly in the Homestead VM.
Am I missing some redis configuration setting that terminates connections after 40 minutes? I'm running a cluster-system where there is 1 seperate - remote - redis-server which the application(s) talk to.
I want to provide more information but I'm not sure what more I could provide that would be helpful, so if you have any thoughts, I'm probably willing to provide most information.
Thanks in advance.
--- EDIT ---
I got a part of the supervisor log where a termination happens right here:
2017-11-28 12:54:34,151 INFO waiting for application_laravel_queue_02 to stop
2017-11-28 12:54:34,151 INFO waiting for application_laravel_queue_03 to stop
2017-11-28 12:54:34,151 INFO waiting for application_laravel_queue_00 to stop
2017-11-28 12:54:34,151 INFO waiting for application_laravel_queue_01 to stop
2017-11-28 12:54:34,155 INFO stopped: application_laravel_queue_02 (terminated by SIGKILL)
2017-11-28 12:54:34,155 INFO stopped: application_laravel_queue_00 (terminated by SIGKILL)
2017-11-28 12:54:34,156 INFO stopped: application_laravel_queue_01 (terminated by SIGKILL)
2017-11-28 12:54:36,158 INFO waiting for application_laravel_queue_03 to stop
2017-11-28 12:54:38,161 INFO waiting for application_laravel_queue_03 to stop
2017-11-28 12:54:40,164 INFO waiting for application_laravel_queue_03 to stop
2017-11-28 12:54:42,170 INFO waiting for application_laravel_queue_03 to stop
2017-11-28 12:54:44,173 WARN killing 'application_laravel_queue_03' (1004) with SIGKILL
2017-11-28 12:54:44,174 INFO waiting for application_laravel_queue_03 to stop
2017-11-28 12:54:44,182 INFO stopped: application_laravel_queue_03 (terminated by SIGKILL)
2017-11-28 12:54:45,193 INFO spawned: 'application_laravel_queue_02' with pid 4235
2017-11-28 12:54:45,196 INFO spawned: 'application_laravel_queue_00' with pid 4236
2017-11-28 12:54:45,199 INFO spawned: 'application_laravel_queue_01' with pid 4237
2017-11-28 12:54:46,378 INFO success: application_laravel_queue_02 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-11-28 12:54:46,379 INFO success: application_laravel_queue_00 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-11-28 12:54:46,379 INFO success: application_laravel_queue_01 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
it looks like it it getting a termination signal (SIGKILL), but I don't know how or from where. The php-fpm logs is cleaned for this timeframe.

Docker stop exit code -1 if the default CMD is a shell script

I am building a tomcat container in Docker with supervisord. If the default command in the Dockerfile is
CMD supervisord -c /etc/supervisord.conf
and when i dispatch docker stop command, the container exits successfully with the exit code 0.
But instead if i have
CMD ["/run"]
and in run.sh,
supervisord -c /etc/supervisord.conf
The docker stop command gives me a exit code -1. On viewing the logs, it seems that the supervisord did not receive the SIGTERM indicating the exit request.
2014-10-06 19:48:54,420 CRIT Supervisor running as root (no user in config file)
2014-10-06 19:48:54,450 INFO RPC interface 'supervisor' initialized
2014-10-06 19:48:54,451 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2014-10-06 19:48:54,451 INFO supervisord started with pid 6
2014-10-06 19:48:55,457 INFO spawned: 'tomcat' with pid 9
2014-10-06 19:48:56,503 INFO success: tomcat entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
as opposed to the previous logs where it receives a sigterm and gracefully exits.
2014-10-06 20:02:59,527 CRIT Supervisor running as root (no user in config file)
2014-10-06 20:02:59,556 INFO RPC interface 'supervisor' initialized
2014-10-06 20:02:59,556 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2014-10-06 20:02:59,557 INFO supervisord started with pid 1
2014-10-06 20:03:00,561 INFO spawned: 'tomcat' with pid 9
2014-10-06 20:03:01,602 INFO success: tomcat entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2014-10-06 20:05:11,690 WARN received SIGTERM indicating exit request
2014-10-06 20:05:11,690 INFO waiting for tomcat to die
2014-10-06 20:05:12,450 INFO stopped: tomcat (exit status 143)
Any help appreciated.
Thanks,
Karthik
UPDATE:
supervisord.conf file
[supervisord]
nodaemon=true
logfile=/var/log/supervisor/supervisord.log
[program:mysql]
command=/usr/bin/pidproxy /var/run/mysqld/mysqld.pid /usr/bin/mysqld_safe --pid-file=/var/run/mysqld/mysqld.pid
stdout_logfile=/tmp/mysql.log
stderr_logfile=/tmp/mysql_err.log
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock
[unix_http_server]
file=/tmp/supervisor.sock ; path to your socket file
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
When you run the process via run.sh, signals are only sent to that process. Unless you are
going out of your way to send signals to child processes, e.g. with trap
sending signals to the process group.
doing exec supervisord ... in run.sh
the child process won't get the signals.

Resources