When i execute direct on terminal
/ins/icenv/bin/gunicorn app.main:app --workers 1 --worker-class uvicorn.workers.UvicornWorker --bind 0.0.0.0:8000 --certfile=/ins/key_ls/fullchain.pem --keyfile=/ins/key_ls/privkey.pem --access-logfile /tmp/null --error-logfile /ins/logs/error_log_gunicorn_ws.log &
gunicorn work perfectly and I receive data from port 8000 but when I run
sudo systemctl start gunicornws
process start and status is active but connection to port 8000 return no data
Here is my gunicornws.service
Description=WebSocket Gunicorn Service
After=network.target
[Service]
User=customuser
Group=www-data
WorkingDirectory=/instant_connection/
ExecStart=/ins/icenv/bin/gunicorn app.main:app --workers 1 --worker-class uvicorn.workers.UvicornWorker --bind 0.0.0.0:8000 --certfile=/ins/key_ls/fullchain.pem --keyfile=/ins/key_ls/privkey.pem --access-logfile /tmp/null --error-logfile /ins/logs/error_log_gunicorn_ws.log
[Install]
WantedBy=multi-user.target
I already tried many settings but I can't find any logical reasons for this.
I will appreciate your support.
NOTE: I have another instance of gunicorn running on .sock and work perfectly with nginx but I need another for WebSockets and a port for internal reasons.
EDIT.
I change my gunicornws.service to and it's running perfectly
Description=WebSocket Gunicorn Service
After=network.target
[Service]
User=jean15151352
Group=jean15151352
WorkingDirectory=/ins/
ExecStart=/ins/icenv/bin/gunicorn \
app.main:app \
--workers 1 \
--worker-class uvicorn.workers.UvicornWorker \
--bind 0.0.0.0:8000 \
--certfile=/ins/key_ls/fullchain.pem \
--keyfile=/ins/key_ls/privkey.pem \
--access-logfile /tmp/null \
--error-logfile /ins/logs/error_log_gunicorn_ws.log
[Install]
WantedBy=multi-user.target
Related
I have:
Windows 10
Docker for Windows with WSL2 integration (using Ubuntu 20.04)
I created a simple tftpd server to run in a Linux (Ubuntu 16.04) container and exposed port 69. Here is the Dockerfile:
FROM ubuntu:16.04
RUN apt-get update && \
apt-get install --no-install-recommends -y \
tftpd-hpa \
nfs-kernel-server && \
# Clean rootfs
apt-get clean all && \
apt-get autoremove -y && \
apt-get purge && \
rm -rf /var/lib/{apt,dpkg,cache,log}
# Export the TFTP server port
EXPOSE 69/udp
WORKDIR /
VOLUME ["/var/lib/tftpboot"]
VOLUME ["/nfs"]
VOLUME ["/etc/dhcp"]
VOLUME ["/etc/default"]
COPY entrypoint.sh /entrypoint.sh
# Set correct entrypoint permission
RUN chmod u+x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
The script extrypoint.sh just start the tftd-hpa service on start-up and then waits until C+C is pressed:
service tftpd-hpa start
echo "Started..."
while true; do sleep 1; done
I spin up a container using a Powershell script this:
docker run --rm -ti --privileged `
-p 69:69/udp `
-v ${PWD}/tftp:/var/lib/tftpboot `
-v ${PWD}/nfs:/nfs `
-v ${PWD}/etc/exports:/etc/exports `
-v ${PWD}/etc/default/nfs-kernel-server:/etc/default/nfs-kernel-server `
-v ${PWD}/etc/network/interfaces:/etc/network/interfaces `
nfs_tftp_server:latest;
As you can see, I am mapping port 69 in the container to 69 on the host. When I do netstat -an | find "UDP" in a command window, I do see entries for port 69. This seems to indicate the port is okay. I checked the status of the tftpd-hpa service in the container, and it is running.
I changed the Firewall settings to allow inbound and outbound traffic on port 69. I also updated the firewall to allow the Trivial File Transfer Protocol App (TFTP.EXE) to communicate through it.
The problem is when I try to do tftp:
tftp 0.0.0.0 GET test.txt
it just hangs and times out. The file does exist.
Why doesn't it work? When I try the same in my Ubuntu Virtual Machine, it works. So this must be related to Docker on Windows.
I am newer in big data domain, and this is my first time using Docker. I just found this amazing project: https://kiwenlau.com/2016/06/26/hadoop-cluster-docker-update-english/ which create a hadoop cluster composed of one master and two slaves using Docker.
After doing all the installation, I just run containers and they work fine. There is start-containers.sh file which give me the hand to lunch the cluster. I decide to install some tools like sqoop to import my local relational data base to Hbase, and that's work fine. After that I stop all Docker container in my pc by tapping
docker stop $(docker ps -a -q)
In the second day, when I tried to relaunch containers by running the same script ./start-container.sh , I found this error:
start hadoop-master container...
start hadoop-slave1 container...
start hadoop-slave2 container...
Error response from daemon: Container
e942e424a3b166452c9d2ea1925197d660014322416c869dc4a982fdae1fb0ad is
not running
even, I lunch this daemon; containers of my cluster cannot connect to each other, and I can't access to data which is stored on Hbase.
First can any one tell me why this daemon don't work.
PS: in the start-container.sh file there is a line which removes containers if they exist before creating them, I delete this line because If I don't delete them, every time I do all things from the beginning.
After searching I found that is preferable to use the docker compose which give me the hand to lunch all container together.
But I can't found how to translate my start-container.sh file to docker-compose.yml file. Is this the best way to lunch all my containers in the same time ? This is the content of start-containers.sh file:
#!/bin/bash
sudo docker network create --driver=bridge hadoop
# the default node number is 3
N=${1:-3}
# start hadoop master container
#sudo docker rm -f hadoop-master &> /dev/null
echo "start hadoop-master container..."
sudo docker run -itd \
--net=hadoop \
-p 50070:50070 \
-p 8088:8088 \
-p 7077:7077 \
-p 16010:16010 \
--name hadoop-master \
--hostname hadoop-master \
spark-hadoop:latest &> /dev/null
# sudo docker run -itd \
# --net=hadoop \
# -p 5432:5432 \
# --name postgres \
# --hostname hadoop-master \
# -e POSTGRES_PASSWORD=0000
# --volume /media/mobelite/0e5603b2-b1ad-4662-9869-8d0873b65f80/postgresDB/postgresql/10/main:/var/lib/postgresql/data \
# sameersbn/postgresql:10-2 &> /dev/null
# start hadoop slave container
i=1
while [ $i -lt $N ]
do
# sudo docker rm -f hadoop-slave$i &> /dev/null
echo "start hadoop-slave$i container..."
port=$(( 8040 + $i ))
sudo docker run -itd \
-p $port:8042 \
--net=hadoop \
--name hadoop-slave$i \
--hostname hadoop-slave$i \
spark-hadoop:latest &> /dev/null
i=$(( $i + 1 ))
done
# get into hadoop master container
sudo docker exec -it hadoop-master bash
Problems with restarting containers
I am not sure if I understood the mentioned problems with restarting containers correctly. Thus in the following, I try to concentrate on potential issues I can see from the script and error messages:
When starting containers without --rm, they will remain in place after being stopped. If one tries to run a container with same port mappings or same name (both the case here!) afterwards that fails due to the container already being existent. Effectively, no container will be started in the process. To solve this problem, one should either re-create containers everytime (and store all important state outside of the containers) or detect an existing container and start it if existent. With names it can be as easy as doing:
if ! docker start hadoop-master; then
docker run -itd \
--net=hadoop \
-p 50070:50070 \
-p 8088:8088 \
-p 7077:7077 \
-p 16010:16010 \
--name hadoop-master \
--hostname hadoop-master \
spark-hadoop:latest &> /dev/null
fi
and similar for the other entries. Note that I do not understand why one would
use the combination -itd (interactive, assign TTY but go to background) for
a service container like this? I'd recommend going with just -d here?
Other general scripting advice: Prefer bash -e (causes the script to stop on unhandled errors).
Docker-Compose vs. Startup Scripts
The question contains some doubt whether docker-compose should be the way to go or if a startup script should be preferred. From my point of view, the most important differences are these:
Scripts are good on flexibility: Whenever there are things that need to be detected from the environment which go beyond environment variables, scripts provide the needed flexibility to execute commands and to be have environment-dependently. One might argue that this goes partially against the spirit of the isolation of containers to be dependent on the environment like this, but a lot of Docker environments are used for testing purposes where this is not the primary concern.
docker-compose provides a few distinct advantages "out-of-the-box". There are commands up and down (and even radical ones like down -v --rmi all) which allow environments to be created and destroyed quickly. When scripting, one needs to implement all these things separately which will often result in less complete solutions. An often-overlooked advantage is also portability concerns: docker-compose exists for Windows as well. Another interesting feature (although not so "easy" as it sounds) is the ability to deploy docker-compose.yml files to Docker clusters. Finally docker-compose also provides some additional isolation (e.g. all containers become part of a network specifically created for this docker-compose instance by default)
From Startup Script to Docker-Compose
The start script at hand is already in a good shape to consider moving to a docker-compose.yml file instead. The basic idea is to define one service per docker run instruction and to transform the commandline arguments into their respective docker-compose.yml names. The Documentation covers the options quite thoroughly.
The idea could be as follows:
version: "3.2"
services:
hadoop-master:
image: spark-hadoop:latest
ports:
- 50070:50070
- 8088:8088
- 7077:7077
- 16010:16010
hadoop-slave1:
image: spark-hadoop:latest
ports:
- 8041:8042
hadoop-slave2:
image: spark-hadoop:latest
ports:
- 8042:8042
hadoop-slave2:
image: spark-hadoop:latest
ports:
- 8043:8042
Btw. I could not test the docker-compose.yml file because the image spark-hadoop:latest does not seem to be available through docker pull:
# docker pull spark-hadoop:latest
Error response from daemon: pull access denied for spark-hadoop, repository does not exist or may require 'docker login'
But the file above might be enough to get an idea.
The below init code works on ubuntu 14 however latest Ubuntu does not support init and need to convert this script to work in systemd.
I figured starting the service part however unable to figure out on how to restrict it to only tty2,3; how can I fix this?
(init-code)
# tty1 - getty
#
# This service maintains a getty on tty1 from the point the system is
# started until it is shut down again.
start on stopped rc RUNLEVEL=[23] and (
not-container or
container CONTAINER=lxc or
container CONTAINER=lxc-libvirt)
stop on runlevel [!23]
respawn
exec /sbin/getty -n -l /etc/init/banner.sh -8 <port> tty1
[Unit]
Description=This service maintains a getty on tty1 from the point the system is started until it is shut down again.
After=network.target
[Service]
Type=simple
ExecStart=/sbin/getty -n -l /etc/init/bapp_banner.sh -8 <Port> tty1
StandardOutput=syslog
StandardError=syslog
[Install]
WantedBy=multi-user.target
There are no runlevels in systemd, and tty1 are available and managed automagically until the system HAS to HANGUP on them.
I have made some changes in the settings for HDFS on an Amazon EMR cluster. I want to restart the namenode and the datanode for the changes to take effect. I am not able to find any start and stop scripts to do so on neither the namenode(master) nor the datanodes. What should be the way to restart the cluster?
On EMR4 , run following on master host -
sudo /sbin/start hadoop-hdfs-namenode
ssh -i <key.pem> <slave-hostname1> "sudo /sbin/restart hadoop-hdfs-datanode"
ssh -i <key.pem> <slave-hostname2> "sudo /sbin/restart hadoop-hdfs-datanode"
ssh -i <key.pem> <slave-hostname3> "sudo /sbin/restart hadoop-hdfs-datanode"
You have to manually restart the cluster. This can be either performed manually or using a simple shell script.
1) Get the list of hostnames or ipaddress of all the nodes,
2) ssh into the node using the key
3) Restart the required service.
If you are good in programming, you can create a general utility that will get the list of ipaddress of all the nodes corresponding to an EMR by using the cluster id and perform the service restart in individual nodes.
Otherwise, get the hostname or ipaddress of all the nodes manually and create a script like the below one and execute from the master node
sudo service hadoop-hdfs-namenode restart
ssh -i <key.pem> <hostname1> "sudo service hadoop-hdfs-datanode restart"
ssh -i <key.pem> <hostname2> "sudo service hadoop-hdfs-datanode restart"
ssh -i <key.pem> <hostname3> "sudo service hadoop-hdfs-datanode restart"
On EMR 5.x this is what I used:
Copy PEM file to your head node and set these values:
CLUSTER_ID="j-XXXXXXXXXXX"
IDENT="cluster.pem"
Run this:
nodes=$(aws emr list-instances \
--cluster-id $ \
--instance-group-types CORE \
--instance-states RUNNING \
--output text \
--query "Instances[*].PublicDnsName" )
for node in nodes; do
ssh -i $IDENT \
-o StrictHostKeyChecking=no \
-o UserKnownHostsFile=/dev/null \
$node "sudo stop hadoop-hdfs-datanode; sudo start hadoop-hdfs-datanode"
done
docker run -itp 26542:26542 stack/vowpall vw -t -i /home/alex/cb.model --daemon --port 26542
when I run this command there is no daemon listening. When I run
docker ps
there are no processes
but when I go to docker container bash and run
vw -t -i /home/alex/cb.model --daemon --port 26542
there is a daemon listening, also visible in docker ps. Any ideas?
The problem is that the daemon is forking to the background and a Docker container only runs as long as its main process. When the daemon forks to the background, the main process ends and so does the container. You just need to keep the application running in the foreground, which probably just means removing the --daemon argument.
Also, you only need the -it arguments if you want a shell, so you can remove them as well. If you want to get the shell on your host back after running the docker command, add -d so that the client disconnects after starting the container e.g:
docker run -d p 26542:26542 stack/vowpall vw -t -i /home/alex/cb.model --port 26542