Spark Docker - Can't access web UI for resource manager - Mac PC - user-interface

Can't access resource manager web ui - Spark docker container - Mac PC
These are the steps that I did:
docker pull sequenceiq/spark:1.6.0
docker run -it -p 8088:8088 -p 8042:8042 -p 4040:4040 -h sandbox sequenceiq/spark:1.6.0 bash
I tested using this: ( Runs fine )
run the spark shell
spark-shell \
--master yarn-client \
--driver-memory 1g \
--executor-memory 1g \
--executor-cores 1
execute the the following command which should return 1000
scala> sc.parallelize(1 to 1000).count()
But can't access the web ui:
I tried:
a. :8088
b: http://sandbox:8088/proxy/application_1458858022274_0002/A
c: localhost:8088
Nothing works.. Any help ??
Thanks in advance!!

You need to expose the ports before publishing them. Either EXPOSE 8088 8042 4040 in the Dockerfile or -e 8088 -e 8042 -e 4040 in your run command. Expose functionality is separated from publish/host mapping functionality because there are cases where one would like to expose to port to other containers without mapping it to the host.

Related

Can't deploy spring application using WSL

The software I'm trying to deploy is https://github.com/gitbitex/gitbitex-new
To sum up, it works. But only on linux machines.
Here is the script I used (from github).
# start zookeeper
docker run -d --name zookeeper-server \
--network host \
-e ALLOW_ANONYMOUS_LOGIN=yes \
bitnami/zookeeper:latest
# start kafka
docker run -d --name kafka-server \
--network host \
-e ALLOW_PLAINTEXT_LISTENER=yes \
-e KAFKA_CFG_ZOOKEEPER_CONNECT=127.0.0.1:2181 \
bitnami/kafka:latest
# start redis
docker run -d --name redis-server \
--network=host \
redis
# start mysql
docker run -d --name mysql \
--network=host \
-e MYSQL_ROOT_PASSWORD=123456 \
mysql:5.7
# Please wait for MySQL to start before executing the following command
docker exec -it mysql mysql -uroot -p123456 -e "create database gitbitex;"
# start gitbitex
# open the browser and visit http://127.0.0.1/trade/BTC-USDT
docker run -d --name gitbitex \
--network=host \
greensheng/gitbitex
Firstly, I've tried to deploy it on vds using guide on github. And it worked without any problem.
Then I've tried to deploy it on my local machine which uses wsl with docker(integration is enabled) and it didn't work. Just can't load the webpage in browser. Curl from wsl also doesn't work, it says: connection refused.
Tried to deploy on a different linux machine, it worked again.
Tried different configuration using docker compose(cause this is what I'll need to do next). I've almost figured out how to setup it without network_mode: host but spring app itself throws exceptions anyway. With network_mode: host it doesn't work even on remote machine(but maybe it's my bad), but all containers are running properly, without errors.
It works on linux machine, but doesn't on Windows. As David Maze told in comments, it's because --network=host doesn't work on docker desktop.
I'll try instead to setup networking put everything into compose and make it work.
UPD: spent a lot of time trying to setup it without --network=host using docker compose and.. it still doesn't work on Windows. Docker destop is weird..

How do I expose ports on Heroku with a Dockerfile?

I am trying to deploy a Docker image on Heroku and am trying to understand how to expose multiple ports. Here is the Docker command that I am trying to run in the Heroku deploy:
docker run \
-p 2222:22 \
-p 33306:3306 \
-p 27017:27017 \
-p 28015:28015 \
-p 29015:29015 \
-p 8080:8080 \
test/db-migration
How do I do this in Heroku?
You can't - you should use the $PORT environment variable which will be randomly assigned and then mapped to port 80 by the Heroku routers. Also, only http requests are accepted. See https://devcenter.heroku.com/articles/container-registry-and-runtime#dockerfile-commands-and-runtime for more details.
You may want to look at Dockhero add-on. It's a good way to deploy supplementary resources alongside your Heroku app, and it supports docker-compose with multi-port mapping. The web app itself should still be running on Heroku dynos.

How to remove docker container using port number

I have Node services which are running in Docker container
I am using shell script to run these services
I want to run three different instances of the same service on 3 different port. say 9011 9022 9033
I also want it to configure it in such a way that after every new deployment it should stop the previous service and remove it
I am using docker rm test-service to remove it but it will remove other instances too.
by this approach only once instance can be running.
Is there any way to remove Docker service running on the specific port.
here is my shell script
#!/bin/bash
ORGANISATION="$1"
SERVICE_NAME="$2"
VERSION="$3"
ENVIRONMENT="$4"
INTERNAL_PORT_NUMBER="$5"
EXTERNAL_PORT_NUMBER="$6"
NETWORK="$7"
docker build -t ${ORGANISATION}/${SERVICE_NAME}:${VERSION} --build-arg PORT=${INTERNAL_PORT_NUMBER} --build-arg ENVIRONMENT=${ENVIRONMENT} --no-cache .
docker stop ${SERVICE_NAME}
docker rm ${SERVICE_NAME}
sudo npm install
sudo npm install -g express
docker run -p ${EXTERNAL_PORT_NUMBER}:${INTERNAL_PORT_NUMBER} --network ${NETWORK} --name ${SERVICE_NAME} --restart always -itd ${ORGANISATION}/${SERVICE_NAME}:${VERSION}
I can not run more than one container with the same name. Can I run the docker service with the same name on 3 different port. if yes what modifications do i need to make in above shell file?
That would be three docker run, each using the same internal port, but mapped to a different host port, with three different names
docker run -p ${EXTERNAL_PORT_NUMBER1}:${INTERNAL_PORT_NUMBER} --name ${SERVICE_NAME1}
docker run -p ${EXTERNAL_PORT_NUMBER2}:${INTERNAL_PORT_NUMBER} --name ${SERVICE_NAME2}
docker run -p ${EXTERNAL_PORT_NUMBER3}:${INTERNAL_PORT_NUMBER} --name ${SERVICE_NAME3}
I want to perform LoadBalance for service
See docker swarm mode
The swarm manager uses ingress load balancing to expose the services you want to make available externally to the swarm.
The swarm manager can automatically assign the service a PublishedPort or you can configure a PublishedPort for the service. You can specify any unused port. If you do not specify a port, the swarm manager assigns the service a port in the 30000-32767 range.
Example:
the following command publishes port 80 in the nginx container to port 8080 for any node in the swarm
$ docker service create \
--name my-web \
--publish 8080:80 \
--replicas 2 \
nginx

Failed to communicate a dockerized process with elastic search with "None of the configured nodes are available"

I have spring boot application which communicate with ElasticSearch 5.0.0 alpha 2.
My application successfully communicate with elastic and preform several queries.
When I try to dockerize my application, it fails to communicate with ElasticSearch, and I get the following error:
None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{127.0.0.1:9300}]
I have spent a lot of time on the internet, but I have found problems when the ElasticSearch is dockerized, but in my case, the client is dockerized, and it is working fine without the docker.
The command I used to create the docker image is: docker build -t my-service .
The DockerFile is:
FROM java:8
VOLUME /tmp
ADD ./build/libs/myjarfile-2.0.0.jar app.jar
EXPOSE 8090
RUN sh -c 'touch /app.jar'
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
To execute the image i use: docker run --name myname -d -p 8090:8090 -t my-service
Can someone share his/her experience with this issue?
Thanks
Guy Hudara
The problem is that your elasticsearch is not available on your dockerized host. When you put something in a docker container it also gets isolated on a network layer and localhost is localhost of the docker container but not the host itself. Therefore if you have elasticsearch also in a docker container use container linking and environment variable injection or reference your host machines address of your main network interface – not loopback – to your app.
Option 1
assuming that elasticsearch exposes 9200 try to run the following
$ docker run -d --name=elasticsearch elasticsearch
$ docker run -d --name=my-app --link elasticsearch:elasticsearch -p 8090:8090 my-app
Then you can define elasticsearch address in your app using env variable ${ELASTICSEARCH_PORT_9200_TCP_ADDR}.
Option 2
assuming your host machine runs on 192.168.1.10 you can also do the following:
$ docker run -d -p 9200:9200 elasticsearch
$ docker run -d -p 8090:8090 my-app
note that the name for the easticsearch container is optional here but the exposing of elasticsearch port mandatory. In this case you'll have to configure your elasticsearch host in your app given address of 192.168.1.10.

Debugging Tomcat in Docker container

I have a CoreOS running in Vagrant. Vagrant private network IP is 192.168.111.1. Inside a CoreOS is a docker container with Tomcat 8.0.32. Pretty much everything works ok (app deployment etc.) just debugging does not. Tomcat is mapped to 8080 port and the JPDA port should be 8000.
Facts
Tomcat JPDA is configured with:
JDPA_OPTS -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000
It starts with catalina.sh jpda start command. The output in the console when running it with docker-compose is:
tomcat | Listening for transport dt_socket at address: 8000
From the container info I assume that ports are mapped as they should:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
dcae1e0148f8 tomcat "/run.sh" 8 minutes ago Up 8 minutes 0.0.0.0:8000->8000/tcp, 0.0.0.0:8080->8080/tcp tomcat
My docker image is based on this Dockerfile.
Problem
When trying to run Remote debug configuration (screenshot below) I get the error Error running Debug: Unable to open debugger port (192.168.111.1:8000): java.net.ConnectException "Connection refused". I've tried everything from changing various configuration but no luck. Am I missing something?
This is the command I use for this:
docker run -it --rm \
-e JPDA_ADDRESS=8000 \
-e JPDA_TRANSPORT=dt_socket \
-p 8888:8080 \
-p 9000:8000 \
-v D:/tc/conf/tomcat-users.xml:/usr/local/tomcat/conf/tomcat-users.xml \
tomcat:8.0 \
/usr/local/tomcat/bin/catalina.sh jpda run
Explanation
-e JPDA_ADDRESS=8000debugging port in container, passed as environment variable
-e JPDA_TRANSPORT=dt_sockettransport type for debugging as socket, passed as environment variable
-p 8888:8080 expose tomcat port 8080 on host as port 8888
-p 9000:8000 expose java debugging port 8000 on host as port 9000
-v {host-file}:{container-file}overwrite tomcat-user.xml with my local on, since I need access to the manager apiomit this line if this isn't necessary for your use case
tomcat:8.0see https://hub.docker.com/_/tomcat/
/usr/local/tomcat/bin/catalina.sh jpda runcommand to run in the container
The accepted answer didn't work for me, apparently because I was using Java 11. It seems that if you're using Java 9 or newer, you need to specify the JPDA address like this:
JPDA_ADDRESS=*:8100
You can always update the Dockerfile to something like the following: -
FROM tomcat:8-jre8
MAINTAINER me
ADD target/app.war /usr/local/tomcat/webapps/app.war
ENV JPDA_ADDRESS="8000"
ENV JPDA_TRANSPORT="dt_socket"
EXPOSE 8080 8000
ENTRYPOINT ["catalina.sh", "jpda", "run"]
This does mean though that your docker file has debug on by default which is probably not suited to a production environment.
Try add to your Dockerfile
ENV JPDA_ADDRESS=8000
ENV JPDA_TRANSPORT=dt_socket
It works for me
You need to make sure that port 8080 is exposed to IntelliJ for connection. That is while running docker you shall require something like docker run -p 8080:8080
For example, I am able to achieve the similar requirement like this by doing below mentioned steps/checks.
This is what my docker run command looks like:
sudo docker run --privileged=true -d -p 63375:63375 -p 63372:8080 -v /tmp/:/usr/local/tomcat/webapps/config <container name>:<tag>
NOTE: I am exposing an extra port 63375 on container and on my host both. The same port I am using in CATALINA_OPTS below.
This is what my entry point (for the image that I am building) looks like. NOTE: I am using CATALINA_OPTS. Also, I am using maven to create image so below is excrept from pom.xml.
<entryPoint>
<shell>cd /usr/local/tomcat/bin; CATALINA_OPTS="-agentlib:jdwp=transport=dt_socket,address=63375,server=y,suspend=n" catalina.sh run</shell>
</entryPoint>
I resolved a similar, if not the same, issue when using docker-compose.
It involved the environment variables not being passed properly from the docker-compose.yml file.
See my stack overflow issue:
For me is cleaner this way:
docker run -e JAVA_TOOL_OPTIONS="-agentlib:jdwp=transport=dt_socket,address=8000,server=y,suspend=n" -p 8000:8000 tomcat:8.5-jdk8
This way you don't have to modify your container Dockerfile.
Explanation: all java version check the JAVA_TOOL_OPTIONS environment variable: https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/envvars002.html
I have similar setup in my local environment. I included JPDA_ADDRESS as environment variable in the Dockerfile and recreated the containers.
ENV JPDA_ADDRESS 8000
#Expose port 8080, JMX port 13333 & Debug port 8000
EXPOSE 8080 13333 8000
CMD ["tail", "-f", "/dev/null"]

Resources