I'm trying to setup a Docker container to build my java project with maven
I have created my Docker file FROM maven:3.2-jdk-7 and built the image.
when I execute with:
docker run -it --rm --name my-maven-project -v "$PWD":/usr/src/app -v
"$HOME"/.m2:/root/.m2 -w /usr/src/app -v "$HOME"/.ssh:/root/.ssh test
mvn clean package -Dmaven.test.skip=true
It takes about 20 minute to complete.. but if I run the the same mvn command on my host it takes 2minutes
I have tried giving more memory to the container by using
-m 4gb
But it didn't change anything, looking at docker stats the container barely used more than 2G
I'm running all this from OSX
Is there anything I need to do to have a maven finish in a decent time? I very very surprised it takes THAT much when on the host it takes 2minutes..
This what docker stats says after maven has been building for 10 min
CPU: 201.13%
Mem usage / limit : 2.508GiB
MEM % : 62.69%
NET I/O: 3.01kB / 861B
BLOCK I/O: 57.7MB / 2.23MB
PIDS: 38
- EDIT -
It turns out Docker for mac does not play well when using mounted volume.
In order to avoid having to git clone the project inside the container I preferred using using -v "$PWD":/usr/src/app
To test I have directly git cloned the app form within the container and now the build takes a normal amount of time (4minutes)
Note that the git clone took... 6 min!!! instead (1min on host) so in total from git clone to final build it still takes 10min which is ridiculous.
So yea OSX and Docker is a big no no when using mounted volume...
I ran into this same issue using the same docker run syntax as you (docker run -v src:dest). A Maven build that took ~30 seconds on my OSX host was taking ~4 minutes in my container. I didn't solve it entirely, but switching to explicitly use a bind mount took my builds from around 4 minutes down to about 1.5 minutes. This still isn't an acceptable increase in build time for my use case, but it may help someone else. Try switching your docker run command to this:
docker run --name=my-maven-project -it \
--mount type=bind,source="$(pwd)",destination=/usr/src/app,consistency=delegated <docker image name>
NOTE: The consistency option at the very end is only valid on OSX, and has two other values, either of which may be more appropriate for your situation. I tried all three out of curiosity and build times were comparable between the delegated and cached options, meanwhile the consistent option nearly as slow as the way I was doing it before (unsurprisingly). Here's the documentation:
https://docs.docker.com/storage/bind-mounts/
So, unfortunately, despite bind mounts being "very performant," they're still apparently at least twice as slow as a native filesystem when it comes to maven builds, at least on OSX. With luck that will improve as time goes on.
Related
I'm using WSL2 with Ubuntu 20.04 distribution, and I was trying create a container in Docker with the Following command:
docker run --hostname=quickstart.cloudera --privileged=true -it -v $PWD:/src --publish-all=true -p 8888:8888 -p 8080:8080 -p 7180:7180 cloudera/quickstart /usr/bin/docker-quickstart
when I run this command, a download started with a weight of about 4.4 GB, (i think that is because because was the first time that I run this container), whe the download was over, I used the following command to check the containers docker ps -a and the status for the container is Exited (139) 6 minutes ago, when check my image list
REPOSITORY TAG IMAGE ID CREATED SIZE
uracilo/hadoop latest 902e5bb989ad 8 months ago 727MB
cloudera/quickstart latest 4239cd2958c6 4 years ago 6.34GB
I think that the image was created successfully, but when I try to run the first command, I keep gettind the Exited (139) in the status and I can't use the container
Apparently the exit code 139 refers to some problem with the system or the hardware, maybe the RAM, but I'm not sure. and I don't know if this problem is because I'm using wsl or my 8GB in ram
not enough to run the image
is there any way to run this image successfully?
You need to create a file named .wslconfig under %userprofile% folder on your Windows and copy the following lines into that file
[wsl2]
kernelCommandLine = vsyscall=emulate
Then just restart your Docker engine.
I fixed this by changing the Docker engine from WSL2 back-end to Hyper-V
https://community.cloudera.com/t5/Support-Questions/docker-exited-with-139-when-running-cloudera-quickstart/td-p/298586
I am migrating to circleci2.0 and after successful build when tests are executed they are randomly failing with following error messages
/usr/local/go/pkg/tool/linux_amd64/link: signal: killed
/usr/local/go/pkg/tool/linux_amd64/link: flushing $WORK/b462/payment_step_svc.test: write $WORK/b462/svc.test: cannot allocate memory
I used the following config
jobs:
build:
docker:
- image: circleci/golang:latest
- image: rabbitmq:3.5.4
- image: redis
working_directory: /go/src/github.com/soniabhishek/taskrunner
environment:
GOOS: linux
GOARCH: amd64
GOPATH: /go
steps:
- checkout
- run:
name: Get dependencies
command: go get -t -d -v ./...
- run:
name: Build all
command: go build ./...
- run:
name: Test all
command: go test -v ./...
I have tried for many golang versions other than latest like (1.10.3).
Although I found the hack for this but I am not sure why is this happening,
all my tests run when I use CGO_ENABLED=0
Would like to know why is this issue occurring and permanent solution for this
There are two clues here that suggest you are consuming too much memory, and the OS is forcibly killing your "rogue" process. They are:
killed
cannot allocate memory
You can confirm this by getting an SSH session at the end of a failed build, and examining the kill history using dmesg. If it gives you a "sacrifice child" message, then you hit a memory limit so severe that the OS was forced to remove a process from memory.
You have several options. I put them here in the order I would suggest you prefer them:
Reduce your memory consumption by making changes to your Go program
Ask CircleCI to set you up with a larger Docker container (this will be a chargeable option). The default is 4G, I believe 8G and 16G are available
Swap to a Machine executor (presently free of charge, subject to change). This gives you 8G of RAM, but you lose some of the flexibility of the Docker approach (e.g. you have to use one of a few preset base images).
What worked for me in this situation was go test -p 1 to tell Go to run just one compilation at a time.
Without that Go will run the number of CPUs you have, which by default on CircleCI is two.
I'm using Docker 1.13.1 for Mac. The Docker client allows you to change the amount of memory provided to Docker using a simple slider interface.
How can I set this value via docker's command line utility?
For added clarity, this is not per container memory, this is the value of "Total Memory" that's returned when you run docker info.
Thank you
With docker (at least version 18.03.1) the settings for the VM are maintained in a special file located at:
/Users/<username>/Library/Group\ Containers/group.com.docker/settings.json
If you close docker you can edit it directly from the command line using sed, for example the command below will replace the 2 GB limit with a 10GB limit, and create a backup file of the original settings at settings.json.bak
sed -i .bak 's/2048/10240/g' /Users/`id -un`/Library/Group\ Containers/group.com.docker/settings.json
When docker restarts, it will now have 10 GB.
On a Mac, Docker actually runs as a Hyperkit virtual machine. The docker command line utility just interfaces with the docker daemon process running inside that virtual machine.
If you run ps auxwww | grep hyperkit on your Mac, you'll see the hyperkit process running with the amount of memory passed as an argument. This is controlled by the Docker Mac client, and I imagine the saved value is stored in a .plist file somewhere.
In order to modify that on the command line, you'd need to find where the Docker Mac client stores the data, modify it, and restart the hyperkit process.
I'm trying to run the Hetionet v1.0 docker container mentioned in this SO post.
I've setup a digitalocean droplet with Docker
I ran docker pull dhimmel/hetionet and it worked
Now I run docker run dhimmel/hetionet and the following happens (and never returns to the interactive shell prompt).
If that completed successfully I think the last thing I'm supposed to do is run sh ~/run-docker.sh. Furthermore nothing is live at my droplet's ip_address:7474.
The error in the screenshot above looks a lot like it could be related to some redundant #Path("/") annotation, as described in this SO post's comment, buried in the docker container but I'm not sure.
Is the output from running docker run dhimmel/hetionet supposed to hang my shell? I'm running a 2 GB Memory / 40 GB Disk Droplet on Ubuntu 16.04 with Docker 1.12.5.
Thanks for your interest in the Hetionet Docker.
The output in 3 is expected. It looks like a Docker container successfully launched, downloaded the Hetionet database, and launched the Neo4j server. I'll look into fixing the warnings, but they're not errors, as Neo4j is still launching.
For production, we use a more advanced Docker run command. Depending on your use case, you may want to use the development docker run command:
docker run \
--publish=7474:7474 \
--publish=7687:7687 \
--volume=$HOME/neo4j/hetionet-data:/data \
--volume=$HOME/neo4j/hetionet-logs:/var/lib/neo4j/logs \
dhimmel/hetionet
Both the production and development command map ports. This will make it so the Neo4j server running inside your Docker container is available at http://localhost:7474/. This is most likely what you want. If you're doing this on DigitalOcean, you would replace http://localhost with the IP address of your droplet.
For an interactive shell session in a dhimmel/hetionet container, you can use:
docker run --interactive --tty dhimmel/hetionet bash
However, that command does not launch the Neo4j server -- it just let's you explore the image.
Does this clear things up?
I installed boot2docker as explained on the docker website. Here are some command runs to show that I have things installed correctly:
$$:~ kv$ boot2docker start
Waiting for VM and Docker daemon to start...
...................ooo
Started.
Writing /Users/kvantum/.boot2docker/certs/boot2docker-vm/ca.pem
Writing /Users/kvantum/.boot2docker/certs/boot2docker-vm/cert.pem
Writing /Users/kvantum/.boot2docker/certs/boot2docker-vm/key.pem
Your environment variables are already set correctly.
$$:~ kv$ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
ubuntu 14.04 b39b81afc8ca 11 days ago 188.3 MB
hello-world latest e45a5af57b00 3 weeks ago 910 B
After this, I ran the following command:
docker run -t -i ubuntu:14.04 /bin/bash
Inside the container, I installed zeromq, and started a zeromq server on port 5555 using tcp.
My questions are following:
If I exit out of the container, will it save all the work I do inside it?
I have no idea how to connect to the server running on port 5555. I read something about exposing a port, but I am not sure how to go about doing that. I did an ifconfig inside the container, and tried to connect to the server from the host like this:
$$:~ kv$ ./zmq_client tcp://container_ip:5555
This did not work. Can someone please lists the steps I need to take in order to connect to the server running within the container.
For completion sake, I am providing the list of my environment variables:
TERM_PROGRAM=Apple_Terminal
TERM=xterm-256color
SHELL=/bin/bash
TMPDIR=/var/folders/km/5kbpdx4s7cg4rmyc6d5q9l9r0000gq/T/
DOCKER_HOST=tcp://192.168.109.103:2376
Apple_PubSub_Socket_Render=/tmp/launch-1tWMHJ/Render
TERM_PROGRAM_VERSION=326
OLDPWD=/Users
TERM_SESSION_ID=262CBC8B-0A74-4B70-9F28-D9FA51FF713C
USER=kv
SSH_AUTH_SOCK=/tmp/launch-ZTWNGL/Listeners
__CF_USER_TEXT_ENCODING=0x1F7:0:0
DOCKER_TLS_VERIFY=1
__CHECKFIX1436934=1
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin
PWD=/Users/kv
DOCKER_CERT_PATH=/Users/kv/.boot2docker/certs/boot2docker-vm
HOME=/Users/kv
SHLVL=1
LOGNAME=kv
LC_CTYPE=UTF-8
DISPLAY=/tmp/launch-rco9zt/org.macosforge.xquartz:0
_=/usr/bin/env
One last question I have is about code performance. So within my Mac OS X, I have a docker container running (which runs Ubuntu). If I run the application, like a zeromq based server inside the container, will it not be slower as compared to running it on Mac OS X directly. Please explain the benefits of using docker in such a scenario..
You should really do some more reading and research before turning to SO, then ask about anything you can't figure out. But:
No. If the container is "exited" you can restart it and your files will still be there, but once it is removed your files are gone. You can use docker commit to save them to an image, but the best bet is to use a Dockerfile.
docker run -p 5000:8000 image will expose port 8000 in the container as port 5000 on the host.
Yes, it will be slower due to the boot2docker VM. It would not be slower if you were running on a Linux host. The advantage is that zeromq is now running in an isolated container with all its dependencies.