In our Jenkins agents we are running about several (around 20) tests whose setup involves running docker-compose up for a "big" number of services/containers (around 14).
From time to time, I'll get the following error:
ERROR: for testdb-data UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).
Haven't been able to reproduce this consistently. And I'm still trying to figure out whether or not there is a correlation with our agent's resources being at full use.
docker -v is 1.10.1 and docker-compose -v is 1.13.1.
Any ideas about what this may be related to?
Restarting docker service:
sudo systemctl restart docker
and setting DOCKER_CLIENT_TIMEOUT and COMPOSE_HTTP_TIMEOUT environment variables:
export DOCKER_CLIENT_TIMEOUT=120
export COMPOSE_HTTP_TIMEOUT=120
are two workarounds for now. But the issues are still open in docker compose github:
https://github.com/docker/compose/issues/3927
https://github.com/docker/compose/issues/4486
https://github.com/docker/compose/issues/3834
I had the same problem. It was solved after change the max-file size value from a number to a string.
Wrong config
logging:
options:
max-file: 10
max-size: 10m
Correct config
logging:
options:
max-file: "10"
max-size: 10m
docker-compose down
Running docker-compose down and then running docker-compose up --build may work. I am working on vscode and when I encountered a similar problem while building docker, this solution worked for me.
Before performing above mentioned command its better you refer to what is the purpose of docker-compose down
docker-compose restart
Fixed my issue. This will restart all stopped and running services.
docker-compose down
export DOCKER_CLIENT_TIMEOUT=120
export COMPOSE_HTTP_TIMEOUT=120
docker-compose up -d
following steps worked
sometimes it could because of docker internal cache, for me following steps worked:
1: docker-compose down -v --rmi all --> to remove all the image and cache
2: docker-compose up --build
Related
I follow the first steps to install Flink.
I can start the cluster without any problem
$ start-cluster.sh
Starting cluster.
Starting standalonesession daemon on host DESKTOP-....
Starting taskexecutor daemon on host DESKTOP-....
But I don't get any status from
$ ps aux | grep flink
I can also not access the dashboard via localhost:8081.
There is an older post having these issues, but the solution didn't work for me, since the described conf files do no longer exist, apparently.
My JAVA_HOME is set as C:\Progra~1\Java\jdk1.8.0_311 to avoid issues with the space in Program Files.
Can you check the logs in the /logs folder? I'm suspecting that C:\Program Files\ could still cause issues because of the space there.
go to download Flink folder and try bash command
$./bin/start-cluster.sh --daemon bootstrap-server localhost:8081
and run code one more
$ ./bin/flink run examples/streaming/WordCount.jar
if you finished run above code which not issue, go to localhost:8081
This still seems to be problematic. I tried to run from Windows Subsystem for Linux (WSL).
I have the following versions: java 11.0.16 and flink 1.15.2.
sudo apt-get update
sudo apt install openjdk-11-jre-headless
export FLINK_HOME=/mnt/c/Projects/Apache/flink-1.15.2
I set the following in flink-conf.yaml
rest.port: 8081
rest.address: localhost
rest.bind-adress: 0.0.0.0
Whereby I changed the bind address for localhost to 0.0.0.0 this seems to have fixed the problem.
$FLINK_HOME/bin/start-cluster.sh
Now I can access the Flink Web Dashboard.
a. Run the below export command in terminal:
export DOCKER_CONTENT_TRUST=1
b. Run :
docker-compose up --build -d
FAILURE:
ERRO[0000] Can't close tar writer: io: read/write on closed pipe
error during connect: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.41/build?buildargs=%7B%22EIS_VERSION%22%3A%222.4%22%2C%22GO_VERSION%22%3A%221.12%22%2C%22HOST_TIME_ZONE%22%3A%22Asia%2FKolkata%22%2C%22HTTPS_PROXY%22%3A%22http%3A%2F%2Fproxy-xyz.com%3A911%22%2C%22HTTP_PROXY%22%3A%22http%3A%2F%2Fproxy-xyz.com%3A911%22%2C%22NO_PROXY%22%3A%22127.0.0.1%2Clocalhost%22%2C%22UBUNTU_IMAGE_VERSION%22%3A%2218.04%22%2C%22http_proxy%22%3A%22http%3A%2F%2Fproxy-xyz.com%3A911%22%2C%22https_proxy%22%3A%22http%3A%2F%2Fproxy-xyz.com%3A911%22%2C%22no_proxy%22%3A%22127.0.0.1%2Clocalhost%22%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=dockerfiles%2FDockerfile.eisbase&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&t=ia_eisbase%3A2.4&target=&ulimits=null&version=1: invalid reference format
Sending build context to Docker daemon 12.55kB
I think DOCKER_CONTENT_TRUST locks some file wrt docker which causes this error.
docker-compose does not have any support for image signing or image signature verification.
— https://github.com/sse-secure-systems/trusted-compose
You need to implement a wrapper for docker-compose or use an existing one in order to work with content trust and docker-compose.
I don't use personnaly trusted-compose but it looks like as a good solution: https://github.com/sse-secure-systems/trusted-compose.
Additional resources:
Bringing Content Trust into the World of docker-compose (medium.com)
Docker Compose not using trusted content during pull (https://github.com/docker/compose)
ERROR [main] 2017-08-04 13:24:21,949 CassandraDaemon.java:638 - Detected unreadable sstables /opt/cassandra/data/some_key_space/ep_lc_events-adc44160dbe611e6953689bcd3ed73aa/mc-547-big-Summary.db, and many others...
That has happened after I upgraded Cassandra to 3 version and after a while downgraded it to 2nd version.
When I run this command: sudo service cassandra status
I have got such message:
could not access pidfile for Cassandra
In /var/log/cassandra/system.log I have logs which I wrote at the beginning.
PS: let me pay your attention that everything is happening on EC2 Amazon instance.
Well, I have just upgraded back to 3rd version, used cassandra-unloader to export all data, then downgraded back to 2nd version and used cassandra-loader to import all data. But if you were lucky and had backups and snapshots it would not be an obstacle for you.
PS. Afterwards, I had to run this command nodetool resetlocalschema to reset local schema and resynchronize.
PPS. This you can find how to do that.
https://github.com/brianmhess/cassandra-loader
I also got the same error, but it was due to switching between cassandra 4.0.0 and version 3.11 and back again while using docker.
Update the version to the right one for the ssltable, or delete the data volume:
docker-compose logs cassandra
docker volumes ls
docker ps
docker-compose down
docker volume rm testapp_cassandra
docker volume ls
docker-compose up
I'm facing the following issue with my Docker containers: When I try to enter the container using
docker exec -it container-id /bin/bash
Docker (I assume it's Docker) complains with the following message:
proxy: unknown scheme: http
I have traced this back to the following environment variable that's set on my host machine, since I'm using a proxy server to access the web:
ALL_PROXY=http://myproxy:8080
The error message seems to come from the net/proxy.go file, which can be found here - the error message is issued on the last line of the file. Why would http not be a registered URL scheme in the Docker case?
As soon as I unset ALL_PROXY on the host, I can enter the container without any issues.
Environment:
Mac OS X v10.11.5
Docker v1.11.1
Docker-Machine v0.7.0
Any idea how to fix this (other than unsetting the variable each time)?
I am facing the same issue with docker 1.11.2. I believe the error is coming from method FromURL.
After checking the relevant commit https://github.com/docker/docker/commit/16effc66c028a7800096ed92174ca4bceba229ad, it turns out from v1.11.0-rc1 up to v1.12.0-rc4 are including this commit.
So the solution for me is to install a lower version (I used v1.10.3) of docker toolbox, after which "docker run hello-world" works.
My goal is to be able to remote debug from Atom.io. into a docker container running go in dlv debugger. This is the first problem:
Update:1. I am running Docker container on a mac, but that should not influence the code signing thing as I am running in a container, right?
Update:2. Codesignig the host, did not help.
Error:
1. root#...:/go/src/app# go get github.com/derekparker/delve/cmd/dlv
2. root#...:/go/src/app# dlv debug hello.go
could not launch process: fork/exec ./debug: operation not permitted
Then tried to
1. root#...:/go/src/app# sudo
2. bash: sudo: command not found
According Delve Issue #515
Docker has security settings preventing ptrace(2) operations by
default with in the container. Pass --security-opt seccomp:unconfined
to docker run when starting.
*confirmation of this in official docker bug tracker https://github.com/docker/docker/issues/21051
It seems to work if you start the container with --privileged. I've tried to figure out if there is a more fine-grained capability but failed.
Also I just found https://github.com/steeve/homebrew-delve which should make things easier on OSX.
Docker has security settings preventing ptrace(2)
See how i fixed it.
if using a docker-compose file to run the container then append seccomp:unconfined in the services section like below
api:
security_opt:
- seccomp:unconfined
if using docker run ...passing seccomp:unconfined works as well
Run Docker container as a command:
docker run -itd -p 2028:22 -p 2345:2345 --dns=10.236.8.8 --privileged=true --name=golang centos7-golang /usr/bin/supervisord
it works for me~