I'm trying to test out concourse on an ubuntu 14.04 ec2 instance. I am attempting to use the containerized version of the software with the docker-compose example shown here in the documentation. However on any attempt the
concourse-web container fails after about 15 seconds. I am just looking for a quick easy setup of concourse on ec2 so I can test it out, how can I get it running using the containerized version of the software?
More info:
Here is the script I am using to get it up and running:
mkdir concourse
cd concourse
mkdir -p keys/web keys/worker
ssh-keygen -t rsa -f ./keys/web/tsa_host_key -N ''
ssh-keygen -t rsa -f ./keys/web/session_signing_key -N ''
ssh-keygen -t rsa -f ./keys/worker/worker_key -N ''
cp ./keys/worker/worker_key.pub ./keys/web/authorized_worker_keys
cp ./keys/web/tsa_host_key.pub ./keys/worker
# for ec2
export CONCOURSE_EXTERNAL_URL=$(wget -q -O - http://instance-data/latest/meta-data/public-ipv4)
#creating docker compose file
echo 'concourse-db:
image: postgres:9.5
environment:
POSTGRES_DB: concourse
POSTGRES_USER: concourse
POSTGRES_PASSWORD: changeme
PGDATA: /database
concourse-web:
image: concourse/concourse
links: [concourse-db]
command: web
ports: ["8080:8080"]
volumes: ["./keys/web:/concourse-keys"]
environment:
CONCOURSE_BASIC_AUTH_USERNAME: concourse
CONCOURSE_BASIC_AUTH_PASSWORD: changeme
CONCOURSE_EXTERNAL_URL: "${CONCOURSE_EXTERNAL_URL}"
CONCOURSE_POSTGRES_DATA_SOURCE: |
postgres://concourse:changeme#concourse-db:5432/concourse?sslmode=disable
concourse-worker:
image: concourse/concourse
privileged: true
links: [concourse-web]
command: worker
volumes: ["./keys/worker:/concourse-keys"]
environment:
CONCOURSE_TSA_HOST: concourse-web' > docker-compose.yml
docker-compose up -d
However about 15 seconds after doing a docker-compose up -d the concorse_concourse-web_1 container stops running and I cannot connect to it through a browser at any point. Here are the docker logs of the container at the end right when it fails (there's more but I cant fit it all so just test it yourself to see the full logs):
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x5e093a]
goroutine 1 [running]:
panic(0xfba6c0, 0xc820016070)
/usr/local/go/src/runtime/panic.go:481 +0x3e6
github.com/concourse/atc/atccmd.(*ATCCommand).constructAPIHandler(0xc82023c608, 0x7ff484d1b5d0, 0xc8200501e0, 0xc82026f0e0, 0xc8202c9300, 0x7ff484d1d858, 0xc82030c5c0, 0x7ff484d1d980, 0xc8202afda0, 0x7ff484d1d958, ...)
/tmp/build/9674af12/concourse/src/github.com/concourse/atc/atccmd/command.go:787 +0x121a
github.com/concourse/atc/atccmd.(*ATCCommand).Runner(0xc82023c608, 0xc820270d30, 0x0, 0x1, 0x0, 0x0, 0x0, 0x0)
/tmp/build/9674af12/concourse/src/github.com/concourse/atc/atccmd/command.go:221 +0xe44
main.(*WebCommand).Execute(0xc82023c608, 0xc820270d30, 0x0, 0x1, 0x0, 0x0)
/tmp/build/9674af12/gopath/src/github.com/concourse/bin/cmd/concourse/web.go:54 +0x297
github.com/concourse/bin/vendor/github.com/vito/twentythousandtonnesofcrudeoil.installEnv.func2(0x7ff484d0b5e0, 0xc82023c608, 0xc820270d30, 0x0, 0x1, 0x0, 0x0)
/tmp/build/9674af12/gopath/src/github.com/concourse/bin/vendor/github.com/vito/twentythousandtonnesofcrudeoil/environment.go:30 +0x81
github.com/concourse/bin/vendor/github.com/jessevdk/go-flags.(*Parser).ParseArgs(0xc8200512c0, 0xc82000a150, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/tmp/build/9674af12/gopath/src/github.com/concourse/bin/vendor/github.com/jessevdk/go-flags/parser.go:312 +0xa34
github.com/concourse/bin/vendor/github.com/jessevdk/go-flags.(*Parser).Parse(0xc8200512c0, 0x0, 0x0, 0x0, 0x0, 0x0)
/tmp/build/9674af12/gopath/src/github.com/concourse/bin/vendor/github.com/jessevdk/go-flags/parser.go:185 +0x9b
main.main()
/tmp/build/9674af12/gopath/src/github.com/concourse/bin/cmd/concourse/main.go:29 +0x10d
Also after trying to stop and remove the containers the concorse_concourse-worker_1 container cannot be removed and shows up in a docker ps -a as Dead. The following error message occurs when attempting to remove it:
ubuntu#ip-172-31-59-167:~/concorse$ docker rm a005503d568b
Error response from daemon: Driver aufs failed to remove root filesystem a005503d568b4931f860334e95ff37265dc0913083d3592f0291e023275bbf20: rename /var/lib/docker/aufs/diff/9bcff3a39934ea3525bf8a06ef900bf9dfba59a5187747beb65e9ba5709ebf75 /var/lib/docker/aufs/diff/9bcff3a39934ea3525bf8a06ef900bf9dfba59a5187747beb65e9ba5709ebf75-removing: device or resource busy
The documentation on this has been updated with more succinct instructions.
Related
Currently, I have a strange issue in that when I try to connect to the Oracle Database container from an Apache Container, I will get "TNS: Connect Timeout Occurred" from oci_error method. My PHP code is as follow:
<?php
$objConnect = oci_connect('SYSTEM', 'xxxxxxxxxx', 'x.x.x.x/xxxxx');
if($objConnect)
{
echo "from Docker Oracle Server Connected" . PHP_EOL;
}
else
{
echo "Cannot connect to Oracle Server" . PHP_EOL;
var_dump( oci_error() );
}
?>
My docker code to run Oracle Database is:
docker run --name orcl_12c_r_1 -p 1521:1521 -p 5500:5500 -e ORACLE_SID=xxxxx oracle/database:12.1.0.2-se2
And I brought my Apache up with this docker-compose.yml:
version: '3'
services:
oraclelinuxphp:
build:
context: ./oraclelinuxphp
dockerfile: Dockerfile # install httpd and PHP here.
ports:
- "8080:80"
volumes:
- ./web:/var/www/html
However, this issue is resolved when I added network_mode host to the docker-compose.yml :
version: '3'
services:
oraclelinuxphp:
build:
context: ./oraclelinuxphp
dockerfile: Dockerfile # install httpd and PHP here.
ports:
- "8080:80"
volumes:
- ./web:/var/www/html
network_mode: "host"
I am still a rookie when it comes to Docker, and from here, I believe I am missing something in Docker. I could connect to the Oracle Database in the container on the server from Oracle SQL Developer app on my desktop without any problem. In addition, I also tried the non-Docker route, and the non-Docker PHP could also connect to this Oracle Database as well.
So I believe this is Docker issue. What am I missing here? How can I make it so the Apache Container could connect to the Oracle Database Container in this case?
Note: I am using :
centos-release-7-7.1908.0.el7.centos.x86_64
Docker version 19.03.8, build afacb8b
Oracle Database 12.1.0.2 from https://github.com/oracle/docker-images
There is some convenient magic happening when utilizing docker-compose vs. docker run. Network will implicitly be established.
An example:
cat > docker-compose.yml <<EOF
version: '3'
services:
c1:
image: alpine
container_name: c1
command: "sleep 1000"
c2:
image: alpine
container_name: c2
command: "sleep 1000"
EOF
# fire up the containers and detach
docker-compose up -d
These containers will be able to talk to each other due to the fact a default network has been established. (folder_name_default)
docker network ls -fname=demo1
NETWORK ID NAME DRIVER SCOPE
e3777f15f5aa demo1_default bridge local
# c1 can talk to c2
docker-compose exec c1 sh -c 'ping -c1 c2'
PING c2 (172.30.0.2): 56 data bytes
64 bytes from 172.30.0.2: seq=0 ttl=64 time=3.741 ms
# c2 can talk to c1
docker-compose exec c2 sh -c 'ping -c1 c1'
PING c1 (172.30.0.3): 56 data bytes
64 bytes from 172.30.0.3: seq=0 ttl=64 time=0.798 ms
Now, your scenario is that your database container is not attached to the network created by docker-compose. Like this:
docker run --rm -it --name c3 alpine sh -c 'ping -c1 c1'
ping: bad address 'c1'
You can define network for your run-command (this will work):
docker run --rm --net demo1_default -it --name c3 alpine sh -c 'ping -c1 c1'
PING c1 (172.30.0.3): 56 data bytes
64 bytes from 172.30.0.3: seq=0 ttl=64 time=0.571 ms
# make sure c3 keeps running while we try to contact it.
docker run --rm --net demo1_default -d -it --name c3 alpine sh -c 'sleep 1000'
# yes it works!
docker-compose exec c1 sh -c 'ping -c1 c3'
PING c3 (172.30.0.4): 56 data bytes
64 bytes from 172.30.0.4: seq=0 ttl=64 time=0.314 ms
To bundle all my different components together, I always define them in docker-compose where I explicitly name the network like this:
cat > docker-compose.yml <<EOF
version: '3'
services:
c1:
image: alpine
container_name: c1
command: "sleep 1000"
networks:
- mynet
c2:
image: alpine
container_name: c2
command: "sleep 1000"
networks:
- mynet
networks:
mynet:
EOF
docker run is just for ad-hoc imperative stuff. I would include the the database service in the docker-compose.yml file or a separate file like docker-compose -f mydb.yml up -d where network-name is defined. From there on your containers will be able to communicate with the database.
Best of luck!
I'm using docker-compose to run three containers. Two of them depends on database so I'm using wait-for-it.sh to make sure they are not run until database is listening.
This is my docker-compose.yml file:
web:
build: ./docker/web
command: ["./wait-for-it.sh", "db:5432", "--", "python", "manage.py", "runserver", "0.0.0.0:8080"]
ports:
- "8080:8080"
depends_on:
- db
- spider
links:
- db
When I run docker-compose up command I get the error:
web_1 | wait-for-it.sh: waiting 15 seconds for db:5432
web_1 | wait-for-it.sh: db:5432 is available after 0 seconds
web_1 | python: can't open file 'manage.py': [Errno 2] No such file or directory
When I add volume .:/src the manage.py is found but wait-for-it.sh isn't:
ERROR: for web Cannot start service web: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"./wait-for-it.sh\": stat ./wait-for-it.sh: no such file or directory": unknown
I added wait-for-it.sh file to the directory where Dockerfile for web service is.
Any idea how can I make this work?
EDIT
Here's the Dockerfile used in docker-compose:
FROM python:3.6
ENV PYTHONUNBUFFERED 1
RUN mkdir /src
COPY . /src
WORKDIR /src
RUN pip install -r requirements.txt
I fixed it by changing approach. Added healthcheck to db service:
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:5432"]
interval: 5s
timeout: 30s
retries: 5
And restart policies to other services:
restart: on-failure
I followed the tutorial
https://docs.docker.com/get-started/part4/#deploy-the-app-on-the-swarm-manager
And created my own registry using
https://github.com/docker/docker-registry/blob/master/README.md#quick-start
https://docs.docker.com/registry/#basic-commands
https://blog.docker.com/2013/07/how-to-use-your-own-registry/
However it fails to deploy on the worker nodes with the error "No such image: 192.168.99.100". What is wrong?
docker run -d -p 5000:5000 --name registry registry:2
docker tag friendlyhello 192.168.99.100:5000/get-started:part2
docker push 192.168.99.100:5000/get-started # Get https://192.168.99.100:5000/v2/: http: server gave HTTP response to HTTPS client
docker tag friendlyhello localhost:5000/get-started:part2
docker push localhost:5000/get-started:part2
docker stack deploy -c docker-compose.yml getstartedlab
docker service ps getstartedlab_web
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR
PORTS
o4nbsqccqlm4 getstartedlab_web.1 192.168.99.100:5000/get-started:part2 default Running Running 17 minutes ago
qcjtq3gqag9j \_ getstartedlab_web.1 192.168.99.100:5000/get-started:part2 myvm1 Shutdown Rejected 17 minutes ago "No such image: 192.168.99.100â?▌"
This is my docker-compose.yml file:
...
image: 192.168.99.100:5000/get-started:part2
...
I tried to use image: localhost:5000/get-started:part2 in the docker-compose.yml file also, but it gave the error No such image: localhost:5000.
docker stack rm getstartedlab
docker stack deploy -c docker-compose.yml getstartedlab
docker service ps getstartedlab_web
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR
PORTS
k2cck1p7wpg1 getstartedlab_web.1 localhost:5000/get-started:part2 default Running Running 10 seconds ago
69km7zabgw6l \_ getstartedlab_web.1 localhost:5000/get-started:part2 myvm1 Shutdown Rejected 21 seconds ago "No such image: localhost:5000â?▌"
Windows 8.1, Docker version 18.03.0-ce, build 0520e24302
I'm trying to test fabric chaincode example02, with docker. I'm newbie :)
This is my docker-compose.yml :
membersrvc:
image: hyperledger/fabric-membersrvc
command: membersrvc
vp0:
image: hyperledger/fabric-peer
environment:
- CORE_PER_ID=vp0
- CORE_PEER_ADDRESSAUTODETECT=true
- CORE_VM_ENDPOINT=http://0.0.0.0:2375
- CORE_LOGGING_LEVEL=DEBUG
command: sh -c "sleep 5; peer node start --peer-chaincodedev"
vp1:
extends:
service: vp0
environment:
- CORE_PEER_ID=vp1
- CORE_PEER_DISCOVERY_ROOTNODE=vp0:7051
links:
- vp0
vp2:
extends:
service: vp0
environment:
- CORE_PEER_ID=vp2
- CORE_PEER_DISCOVERY_ROOTNODE=vp0:7051
links:
- vp0
and I run (I refered to Fabric chaincode setup page):
Terminal 1 :
$ docker-compose up
Terminal 2 :
$ cd /hyperledger/examples/chaincode/go/chaincode_example02
$ CORE_CHAINCODE_ID_NAME=mycc CORE_PEER_ADDRESS=0.0.0.0:7051 ./chaincode_example02
Terminal 3 :
$ peer chaincode deploy -n mycc -c '{"Args": ["init", "a","100", "b", "200"]}'
It works well in terminal 1,2. But terminal 3 shows connection error.
2016/10/21 04:39:15 grpc: addrConn.resetTransport failed to create client
transport: connection error: desc = "transport: dial tcp 0.0.0.0:7051:
getsockopt: connection refused"; Reconnecting to {"0.0.0.0:7051" <nil>}
Error: Error building chaincode: Error trying to connect to local peer:
grpc: timed out when dialing
What's the problem?
It seems you are missing the compose statements to map the required ports from the docker container to the host machine (where you are trying out the peer command ). So its possible that the peer process is listening on port 7051 inside your peer docker container, but this connection is not available to the peer command used outside of this container in terminal 3.
You can map ports using the 'ports' tag. eg:
membersrvc:
image: hyperledger/fabric-membersrvc
ports:
- "7054:7054"
command: membersrvc
vp0:
image: hyperledger/fabric-peer
ports:
- "7050:7050"
- "7051:7051"
- "7053:7053"
environment:
- CORE_PER_ID=vp0
- CORE_PEER_ADDRESSAUTODETECT=true
- CORE_VM_ENDPOINT=http://0.0.0.0:2375
- CORE_LOGGING_LEVEL=DEBUG
command: sh -c "sleep 5; peer node start --peer-chaincodedev"
Before you do peer chaincode deploy ...in terminal 3, you can check if a the peer process is listening on port 7051 using
netstat -lnptu |grep 7051
I'm trying to connect golang and reds through Docker using docker-compose but I'm not having much luck. I have published my attempt at https://github.com/davidwilde/docker-compose-golang-redis/tree/stackoverflow_question and listed the logs below.
Redis says it is ready to accept connections but my golang app using gopkg.in/redis.v3 says no.
~/workspace/composetest master ● docker-compose up
Starting composetest_db_1...
Starting composetest_web_1...
.
.
.
ur kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
db_1 | 1:M 20 Nov 05:58:33.371 * DB loaded from disk: 0.000 seconds
db_1 | 1:M 20 Nov 05:58:33.371 * The server is now ready to accept connections on port 6379
web_1 | panic: dial tcp [::1]:6379: getsockopt: connection refused
web_1 |
web_1 | goroutine 1 [running]:
web_1 | main.main()
web_1 | /go/src/app/app.go:19 +0x131
web_1 |
web_1 | goroutine 17 [syscall, locked to thread]:
web_1 | runtime.goexit()
web_1 | /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1
web_1 | panic: dial tcp [::1]:6379: getsockopt: connection refused
web_1 |
web_1 | goroutine 1 [running]:
web_1 | main.main()
web_1 | /go/src/app/app.go:19 +0x131
web_1 |
web_1 | goroutine 17 [syscall, locked to thread]:
web_1 | runtime.goexit()
web_1 | /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1
So we have two different containers which means two different "localhost" in this case.
client := redis.NewClient(&redis.Options{
Addr: "localhost:6379",
Password: "",
DB: 0,
})
So, your app is making requests to its own sandboxed container, not to your "other" sandboxed container which includes redis.
You have two options;
Give a mapping in your compose file like redisdb:db and pass that information instead of localhost.
Or, use the "--net=host" option in order to provide common networking for your containers without changing your code.
edit: typo
The answer from #Gladmir is great. Just to expand on his/her answer, I didn't need to remove localhost from my Golang code:
client := redis.NewClient(&redis.Options{
Addr: "localhost:6379",
Password: "",
DB: 0,
})
I changeed my Docker Compose file to use network_mode: "host":
version: "3.9"
services:
web:
build:
context: .
network_mode: "host"
redis:
container_name: "redis"
image: "redis:alpine"
command: redis-server /usr/local/etc/redis/redis.conf
ports:
- "6379:6379"
volumes:
- $PWD/configs/redis.conf:/usr/local/etc/redis/redis.conf