Using Elasticsearch:latest (v5.1) from the Docker public repo, I created my own image containing Cerebro. I am now attempting to get Elasticsearch networking properly configured so that I can connect to Elasticsearch from Cerebro. Cerebro running inside of the container I created, renders properly on my host at: http://localhost:9000.
After committing my image, I created my Docker container with the following:
sudo docker run -d -it --privileged --name es5.1 --restart=always \
-p 9200:9200 \
-p 9300:9300 \
-p 9000:9000 \
-v ~/elasticsearch/5.1/config:/usr/share/elasticsearch/config \
-v ~/elasticsearch/5.1/data:/usr/share/elasticsearch/data \
-v ~/elasticsearch/5.1/cerebro/conf:/root/cerebro-0.4.2/conf \
elasticsearch_cerebro:5.1 \
/root/cerebro-0.4.2/bin/cerebro
my elasticsearch.yml in ~/elasticsearch/5.1/config currently has the following network and discovery entries specified:
network.publish_host: 192.168.1.26
discovery.zen.ping.unicast.hosts: ["192.168.1.26:9300"]
I have also tried 0.0.0.0 and not specifying the values to default to the loopback for these settings. In addition, I've tried specifying network.host with a combination of values. No matter how I set this, elasticsearch logs on startup:
[info] play.api.Play - Application started (Prod)
[info] p.c.s.NettyServer - Listening for HTTP on /0:0:0:0:0:0:0:0:9000
[error] p.c.s.n.PlayDefaultUpstreamHandler - Cannot invoke the action
java.net.ConnectException: Connection refused: localhost/127.0.0.1:9200
… cascading errors because of this connection refusal...
No matter how I set the elasticsearch.yml networking, the error message on Elasticsearch startup does not change. I verified that the elasticsearch.yml is being picked-up inside of the Docker container. Please let me know were I'm going wrong with this configuration.
Well, it looks like I"m answering my own question after a days-worth of battle with this! The issue was that elasticsearch wasn't started inside of the container. To determine this, I got a terminal into the container:
docker exec -it es5.1 bash
Once in the container, I checked service status:
service elasticsearch status
To this, the OS responded with:
[FAIL] elasticsearch is not running ... failed!
I started it with:
service elasticsearch start
I add a single script that I'll call from docker run to start elasticsearch and cerebro and that should do the trick. However, I would still like to hear if there is a better way to configure this.
I made a github docker-compose repo that will spin up a elasticsearch, kibana, logstash, cerebro cluster
https://github.com/Shuliyey/elkc
========================================================================
On the other hand, in regard to the actual problem (elasticsearch_cerebro not working).
To get the elasticsearch and cerebro working in one docker container. Need to use supervisor
https://docs.docker.com/engine/admin/using_supervisord/
will update with more details
No need to use supervisor at all. A very simple way to solve this is to use docker-compose and bundle Elasticsearch and Cerebro together, like this:
docker-compose.yml:
version: '2'
services:
elasticsearch:
build: elasticsearch
volumes:
- ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
- ./elasticsearch/data:/usr/share/elasticsearch/data
ports:
- "9200:9200"
- "9300:9300"
environment:
ES_JAVA_OPTS: "-Xmx1500m -Xms1500m"
networks:
- elk
cerebro:
build: cerebro
volumes:
- ./cerebro/config/application.conf:/opt/cerebro/conf/application.conf
ports:
- "9000:9000"
networks:
- elk
depends_on:
- elasticsearch
networks:
elk:
driver: bridge
elasticsearch/Dockerfile:
FROM docker.elastic.co/elasticsearch/elasticsearch:5.5.1
cerebro/Dockerfile:
FROM yannart/cerebro
Then you run docker-compose build and docker-compose up. When everything is started, you can access ES at http://localhost:9200 and Cerebro at http://localhost:9000
Related
I'm running an "elasticsearch" container. I can curl the container and get results but when I try to communicate with the container from within my "web" container it refuses the connection.
docker-compose up
curl localhost:9200 // works.
curl docker-compose run web curl localhost:9200 // connection refused.
docker-compose.yml
version: '2'
services:
web:
build:
context: .
dockerfile: Dockerfile
volumes:
- .:/src
ports:
- "5000:5000"
depends_on:
- elasticsearch
elasticsearch:
image: elasticsearch:5.1.2
ports:
- "9200:9200"
Dockerfile
FROM python:3.5
ADD . /src
WORKDIR /src
RUN pip install -r requirements.txt
CMD python project/wsgi.py
You cannot use localhost:9200 from within the web container to connect to the elasticsearch container. You could define a link or just use the service name (which is mapped by default):
curl elasticsearch:9200
Links allow you to define extra aliases by which a service is reachable from another service. They are not required to enable services to communicate - by default, any service can reach any other service at that service’s name.
Also see Docker Compose Links
You should be trying to curl elasticsearch:9200, not localhost:9200. The hostname elasticsearch should be in your hosts file on the web container.
I'm working with docker for the first time.
I successfully installed elasticsearch and kibana on docker, but when I try to connect kibana with elastic I get a red status with the following errors:
ui settings Elasticsearch plugin is red
plugin:elasticsearch#5.1.1 Authentication Exception
I'm not sure but I think the problem is kibana doesn't pass elastic x-pack authentication.
Now, I'm trying to disable this authentication via elastic yml file, according to the instructions here.
But I can't find the yml file anywhere (I searched /usr/share/elasticsearch but I can't find either config directory or elasticsearch.yml file).
How do I config elastic with docker?
P.S.
I'm working with ubuntu 16.04
For Debian/Ubuntu/Mint, you can find the config files under /etc folder.
/etc/elasticsearch/elasticsearch.yml
Take a look at: https://www.elastic.co/guide/en/elasticsearch/reference/2.4/setup-dir-layout.html
I'm wondering why this is even happening. With the following docker-compose.yml it's working fine for me with security enabled:
---
version: '2'
services:
kibana:
image: docker.elastic.co/kibana/kibana:5.1.1
links:
- elasticsearch
ports:
- 5602:5601
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:5.1.1
cap_add:
- IPC_LOCK
volumes:
- esdata1:/usr/share/elasticsearch/data
ports:
- 9201:9200
volumes:
esdata1:
driver: local
I successfully run elastic and kibana using the official elastic docker. Somehow, the container version in the official elastic documention didn't work for me.
If you prefer to start a container using docker run and not through compose file. (only use this for dev envs, not recommended on prod envs)
docker network create elastic
docker run --network=elastic --name=elasticsearch docker.elastic.co/elasticsearch/elasticsearch:5.2.2
docker run --network=elastic -p 5601:5601 docker.elastic.co/kibana/kibana:5.2.2
A brief description can be found here:
https://discuss.elastic.co/t/kibana-docker-image-doesnt-connect-to-elasticsearch-image/79511/4
I have the following docker containers running on my box...
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5da7523e527b kibana "/docker-entrypoint.s" About a minute ago Up About a minute 0.0.0.0:5601->5601/tcp elated_lovelace
20aea0e545ca elasticsearch "/docker-entrypoint.s" 3 hours ago Up 3 hours 0.0.0.0:9200->9200/tcp, 9300/tcp sad_meitner
My aim was to get kibana to link to my elasticsearch container however when I hit kibana it's telling me that I do not have any document stores. I know this is not right because I definitely have documents in elasticsearch. I'm guessing my link command is wrong.
This is the docker command I used to start the kibana container.
docker run -p 5601:5601 --link sad_meitner:elasticsearch -d kibana
Can someone tell me what I've done wrong?
thanks
First of all, Linking is a legacy feature, Create a user defined network first:
docker network create mynetwork --driver=bridge
Now use mynetwork for containers you want to be able to communicate with each other.
docker run -p 5601:5601 --name kibana -d --network mynetwork kibana
docker run -p 9200:9200 -p 9300:9300 --name elasticsearch -d --network mynetwork elasticsearch
Docker will run a dns server for your user defined network, so you can ping other container by name.
docker exec -it kibana /bin/bash
ping elasticsearch
You can use telnet or curl to verify kibana->elasticsearch connectivity from kibana container.
p.s I used official (library) docker images for ELK stack with user defined networking recently and it worked like a charm.
you can add ENV ELASTICSEARCH_URL=elasticsearch:9200 to your Dockerfile before build kibana, then use docker-compose to run elasticsearch with kibana like this:
version: '2'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:5.3.0
container_name: elasticsearch
ports:
- "9200:9200"
- "9300:9300"
environment:
ES_JAVA_OPTS: "-Xmx256m -Xms256m"
kibana:
image: docker.elastic.co/kibana/kibana:5.3.0
container_name: kibana
ports:
- "5601:5601"
depends_on:
- elasticsearch
I tried to create Kibana and Elasticsearch and it seems that Kibana is having trouble identifying Elasticsearch.
Here are my steps:
1) Create network
docker network create mynetwork --driver=bridge
2) Run Elasticsearch Container
docker run -d -p 9200:9200 -p 9300:9300 --name elasticsearch_2_4 --network mynetwork elasticsearch:2.4
3) Run Kibana Container
docker run -i --network mynetwork -p 5601:5601 kibana:4.6
I get a JSON output when I connect to Elasticsearch via http://localhost:9200/ through my browser.
But when I open http://localhost:5601/ I get
Unable to connect to Elasticsearch at http://elasticsearch:9200.
Alternate Approach,
I still get a similar error when I try
docker run -d -e ELASTICSEARCH_URL=http://127.0.0.1:9200 -p 5601:5601 kibana:4.6
where I get the error
Unable to connect to Elasticsearch at http://127.0.0.1:9200.
My blog post based on the accepted answer: https://gunith.github.io/docker-kibana-elasticsearch/
There is some misunderstanding about what localhost or 127.0.0.1 means when running a command inside a container. Because every container has its own networking, localhost is not your real host system but either the container itself. So when you are running kibana and pointing the ELASTICSEARCH_URL variable to localhost:9200 the kibana process will look for elasticsearch inside the kibana container which of course isn't running there.
You already introduced some custom network that you referenced when starting the containers. All containers running in the same network can reference each other via name on their exposed ports (see Dockerfiles). As you named your elasticsearch container elasticsearch_2_4, you can reference the http endpoint of elasticsearch as http://elasticsearch_2_4:9200.
docker run -d --network mynetwork -e ELASTICSEARCH_URL=http://elasticsearch_2_4:9200 -p 5601:5601 kibana:4.6
As long as you don't need to access the elasticsearch instance directly, you can even omit mapping the ports 9200 and 9300 to your host.
Instead of starting all containers on their own, I would also suggest to use docker-compose to manage all services and parameters. You should also consider mounting a local folder as volume to have the data persisted. This could be your compose file. Add the networks, if you need to have the external network, otherwise this setup just creates a network for you.
version: "2"
services:
elasticsearch:
image: elasticsearch:2.4
ports:
- "9200:9200"
volumes:
- ./esdata/:/usr/share/elasticsearch/data/
kibana:
image: kibana:4.6
ports:
- "5601:5601"
environment:
- ELASTICSEARCH_URL=http://elasticsearch:9200
Test:
docker run -d -e ELASTICSEARCH_URL=http://yourhostip:9200 -p 5601:5601 kibana:4.6
You can test with your host ip or the ip identified by docker0 in ifconfig
Regards
I changed network configuration for Kibana container and after this it works fine:
I want to run 2 instances of Elasticsearch on 2 different hosts.
I have built my own Docker image based on Ubuntu 14.04 and the 1.3.2 version of Elasticsearch. If I run 2 ES containers on 1 host, each instance can see and communicate with the other; but when I run 2 instances of ES on 2 different hosts, it didn't work. The 9300 port of the container is bind to the 9300 host's port.
Is it possible to create an ES cluster with my configuration?
I was able to get clustering working using unicast across two docker hosts. I just happen to be using the ehazlett/elasticsearch image, but I do not think this should matter all that much. The really important bit seems to be setting the network.publish_host setting to a public or routable IP its docker host.
Configuration
docker-host-01
eth0: 192.168.1.10
Docker version 1.4.1, build 5bc2ff8/1.4.1
docker-host-02
eth0: 192.168.1.20
Docker version 1.4.1, build 5bc2ff8/1.4.1
Building the Cluster
On Docker Host 01
docker run -d \
-p 9200:9200 \
-p 9300:9300 \
ehazlett/elasticsearch \
--cluster.name=unicast \
--network.publish_host=192.168.1.10 \
--discovery.zen.ping.multicast.enabled=false \
--discovery.zen.ping.unicast.hosts=192.168.1.20 \
--discovery.zen.ping.timeout=3s \
--discovery.zen.minimum_master_nodes=1
On Docker Host 02
docker run -d \
-p 9200:9200 \
-p 9300:9300 \
ehazlett/elasticsearch \
--cluster.name=unicast \
--network.publish_host=192.168.1.20 \
--discovery.zen.ping.multicast.enabled=false \
--discovery.zen.ping.unicast.hosts=192.168.1.10 \
--discovery.zen.ping.timeout=3s \
--discovery.zen.minimum_master_nodes=1
Using docker-compose is much easier than running it manually in command line:
elasticsearch_master:
image: elasticsearch:latest
command: "elasticsearch -Des.cluster.name=workagram -Des.node.master=true -Des.node.data=false"
environment:
- ES_HEAP_SIZE=512m
ports:
- "9200:9200"
- "9300:9300"
elasticsearch1:
image: elasticsearch:latest
command: "elasticsearch -Des.cluster.name=workagram -Des.discovery.zen.ping.unicast.hosts=elasticsearch_master"
links:
- elasticsearch_master
volumes:
- "/opt/elasticsearch/data"
environment:
- ES_HEAP_SIZE=512m
elasticsearch2:
image: elasticsearch:latest
command: "elasticsearch -Des.cluster.name=workagram -Des.discovery.zen.ping.unicast.hosts=elasticsearch_master"
links:
- elasticsearch_master
volumes:
- "/opt/elasticsearch/data"
environment:
- ES_HEAP_SIZE=512m
You should be able to communicate the two containers running in different hosts as far as the host machines are accessible between them in the ports needed. I think your problem is that you are trying to use ElasticSearch multicast discovery, but if then you need to expose also port 54328 of the containers. If it doesn't work you can also try to configure ElasticSearch using unicast, setting adequately the machines IP's in your elasticsearch.yml.