How can I get Elasticsearch running on CoreOS? - elasticsearch

Question

I've a CoreOS cluster with three servers (on Digital Ocean), at this moment running MongoDB. Now I want to start Elasticsearch on this cluster with 1 replica (not using the Mongo river).
I followed the description as outlined here.
Resulting in two services, elasticsearch#.service & elasticsearch-discovery#.service.
elasticsearch#.service
[Unit]
Description=ElasticSearch service
After=etcd.service
After=docker.service
Before=elasticsearch-discovery#%i.service
Requires=elasticsearch-discovery#%i.service
[Service]
KillMode=none
TimeoutStartSec=0
TimeoutStopSec=360
EnvironmentFile=/etc/environment
ExecStartPre=-/usr/bin/docker kill %p-%i
ExecStartPre=-/usr/bin/docker rm %p-%i
ExecStartPre=-/usr/bin/bash -c "echo PreKill and rm done;"
ExecStartPre=/usr/bin/mkdir -p /data/elasticsearch
ExecStartPre=/usr/bin/docker pull dockerfile/elasticsearch
ExecStartPre=/usr/bin/bash -c "echo mkdir and docker pull done;"
ExecStart=/bin/bash -c "\
echo StartingUp; \
curl -f ${COREOS_PUBLIC_IPV4}:4001/v2/keys/services/elasticsearch; \
if [ $? = 0 ]; then \
UNICAST_HOSTS = $(etcdctl ls --recursive /services/elasticsearch | sed 's/\/services\/elasticsearch\///g' | sed 's/$/:9300/' | paste -s -d ','); \
echo Key found; \
else \
UNICAST_HOSTS=''; \
echo No Key found; \
fi;"
ExecStartPost=/bin/bash -c "\
echo Starting Docker; \
/usr/bin/docker run \
--name %p-%i \
--publish 9200:9200 \
--publish 9300:9300 \
--volume /data/elasticsearch:/data \
dockerfile/elasticsearch \
/elasticsearch/bin/elasticsearch \
--node.name=%p-%i \
--cluster.name=nvssearch \
--network.publish_host=${COREOS_PUBLIC_IPV4} \
--discovery.zen.ping.multicast.enabled=false \
--discovery.zen.ping.unicast.hosts=$UNICAST_HOSTS;"
ExecStop=/bin/bash/ -c "/usr/bin/docker kill %p-%i"
Restart=on-failure
[X-Fleet]
X-Conflicts=%p#*.service
elasticsearch-discovery#.service
[Unit]
Description=ElasticSearch discovery service
BindsTo=elasticsearch#%i.service
After=elasticsearch#%i.service
[Service]
EnvironmentFile=/etc/environment
ExecStart=/bin/bash -c '\
while true; do \
curl -f ${COREOS_PUBLIC_IPV4}:9200; \
if [ "$?" = "0" ]; then \
etcdctl set /services/elasticsearch/${COREOS_PUBLIC_IPV4} \ '{"http_port": 9200, "transport_port": 9300}\' --ttl 60; \
else \
etcdctl rm /services/elasticsearch/${COREOS_PUBLIC_IPV4}; \
fi; \
sleep 45; \
done'
ExecStop=/usr/bin.etcdctl rm /services/elasticsearch/${COREOS_PUBLIC_IPV4}
[X-Fleet]
X-ConditionMachineOf=elasticsearch#%i.service
But if I try to run the service (fleetctl submit / load / start elasticsearch#1.service), it immediately dies:
elasticsearch#1.service 475f6273.../IP inactive dead
Running fleetctl journal elasticsearch#1 results in the following message:
Mar 17 09:17:04 nvs-1 systemd[1]: Stopped ElasticSearch service.
That's all, no echoes I've added (to the service) are shown, or whatsoever. Anyone any ideas on how to get me further?

Answer 1

It is somewhat simpler to get Elasticsearch running with Weave. My blog post shows how to run it on Vagrant, however moving the same setup to cloud should be pretty straight-forward.

Answer 2

I've followed the same article you did and ran into the same issue. I tracked it down to the docker container not being available anymore, so I change the docker run command in the elasticsearch#service unit:
[Unit]
Description=ElasticSearch service
After=docker.service
Requires=docker.service
[Service]
TimeoutSec=180
EnvironmentFile=/etc/environment
ExecStartPre=/usr/bin/mkdir -p /data/elasticsearch
ExecStartPre=/usr/bin/docker pull elasticsearch
ExecStart=/bin/bash -c '\
curl -f ${COREOS_PRIVATE_IPV4}:4001/v2/keys/services/elasticsearch; \
if [ "$?" = "0" ]; then \
UNICAST_HOSTS=$(etcdctl ls --recursive /services/elasticsearch \
| sed "s/\/services\/elasticsearch\///g" \
| sed "s/$/:9300/" \
| paste -s -d","); \
else \
UNICAST_HOSTS=""; \
fi; \
/usr/bin/docker run \
--rm \
--name %p-%i \
--publish 9200:9200 \
--publish 9300:9300 \
--volume /data/elasticsearch:/data \
elasticsearch \
--node.name=%p-%i \
--cluster.name=logstash \
--network.publish_host=${COREOS_PRIVATE_IPV4} \
--discovery.zen.ping.multicast.enabled=false \
--discovery.zen.ping.unicast.hosts=$UNICAST_HOSTS'
ExecStop=/usr/bin/docker stop %p-%i
ExecStop=/usr/bin/docker rm %p-%i
[X-Fleet]
X-Conflicts=%p#*.service
Apart from that I think, on your original post, you're missing some quotes, double quotes and have some wrong blank spaces.
This is my current elastic-discovery#service for reference:
[Unit]
Description=ElasticSearch discovery service
BindsTo=elasticsearch#%i.service
[Service]
EnvironmentFile=/etc/environment
ExecStart=/bin/bash -c '\
while true; do \
curl -f ${COREOS_PRIVATE_IPV4}:9200; \
if [ "$?" = "0" ]; then \
etcdctl set /services/elasticsearch/${COREOS_PRIVATE_IPV4} \'{"http_port": 9200, "transport_port": 9300}\' --ttl 60; \
else \
etcdctl rm /services/elasticsearch/${COREOS_PRIVATE_IPV4}; \
fi; \
sleep 45; \
done'
ExecStop=/usr/bin/etcdctl rm /services/elasticsearch/${COREOS_PRIVATE_IPV4}
[X-Fleet]
X-ConditionMachineOf=elasticsearch#%i.service