Changing hostname breaks Rabbitmq when running on Kubernetes - amazon-ec2

Question

I'm trying to run Rabbitmq using Kubernetes on AWS. I'm using the official Rabbitmq docker container. Each time the pod restarts the rabbitmq container gets a new hostname. I've setup a service (of type LoadBalancer) for the pod with a resolvable DNS name.
But when I use an EBS to make the rabbit config/messsage/queues persistent between restarts it breaks with:
exception exit: {{failed_to_cluster_with,
['rabbitmq#rabbitmq-deployment-2901855891-nord3'],
"Mnesia could not connect to any nodes."},
{rabbit,start,[normal,[]]}}
in function application_master:init/4 (application_master.erl, line 134)
rabbitmq-deployment-2901855891-nord3 is the previous hostname rabbitmq container. It is almost like Mnesia saved the old hostname :-/
The container's info looks like this:
Starting broker...
=INFO REPORT==== 25-Apr-2016::12:42:42 ===
node : rabbitmq#rabbitmq-deployment-2770204827-cboj8
home dir : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash : XXXXXXXXXXXXXXXX
log : tty
sasl log : tty
database dir : /var/lib/rabbitmq/mnesia/rabbitmq
I'm only able to set the first part of the node name to rabbitmq using the RABBITMQ_NODENAME environment variable.
Setting RABBITMQ_NODENAME to a resolvable DNS name breaks with:
Can't set short node name!\nPlease check your configuration\n"
Setting RABBITMQ_USE_LONGNAME to true breaks with:
Can't set long node name!\nPlease check your configuration\n"
Update:
Setting RABBITMQ_NODENAME to rabbitmq#localhost works but that negates any possibility to cluster instances.
Starting broker...
=INFO REPORT==== 26-Apr-2016::11:53:19 ===
node : rabbitmq#localhost
home dir : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash : 9WtXr5XgK4KXE/soTc6Lag==
log : tty
sasl log : tty
database dir : /var/lib/rabbitmq/mnesia/rabbitmq#localhost
Setting RABBITMQ_NODENAME to the service name, in this case rabbitmq-service like so rabbitmq#rabbitmq-service also works since kubernetes service names are internally resolvable via DNS.
Starting broker...
=INFO REPORT==== 26-Apr-2016::11:53:19 ===
node : rabbitmq#rabbitmq-service
home dir : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash : 9WtXr5XgK4KXE/soTc6Lag==
log : tty
sasl log : tty
database dir : /var/lib/rabbitmq/mnesia/rabbitmq#rabbitmq-service
Is this the right way though? Will I still be able to cluster multiple instances if the node names are the same?

Answer 1

The idea is to use a different 'service' and 'deployment' for each of the node you want to create.
As you said, you have to create a custom NODENAME for each i.e:
RABBITMQ_NODENAME=rabbit#rabbitmq-1
Also rabbitmq-1,rabbitmq-2,rabbitmq-3 have to be resolved from each nodes. For that you can use kubedns. The /etc/resolv.conf will look like:
search rmq.svc.cluster.local
and /etc/hosts must contains:
127.0.0.1 rabbitmq-1 # or rabbitmq-2 on node 2...
The services are here to create a stable network identity for each nodes
rabbitmq-1.svc.cluster.local
rabbitmq-2.svc.cluster.local
rabbitmq-3.svc.cluster.local
The different deployments resources will allow you to mount a different volume on each node.
I'm working on a deployment tool to simplify those actions:
I've done a demo on how I scale and deploy rabbitmq from 1 to 3 nodes on kubernetes:
https://asciinema.org/a/2ktj7kr2d2m3w25xrpz7mjkbu?speed=1.5
More generally, the complexity your facing to deploy a clustered application is addressed in the 'petset proposal': https://github.com/kubernetes/kubernetes/pull/18016

Answer 2

In addition to the first reply by #ant31:
Kubernetes now allows to setup a hostname, e.g. in yaml:
template:
metadata:
annotations:
"pod.beta.kubernetes.io/hostname": rabbit-rc1
See https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns # A Records and hostname Based on Pod Annotations - A Beta Feature in Kubernetes v1.2
It seems that the whole configuration alive multiple restarts or re-schedules. I've not setup a cluster however I'm going to follow the tutorial for mongodb, see https://www.mongodb.com/blog/post/running-mongodb-as-a-microservice-with-docker-and-kubernetes
The approach will be probably almost same from kubernetes point of view.