Gossip encryption not working fine - consul

Question

I have created a master token using the below command:
$ consul keygen
G74SM8N9NUc4meaHfA7CFg==
Then, I bootstrapped the server with the following config.json:
{
"server": true,
"datacenter": "consul",
"data_dir": "/var/consul",
"log_level": "INFO",
"enable_syslog": true,
"disable_update_check": true,
"client_addr": "0.0.0.0",
"bootstrap": true,
"leave_on_terminate": true,
"encrypt": "G74SM8N9NUc4meaHfA7CFg=="
}
The output of the bootstrap server is as follows:
Node name: 'abcd'
Datacenter: 'consul'
Server: true (bootstrap: true)
Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600, RPC: 8400)
Cluster Addr: x.x.x.x (LAN: 8301, WAN: 8302)
Gossip encrypt: true, RPC-TLS: false, TLS-Incoming: false
Atlas: <disabled>
Then, I added a new server as a regular consul server which has the following config.json:
{
"server": true,
"datacenter": "consul",
"data_dir": "/var/consul",
"log_level": "INFO",
"enable_syslog": true,
"disable_update_check": true,
"client_addr": "0.0.0.0",
"bootstrap": false,
"leave_on_terminate": true,
"ui_dir": "/usr/local/bin/consul_ui",
"check_update_interval": "0s",
"ports": {
"dns": 8600,
"http": 8500,
"https": 8700,
"rpc": 8400,
"serf_lan": 8301,
"serf_wan": 8302,
"server": 8300
},
"dns_config": {
"allow_stale": true,
"enable_truncate": true,
"only_passing": true,
"max_stale": "02s",
"node_ttl": "30s",
"service_ttl": {
"*": "10s"
}
},
"advertise_addr": "y.y.y.y",
"encrypt": "G74SM8N9NUc4meaHfA7CFg==",
"retry_join": [
"x.x.x.x",
"y.y.y.y"
]
}
Note: Here, x.x.x.x is IP address of the bootstrap server, y.y.y.y is IP address of the regular server.
For testing purpose, I changed the encrypt key on one of the servers. And, when I do consul members, I can still see the all IPs which means that the servers are still able to communicate even with the different encrypt key. It seems that the gossip encryption is not working fine.

Answer 1

A Consul instance will cache the initial key and re-use it. It is stored in the serf folder in the file local.keyring.
This is counter-intuitive, but it is documented at least in one place together with the encrypt option.
You'll need to delete this file and restart Consul in order to get the expected behaviour.