Elastic 2.3.4. Node Startup Quiet Failure - elasticsearch

Question

We are using a 5 node cluster hosted in Google Cloud (Ubuntu 16.04 LTS) and we noticed that one of the node's disk space was at 90%+ so we shut down the node with:
sudo service elasticsearch stop
Then stopping the instance in the GCP console.
After upgrading the node's disk space, we tried starting elastic again using:
sudo service elasticsearch start
This command seems to fail silently, and the SSH session terminates after freezing momentarily. Nothing shows in the node's elasticsearch logs, and nothing shows up in the current cluster's master elasticsearch logs either. The only hint we can find of something going wrong is in the node's syslog:
Jan 25 15:48:29 elasticsearch-1-vm systemd[1]: Started Cleanup of Temporary Directories.
Jan 25 15:48:29 elasticsearch-1-vm systemd[1]: Starting Elasticsearch...
Jan 25 15:48:29 elasticsearch-1-vm systemd[1]: Started Elasticsearch.
Jan 25 15:48:30 elasticsearch-1-vm kernel: [ 919.597729] kernel tried to execute NX-protected page - exploit attempt? (uid: 113)
Jan 25 15:48:30 elasticsearch-1-vm kernel: [ 919.605545] BUG: unable to handle kernel paging request at 00007f896d5467c0
Jan 25 15:48:30 elasticsearch-1-vm kernel: [ 919.612621] IP: 0x7f896d5467c0
Jan 25 15:48:30 elasticsearch-1-vm kernel: [ 919.615779] PGD 80000003050ee067
Jan 25 15:48:30 elasticsearch-1-vm kernel: [ 919.615780] P4D 80000003050ee067
Jan 25 15:48:30 elasticsearch-1-vm kernel: [ 919.619199] PUD 30508d067
Jan 25 15:48:30 elasticsearch-1-vm kernel: [ 919.622626] PMD 305162067
Jan 25 15:48:30 elasticsearch-1-vm kernel: [ 919.625438] PTE 80000003df15b867
Jan 25 15:48:30 elasticsearch-1-vm kernel: [ 919.628245]
Jan 25 15:48:30 elasticsearch-1-vm kernel: [ 919.633174] Oops: 0011 [#1] SMP PTI
The cluster health with 4 nodes is green, and we can't seem to figure out why this may be happening.
Any ideas on why this may be happening would be very helpful.
Here is our config located in /etc/default/elasticsearch:
https://gist.github.com/deppi/58826c38ea8414d301eb034e9a29cd54
Also here is our /etc/elasticsearch/elasticsearch.yml
https://gist.github.com/deppi/17b1f28e649ee528b0fe2ca93a2ff19c
The only thing I can think that might be causing this issue is discovery.zen.minimum_master_nodes: 2
When maybe it should be configured as
discovery.zen.minimum_master_nodes: 3
But we are uncertain this is the issue and don't want to risk further breaking out elasticsearch cluster

Answer 1

By experience, I know that shutting down the cluster using the elasticsearch command was not the best, we had issues with nodes not entirely down, and trying to take the master level. That's maybe why you can see 2 nodes, but your node is not part of it anymore.
What you should do, is shutting down the elasticsearch process on each nodes, unless you still index on the two nodes. In this case shut your cluster properly :
Stop the collect first everytime you need to stop elasticsearch, so logstash if you are using the stack
Then stop elasticsearch itself https://www.elastic.co/guide/en/elasticsearch/reference/master/stopping-elasticsearch.html
Start your first nodes as you let the protocol take place
Start elastic on the other nodes => see if all the nodes enter in
If not your config might be the problem, as I would use 1 master node and 3 slaves, and use another data path. When you need to shut down your cluster, stop the collect, stop the queuing, stop the storage (elastic), node by node

Answer 2

This seems to be an issue with a new kernel that has been deployed on GCP for the Ubuntu 16.04 LTS OS.
Problem Kernel:
uname -a
Linux elasticsearch-1-vm 4.13.0-1007-gcp #10-Ubuntu SMP Fri Jan 12 13:56:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Proper Kernel:
uname -a
Linux elasticsearch-1-vm 4.13.0-1006-gcp #9-Ubuntu SMP Mon Jan 8 21:13:15 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
To fix the issue with the GCP instances, I ran:
sudo apt remove 4.13.0-1007-gcp
sudo apt install 4.13.0-1006-gcp
exit
Then in google cloud console, restart the instance, then SSH back in then:
sudo service elasticsearch start