Consul - Alert if drive is full - consul

Question

In the demo of consul, there are checks for disk utilization and memory utilization.
http://demo.consul.io/ui/#/ams2/nodes/ams2-server-1
How could you write a configuration to do what the demo shows? Warning at 10% and critical erros at 5% ?
Here is what I am trying
{
"check": {
"name": "Disk Util",
"script": "disk_util=$(df -k | grep '/dev/sda1' | awk '{print $5}' | sed 's/[^0-9]*//g' ) | if [ $disk_util > 90 ] ; then echo 'Disk /dev/sda above 90% full' && exit 1; elif [ $disk_util > 80 ] ; then echo 'Disk /dev/sda above 80%' && exit 3; else exit 0; fi",
"interval": "2m"
}
}
Here is the same script, but more human readable
disk_util=$(df -k | grep '/dev/sda1' | awk '{print $5}' | sed 's/[^0-9]*//g' ) |
if [ $disk_util > 90 ]
then echo 'Disk /dev/sda above 90% full' && exit 1
elif [ $disk_util > 80 ]
then echo 'Disk /dev/sda above 80%' && exit 3
else exit 0; fi
It seems like the check is working, but it doesn't print out any text. How can I verify this is working, and print output?

Answer 1

The output that you are seeing is produced by Nagios plugin check_disk (https://www.monitoring-plugins.org/doc/man/check_disk.html)
The "Output" field gets populated by stdout of the check. Your check runs cleanly and produces no output. So you see nothing.
To add some notes just add a "notes" field in the check definition as outlined in the documentation: https://www.consul.io/docs/agent/checks.html
Your check json file would look something like this:
{
"check": {
"name": "disks",
"notes": "Critical 5%, warning 10% free",
"script": "/path/to/check_disk -w 10% -c 5%",
"interval": "2m"
}
}

Answer 2

Exit code for your warning state should be 1, for critical, 2 or higher. (See "Check Scripts" at https://www.consul.io/docs/agent/checks.html), so you likely want to swap your exit lines.
Your 'OK' state (disk use < 80%) does not give any output, which is most likely why you see blank output.
I second the notion of using nagios plugins rather than rolling your own. Many OSes will have a nagios-plugins package(s) that are a yum/apt install away.

Answer 3

Health checks rely on the exit code of the check. To test if the health checks are being read by the Consul server you could write a script that always exits with a 1, and then you will see the health check as failed. Then replace it with a script that always returns 0 and you should see the health check as passed.
If you want to return text to the ui, add an output field to the json.

Answer 4

It seem consul analyse stdout only and not stderr. I have tested with redirect ( 2>&1 ) in service check file configuration. That seem work !
JSON config
{
"check": {
"name": "disks",
"notes": "Critical 5%, warning 10% free",
"script": "/path/to/check_disk -w 10% -c 5% 2>&1",
"interval": "2m"
}
}
Output result