Script to Monitor IPMI SEL in real time - bash

Question

I want to monitor IPMI System Event Log(SEL) in real time. What I want that is that whenever a event is generated in SEL , automatically a mail alert should be generated .
One way for me to achieve this is that I can write a script and schedule it in cron. The script will run 3 or 4 times a day so whenever a new event is generated , a mail alert will be send to me.
I want the monitoring to be active. Like whenever a event is generated , a mail should be sent to me instead of checking at regular intervals.
The SEL Log format is as follows:
server-001% sudo ipmitool sel list
b4 | 05/27/2009 | 13:38:32 | Fan #0x37 | Upper Critical going high
c8 | 05/27/2009 | 13:38:35 | Fan #0x37 | Upper Critical going high
dc | 08/15/2009 | 07:07:50 | Fan #0x37 | Upper Critical going high
So , for the above case whenever a new event is generated , automatically a mail alert should be send to me with the event.
How can I achieve this with a bash script . Any pointers will be highly appreciated.

Answer 1

I believe some vendors have special extensions in their firmware for exactly what you are describing (i.e. you just configure an e-mail address in the service processor), but I can't speak to each vendor's support. You'll have to look for your motherboard's documentation for that.
In terms of a standard mechanism, you are probably looking for IPMI PET (platform event trap) support. With PET, when certain SEL events are generated, it will generate a SNMP trap. The SNMP trap, once received by an SNMP daemon can do whatever you want, such as send an e-mail out.
A user of FreeIPMI wrote up his experiences in a doc and posted his scripts, which you can find here:
http://www.gnu.org/software/freeipmi/download.html
(Disclaimer: I maintain FreeIPMI so I know FreeIPMI better, unsure of support in other IPMI software.)
As an FYI, several IPMI SEL logging daemons (FreeIPMI's ipmiseld and ipmitool's ipmievtd are two I know) poll the SEL based on a configurable number of seconds and log the SEL information to syslog. A mail alert could also be configured in syslog to send out an e-mail when an event occurs. These daemons are still polling based instead of real-time, but the daemons will probably handle many IPMI corner cases that your cron script may not be aware of.

Answer 2

Monitoring of IPMI SEL events can be achieved using ipmievd tool . It is a part of ipmitool package.
# rpm -qf /usr/sbin/ipmievd
ipmitool-1.8.11-12.el6.x86_64
To send SEL events to syslog , execute the following command.
ipmievd sel daemon
Now , to simulate generation of SEL events , we will execute the following command.
ipmitool event 2
This will generate the following event
` Voltage Threshold - Lower Critical - Going Low`
To get the list of SEL events that can be generated are , try
# ipmitool event
usage: event <num>
Send generic test events
1 : Temperature - Upper Critical - Going High
2 : Voltage Threshold - Lower Critical - Going Low
3 : Memory - Correctable ECC
The event will be notified to /var/log/messages. Following message was generated in log file.
Oct 21 15:12:32 mgthost ipmievd: Voltage sensor - Lower Critical going low

Answer 3

Just in case it helps anyone else...
I created a shell script to record data in this format, and I parse it with php and use google's chart api to make a nice line graph.
2016-05-25 13:33:15, 20 degrees C, 23 degrees C
2016-05-25 13:53:06, 21.50 degrees C, 24 degrees C
2016-05-25 14:34:39, 19 degrees C, 22.50 degrees C
#!/bin/sh
DATE=`date '+%Y-%m-%d %H:%M:%S'`
temp0=$(ipmitool sdr type Temperature | grep "CPU0 Diode" | cut -f5 -d"|")
temp1=$(ipmitool sdr type Temperature | grep "CPU1 Diode" | cut -f5 -d"|")
echo "$DATE,$temp0,$temp1" >> /events/temps.dat
The problem I'm having now is getting the cron job to access the data properly, even though it's set in the root crontab.