Zabbix custom LLD to obtain passenger performance counters - passenger

Question

I have written a python script to parse the passenger-status output while I need to setup LLD for the number of process that passenger is running.
[root#ip-10-1-1-1 cronjob]# ./passenger_status.py discovery trapper
- passenger.threads.discovery {"data": [{"{#THREADNO}":"1"},{"{#THREADNO}":"2"},{"{#THREADNO}":"3"},{"{#THREADNO}":"4"},{"{#THREADNO}":"5"},{"{#THREADNO}":"6"},{"{#THREADNO}":"7"},{"{#THREADNO}":"8"}]}
Then I have tried to send the discover to zabbix server using zabbix trapper as:-
#!/bin/bash
export HOME=/usr/cronjob
tmpFile=`/bin/mktemp`
$HOME/passenger_status.py discovery trapper > ${tmpFile}
/usr/bin/zabbix_sender -vv -c /etc/zabbix/zabbix_agentd.conf -i ${tmpFile}
rm ${tmpFile}
And then I setup a discovery rule
When I run the command, it success:-
[root#ip-10-1-1-1 cronjob]# ./passenger_discovery.sh
zabbix_sender [12836]: DEBUG: answer [{
"response":"success",
"info":"processed: 1; failed: 0; total: 1; seconds spent: 0.010336"}]
info from server: "processed: 1; failed: 0; total: 1; seconds spent: 0.010336"
sent: 1; skipped: 0; total: 1
Then I assume everything is OK, and then I setup some testing key.
And then I tried to send the server some data for testing.
[root#ip-10-1-1-1 cronjob]# ./passenger_status.sh
zabbix_sender [13041]: DEBUG: answer [{
"response":"success",
"info":"processed: 0; failed: 250; total: 250; seconds spent: 0.002045"}]
info from server: "processed: 0; failed: 250; total: 250; seconds spent: 0.002045"
zabbix_sender [13042]: DEBUG: answer [{
"response":"success",
"info":"processed: 0; failed: 46; total: 46; seconds spent: 0.000431"}]
info from server: "processed: 0; failed: 46; total: 46; seconds spent: 0.000431"
sent: 296; skipped: 0; total: 296
It simply tell me that no key were being accepted.
For debug purpose, I dumped the server log below when I send the discovery trapper:-
3345:20170116:195020.919 trapper #8 [processing data]
3345:20170116:195020.919 trapper got '{
"request":"sender data",
"data":[
{
"host":"ec2-59-56-61-222.ap-northeast-1.compute.amazonaws.com",
"key":"passenger.threads.discovery",
"value":"{\"data\": [{\"{#THREADNO}\":\"1\"},{\"{#THREADNO}\":\"2\"},{\"{#THREADNO}\":\"3\"},{\"{#THREADNO}\":\"4\"},{\"{#THREADNO}\":\"5\"},{\"{#THREADNO}\":\"6\"},{\"{#THREADNO}\":\"7\"},{\"{#THREADNO}\":\"8\"}]}"}]}'
3345:20170116:195020.919 In recv_agenthistory()
3345:20170116:195020.919 In process_hist_data()
3345:20170116:195020.919 In process_mass_data()
3345:20170116:195020.919 In substitute_simple_macros() data:EMPTY
3345:20170116:195020.919 In lld_process_discovery_rule() itemid:30048
3345:20170116:195020.919 query [txnlev:0] [select hostid,key_,state,filter,error,lifetime from items where itemid=30048]
3345:20170116:195020.920 In substitute_simple_macros() data:'30'
3345:20170116:195020.920 In lld_rows_get()
3345:20170116:195020.920 lld_rows_get() f_macro:'{#THREADNO}' f_regexp:'# The no. of threads that the passenger generated'
3345:20170116:195020.920 In lld_check_record() jp_row:'{"{#THREADNO}":"1"}'
3345:20170116:195020.920 End of lld_check_record():FAIL
3345:20170116:195020.920 In lld_check_record() jp_row:'{"{#THREADNO}":"2"}'
3345:20170116:195020.920 End of lld_check_record():FAIL
3345:20170116:195020.920 In lld_check_record() jp_row:'{"{#THREADNO}":"3"}'
3345:20170116:195020.920 End of lld_check_record():FAIL
3345:20170116:195020.920 In lld_check_record() jp_row:'{"{#THREADNO}":"4"}'
3345:20170116:195020.920 End of lld_check_record():FAIL
3345:20170116:195020.920 In lld_check_record() jp_row:'{"{#THREADNO}":"5"}'
3345:20170116:195020.920 End of lld_check_record():FAIL
3345:20170116:195020.920 In lld_check_record() jp_row:'{"{#THREADNO}":"6"}'
3345:20170116:195020.920 End of lld_check_record():FAIL
3345:20170116:195020.920 In lld_check_record() jp_row:'{"{#THREADNO}":"7"}'
3345:20170116:195020.920 End of lld_check_record():FAIL
3345:20170116:195020.920 In lld_check_record() jp_row:'{"{#THREADNO}":"8"}'
3345:20170116:195020.920 End of lld_check_record():FAIL
3345:20170116:195020.920 End of lld_rows_get():SUCCEED
3345:20170116:195020.920 In lld_update_items()
3345:20170116:195020.920 query [txnlev:0] [select i.itemid,i.name,i.key_,i.type,i.value_type,i.data_type,i.delay,i.delay_flex,i.history,i.trends,i.status,i.trapper_hosts,i.units,i.multiplier,i.delta,i.formula,i.logtimefmt,i.valuemapid,i.params,i.ipmi_sensor,i.snmp_community,i.snmp_oid,i.port,i.snmpv3_securityname,i.snmpv3_securitylevel,i.snmpv3_authprotocol,i.snmpv3_authpassphrase,i.snmpv3_privprotocol,i.snmpv3_privpassphrase,i.authtype,i.username,i.password,i.publickey,i.privatekey,i.description,i.interfaceid,i.snmpv3_contextname from items i,item_discovery id where i.itemid=id.itemid and id.parent_itemid=30048]
3345:20170116:195020.921 In lld_items_get()
3345:20170116:195020.921 query [txnlev:0] [select id.itemid,id.key_,id.lastcheck,id.ts_delete,i.name,i.key_,i.type,i.value_type,i.data_type,i.delay,i.delay_flex,i.history,i.trends,i.trapper_hosts,i.units,i.multiplier,i.delta,i.formula,i.logtimefmt,i.valuemapid,i.params,i.ipmi_sensor,i.snmp_community,i.snmp_oid,i.port,i.snmpv3_securityname,i.snmpv3_securitylevel,i.snmpv3_authprotocol,i.snmpv3_authpassphrase,i.snmpv3_privprotocol,i.snmpv3_privpassphrase,i.authtype,i.username,i.password,i.publickey,i.privatekey,i.description,i.interfaceid,i.snmpv3_contextname from item_discovery id join items i on id.itemid=i.itemid where id.parent_itemid=30049]
3345:20170116:195020.922 End of lld_items_get()
3345:20170116:195020.922 In lld_items_validate()
3345:20170116:195020.922 End of lld_items_validate()
3345:20170116:195020.922 In lld_applications_make()
3345:20170116:195020.922 In lld_applications_get()
3345:20170116:195020.922 query [txnlev:0] [select applicationid from items_applications where itemid=30049]
3345:20170116:195020.922 End of lld_applications_get() values_num:0
3345:20170116:195020.922 End of lld_applications_make()
3345:20170116:195020.922 In lld_items_save()
3345:20170116:195020.922 End of lld_items_save()
3345:20170116:195020.922 In lld_items_get()
3345:20170116:195020.923 query [txnlev:0] [select id.itemid,id.key_,id.lastcheck,id.ts_delete,i.name,i.key_,i.type,i.value_type,i.data_type,i.delay,i.delay_flex,i.history,i.trends,i.trapper_hosts,i.units,i.multiplier,i.delta,i.formula,i.logtimefmt,i.valuemapid,i.params,i.ipmi_sensor,i.snmp_community,i.snmp_oid,i.port,i.snmpv3_securityname,i.snmpv3_securitylevel,i.snmpv3_authprotocol,i.snmpv3_authpassphrase,i.snmpv3_privprotocol,i.snmpv3_privpassphrase,i.authtype,i.username,i.password,i.publickey,i.privatekey,i.description,i.interfaceid,i.snmpv3_contextname from item_discovery id join items i on id.itemid=i.itemid where id.parent_itemid=30050]
3345:20170116:195020.923 End of lld_items_get()
3345:20170116:195020.923 In lld_items_validate()
3345:20170116:195020.923 End of lld_items_validate()
3345:20170116:195020.923 In lld_applications_make()
3345:20170116:195020.923 In lld_applications_get()
3345:20170116:195020.923 query [txnlev:0] [select applicationid from items_applications where itemid=30050]
3345:20170116:195020.923 End of lld_applications_get() values_num:0
3345:20170116:195020.923 End of lld_applications_make()
3345:20170116:195020.923 In lld_items_save()
3345:20170116:195020.923 End of lld_items_save()
3345:20170116:195020.923 In lld_items_get()
3345:20170116:195020.923 query [txnlev:0] [select id.itemid,id.key_,id.lastcheck,id.ts_delete,i.name,i.key_,i.type,i.value_type,i.data_type,i.delay,i.delay_flex,i.history,i.trends,i.trapper_hosts,i.units,i.multiplier,i.delta,i.formula,i.logtimefmt,i.valuemapid,i.params,i.ipmi_sensor,i.snmp_community,i.snmp_oid,i.port,i.snmpv3_securityname,i.snmpv3_securitylevel,i.snmpv3_authprotocol,i.snmpv3_authpassphrase,i.snmpv3_privprotocol,i.snmpv3_privpassphrase,i.authtype,i.username,i.password,i.publickey,i.privatekey,i.description,i.interfaceid,i.snmpv3_contextname from item_discovery id join items i on id.itemid=i.itemid where id.parent_itemid=30052]
3345:20170116:195020.924 End of lld_items_get()
3345:20170116:195020.924 In lld_items_validate()
3345:20170116:195020.924 End of lld_items_validate()
3345:20170116:195020.924 In lld_applications_make()
3345:20170116:195020.924 In lld_applications_get()
3345:20170116:195020.924 query [txnlev:0] [select applicationid from items_applications where itemid=30052]
3345:20170116:195020.924 End of lld_applications_get() values_num:0
3345:20170116:195020.924 End of lld_applications_make()
3345:20170116:195020.924 In lld_items_save()
3345:20170116:195020.924 End of lld_items_save()
3345:20170116:195020.924 End of lld_update_items()
3345:20170116:195020.924 In lld_update_triggers()
3345:20170116:195020.924 query [txnlev:0] [select distinct t.triggerid,t.description,t.expression,t.status,t.type,t.priority,t.comments,t.url from triggers t,functions f,items i,item_discovery id where t.triggerid=f.triggerid and f.itemid=i.itemid and i.itemid=id.itemid and id.parent_itemid=30048]
3345:20170116:195020.926 End of lld_update_triggers()
3345:20170116:195020.926 In lld_update_graphs()
3345:20170116:195020.926 query [txnlev:0] [select distinct g.graphid,g.name,g.width,g.height,g.yaxismin,g.yaxismax,g.show_work_period,g.show_triggers,g.graphtype,g.show_legend,g.show_3d,g.percent_left,g.percent_right,g.ymin_type,g.ymin_itemid,g.ymax_type,g.ymax_itemid from graphs g,graphs_items gi,items i,item_discovery id where g.graphid=gi.graphid and gi.itemid=i.itemid and i.itemid=id.itemid and id.parent_itemid=30048]
3345:20170116:195020.928 End of lld_update_graphs()
3345:20170116:195020.928 In lld_update_hosts()
3345:20170116:195020.928 query [txnlev:0] [select h.proxy_hostid,h.ipmi_authtype,h.ipmi_privilege,h.ipmi_username,h.ipmi_password from hosts h,items i where h.hostid=i.hostid and i.itemid=30048]
3345:20170116:195020.928 query [txnlev:0] [select hi.interfaceid,hi.type,hi.main,hi.useip,hi.ip,hi.dns,hi.port from interface hi,items i where hi.hostid=i.hostid and i.itemid=30048]
3345:20170116:195020.929 In lld_hostmacros_get()
3345:20170116:195020.929 query [txnlev:0] [select hm.macro,hm.value from hostmacro hm,items i where hm.hostid=i.hostid and i.itemid=30048]
3345:20170116:195020.929 End of lld_hostmacros_get()
3345:20170116:195020.929 query [txnlev:0] [select h.hostid,h.host,h.name,h.status,hi.inventory_mode from hosts h,host_discovery hd left join host_inventory hi on hd.hostid=hi.hostid where h.hostid=hd.hostid and hd.parent_itemid=30048]
3345:20170116:195020.930 End of lld_update_hosts()
3345:20170116:195020.930 End of lld_process_discovery_rule()
3345:20170116:195020.930 End of process_mass_data()
3345:20170116:195020.930 End of process_hist_data():SUCCEED
3345:20170116:195020.930 In zbx_send_response()
3345:20170116:195020.930 zbx_send_response() '{
"response":"success",
"info":"processed: 1; failed: 0; total: 1; seconds spent: 0.010795"}'
3345:20170116:195020.930 End of zbx_send_response():SUCCEED
3345:20170116:195020.930 End of recv_agenthistory()
3345:20170116:195020.930 trapper #8 [processed data in 0.011424 sec, waiting for connection]
I tells me failed but didn't say any reason? I also dumped the server log during the trapper data send below:-
3338:20170116:195647.884 trapper #1 [processing data]
3338:20170116:195647.885 trapper got '{
"request":"sender data",
"data":[
{
"host":"ec2-59-56-61-222.ap-northeast-1.compute.amazonaws.com",
"key":"passenger.threads[8,total_requests_begun]",
"value":"124"},
{
"host":"ec2-59-56-61-222.ap-northeast-1.compute.amazonaws.com",
"key":"passenger.threads[7,total_requests_begun]",
"value":"124"},
{
"host":"ec2-59-56-61-222.ap-northeast-1.compute.amazonaws.com",
"key":"passenger.threads[6,total_requests_begun]",
"value":"124"},
{
"host":"ec2-59-56-61-222.ap-northeast-1.compute.amazonaws.com",
"key":"passenger.threads[5,total_requests_begun]",
"value":"124"},
{
"host":"ec2-59-56-61-222.ap-northeast-1.compute.amazonaws.com",
"key":"passenger.threads[4,total_requests_begun]",
"value":"124"},
{
"host":"ec2-59-56-61-222.ap-northeast-1.compute.amazonaws.com",
"key":"passenger.threads[3,total_requests_begun]",
"value":"124"},
{
"host":"ec2-59-56-61-222.ap-northeast-1.compute.amazonaws.com",
"key":"passenger.threads[2,total_requests_begun]",
"value":"124"}]}'
3338:20170116:195647.885 In recv_agenthistory()
3338:20170116:195647.885 In process_hist_data()
3338:20170116:195647.887 In process_mass_data()
3338:20170116:195647.887 End of process_mass_data()
3338:20170116:195647.887 End of process_hist_data():SUCCEED
3338:20170116:195647.887 In zbx_send_response()
3338:20170116:195647.887 zbx_send_response() '{
"response":"success",
"info":"processed: 0; failed: 250; total: 250; seconds spent: 0.002063"}'
3338:20170116:195647.888 End of zbx_send_response():SUCCEED
3338:20170116:195647.888 End of recv_agenthistory()
3338:20170116:195647.888 trapper #1 [processed data in 0.003354 sec, waiting for connection]
3346:20170116:195647.889 trapper #9 [processing data]
3346:20170116:195647.889 trapper got '{
"request":"sender data",
"data":[
{
"host":"ec2-59-56-61-222.ap-northeast-1.compute.amazonaws.com",
"key":"passenger.threads[1,total_requests_begun]",
"value":"124"}]}'
3346:20170116:195647.889 In recv_agenthistory()
3346:20170116:195647.889 In process_hist_data()
3346:20170116:195647.890 In process_mass_data()
3346:20170116:195647.890 End of process_mass_data()
3346:20170116:195647.890 End of process_hist_data():SUCCEED
3346:20170116:195647.890 In zbx_send_response()
3346:20170116:195647.890 zbx_send_response() '{
"response":"success",
"info":"processed: 0; failed: 46; total: 46; seconds spent: 0.000419"}'
3346:20170116:195647.890 End of zbx_send_response():SUCCEED
3346:20170116:195647.890 End of recv_agenthistory()
3346:20170116:195647.890 trapper #9 [processed data in 0.000970 sec, waiting for connection]
Then I read from the internet that discovery won't work with sender? Is that true? But anyway, I also performed some changes which is to include the discovery inside the UserParam to be monitored with zabbix_agent but it didn't work as well.
Here comes a list of testing item prototypes:-
But I have saw from the internet that there are a lot of successful case with custom LLD, what have I done wrong? The zabbix version that I am using is 2.2.8, would that be a possible cause?
#Richlv is correct, finally I removed the macro and regex and it did work now.
and BTW I have make some other changes, bcoz the way how zabbix works.... coz some of the reading for some thread result in None.... (bcoz I parse using python) and zabbix got this and rejected the same key for all other thread even though they are valid no..... I have modified the python script so that it will return a 0 in case the setting is None in python. Now it works. Thx very much for everyone's help. Although I am not fully understand why, while the key is filter and regex got to be empty.