Unable to access data inside alert section of elastalert - elasticsearch

Question

I have been trying to set up elastalert monitoring on my ELK stack. For the beginning I want to set up a simple rule which will generate a notification if any disk on the file system has reached 80% usage. The rule seems to be working correctly but in the alert section I am not able to pass the data to python script. The uncommented command in the alert section gives following error
ERROR:root:Error while running alert command: Error formatting command: 'system.filesystem.mount_point' error.
Here is my rule file. Please excuse the formatting of the yaml.
name: Metricbeat high FS percentage
type: metric_aggregation
es_host: localhost
es_port: 9200
index: metricbeat-*
buffer_time:
minutes: 1
metric_agg_key: system.filesystem.used.pct
metric_agg_type: max
query_key: beat.name.keyword
doc_type: metricsets
bucket_interval:
minutes: 1
realert:
minutes: 2
sync_bucket_interval: true
#allow_buffer_time_overlap: true
#use_run_every_query_size: true
max_threshold: 0.8
filter:
- query:
query_string:
query: "system.filesystem.device_name: dev"
analyze_wildcard: true
- term:
metricset.name: filesystem
# (Required)
# The alert is use when a match is found
alert:
- debug
- command
command: ["/home/ubuntu/sendToSlack.py","beat-name","%(beat.name.keyword)s","used_pc","%(system.filesystem.used.pct_max)s","mount_point","%(system.filesystem.mount_point)s"]
# command: ["/home/ubuntu/sendToSlack.py","--beat-name","{match[beat.name.keyword]}","--mount_point","{match[system.filesystem.mount_point]}"]
# command: ["/home/ubuntu/sendToSlack.py","--beat-name","{match[beat][name]}","--mount_point","{match[system][filesystem][mount_point]}"]
#pipe_match_json: true
#- command:
# command: ["/home/ubuntu/sendToSlack.py","%(system.filesystem.used.bytes)s"]
Some observations:
On testing the rule file using the command python -m elastalert.test_rule rules/high_fs.yaml I get the output
Successfully loaded Metricbeat high FS percentage
Got 149161 hits from the last 1 day
Available terms in first hit:
tags
beat.hostname
beat.name
beat.version
type
#timestamp
system.filesystem.available
system.filesystem.files
system.filesystem.mount_point
system.filesystem.free_files
system.filesystem.free
system.filesystem.device_name
system.filesystem.used.bytes
system.filesystem.used.pct
system.filesystem.total
host
#version
metricset.rtt
metricset.name
metricset.module
I should be able to access any of the fields mentioned above. When I run this rule using python -m elastalert.elastalert --verbose --rule rules/high_fs.yaml a list is printed on the screen
#timestamp: 2017-10-18T17:15:00Z
beat.name.keyword: my_server_name
num_hits: 98
num_matches: 5
system.filesystem.used.pct_max: 0.823400020599
I am able to access all the key value pairs in this list. Anything thats outside the list fails with the formatting error. Been stuck over this for long. Any help is appreciated.

Answer 1

UPDATE: A reply for the same problem on elastalert's github repo says that certain query types do not contain the full field data.
While I am not sure if this is the correct way to achieve what I was looking for but I was able to get the the desired output using the rule type any and writing my own filters. Here is how one of my rules file looks currently.
name: High CPU percentage
type: any
es_host: localhost
es_port: 9200
index: consumer-*
query_key:
- beat.name
filter:
- range:
system.cpu.total_norm_pct:
from: 0.95
to: 10.0
realert:
minutes: 60
alert:
- command:
command: ["/home/ubuntu/slackScripts/sendCPUDetails.py","{match[beat][name]}","{match[system][cpu][total_norm_pct]}"]
new_style_string_format: true
Hope it helps someone.