Elasticsearch Query "must match" in log - elasticsearch

Question

I have the following in my log that i would like to use ElasticSearch query to find:
2014-07-02 20:52:39 INFO home.helloworld: LOGGER/LOG:ID1234 has successfully been received, {"uuid"="abc123"}
2014-07-02 20:52:39 INFO home.helloworld: LOGGER/LOG:ID1234 has successfully been transferred, {"uuid"="abc123"}
2014-07-02 20:52:39 INFO home.byebyeworld: LOGGER/LOG:ID1234 has successfully been processed, {"uuid"="abc123"}
2014-07-02 20:52:39 INFO home.byebyeworld: LOGGER/LOG:ID1234 has exited, {"uuid"="abc123"}
2014-07-02 20:53:00 INFO home.helloworld: LOGGER/LOG:ID1234 has successfully been received, {"uuid"="def123"}
2014-07-02 20:53:00 INFO home.helloworld: LOGGER/LOG:ID1234 has successfully been transferred, {"uuid"="def123"}
2014-07-02 20:53:00 INFO home.byebyeworld: LOGGER/LOG:ID1234 has successfully been processed, {"uuid"="def123"}
2014-07-02 20:53:00 INFO home.byebyeworld: LOGGER/LOG:ID1234 has exited, {"uuid"="def123"}
Since each of above line is represented as single "message" in elasticsearch, i have a hard time querying it using POST rest calls. I tried using "must match" like below to only get line 1 of my log but it is not consistent, sometimes it returns multiple hits instead of just 1 hit:
{
"query" : {
"constant_score" : {
"filter" : {
"bool" : {
"must" : [
{ "match_phrase_prefix" : {"message" : "home.helloworld:"}},
{ "match_phrase_prefix" : {"message" : "LOGGER/LOG:ID1234"}},
{ "match" : {"message" : "received, {\"uuid\"=\"abc123\"}"}}
]
}
}
}
}
}
am i doing anything wrong with above elasticsearch query? i thought "must" is equal to AND, and "match" is more of CONTAINS, "match_phrase_prefix" is STARTSWITH? can someone please show me how to properly query a log filled with above logs with different uuid number and only return the single hit? originally i thought i got the query down with above, it first returned just 1 hit but then it returned 2 then a lot more.. .which to me is not consistent. Thank you in advance!!

Answer 1

The problem is with the 3-rd clause of your bool query. Let me give you couple queries which will just work for you and I'll explain why they do the job.
First Query
curl -XGET http://localhost:9200/my_logs/_search -d '
{
"query" : {
"constant_score" : {
"filter" : {
"bool" : {
"must" : [
{ "match_phrase_prefix" : {"message" : "home.helloworld:"}},
{ "match_phrase_prefix" : {"message" : "LOGGER/LOG:ID1234"}},
{ "match" : {
"message" : {
"query": "received, {\"uuid\"=\"abc123\"",
"operator": "and"
}
}
}
]
}
}
}
}
}'
Explanation
Let's make sure we are on the same page about indexing. By default the indexer will pass your data through standard chain of analysis. I.e. splitting by whitespace, reducing special characters, making lower-casing, etc. So in the indices we will just have tokens with its positions.
Match query as full-text query will take your query text "received, {\"uuid\"=\"abc123\"" and will pass through analysis as well. By default this analysis just splits your text by whitespace, reduces special characters, makes lower-casing, etc. The result of this analysis would look similar to this (simplified): received, uuid, abc123.
What match query will do - it will combine those tokens against message field using default operator (which is or). So as a logical expression the very last clause (match-query) would look like this: message:received OR message:uuid OR message:abc123.
This is why first 4 log-entries will match. I was able to reproduce it.
Second Query
curl -XGET http://localhost:9200/my_logs/_search -d '
{
"query" : {
"constant_score" : {
"filter" : {
"bool" : {
"must" : [
{ "match_phrase_prefix" : {"message" : "home.helloworld:"}},
{ "match_phrase_prefix" : {"message" : "LOGGER/LOG:ID1234"}},
{ "match_phrase_prefix" : {"received, {\"uuid\"=\"abc123\""}}
]
}
}
}
}
}'
Explanation
This one a little bit simpler. Remember: our indexing process left tokens and their positions in the index.
What actually Match Phrase Prefix query is doing - it takes the input query (let's take "received, {\"uuid\"=\"abc123\"" as an example), makes exactly the same step of query text analysis. And tries to find tokens received, uuid, abc123 at neighboring positions in the index. Just in the same exact ordering: received -> uuid -> abc123 (almost).
Except the very last token, which in our case is abc123. To be precisely, it will make the wildcard for the last token. I.e. received -> uuid -> abc123*.
To be perfectionist I'd add that received -> uuid -> abc123 (i.e. without wildcard in the end) - is what actually Match Phrase query is doing. It also counts the positions in the index, i.e. tries to match the 'phrase', not just separate tokens in random positions.