Accurate query confusion on Elasticsearch - elasticsearch

Question

Here I simulated batch of mock data:
{
"took": 35,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 9000009,
"max_score": 1,
"hits": [
{
"_index": "boss-mock",
"_type": "soap-mock",
"_id": "AV-P15lDpN_SAQa2UP7B",
"_score": 1,
"_source": {
"operation_name": "ADD_IFC",
"hlrsn": "51",
"user_name": "boss2",
"business_type": "VoLTE",
"task_id": "a-ec0fe200-6219-46fa-8f9b-d23d3fc367a0#1509946767871",
"response_time": "2017-11-06T05:39:27.871Z",
"imsi": "460082279570892",
"msisdn": "8618882291205",
"content": """2017-11-06 05:39:27,871|User:boss2| id:a-ec0fe200-6219-46fa-8f9b-d23d3fc367a0#1509946767871 |{"HLRSN":"51","operationName":"ADD_IFC","ISDN":"8618882291205"}"""
}
},
{
"_index": "boss-mock",
"_type": "soap-mock",
"_id": "AV-P15lDpN_SAQa2UP7C",
"_score": 1,
"_source": {
"operation_name": "BAT_RMV_EPSDATA",
"hlrsn": "50",
"user_name": "boss3",
"business_type": "OVERHEAD",
"task_id": "a-6dbf64ee-81e9-4ef4-8b05-664a7fc3f47b#1509946767871",
"response_time": "2017-11-06T05:39:27.871Z",
"imsi": "460050840482507",
"msisdn": "8618178395664",
"content": """2017-11-06 05:39:27,871|User:boss3| id:a-6dbf64ee-81e9-4ef4-8b05-664a7fc3f47b#1509946767871 |{"HLRSN":"50","operationName":"BAT_RMV_EPSDATA","ISDN":"8618178395664"}"""
}
},
...
I want to query data according to a specific task_id :
GET /boss-mock/soap-mock/_search
{
"query": {
"match": {
"task_id": "a-ec0fe200-6219-46fa-8f9b-d23d3fc367a0#1509946767871"
}
}
}
response:
{
"took": 66,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 9000009,
"max_score": 68.65554,
"hits": [
{
"_index": "boss-mock",
"_type": "soap-mock",
"_id": "AV-P15lDpN_SAQa2UP7B",
"_score": 68.65554,
"_source": {
"operation_name": "ADD_IFC",
"hlrsn": "51",
"user_name": "boss2",
"business_type": "VoLTE",
"task_id": "a-ec0fe200-6219-46fa-8f9b-d23d3fc367a0#1509946767871",
"response_time": "2017-11-06T05:39:27.871Z",
"imsi": "460082279570892",
"msisdn": "8618882291205",
"content": """2017-11-06 05:39:27,871|User:boss2| id:a-ec0fe200-6219-46fa-8f9b-d23d3fc367a0#1509946767871 |{"HLRSN":"51","operationName":"ADD_IFC","ISDN":"8618882291205"}"""
}
},
{
"_index": "boss-mock",
"_type": "soap-mock",
"_id": "AV-P15lDpN_SAQa2UP7K",
"_score": 20.13632,
"_source": {
"operation_name": "ADD_TPLSUB",
"hlrsn": "53",
"user_name": "boss1",
"business_type": "OVERHEAD",
"task_id": "a-931b0935-a0d4-46fa-b403-7c1075a1d7a7#1509946767871",
"response_time": "2017-11-06T05:39:27.871Z",
"imsi": "",
"msisdn": "8618509192307",
"content": """2017-11-06 05:39:27,871|User:boss1| id:a-931b0935-a0d4-46fa-b403-7c1075a1d7a7#1509946767871 |{"HLRSN":"53","operationName":"ADD_TPLSUB","ISDN":"8618509192307"}"""
}
},
{
"_index": "boss-mock",
"_type": "soap-mock",
"_id": "AV-P06bepN_SAQa2S9uQ",
"_score": 17.619738,
"_source": {
"operation_name": "DEA_BOICEXHC",
"hlrsn": "52",
"user_name": "boss3",
"business_type": "VOICE",
"task_id": "a-cc771389-8712-46fa-8f9b-0e64e4fc38e6#1509946485051",
"response_time": "2017-11-06T05:34:45.051Z",
"imsi": "",
"msisdn": "8618914540349",
"content": """2017-11-06 05:34:45,051|User:boss3| id:a-cc771389-8712-46fa-8f9b-0e64e4fc38e6#1509946485051 |{"HLRSN":"52","operationName":"DEA_BOICEXHC","ISDN":"8618914540349"}"""
}
},
{
"_index": "boss-mock",
"_type": "soap-mock",
"_id": "AV-P15kQpN_SAQa2UP6w",
"_score": 12.451507,
"_source": {
"operation_name": "LST_STNSR",
"hlrsn": "51",
"user_name": "boss1",
"business_type": "",
"task_id": "a-30e82392-8817-48ed-8c3d-f4aee6e6c61d#1509946767871",
"response_time": "2017-11-06T05:39:27.871Z",
"imsi": "",
"msisdn": "8618871203019",
"content": """2017-11-06 05:39:27,871|User:boss1| id:a-30e82392-8817-48ed-8c3d-f4aee6e6c61d#1509946767871 |{"HLRSN":"51","operationName":"LST_STNSR","ISDN":"8618871203019"}"""
}
...
```
It seems that ES returned all the data,but the first piece is what I query.
Then I try to use `term` query:
```
GET /boss-mock/soap-mock/_search
{
"query": {
"term": {
"task_id": "a-ec0fe200-6219-46fa-8f9b-d23d3fc367a0#1509946767871"
}
}
}
But I get nothing:
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 0,
"max_score": null,
"hits": []
}
}
However,it works for other field ,which has 'shorter' data length,such as msisdn:
GET /boss-mock/soap-mock/_search
{
"query": {
"term": {
"msisdn": "8618882291205"
}
}
}
response:
{
"took": 35,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 9000009,
"max_score": 1,
"hits": [
{
"_index": "boss-mock",
"_type": "soap-mock",
"_id": "AV-P15lDpN_SAQa2UP7B",
"_score": 1,
"_source": {
"operation_name": "ADD_IFC",
"hlrsn": "51",
"user_name": "boss2",
"business_type": "VoLTE",
"task_id": "a-ec0fe200-6219-46fa-8f9b-d23d3fc367a0#1509946767871",
"response_time": "2017-11-06T05:39:27.871Z",
"imsi": "460082279570892",
"msisdn": "8618882291205",
"content": """2017-11-06 05:39:27,871|User:boss2| id:a-ec0fe200-6219-46fa-8f9b-d23d3fc367a0#1509946767871 |{"HLRSN":"51","operationName":"ADD_IFC","ISDN":"8618882291205"}"""
}
},
{
"_index": "boss-mock",
"_type": "soap-mock",
"_id": "AV-P15lDpN_SAQa2UP7C",
"_score": 1,
"_source": {
"operation_name": "BAT_RMV_EPSDATA",
"hlrsn": "50",
"user_name": "boss3",
"business_type": "OVERHEAD",
"task_id": "a-6dbf64ee-81e9-4ef4-8b05-664a7fc3f47b#1509946767871",
"response_time": "2017-11-06T05:39:27.871Z",
"imsi": "460050840482507",
"msisdn": "8618178395664",
"content": """2017-11-06 05:39:27,871|User:boss3| id:a-6dbf64ee-81e9-4ef4-8b05-664a7fc3f47b#1509946767871 |{"HLRSN":"50","operationName":"BAT_RMV_EPSDATA","ISDN":"8618178395664"}"""
}
},
{
"_index": "boss-mock",
"_type": "soap-mock",
"_id": "AV-P15lDpN_SAQa2UP7J",
"_score": 1,
"_source": {
"operation_name": "MOD_EPS_CONTEXT",
"hlrsn": "52",
"user_name": "boss2",
"business_type": "LTE",
"task_id": "a-b0bed660-3fca-4201-a90c-e4103f6289c5#1509946767871",
"response_time": "2017-11-06T05:39:27.871Z",
"imsi": "460039208697055",
"msisdn": "8618275883802",
"content": """2017-11-06 05:39:27,871|User:boss2| id:a-b0bed660-3fca-4201-a90c-e4103f6289c5#1509946767871 |{"HLRSN":"52","operationName":"MOD_EPS_CONTEXT","ISDN":"8618275883802"}"""
}
},
{
"_index": "boss-mock",
"_type": "soap-mock",
"_id": "AV-P15lDpN_SAQa2UP7L",
"_score": 1,
"_source": {
"operation_name": "DEA_BAIC",
"hlrsn": "53",
"user_name": "boss3",
"business_type": "VOICE",
"task_id": "a-c5cc2332-9d81-476c-ad0a-0809c23cfe49#1509946767871",
"response_time": "2017-11-06T05:39:27.871Z",
"imsi": "",
"msisdn": "",
"content": """2017-11-06 05:39:27,871|User:boss3| id:a-c5cc2332-9d81-476c-ad0a-0809c23cfe49#1509946767871 |{"HLRSN":"53","operationName":"DEA_BAIC","ISDN":"8618886204829"}"""
}
},
{
"_index": "boss-mock",
"_type": "soap-mock",
"_id": "AV-P15lDpN_SAQa2UP7O",
"_score": 1,
"_source": {
"operation_name": "LST_SIFC",
"hlrsn": "51",
"user_name": "boss3",
"business_type": "",
"task_id": "a-b0f2a526-8757-4b2c-9011-674cc714fedc#1509946767871",
"response_time": "2017-11-06T05:39:27.871Z",
"imsi": "",
"msisdn": "",
"content": """2017-11-06 05:39:27,871|User:boss3| id:a-b0f2a526-8757-4b2c-9011-674cc714fedc#1509946767871 |{"HLRSN":"51","operationName":"LST_SIFC","ISDN":"8618258093284"}"""
}
},
{
"_index": "boss-mock",
"_type": "soap-mock",
"_id": "AV-P15lDpN_SAQa2UP7R",
"_score": 1,
"_source": {
"operation_name": "LST_COLR",
"hlrsn": "52",
"user_name": "boss2",
"business_type": "",
"task_id": "a-348463b7-eb49-45e2-bffb-1068e706802b#1509946767872",
"response_time": "2017-11-06T05:39:27.872Z",
"imsi": "",
"msisdn": "8618557891401",
"content": """2017-11-06 05:39:27,872|User:boss2| id:a-348463b7-eb49-45e2-bffb-1068e706802b#1509946767872 |{"HLRSN":"52","operationName":"LST_COLR","ISDN":"8618557891401"}"""
}
},
{
"_index": "boss-mock",
"_type": "soap-mock",
"_id": "AV-P15lDpN_SAQa2UP7W",
"_score": 1,
"_source": {
"operation_name": "BAT_ADD_TPLSUB",
"hlrsn": "52",
"user_name": "boss2",
"business_type": "OVERHEAD",
"task_id": "a-db3748af-0359-40d3-b5fd-eb09cc53ba56#1509946767872",
"response_time": "2017-11-06T05:39:27.872Z",
"imsi": "460017353100210",
"msisdn": "8618219821848",
"content": """2017-11-06 05:39:27,872|User:boss2| id:a-db3748af-0359-40d3-b5fd-eb09cc53ba56#1509946767872 |{"HLRSN":"52","operationName":"BAT_ADD_TPLSUB","ISDN":"8618219821848"}"""
}
},
{
"_index": "boss-mock",
"_type": "soap-mock",
"_id": "AV-P15lDpN_SAQa2UP7d",
"_score": 1,
"_source": {
"operation_name": "ACT_BAOC",
"hlrsn": "51",
"user_name": "boss2",
"business_type": "VOICE",
"task_id": "a-80d105e7-138f-4c48-99df-e1b6ea404f43#1509946767872",
"response_time": "2017-11-06T05:39:27.872Z",
"imsi": "",
"msisdn": "",
"content": """2017-11-06 05:39:27,872|User:boss2| id:a-80d105e7-138f-4c48-99df-e1b6ea404f43#1509946767872 |{"HLRSN":"51","operationName":"ACT_BAOC","ISDN":"8618881023802"}"""
}
},
{
"_index": "boss-mock",
"_type": "soap-mock",
"_id": "AV-P15lDpN_SAQa2UP7f",
"_score": 1,
"_source": {
"operation_name": "SND_CANCELC",
"hlrsn": "53",
"user_name": "boss1",
"business_type": "LOCATION",
"task_id": "a-1a26292d-0f6d-416b-ab3b-47b0c888843f#1509946767872",
"response_time": "2017-11-06T05:39:27.872Z",
"imsi": "",
"msisdn": "8618571785343",
"content": """2017-11-06 05:39:27,872|User:boss1| id:a-1a26292d-0f6d-416b-ab3b-47b0c888843f#1509946767872 |{"HLRSN":"53","operationName":"SND_CANCELC","ISDN":"8618571785343"}"""
}
},
{
"_index": "boss-mock",
"_type": "soap-mock",
"_id": "AV-P15lDpN_SAQa2UP7g",
"_score": 1,
"_source": {
"operation_name": "MOD_MEDIAID",
"hlrsn": "53",
"user_name": "boss2",
"business_type": "VoLTE",
"task_id": "a-8d2b037b-d346-4b89-9ab7-8f828b1bb783#1509946767872",
"response_time": "2017-11-06T05:39:27.872Z",
"imsi": "",
"msisdn": "",
"content": """2017-11-06 05:39:27,872|User:boss2| id:a-8d2b037b-d346-4b89-9ab7-8f828b1bb783#1509946767872 |{"HLRSN":"53","operationName":"MOD_MEDIAID","ISDN":"8618458567583"}"""
}
}
]
}
}
So,what's going on here?Can't I just query on task_id?
By the way,I have background for SQL.
I need to query data like:
select * from table where task_id = ?
mapping:
```
{
"boss-mock": {
"mappings": {
"soap-mock": {
"properties": {
"business_type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"content": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"hlrsn": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"imsi": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"msisdn": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"operation_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"response_time": {
"type": "date"
},
"task_id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"user_name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}
```

Answer 1

It's happening because you're executing analyzed query against analyzed field. So let me briefly explain what I mean. Each field with type text is analyzed and stored as a set of tokens just for the sake of full text search functionality.
You can check how analyzer will process your data by sending
POST http://localhost:9200/_analyze HTTP/1.1
Content-type: application/json
{
"tokenizer": "standard",
"text": "a-ec0fe200-6219-46fa-8f9b-d23d3fc367a0#1509946767871"
}
Response will inform you that ES would index a list of following tokens
["a", "ec0fe200", "6219", "46fa", "8f9b", "d23d3fc367a0", "1509946767871"]
Same happens to your match which query is analyzed query so in fact you're querying
["a","ec0fe200","6219","46fa","8f9b","d23d3fc367a0","1509946767871"]
against
["a","ec0fe200","6219","46fa","8f9b","d23d3fc367a0","1509946767871"]
["a","931b0935","a0d4","46fa","b403","7c1075a1d7a7","1509946767871"]
["a","cc771389","8712","46fa","8f9b","0e64e4fc38e6","1509946485051"]
["a","30e82392","8817","48ed","8c3d","f4aee6e6c61d","1509946767871"]
What is more, match query's default operator is OR so you'll get a result if at least one token from your query matches with indexed ones ("46fa", "1509946767871" ...).
Then you tried term query which is not analyzed so the problem is quite opposite. You were trying to match
"a-ec0fe200-6219-46fa-8f9b-d23d3fc367a0#1509946767871"
as one string against the same lists of tokens. That's why you get empty results.
So the short answer for your problem is that if you want to have something similar to where from sql, you shouldn't use analyzed fields for these properties or maintain both analyzed and non-analyzed fields.
To solve this issue you should drop your index, define mapping statically like below, index your data again and then use terms query to match entire strings. Keyword data type would be the most relevant one here. You can read more here
PUT http://localhost:9200/boss_mock
Content-type: application/json
{
"mappings": {
"soap-mock": {
"properties": {
"task_id": {
"type": "keyword"
},
"msisdn": {
"type": "keyword"
}
//whatever else you need
}
}
}
}
Please note that you don't have to define mapping statically for all your properties (others will be added dynamically as texts).