How to query IP range in Elastic search? - elasticsearch

I want to query IP range from:172.16.0.0 to 172.31.0.0 in ELK
I try two query methods, but fail.
{
"query": {
"bool": {
"should": [
{
"regexp": {
"DstIP": "172.(3[0-1]|1[6-9]|2[0-9]).*"
}
}
],
"minimum_should_match": 1
}
}
}
{
"query": {
"range": {
"DstIP": {
"gte": "172.16.0.0",
"lte": "172.31.0.0"
}
}
}
}
How can query IP range in ELK?

For range queries to work correctly on IP values it is necessary to define the field data type as ip.
Below is the working example with mapping, sample docs, and search query.
Mapping:
{
"mappings": {
"properties": {
"dest": {
"type": "ip"
}
}
}
}
Index data:
Then I've taken a couple of sample documents like this:
{ "dest":"172.16.0.0"}
{ "dest":"172.31.0.0"}
{ "dest":"172.21.0.0"}
{ "dest":"172.1.0.0" }
{ "dest":"172.12.0.0"}
Search Query :
{
"query": {
"range": {
"dest": {
"gte": "172.16.0.0",
"lte": "172.31.0.0"
}
}
}
}
Search Result :
"hits": [
{
"_index": "foo4",
"_type": "_doc",
"_id": "1",
"_score": 1.0,
"_source": {
"dest": "172.16.0.0"
}
},
{
"_index": "foo4",
"_type": "_doc",
"_id": "2",
"_score": 1.0,
"_source": {
"dest": "172.31.0.0"
}
},
{
"_index": "foo4",
"_type": "_doc",
"_id": "3",
"_score": 1.0,
"_source": {
"dest": "172.21.0.0"
}
}
]

Related

Elastic search Query - How to pass list of queries

I have created an Index with 10000+ documents. Here is the sample from that:
{
"_index": "index_1",
"_type": "_doc",
"_id": "48a454f9-71d2-41a0-9e62-08c149366f05",
"_score": 13.977877,
"_source": {
"customer_id":10,
"customer_name": Mike,
"customer_phone": 1111111111,
"customer_address": "XYZ"
}
},
{
"_index": "index_1",
"_type": "_doc",
"_id": "48a454f9-71d2-41a0-9e62-08c149366f71",
"_score": 12.977861,
"_source": {
"customer_id":20,
"customer_name": Angie,
"customer_phone": 2222222222,
"customer_address": "ABC"
}
},
{
"_index": "index_1",
"_type": "_doc",
"_id": "48a454f9-71d2-41a0-9e62-08c149366f62",
"_score": 10.978777,
"_source": {
"customer_id":30,
"customer_name": John,
"customer_phone": 3333333333,
"customer_address": "PQR"
}
},
{
"_index": "index_1",
"_type": "_doc",
"_id": "48a454f9-71d2-41a0-9e62-08c149366f54",
"_score": 11.817877,
"_source": {
"customer_id":40,
"customer_name": Andy,
"customer_phone": 4444444444,
"customer_address": "MNO"
}
},
{
"_index": "index_1",
"_type": "_doc",
"_id": "48a454f9-71d2-41a0-9e62-08c149366f32",
"_score": 14.457877,
"_source": {
"customer_id": 50,
"customer_name": Nick,
"customer_phone": 5555555555,
"customer_address": "CDE"
}
},
{
"_index": "index_1",
"_type": "_doc",
"_id": "48a454f9-71d2-41a0-9e62-08c149366f21",
"_score": 16.487877,
"_source": {
"customer_id":60,
"customer_name": Atlas,
"customer_phone": 6666666666,
"customer_address": "DFE"
}
}
I want to pass multiple queries at once as list in json body and get the result also in list format:
For example: -> I want to pass below 3 queries in the search condition at the same time:
1) customer_id = 10, customer_name = Mike, customer_phone = 1111111111
2) customer_id = 40, customer_name = Andy, customer_phone = 4444444444
3) customer_id = 50, customer_name = Nick, customer_phone = 5555555555
Although, I can combine these 3 queries using 'AND' and 'OR' like below:
{
"query": {
"query_string": {
"query": "(customer_id: 10 AND customer_name: Mike AND customer_phone: 1111111111) OR (customer_id: 40 AND customer_name: Andy AND customer_phone: 4444444444) OR (customer_id: 50 AND customer_name: Nick AND customer_phone: 5555555555)"
}
}
}
Other than combining the queries as above, is there any other better way to achieve the same (like passing the queries as list).
You can combinate should and must query.
{
"query": {
"bool": {
"should": [
{
"bool": {
"must": [
{
"term": {
"customer_id": {
"value": 10
}
}
},
{
"match": {
"custumer_name": "Mike"
}
},
{
"term": {
"customer_phone": {
"value": 1111111111
}
}
}
]
}
},
{
"bool": {
"must": [
{
"term": {
"customer_id": {
"value": 50
}
}
},
{
"match": {
"custumer_name": "Nick"
}
},
{
"term": {
"customer_phone": {
"value": 5555555555
}
}
}
]
}
},
{
"bool": {
"must": [
{
"term": {
"customer_id": {
"value": 40
}
}
},
{
"match": {
"custumer_name": "Andy"
}
},
{
"term": {
"customer_phone": {
"value": 4444444444
}
}
}
]
}
}
]
}
}
}

elk's elastic search dsl case sensitive

I'm doing an Elasticsearch Query DSL query on ELK such as:
{
"query": {
"wildcard": {
"url.path": {
"value": "*download*",
"boost": 1,
"rewrite": "constant_score"
}
}
}
}
but it seems is case sensitive (so show only info with "download", not "Download" or "DOWNLOAD").
i.e. is case sensitive.
can I disable this? and search case insensitive?
Version used: 7.9.1
The below query will help you perform case-insensitive search as it will fetch results for *download, *Download and *DOWNLOAD. You may replace with your index and with the field you would like to perform this search.
Search Query
GET /<my-index>/_search
{
"query" : {
"bool" : {
"must" : {
"query_string" : {
"query" : "*download",
"fields": ["<field1>"]
}
}
}
}
}
If you wish to perform the same search on multiple fields, you can add the same in list.
Search on multiple fields
GET /<my-index>/_search
{
"query" : {
"bool" : {
"must" : {
"query_string" : {
"query" : "*download",
"fields": ["<field1>","<field2>","field3>"]
}
}
}
}
}
There is a case_insensitive parameter available for wildcard query, but it was introduced in Elasticsearch 7.10.0, so you need to upgrade if you are still on 7.9.1.
If you can upgrade to 7.10.0 or higher:
Ideally, in index mapping field should use wildcard type:
{
"mappings": {
"properties": {
"url.path": {
"type": "wildcard"
}
}
}
}
Then a wildcard query with case insensitivity enabled will find all the variants ("download", "DOWNLOAD", "download", etc)
{
"query": {
"wildcard": {
"url.path": {
"value": "*download*",
"boost": 1,
"rewrite": "constant_score",
"case_insensitive": true
}
}
}
}
If you must remain at 7.9.1:
Define your mapping in such a way that Elasticsearch treats the field contents as lowercase. The following will mimic wildcard type (it's a keyword, so only one token) indexed as lowercase.
{
"mappings": {
"properties": {
"url": {
"type": "text",
"analyzer": "lowercase-keyword"
}
}
},
"settings": {
"analysis": {
"analyzer": {
"lowercase-keyword": {
"type": "custom",
"tokenizer": "keyword",
"filter": "lowercase"
}
}
}
}
}
The query, without the case_insensitive parameter which is unsupported in this version:
{
"query": {
"wildcard": {
"url": {
"value": "*download*",
"boost": 1,
"rewrite": "constant_score"
}
}
}
}
Example results (note that searching for "*download*" and "*DoWnLoAd*" with both work in the same way):
{
"hits": {
"total": {
"value": 3,
"relation": "eq"
},
"max_score": 1.0,
"hits": {
"total": {
"value": 3,
"relation": "eq"
},
"max_score": 1.0,
"hits": [
{
"_index": "my-index",
"_type": "_doc",
"_id": "PtbQe3wByTvslqtrs7Cn",
"_score": 1.0,
"_source": {
"url": "http://example.com/download"
}
},
{
"_index": "my-index",
"_type": "_doc",
"_id": "P9bQe3wByTvslqtrvbDt",
"_score": 1.0,
"_source": {
"url": "http://example.com/Download"
}
},
{
"_index": "my-index",
"_type": "_doc",
"_id": "QNbQe3wByTvslqtrzbDw",
"_score": 1.0,
"_source": {
"url": "http://example.com/DOWNLOAD"
}
}
]
}
}
You can use case_insensitive parameter for wildcard query. This parameter was introduced in 7.10.0 version
Adding a working example with index data, mapping, search query, and search result
Index Mapping:
{
"mappings": {
"properties": {
"url": {
"properties": {
"path": {
"type": "wildcard"
}
}
}
}
}
}
Index Data:
{
"url":{
"path":"xx/download"
}
}
Search Query:
{
"query": {
"wildcard": {
"url.path": {
"value": "*Download*",
"boost": 1,
"rewrite": "constant_score",
"case_insensitive": false
}
}
}
}
Search Result:
No results will be there when you are searching for *Download* or *DOWNLOAD*
Update:
You can use the wildcard query with "case_insensitive": true parameter
Adding a sample index data, search query, and search result
Index Data:
{
"url": {
"path": "download"
}
}
{
"url": {
"path": "DOWNLOAD"
}
}
{
"url": {
"path": "Download"
}
}
Search Query:
{
"query": {
"wildcard": {
"url.path": {
"value": "*DOWNLOAD*",
"boost": 1,
"rewrite": "constant_score",
"case_insensitive": true
}
}
}
}
Search Result:
"hits": [
{
"_index": "67210888",
"_type": "_doc",
"_id": "1",
"_score": 1.0,
"_source": {
"url": {
"path": "download"
}
}
},
{
"_index": "67210888",
"_type": "_doc",
"_id": "2",
"_score": 1.0,
"_source": {
"url": {
"path": "Download"
}
}
},
{
"_index": "67210888",
"_type": "_doc",
"_id": "3",
"_score": 1.0,
"_source": {
"url": {
"path": "DOWNLOAD"
}
}
}
]

Elasticsearch range query with multiple condition

I have to fetch records from Elastic Search on the basis of date it is updated and created. I have these two fields updatedDate and createdDate and the condition should be:
To fetch records that has updatedDate within the range of past 3 years.
If updatedDate is null, fetch records that has createdDate within the range of past 3 years.
I have written the query in java for fetching the records on the basis of record createdDate:
.must(QueryBuilders.rangeQuery("createdDate").from(startDate,true).to(endDate,true));
startDate and endDate holds the date range.
I am new to Elastic Search, don't know how to implement the above condition.
Since you have not provided any index data, so adding a working example with sample index data, mapping, search query and search result that satisfies all the conditions required for your use case.
Index Mapping:
{
"mappings": {
"properties": {
"createdDate": {
"format": "yyyy-MM-dd'T'HH:mm:ss'Z'",
"type": "date"
},
"updatedDate": {
"format": "yyyy-MM-dd'T'HH:mm:ss'Z'",
"type": "date"
}
}
}
}
Index Data:
{
"createdDate": "2020-08-15T00:00:00Z"
}
{
"createdDate": "2019-08-15T00:00:00Z"
}
{
"createdDate": "2010-08-15T00:00:00Z"
}
{
"updatedDate": "2021-08-15T00:00:00Z",
"createdDate": "2002-08-15T00:00:00Z"
}
{
"updatedDate": "2018-08-15T00:00:00Z",
"createdDate": "2020-09-15T00:00:00Z"
}
{
"updatedDate": "2000-08-15T00:00:00Z",
"createdDate": "2020-09-15T00:00:00Z"
}
Search Query:
{
"query": {
"bool": {
"should": [
{
"bool": {
"must": [
{
"bool": {
"filter": {
"range": {
"createdDate": {
"gte": "now-3y",
"lte": "now"
}
}
},
"must_not": {
"exists": {
"field": "updatedDate"
}
}
}
}
]
}
},
{
"bool": {
"filter": {
"range": {
"updatedDate": {
"gte": "now-3y",
"lte": "now"
}
}
}
}
}
],
"minimum_should_match": 1
}
}
}
Search Result:
"hits": [
{
"_index": "64965551",
"_type": "_doc",
"_id": "1",
"_score": 0.0,
"_source": {
"createdDate": "2020-08-15T00:00:00Z"
}
},
{
"_index": "64965551",
"_type": "_doc",
"_id": "2",
"_score": 0.0,
"_source": {
"createdDate": "2019-08-15T00:00:00Z"
}
},
{
"_index": "64965551",
"_type": "_doc",
"_id": "5",
"_score": 0.0,
"_source": {
"updatedDate": "2018-08-15T00:00:00Z",
"createdDate": "2020-09-15T00:00:00Z"
}
}
]

Elastic search query for name / value pair columns pull

We have one document in elastic search with multiple sections of name/value pair and we want to fetch value's only based on name column value.
"envelopeData": {
"envelopeName": "Bills",
"details": {
"detail": [
{
"name": "UC_CORP",
"value": "76483"
},
{
"name": "UC_CYCLE",
"value": "V"
}
We are expecting only 76483 as result based on name equals to UC_CORP
If the field envelopeData.details.detail is nested type then you can perform a match query for the desired name on the nested path and can use inner_hits to get just the value.
Map the field envelopeData.details.detail as nested(if not nested):
PUT stackoverflow
{
"mappings": {
"_doc": {
"properties": {
"envelopeData.details.detail": {
"type": "nested"
}
}
}
}
}
then you can perform the following query to get value using inner_hits:
GET stackoverflow/_search
{
"_source": "false",
"query": {
"nested": {
"path": "envelopeData.details.detail",
"query": {
"match": {
"envelopeData.details.detail.name.keyword": "UC_CORP"
}
},
"inner_hits": {
"_source": "envelopeData.details.detail.value"
}
}
}
}
which outputs:
{
"_index": "stackoverflow",
"_type": "_doc",
"_id": "W5GUW2gB3GnGVyg-Sf4T",
"_score": 0.6931472,
"_source": {},
"inner_hits": {
"envelopeData.details.detail": {
"hits": {
"total": 1,
"max_score": 0.6931472,
"hits": [
{
"_index": "stackoverflow",
"_type": "_doc",
"_id": "W5GUW2gB3GnGVyg-Sf4T",
"_nested": {
"field": "envelopeData.details.detail",
"offset": 0
},
"_score": 0.6931472,
"_source": {
"value": "76483" -> Outputs value only
}
}
]
}
}
}
}

Elasticsearch search for Turkish characters

I have some documents that i am indexing with elasticsearch. But some of the documents are written with upper case and Tukish characters are changed. For example "kürşat" is written as "KURSAT".
I want to find this document by searching "kürşat". How can i do that?
Thanks
Take a look at the asciifolding token filter.
Here is a small example for you to try out in Sense:
Index:
DELETE test
PUT test
{
"settings": {
"analysis": {
"filter": {
"my_ascii_folding": {
"type": "asciifolding",
"preserve_original": true
}
},
"analyzer": {
"turkish_analyzer": {
"tokenizer": "standard",
"filter": [
"lowercase",
"my_ascii_folding"
]
}
}
}
},
"mappings": {
"test": {
"properties": {
"name": {
"type": "string",
"analyzer": "turkish_analyzer"
}
}
}
}
}
POST test/test/1
{
"name": "kürşat"
}
POST test/test/2
{
"name": "KURSAT"
}
Query:
GET test/_search
{
"query": {
"match": {
"name": "kursat"
}
}
}
Response:
"hits": {
"total": 2,
"max_score": 0.30685282,
"hits": [
{
"_index": "test",
"_type": "test",
"_id": "2",
"_score": 0.30685282,
"_source": {
"name": "KURSAT"
}
},
{
"_index": "test",
"_type": "test",
"_id": "1",
"_score": 0.30685282,
"_source": {
"name": "kürşat"
}
}
]
}
Query:
GET test/_search
{
"query": {
"match": {
"name": "kürşat"
}
}
}
Response:
"hits": {
"total": 2,
"max_score": 0.4339554,
"hits": [
{
"_index": "test",
"_type": "test",
"_id": "1",
"_score": 0.4339554,
"_source": {
"name": "kürşat"
}
},
{
"_index": "test",
"_type": "test",
"_id": "2",
"_score": 0.09001608,
"_source": {
"name": "KURSAT"
}
}
]
}
Now the 'preserve_original' flag will make sure that if a user types: 'kürşat', documents with that exact match will be ranked higher than documents that have 'kursat' (Notice the difference in scores for both query responses).
If you want the score to be equal, you can put the flag on false.
Hope I got your problem right!

Resources