Using a Kibana view query from application - elasticsearch

I used the following filter and then searched for query string using Lucene to get the view that I was looking for.
{
"query": {
"match": {
"eventSource": {
"query": "ec2.amazonaws.com",
"type": "phrase"
}
}
}
}
I do not want to return event names those start with the word describe or get. Rest of the event names from ec2 event source should be returned.
!(eventName.keyword: Describe* OR eventName.keyword:
Get* )
The question is how to combine these 2 search requests into one?
I need to use that query from my application.
Update:
The Inspect menu of Kibana Discover tab generates this query. I am just trying to rewrite query_string part with usual match or match_phrase using boolean OR clause.
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "!(eventName.keyword: Describe* OR eventName.keyword: Get* )",
"analyze_wildcard": true
}
},
{
"match_phrase": {
"eventSource": {
"query": "ec2.amazonaws.com"
}
}
},
{
"range": {
"#timestamp": {
"format": "strict_date_optional_time",
"gte": "2020-07-09T08:39:15.947Z",
"lte": "2020-07-24T08:39:15.947Z"
}
}
}
],
"filter": [],
"should": [],
"must_not": []
}
}

You can easily use the boolean query's must_not clause to exclude the documents which you don't want in your search result and you can add as many as must_not as you want, it's fairly easy to do and can be done in a single query.
Please refer the example in the same link to get more info. Created sample in my local to show your the correct query, Please note instead of wildcard I am using the prefix query which is better and server your use-case.
Create index mapping
{
"mappings": {
"properties": {
"eventName": {
"type": "keyword"
}
}
}
}
Index sample doc
{
"eventName" : "Describe the events"
}
{
"eventName" : "the Describe events"
}
{
"eventName" : "Get the event"
}
{
"eventName" : "event Get"
}
Now search query to get only 2 and 3rd doc according to your req
{
"query": {
"bool": {
"must_not": [
{
"prefix": {
"eventName": "Desc"
}
},
{
"prefix": {
"eventName": "Get"
}
}
]
}
}
}
Search result
"hits": [
{
"_index": "ngramkey",
"_type": "_doc",
"_id": "2",
"_score": 0.0,
"_source": {
"eventName": "the Describe events"
}
},
{
"_index": "ngramkey",
"_type": "_doc",
"_id": "4",
"_score": 0.0,
"_source": {
"eventName": "event Get"
}
}
]

As suggested by the user "Opster Elasticsearch Ninja", I have merged must not boolean query like this...
{
"query": {
"bool": {
"must": [
{
"bool": {
"must_not": [
{
"prefix": {
"eventName.keyword": "Desc"
}
},
{
"prefix": {
"eventName.keyword": "Get"
}
}
]
}
},
{
"match_phrase": {
"eventSource": {
"query": "ec2.amazonaws.com"
}
}
},
{
"range": {
"#timestamp": {
"format": "strict_date_optional_time",
"gte": "2020-07-09T08:39:15.947Z",
"lte": "2020-07-24T08:39:15.947Z"
}
}
}
],
"filter": [],
"should": [],
"must_not": []
}
}
}

Related

Elastic Search Query on String Array Field

I'm working on Elastic Search and facing an issue regarding Array field. I've index named test-index with following mapping.
{
"test-index": {
"mappings": {
"properties": {
"courses": {
"type": "keyword"
}
}
}
}
}
My elastic search documents looks like this.
"hits": [
{
"_index": "test-index",
"_id": "1ac:0000000000_1",
"_score": 1,
"_source": {
"courses": [
"Course-1A",
"Course-1B",
"Course-1C",
"Course-1D",
"Course-1E",
"Course-1F"
]
}
},
{
"_index": "test-index",
"_id": "1ac:0000000000_2",
"_score": 1,
"_source": {
"courses": [
"Course-2A",
"Course-2B",
"Course-2C",
"Course-1A"
]
}
}
]
The document _id is my student ID. I want to get results with the maximum/highest relevance at the top and lowest on the bottom.
e.g
If I'm searching for courses ["Course-2A","Course-2B","Course-1C"] then user 1ac:0000000000_2 should appear at the top and user 1ac:0000000000_1 at the bottom.
I've tried following queries.
GET test-index/_search
{
"query": {
"bool": {
"must": [
{
"terms": {
"courses": [
"Course-1A",
"Course-2A",
"Course-2B"
]
}
}
]
}
}
}
User 1ac:0000000000_1 at the top and other at the bottom.
GET test-index/_search
{
"query": {
"bool": {
"should": [
{
"term": {
"courses": "Course-1A",
}
},
{
"term": {
"courses": "Course-2A",
}
},
{
"term": {
"courses": "Course-2B",
}
}
],
"minimum_should_match": "70%"
}
}
}
This gives me some desired results but not sure for larger dataset.

How to search on multiple fields of array in elasticsearch

I have a index in elastic search called professor
If for cross field i need "AND" condition
for same field array i need to OR condition
I need to search subject which is Physics or Accounting this is array of fields(OR) statement
AND
I need to search type is Permanent or GUEST condition this is array of fields(OR) statement
AND
I need to search Location is NY(&) condition
test = [{'id':1,'name': 'A','subject': ['Maths','Accounting'],'type':'Contract', 'Location':'NY'},
{ 'id':2,'name': 'AB','subject': ['Physics','Engineering'],'type':'Permanent','Location':'NY'},
{'id':3,'name': 'ABC','subject': ['Maths','Engineering'],'type':'Permanent','Location':'NY'},
{'id':4,'name':'ABCD','subject': ['Physics','Engineering'],'type':['Contract','Guest'],'Location':'NY'}]
Query is below,3rd one got it, How to add 1 and 2
content_search = es.search(index="professor", body={
"query": {
"bool": {
"must": {
"match_all": {}
},
"filter": [
{
"term": {
"Location.keyword": "NY"
}
}
]
}
}
})
content_search ['hits']['hits']
Expected out is id [{ 'id':2,'name': 'AB','subject': ['Physics','Engineering'],'type':'Permanent','Location':'NY'},{'id':4,'name':'ABCD','subject': ['Physics','Engineering'],'type':['Contract','Guest'],'Location':'NY'}]
The filter clause (query) must appear in matching documents. However
unlike must the score of the query will be ignored. Filter clauses are
executed in filter context, meaning that scoring is ignored and
clauses are considered for caching.
Please go through this Elasticsearch documentation on bool queries, to get a detailed understanding about it.
Adding a working example with index data(same as that in question), search query, and search result
Search Query:
{
"query": {
"bool": {
"must": {
"match": {
"Location.keyword": "NY"
}
},
"filter": [
{
"bool": {
"should": [
{
"match": {
"subject.keyword": "Accounting"
}
},
{
"match": {
"subject.keyword": "Physics"
}
}
]
}
},
{
"bool": {
"should": [
{
"match": {
"type.keyword": "Permanent"
}
},
{
"match": {
"type.keyword": "Guest"
}
}
]
}
}
]
}
}
}
Search Result:
"hits": [
{
"_index": "stof_64370980",
"_type": "_doc",
"_id": "2",
"_score": 0.10536051,
"_source": {
"id": 2,
"name": "AB",
"subject": [
"Physics",
"Engineering"
],
"type": "Permanent",
"Location": "NY"
}
},
{
"_index": "stof_64370980",
"_type": "_doc",
"_id": "4",
"_score": 0.10536051,
"_source": {
"id": 4,
"name": "ABCD",
"subject": [
"Physics",
"Engineering"
],
"type": [
"Contract",
"Guest"
],
"Location": "NY"
}
}
]
Another Search Query:
You can even use terms query that returns documents that contain
one or more exact terms in a provided field.The terms query is the
same as the term query, except you can search for multiple values.
{
"query": {
"bool": {
"must": [
{
"terms": {
"subject.keyword": [
"Physics",
"Accounting"
]
}
},
{
"terms": {
"type.keyword": [
"Guest",
"Permanent"
]
}
},
{
"match": {
"Location.keyword": "NY"
}
}
]
}
}
}
Update 1:
{
"query": {
"bool": {
"must": [
{
"terms": {
"subject.keyword": [
"Physics",
"Accounting"
]
}
},
{
"terms": {
"type.keyword": [
"Guest",
"Permanent"
]
}
},
{
"match": {
"Location.keyword": "NY"
}
},
{
"query_string": {
"query": "ABCD"
}
}
]
}
}
}

How to search array of fields in elasticsearch

I have a index in elastic search called professor
If for cross field i need "AND" condition
for same field array i need to OR condition
I need to search subject which is Physics or Accounting this is array of fields(OR) statement
I need to search type is Permanent(&) condition
I need to search Location is NY(&) condition
There is chance that {'type':['Contract','Guest']} type also coming as list
test = [{'id':1,'name': 'A','subject': ['Maths','Accounting'],'type':'Contract', 'Location':'NY'},
{ 'id':2,'name': 'AB','subject': ['Physics','Engineering'],'type':'Permanent','Location':'NY'},
{'id':3,'name': 'ABC','subject': ['Maths','Engineering'],'type':'Permanent','Location':'NY'}]
Query is below,3rd one got it, How to add 1 and 2
content_search = es.search(index="professor", body={
"query": {
"bool": {
"must": {
"match_all": {}
},
"filter": [
{
"term": {
"Location.keyword": "NY"
}
}
]
}
}
})
content_search ['hits']['hits']
Expected out is id [{ 'id':2,'name': 'AB','subject': ['Physics','Engineering'],'type':'Permanent','Location':'NY'}]
You need to use the bool query, to wrap all your conditions
Adding a working example with index data(same as that in question), search query, and search result
Search Query:
{
"query": {
"bool": {
"must": [
{
"match": {
"type.keyword": "Permanent"
}
},
{
"match": {
"Location.keyword": "NY"
}
}
],
"should": [
{
"match": {
"subject.keyword": "Accounting"
}
},
{
"match": {
"subject.keyword": "Physics"
}
}
],
"minimum_should_match": 1,
"boost": 1.0
}
}
}
Search Result:
"hits": [
{
"_index": "stof_64370980",
"_type": "_doc",
"_id": "2",
"_score": 1.8365774,
"_source": {
"id": 2,
"name": "AB",
"subject": [
"Physics",
"Engineering"
],
"type": "Permanent",
"Location": "NY"
}
}
]

How to use filter in match_all in elastic search

Query is below
{
"from" : 0,
"size" : 100,
"query": {
"match_all": {}
}
}
I need to filter from the match_all if name is test
i tried with
{
"from" : 0,
"size" : 100,
"query": {
"match_all": {}
},
"filter": [ "term": { "name": "test" }}]
}
I got error 'Unknown key for a START_ARRAY in [filter].')
You will need to wrap your query in a bool query , try out this search query:
{
"from":0,
"size":10,
"query": {
"bool": {
"must": {
"match_all": {}
},
"filter": [
{
"term": {
"grocery_name": "elastic"
}
}
]
}
}
}
Update 1:
According to the comment mentioned by #Nons
Search Query:
Terms query return documents that contain an exact term in a provided
field.
{
"from":0,
"size":10,
"query": {
"bool": {
"must": {
"match_all": {}
},
"filter": [
{
"term": {
"parentName.keyword": "Developer" <-- note this
}
}
]
}
}
}
Search Result:
"hits": [
{
"_index": "stof_64275684",
"_type": "_doc",
"_id": "1",
"_score": 1.0,
"_source": {
"id": "1",
"name": "A",
"parentName": "Developer",
"Data": [
{
"id": "455",
"name": "Google",
"lastUpdatedDate": "2020-09-10",
"parent_id": "1"
}
],
"Function": [
{
"id": "1",
"name": "Major"
}
]
}
}
]
You can even use a match query where the provided text is analyzed
before matching.
{
"from": 0,
"size": 10,
"query": {
"bool": {
"must": {
"match": {
"parentName": "developer"
}
}
}
}
}
I would recommend to use the Chrome ElasticSearch Head plugin. It allows to test and run searches against Elastic very easily (functionality is similar to MySql Workbech).
Please find example of usage of plugin below (combination of condition and aggregation).

Empty inner_hits in compound Elasticsearch filter

I'm seeing what appears to be aberrant behavior in inner_hits results within nested boolean queries.
Test data (abbreviated for brevity):
# MAPPING
PUT unit_testing
{
"mappings": {
"document": {
"properties": {
"display_name": {"type": "text"},
"metadata": {
"properties": {
"NAME": {"type": "text"}
}
}
}
},
"paragraph": {
"_parent": {"type": "document"},
"_routing": {"required": true},
"properties": {
"checksum": {"type": "text"},
"sentences": {
"type": "nested",
"properties": {
"text": {"type": "text"}
}
}
}
}
}
}
# DOCUMENT X 2 (d0, d1)
PUT unit_testing/document/doc_id_d0
{
"display_name": "Test Document d0",
"paragraphs": [
"para_id_d0p0",
"para_id_d0p1"
],
"metadata": {"NAME": "Test Document d0 Metadata"}
}
# PARAGRAPH X 2 (d0p0, d1p0)
PUT unit_testing/paragraph/para_id_d0p0?parent=doc_id_d0
{
"checksum": "para_checksum_d0p0",
"sentences": [
{"text": "Test sentence d0p0s0"},
{"text": "Test sentence d0p0s1 ODD"},
{"text": "Test sentence d0p0s2 EVEN"},
{"text": "Test sentence d0p0s3 ODD"},
{"text": "Test sentence d0p0s4 EVEN"}
]
}
This initial query behaves as I would expect (I'm aware that the metadata filter isn't actually necessary in this example case):
GET unit_testing/paragraph/_search
{
"_source": "false",
"query": {
"bool": {
"must": [
{
"has_parent": {
"query": {
"match_phrase": {
"metadata.NAME": "Test Document d0 Metadata"
}
},
"type": "document"
}
},
{
"nested": {
"inner_hits": {},
"path": "sentences",
"query": {
"match": {
"sentences.text": "d0p0s0"
}
}
}
}
]
}
}
}
It yields an inner_hits object containing the one sentence that matched the predicate (some fields removed for clarity):
{
"hits": {
"hits": [
{
"_source": {},
"inner_hits": {
"sentences": {
"hits": {
"hits": [
{
"_source": {
"text": "Test sentence d0p0s0"
}
}
]
}
}
}
}
]
}
}
The following query is an attempt to embed the query above within a parent "should" clause, to create a logical OR between the initial query, and an additional query that matches a single sentence:
GET unit_testing/paragraph/_search
{
"_source": "false",
"query": {
"bool": {
"should": [
{
"bool": {
"must": [
{
"has_parent": {
"query": {
"match_phrase": {
"metadata.NAME": "Test Document d0 Metadata"
}
},
"type": "document"
}
},
{
"nested": {
"inner_hits": {},
"path": "sentences",
"query": {
"match": {
"sentences.text": "d0p0s0"
}
}
}
}
]
}
},
{
"nested": {
"inner_hits": {},
"path": "sentences",
"query": {
"match": {
"sentences.text": "d1p0s0"
}
}
}
}
]
}
}
}
While the "d1" query outputs the result one would expect, with an inner_hits object containing the matching sentence, the original "d0" query now yields an empty inner_hits object:
{
"hits": {
"hits": [
{
"_source": {},
"inner_hits": {
"sentences": {
"hits": {
"total": 0,
"hits": []
}
}
}
},
{
"_source": {},
"inner_hits": {
"sentences": {
"hits": {
"hits": [
{
"_source": {
"text": "Test sentence d1p0s0"
}
}
]
}
}
}
}
]
}
}
Although I'm using the elasticsearch_dsl Python library to build and combine these queries, and I'm something of a novice with respect to the Query DSL, the query format looks solid to me.
What am I missing?
I think what is missing is the name parameter for inner_hits - you have two inner_hits clauses at two different queries that would end up with the same name. Try giving the inner_hits a name parameter (0).
0 - https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-inner-hits.html#_options

Resources