how to make a query on a field I have not defined a mapping for - elasticsearch

I have a field current_country that I am adding to brands, and which has not been defined in my elasticsearch mapping.
I would like to do a filtered query on this, since it is not defined I suppose it is not analyzed and a term query should work.
This is the query I am doing
{
"index": "products",
"type": "brand",
"body": {
"from": 0,
"size": 100,
"sort": [
{
"n_name": "asc"
}
],
"query": {
"filtered": {
"query": {
"function_score": {
"filter": {
"bool": {
"must": [
{
"term": {
"current_country": "DK"
}
}
]
}
}
}
}
}
}
}
}
which returns no documents from the index.
I run the following query to check if current country exists
{
"index": "products",
"type": "brand",
"body": {
"from": 0,
"size": 100,
"sort": [
{
"n_name": "asc"
}
],
"query": {
"filtered": {
"query": {
"function_score": {
"filter": {
"bool": {
"must": [
{
"exists": {
"field": "current_country"
}
}
]
}
}
}
}
}
}
}
}
which returns a total of 693 documents.
here is an example document from the index, returned when I ran the query above.
{
"_index": "products",
"_type": "brand",
"_id": "195da951241478LuxoLivingbrand",
"_score": null,
"_source": {
"categories": [
"Bordlamper og designer bordlamper der giver liv og lys"
],
"image": "http://www.fotoagent.dk/single_picture/11385/138/mega/and_tradition_flowerpot_bordlampe_lilla.jpg",
"top_price": 1695,
"low_price": 1695,
"n_name": "&Tradition",
"name": "&Tradition",
"current_country": "DK",
"current_currency": "DKK"
}
}
How can I query against current_country (preferably a filtered query).

If you do not define any mapping for a field, elasticsearch tries to detect the field as string/date/numeric. If it detects the field as string then it will use the default analyzer (standard analyzer) to analyze your input. Since standard analyzer uses lowercase token filter your input string is indexed as "dk". As term filters does not analyze the input, "DK" won't match "dk".
It can be solved by various means.
(hack) You can lowercase your input filter term. this won't work for phrases.
(better) define a mapping for your input. You can dynamically change mapping/ add new mapping easily

Related

Using a Kibana view query from application

I used the following filter and then searched for query string using Lucene to get the view that I was looking for.
{
"query": {
"match": {
"eventSource": {
"query": "ec2.amazonaws.com",
"type": "phrase"
}
}
}
}
I do not want to return event names those start with the word describe or get. Rest of the event names from ec2 event source should be returned.
!(eventName.keyword: Describe* OR eventName.keyword:
Get* )
The question is how to combine these 2 search requests into one?
I need to use that query from my application.
Update:
The Inspect menu of Kibana Discover tab generates this query. I am just trying to rewrite query_string part with usual match or match_phrase using boolean OR clause.
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "!(eventName.keyword: Describe* OR eventName.keyword: Get* )",
"analyze_wildcard": true
}
},
{
"match_phrase": {
"eventSource": {
"query": "ec2.amazonaws.com"
}
}
},
{
"range": {
"#timestamp": {
"format": "strict_date_optional_time",
"gte": "2020-07-09T08:39:15.947Z",
"lte": "2020-07-24T08:39:15.947Z"
}
}
}
],
"filter": [],
"should": [],
"must_not": []
}
}
You can easily use the boolean query's must_not clause to exclude the documents which you don't want in your search result and you can add as many as must_not as you want, it's fairly easy to do and can be done in a single query.
Please refer the example in the same link to get more info. Created sample in my local to show your the correct query, Please note instead of wildcard I am using the prefix query which is better and server your use-case.
Create index mapping
{
"mappings": {
"properties": {
"eventName": {
"type": "keyword"
}
}
}
}
Index sample doc
{
"eventName" : "Describe the events"
}
{
"eventName" : "the Describe events"
}
{
"eventName" : "Get the event"
}
{
"eventName" : "event Get"
}
Now search query to get only 2 and 3rd doc according to your req
{
"query": {
"bool": {
"must_not": [
{
"prefix": {
"eventName": "Desc"
}
},
{
"prefix": {
"eventName": "Get"
}
}
]
}
}
}
Search result
"hits": [
{
"_index": "ngramkey",
"_type": "_doc",
"_id": "2",
"_score": 0.0,
"_source": {
"eventName": "the Describe events"
}
},
{
"_index": "ngramkey",
"_type": "_doc",
"_id": "4",
"_score": 0.0,
"_source": {
"eventName": "event Get"
}
}
]
As suggested by the user "Opster Elasticsearch Ninja", I have merged must not boolean query like this...
{
"query": {
"bool": {
"must": [
{
"bool": {
"must_not": [
{
"prefix": {
"eventName.keyword": "Desc"
}
},
{
"prefix": {
"eventName.keyword": "Get"
}
}
]
}
},
{
"match_phrase": {
"eventSource": {
"query": "ec2.amazonaws.com"
}
}
},
{
"range": {
"#timestamp": {
"format": "strict_date_optional_time",
"gte": "2020-07-09T08:39:15.947Z",
"lte": "2020-07-24T08:39:15.947Z"
}
}
}
],
"filter": [],
"should": [],
"must_not": []
}
}
}

Multi match query with terms lookup searching multiple indices elasticsearch 6.x

All,
I am working on building a NEST 6.x query that takes a serach term and looks in different fields in different indices.
This is the one I got so far but is not returning any results that I am expecting.
Please see the details below
Indices used
dev-sample-search
user-agents-search
The way the search should work is as follows.
The value in the query field(27921093) is searched against the
fields agentNumber, customerName, fileNumber, documentid(These are all
analyzed fileds).
The search should limit the documents to the agentNumbers the user
sampleuser#gmail.com has access to( sample data for
user-agents-search) is added below.
agentNumber, customerName, fileNumber, documentid and status are
part of the index dev-sample-search.
status field is defined as a keyword.
The fields in the user-agents-search index are all keywords
Sample user-agents-search index data:
{
"id": "sampleuser#gmail.com"",
"user": "sampleuser#gmail.com"",
"agentNumber": [
"123.456.789",
"1011.12.13.14"
]
}
Sample dev-sample-search index data:
{
"agentNumber": "123.456.789",
"customerName": "Bank of america",
"fileNumber":"test_file_1123",
"documentid":"1234456789"
}
GET dev-sample-search/_search
{
"from": 0,
"size": 10,
"query": {
"bool": {
"must": [
{
"multi_match": {
"type": "best_fields",
"query": "27921093",
"operator": "and",
"fields": [
"agentNumber",
"customerName",
"fileNumber",
"documentid^10"
]
}
}
],
"filter": [
{
"bool": {
"must": [
{
"terms": {
"agentNumber": {
"index": "user-agents-search",
"type": "_doc",
"user": "sampleuser#gmail.com",
"path": "agentNumber"
}
}
},
{
"bool": {
"must_not": [
{
"terms": {
"status": {
"value": "pending"
}
}
},
{
"term": {
"status": {
"value": "cancelled"
}
}
},
{
"term": {
"status": {
"value": "app cancelled"
}
}
}
],
"should": [
{
"term": {
"status": {
"value": "active"
}
}
},
{
"term": {
"status": {
"value": "terminated"
}
}
}
]
}
}
]
}
}
]
}
}
}
I see a couple of things that you may want to look at:
In the terms lookup query, "user": "sampleuser#gmail.com", should be "id": "sampleuser#gmail.com",.
If at least one should clause in the filter clause should match, set "minimum_should_match" : 1 on the bool query containing the should clause

elasticsearch multi field query is not working as expected

I've been facing some issues with multi field elasticsearch query. I am trying to query all the documents which matches the field called func_name to two hard coded strings, even though my index has documents with both these function names, but the query result is always fetching only one func_name. So far I have tried following queries.
1) Following returns only one function match, even though the documents have another function as well
GET /_search
{
"query": {
"multi_match": {
"query": "FEM_DS_GetTunerStatusInfo MDM_TunerStatusPrint",
"operator": "OR",
"fields": [
"func_name"
]
}
}
}
2) following intermittently gives me both the functions.
GET /_search
{
"query": {
"match": {
"func_name": {
"query": "MDM_TunerStatusPrint FEM_DS_GetTunerStatusInfo",
"operator": "or"
}
}
}
}
3) Following returns only one function match, even though the documents have another function as well
{
"query": {
"bool": {
"should": [
{ "match": { "func_name": "FEM_DS_GetTunerStatusInfo" }},
{ "match": { "func_name": "MDM_TunerStatusPrint" }}
]
}
}
}
Any help is much appreciated.
Thanks for your reply. Lets assume that I have following kind of documents in my elasticsearch. I want my search to return first two documents out of all as they matches my func_name.
{
"_index": "diag-178999",
"_source": {
"severity": "MIL",
"t_id": "03468500",
"p_id": "000007c6",
"func_name": "MDM_TunerStatusPrint",
"timestamp": "2017-06-01T02:04:51.000Z"
}
},
{
"_index": "diag-344563",
"_source": {
"t_id": "03468500",
"p_id": "000007c6",
"func_name": "FEM_DS_GetTunerStatusInfo",
"timestamp": "2017-07-20T02:04:51.000Z"
}
},
{
"_index": "diag-101010",
"_source": {
"severity": "MIL",
"t_id": "03468500",
"p_id": "000007c6",
"func_name": "some_func",
"timestamp": "2017-09-15T02:04:51.000Z"
}
The "two best ways" to request your ES is to filter by terms on a particular field or to aggregate your queries so that you can rename the field, apply multiple rules, and give a more understandable format to your response
See : https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html and the other doc page is here, very useful :
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html
In your case, you should do :
{
"from" : 0, "size" : 2,
"query": {
"filter": {
"bool": {
"must": {
"term": {
"func_name" : "FEM_DS_GetTunerStatusInfo OR MDM_TunerStatusPrint",
}
}
}
}
}
}
OR
"aggs": {
"aggregationName": {
"terms": {
"func_name" : "FEM_DS_GetTunerStatusInfo OR MDM_TunerStatusPrint"
}
}
}
}
The aggregation at the end is just here to show you how to do the same thing as your query filter. Let me know if it's working :)
Best regards
As I understand, you should use filtered query to match any document with one of the values of func_name mentioned above:
{
"query": {
"filtered": {
"filter": {
"bool": {
"must": [
{
"terms": {
"func_name": [
"FEM_DS_GetTunerStatusInfo",
"MDM_TunerStatusPrint"
]
}
}
]
}
}
}
}
}
See:
Filtered Query, Temrs Query
UPDATE in ES 5.0:
{
"query": {
"bool": {
"must": [
{
"terms": {
"func_name": [
"FEM_DS_GetTunerStatusInfo",
"MDM_TunerStatusPrint"
]
}
}
]
}
}
}
See: this answer

Aggregation with fuzzy filter

Is possible in Elastisearch to have an aggregation which will have a filter/query including fuzzy?
ATM i have documents which contains nested object[]. What I want to achieve:
- select from each document 0..n nested objects which match a filter
- from this array of nested objects take the distinct one
- sort them by _score
- take the top 5 or X
- use the terms for an autocomplete/suggestions (should work more as a "like" and not autocomplete)
Until now I tried different types of aggregations like: significant_terms, top_hits but not in a good combination so I don't get the desired result.
Problems:
significant_terms doesn't return a value until he figures out when a term is significant (maybe i did not use a good analyzer)
top-hits returns any nested obj from the selected document and also contains duplicates
Here is an example of my query
GET customerinsights/_search
{
"query": {
"bool": {
"must": [
{
"nested": {
"path": "CustomerInsightTargets",
"query": {
"bool": {
"must": [
{
"match": {
"CustomerInsightTargets.CustomerInsightValue": {
"query": "2017",
"operator": "AND",
"fuzziness": 2
}
}
}
]
}
}
}
}
]
}
} ,
"aggs": {
"root": {
"nested": {
"path": "CustomerInsightTargets"
},
"aggs": {
"top_tags": {
"terms": {
"field": "CustomerInsightTargets.CustomerInsightSource.keyword"
},
"aggs": {
"top_tag_hits": {
"top_hits": {
"sort": [
{
"_score": {
"order": "desc"
}
}
],
"size": 5,
"_source": "CustomerInsightTargets"
}
}
}
}
}
}
},
"size": 0,
"_source": "CustomerInsightTargets"
}

Elasticsearch. filtered query with partial_fields possible?

Is it possible to exclude certain field from result ? I'm using filtered query like this:
{
"size": 10,
"query": {
"filtered": {
"query": {
"bool": {
"should": [
{
"text": {
"name": {
"query": "list",
"operator": "or",
"boost": 30
}
}
},
{
"text": {
"field2": {
"query": "list",
"operator": "or",
"boost": 0.2
}
}
},
{
"text": {
"field1": {
"query": "list",
"operator": "or",
"boost": 0.02
}
}
}
]
}
},
"filter": {
"and": [
{
"term": {
"_type": "product"
}
}
]
}
},
"filter": {
"partial_fields": {
"exclude": "field3"
}
}
},
"sort": [
{
"_score": "desc"
}
]
}
I've added filter partial_fields but it does not seem to have any effect. I'm using ES 0.9
Keep in mind that partial_fields support has been deprecated as of 1.0.0beta -
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-fields.html#partial
I know you're on 0.9 but at some point you'll need to upgrade and this approach won't work. I'd suggest upgrading to a 1.x release and using source filtering instead:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-source-filtering.html
partial_fields can return a partial representation of _source based on include and exclude patterns
So i guess you should specify a wildcard pattern for field name in exclude. If your field name is DATA then the exclude pattern should be DAT*..

Resources