This is the mapping.
curl -XPUT 'localhost:9200/products/' -d '{
"settings" : {
"index" : {
"number_of_shards" : 6,
"number_of_replicas" : 1
}
},
"mappings" : {
"product" : {
"_all":{ "enabled": true },
"properties":{
"id" : { "type" : "string", "index" : "not_analyzed", "include_in_all": true },
"description" : { "type" : "string" },
"title" : { "type" : "string", "boost" : 2 },
}
}
}
}'
I don't want to get the ads which have no description. but as you can see in mapping "description" have an index.
So how do I use not query in description?
please help me out.
I seen the doc of elasticsearch and I use this query.
**query => {
filtered => {
filter => {
not => {
filter => {
term => {description => ''}
}
}
},
query => {
match => { _all => $q }
}
}
}**
But it's not working, I think because description have index right?
For 2.4 this would be the correct syntax and query approach:
{
"query": {
"bool": {
"must": [
{"match_all": {}}
],
"filter": {
"bool": {
"must": [
{
"exists": {
"field": "description"
}
},
{
"wildcard": {
"description": "*"
}
}
]
}
}
}
}
}
Instead of filtered you have a bool with must as query and filter as filter. What's inside ofmustis what you have as query and what's inside offilteris what you have as filter. The approach you used withfiltered` is deprecated in ES 2.x.
Related
I have two fields in my elastic search which is lowest_local_price and lowest_global_price.
I want to map dynamic value to third field price on run time based on local or global country.
If local country matched then i want to map lowest_local_price value to price field.
If global country matched then i want to map lowest_global_price value to price field.
If local or global country matched then i want to apply range query on the price field and boost that doc by 2.0.
Note : This is not compulsary filter or query, if matched then just want to boost the doc.
I have tried below solution but does not work for me.
Query 1:
$params["body"] = [
"runtime_mappings" => [
"price" => [
"type" => "double",
"script" => [
"source" => "if (params['_source']['country_en_name'] == '$country_name' ) { emit(params['_source']['lowest_local_price']); } else { emit( params['_source']['global_rates']['$country->id']['lowest_global_price']); }"
]
]
],
"query" => [
"bool" => [
"filter" => [
"range" => [ "price" => [ "gte" => $min_price]]
],
"boost" => 2.0
]
]
];
Query 2:
$params["body"] = [
"runtime_mappings" => [
"price" => [
"type" => "double",
"script" => [
"source" => "if (params['_source']['country_en_name'] == '$country_name' ) { emit(params['_source']['lowest_local_price']); } else { emit( params['_source']['global_rates']['$country->id']['lowest_global_price']); }"
]
]
],
"query" => [
"bool" => [
"filter" => [
"range" => [ "price" => [ "gte" => $min_price, "boost" => 2.0]]
],
]
]
];
None of them working for me, because it can boost the doc. I know filter does not work with boost, then what is the solution for dynamic field mapping with range query and boost?
Please help me to solve this query.
Thank you in advance!
You can (most likely) achieve what you want without runtime_mappings by using a combination of bool queries, here's how.
Let's define test mapping
We need to clarify what mapping we are working with, because different field types require different query types.
Let's assume that your mapping looks like this:
PUT my-index-000001
{
"mappings": {
"dynamic": "runtime",
"properties": {
"country_en_name": {
"type": "text"
},
"lowest_local_price": {
"type": "float"
},
"global_rates": {
"properties": {
"UK": {
"properties":{
"lowest_global_price": {
"type": "float"
}
}
},
"FR": {
"properties":{
"lowest_global_price": {
"type": "float"
}
}
},
"US": {
"properties":{
"lowest_global_price": {
"type": "float"
}
}
}
}
}
}
}
}
Note that country_en_name is of type text, in general such fields should be indexed as keyword but for the sake of demonstration of the use of runtime_mappings I kept it text and will show later how to overcome this limitation.
bool is the same as if for Elasticsearch
The query without runtime mappings might look like this:
POST my-index-000001/_search
{
"query": {
"bool": {
"should": [
{
"match_all": {}
},
{
"bool": {
"should": [
{
"bool": {
"must": [
{
"match": {
"country_en_name": "UK"
}
},
{
"range": {
"lowest_local_price": {
"gte": 1000
}
}
}
]
}
},
{
"range": {
"global_rates.UK.lowest_global_price": {
"gte": 1000
}
}
}
],
"boost": 2
}
}
]
}
}
}
This can be interpreted as the following:
Any document
OR (
(document with country_en_name=UK AND lowest_local_price > X)
OR
(document with global_rates.UK.lowest_global_price > X)
)[boost this part of OR]
The match_all is needed to return also documents that do not match the other queries.
How will the response of the query look like?
Let's put some documents in the ES:
POST my-index-000001/_doc/1
{
"country_en_name": "UK",
"lowest_local_price": 1500,
"global_rates": {
"FR": {
"lowest_global_price": 1000
},
"US": {
"lowest_global_price": 1200
}
}
}
POST my-index-000001/_doc/2
{
"country_en_name": "FR",
"lowest_local_price": 900,
"global_rates": {
"UK": {
"lowest_global_price": 950
},
"US": {
"lowest_global_price": 1500
}
}
}
POST my-index-000001/_doc/3
{
"country_en_name": "US",
"lowest_local_price": 950,
"global_rates": {
"UK": {
"lowest_global_price": 1100
},
"FR": {
"lowest_global_price": 1000
}
}
}
Now the result of the search query above will be something like:
{
...
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : 4.9616585,
"hits" : [
{
"_index" : "my-index-000001",
"_type" : "_doc",
"_id" : "1",
"_score" : 4.9616585,
"_source" : {
"country_en_name" : "UK",
"lowest_local_price" : 1500,
...
}
},
{
"_index" : "my-index-000001",
"_type" : "_doc",
"_id" : "3",
"_score" : 3.0,
"_source" : {
"country_en_name" : "US",
"lowest_local_price" : 950,
"global_rates" : {
"UK" : {
"lowest_global_price" : 1100
},
...
}
}
},
{
"_index" : "my-index-000001",
"_type" : "_doc",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"country_en_name" : "FR",
"lowest_local_price" : 900,
"global_rates" : {
"UK" : {
"lowest_global_price" : 950
},
...
}
}
}
]
}
}
Note that document with _id:2 is on the bottom because it didn't match any of the boosted queries.
Will runtime_mappings be of any use?
Runtime mappings are useful in case there's an existing mapping with data types that do not permit to execute a certain type of query. In previous versions (before 7.11) one would have to do a reindex in such cases, but now it is possible to use runtime mappings (but the query is more expensive).
In our case, we have got country_en_name indexed as text which is suited for full-text search and not for exact lookups. We should rather use keyword instead. This is how the query may look like with the help of runtime_mappings:
POST my-index-000001/_search
{
"runtime_mappings": {
"country_en_name_keyword": {
"type": "keyword",
"script": {
"source": "emit(params['_source']['country_en_name'])"
}
}
},
"query": {
"bool": {
"should": [
{
"match_all": {}
},
{
"bool": {
"should": [
{
"bool": {
"must": [
{
"term": {
"country_en_name_keyword": "UK"
}
},
{
"range": {
"lowest_local_price": {
"gte": 1000
}
}
}
]
}
},
{
"range": {
"global_rates.UK.lowest_global_price": {
"gte": 1000
}
}
}
],
"boost": 2
}
}
]
}
}
}
Notice how we created a new runtime field country_en_name_keyword with type keyword and used a term lookup instead of match query.
I have the following mapping
"mappings":{
"properties":{
"name": {
"type": "text"
},
"age": {
"type": "integer"
},
"customProps":{
"type" : "nested",
"properties": {
"key":{
"type": "keyword"
},
"value": {
"type" : "keyword"
}
}
}
}
}
example data
{
"name" : "person1",
"age" : 10,
"customProps":[
{"hairColor":"blue"},
{"height":"120"}
]
},
{
"name" : "person2",
"age" : 30,
"customProps":[
{"jobTitle" : "software engineer"},
{"salaryAccount" : "AvGhj90AAb"}
]
}
so i want to be able to search for document by salary account case insensitive, i am also searching using wild card
example query is
{
"query": {
"bool": {
"should": [
{
"nested": {
"path": "customProps",
"query": {
"bool": {
"must": [
{ "match": { "customProps.key": "salaryAccount" } },
{ "wildcard": { "customProps.value": "*AvG*"
}
}
]}}}}]}}}
i tried adding analyzer with PUT using the following syntax
{
"settings":{
"index":{
"analysis":{
"analyzer":{
"analyzer_case_insensitive" : {
"tokenizer":"keyword",
"filter":"lowercase"
}
}
}
}
},
"mappings":{
"people":{
"properties":{
"customProps":{
"properties":{
"value":{
"type": "keyword",
"analyzer": "analyzer_case_insensitive"
}
}
}
}
}
}
}
im getting the following error
"type" : "mapper_parsing_exception",
"reason" : "Root mapping definition has unsupported parameters: [people: {properties={customProps={properties={value={analyzer=analyzer_case_insensitive, type=keyword}}}}}]"
any idea how to do the analyzer for the salary account object in the array when it exists?
Your use case is quite clear, that you want to search on the value of salaryAccount only when this key exists in customProps array.
There are some issues with your mapping definition :
You cannot define a custom analyzer for keyword type field, instead you can use a normalizer
Based on the mapping definition you added at the beginning of the question, it seems that you are using elasticsearch version 7.x. But the second mapping definition that you provided, in that you have added mapping type also (i.e people), which is deprecated in 7.x
There is no need to add the key and value fields in the index mapping.
Adding a working example with index mapping, search query, and search result
Index Mapping:
PUT myidx
{
"mappings": {
"properties": {
"customProps": {
"type": "nested"
}
}
}
}
Search Query:
You need to use exists query, to check whether a field exists or not. And case_insensitive param in Wildcard query is available since elasticsearch version 7.10. If you are using a version below this, then you need to use a normalizer, to achieve case insensitive scenarios.
POST myidx/_search
{
"query": {
"bool": {
"should": [
{
"nested": {
"path": "customProps",
"query": {
"bool": {
"must": [
{
"exists": {
"field": "customProps.salaryAccount"
}
},
{
"wildcard": {
"customProps.salaryAccount.keyword": {
"value": "*aVg*",
"case_insensitive": true
}
}
}
]
}
}
}
}
]
}
}
}
Search Result:
"hits" : [
{
"_index" : "myidx",
"_type" : "_doc",
"_id" : "2",
"_score" : 2.0,
"_source" : {
"name" : "person2",
"age" : 30,
"customProps" : [
{
"jobTitle" : "software engineer"
},
{
"salaryAccount" : "AvGhj90AAb"
}
]
}
}
]
I have an index with a mapping something like this:
"email" : {
"type" : "nested",
"properties" : {
"from" : {
"type" : "text",
"analyzer" : "lowercase_keyword",
"fielddata" : true
},
"subject" : {
"type" : "text",
"analyzer" : "lowercase_keyword",
"fielddata" : true
},
"to" : {
"type" : "text",
"analyzer" : "lowercase_keyword",
"fielddata" : true
}
}
},
"textExact" : {
"type" : "text",
"analyzer" : "lowercase_standard",
"fielddata" : true
}
I want to use query_string to search for matches in both the nested and the non-nested field at the same time, e.g.
email.to:foo#example.com AND textExact:bar
But I can't figure out how to write a query that will search both fields at once. The following doesn't work, because query_string searches do not return nested documents:
"query": {
"query_string": {
"fields": [
"textExact",
"email.to"
],
"query": "email.to:foo#example.com AND textExact:bar"
}
}
I can write a separate nested query, but that will only search against nested fields. Is there any way I can use query_string to match both nested and non-nested fields at the same time?
I am using Elasticsearch 6.8. Cross-posted on the Elasticsearch forums.
Nested documents can only be queried with the nested query.
You can follow below two approaches.
1. You can combine nested and normal query in must clause, which works like "and" for different queries.
{
"query": {
"bool": {
"must": [
{
"nested": {
"path": "email",
"query": {
"term": {
"email.to": "foo#example.com"
}
}
}
},
{
"match": {
"textExact": "bar"
}
}
]
}
}
}
2. copy-to
The copy_to parameter allows you to copy the values of multiple fields into a group field, which can then be queried as a single field.
{
"mappings": {
"properties": {
"textExact":{
"type": "text"
},
"to_email":{
"type": "keyword"
},
"email":{
"type": "nested",
"properties": {
"to":{
"type":"keyword",
"copy_to": "to_email" --> copies to non-nested field
},
"from":{
"type":"keyword"
}
}
}
}
}
}
Query
{
"query": {
"query_string": {
"fields": [
"textExact",
"to_email"
],
"query": "to_email:foo#example.com AND textExact:bar"
}
}
}
Result
"_source" : {
"textExact" : "bar",
"email" : [
{
"to" : "sdfsd#example.com",
"from" : "a#example.com"
},
{
"to" : "foo#example.com",
"from" : "sdfds#example.com"
}
]
}
The mapping contains nested fields which shouldn't be analyzed (not sure if the 'not_analyzed' value is accurate). Is it possible to do an exact match on a nested field? In the query below the "metadata.value": "2014.NWJSD.47" still gets analyzed. Elasticsearch breaks up the string into several terms ("2014", "NWJSD", "47"). I tried to use "term" instead of "match" but this didn't return any result.
"mappings" : {
"metadata" : {
"type" : "nested",
"properties" : {
"name" : {
"type" : "text",
"index" : "not_analyzed"
},
"value" : {
"type" : "text",
"index" : "not_analyzed"
}
}
}
}
The Query:
"query": {
"bool": {
"must": [
{
"nested": {
"path": "metadata",
"query": {
"bool": {
"must": [
{
"match": {
"metadata.name": "number"
}
},
{
"match": {
"metadata.value": "2014.NWJSD.47"
}
}
]
}
}
}
}
]
}
}
Try to use keyword instead of text in your mapping like:
{
"mappings": {
"your_type_name": {
"properties": {
"metadata" : {
"type" : "nested",
"properties" : {
"name" : {
"type" : "keyword"
},
"value" : {
"type" :"keyword"
}
}
}
}
}
}
}
These fields won't be analyzed. Then you should reindex your data and to query your data you should replace match (which is analyzed query) with term.
{
"query": {
"bool": {
"must": [
{
"nested": {
"path": "metadata",
"query": {
"bool": {
"must": [
{
"term": {
"metadata.name": "number"
}
},
{
"term": {
"metadata.value": "2014.NWJSD.47"
}
}
]
}
}
}
}
]
}
}
}
I think you are looking for a query string query.
You can freely disable "analyze" option for that field in mapping option and reindex everything again but you could also check this query out:
as written here:
GET /_search
{
"query": {
"query_string" : {
"query" : "your string"
}
}
}
Overview
I have apartments which have reservations. My index has the reservations as nested fields with date fields for start_date and end_date.
I'm using the chewy ruby gem - but this doesn't matter at this time i think. Just need to get my query right.
Goal
I want to fetch all available apartments which have no reservation at the given date or no reservations at all.
Current query
Unfortunately returns all apartments:
:query => {
:bool => {
:must_not => [
{
:range => {:"reservations.start_date" => {:gte => "2017-02-10"}}
},
{
:range => {:"reservations.end_date" => {:lte => "2017-02-12"}}
}
]
}
}
Index Settings
{
"apartments" : {
"aliases" : { },
"mappings" : {
"apartment" : {
"properties" : {
"city" : {
"type" : "string"
},
"coordinates" : {
"type" : "geo_point"
},
"email" : {
"type" : "string"
},
"reservations" : {
"type" : "nested",
"properties" : {
"end_date" : {
"type" : "date",
"format" : "yyyy-MM-dd"
},
"start_date" : {
"type" : "date",
"format" : "yyyy-MM-dd"
}
}
},
"street" : {
"type" : "string"
},
"zip" : {
"type" : "string"
}
}
}
},
"settings" : {
"index" : {
"creation_date" : "1487289727161",
"number_of_shards" : "5",
"number_of_replicas" : "1",
"uuid" : "-rM79OUvQ3qkkLJmQCsoCg",
"version" : {
"created" : "2040499"
}
}
},
"warmers" : { }
}
}
We have to list free apartments and those apartment that will be available in the desired period (start_date, end_date variables)
So it should be a or query: free_aparments or available_aparments
The free apartments (those that haven't any value in reservations field) should be easy to query with a missing filter, but this is a nested field and we have to deal with.
If we perform the query with a missing filter all docs will be returned. It's weird but it happens. Here there's the explained solution: https://gist.github.com/Erni/7484095 and here is the issue: https://github.com/elastic/elasticsearch/issues/3495 The gist snnipet works with all elasticsearch versions.
The other part of the or query are available apartments. I've solved this part performing a not query. Return me those apartments that NOT have a reservation, thought a list of range that match with those aparments that do have a reservation and then negate the result using must_not filter
elasticsearch_query = {
"query": {
"filtered": {
"filter": {
"bool": {
"should": [
{
"nested": {
"filter": {
"bool": {
"must_not" : [
{
"range": {
"start_date": {
"gte" : start_date,
"lt" :end_date
}
}
},
{
"range": {
"end_date": {
"gte" : end_date,
#"lte" :end_date
}
}
}
]
}
},
"path": "reservations"
}
},
{
#{ "missing" : { "field" : "reservations"} }
"not": {
"nested": {
"path": "reservations",
"filter": {
"match_all": {}
}
}
}
}
],
}
}
},
},
"sort" : {"id":"desc"}
}
You can have a look to my solution in this notebook
I've created and example, populating a sample index and searching for desired apartments with this query
Comments answers:
Prefix: Since nested filter is performed setting path will be queried, prefix is no needed at all (at least in my tested version). And yes, you can add a field names start_date at document level or at another nested field
Apartment matches: Yes, it matches with 91 sample apartments, but since I did a search with default size parameter, only 10 are returned (I didn't specified its value, its default value). If you need to get ALL of them, use a scroll search
(notebook has been modified to clarify this points)
First of all, I think you must use the nested query.
I am not familiar with chewy-gem but the query would look something like:
:query => {
:nested: => {
:path: => "reservations",
:query => {
:bool => {
:must_not => [
{
:range => {:"reservations.start_date" => {:gte => "2017-02-10"}}
},
{
:range => {:"reservations.end_date" => {:lte => "2017-02-12"}}
}
]
}
}
}
}
But it might also not work as if there is a reservation in 2018, the fisrt bool query will be true (as the start date will be > 2017-02-10), therefore the appartment will not be returned, if I'm correct.
I would do something like:
:query => {
:nested: => {
:path: => "reservations",
:query => {
:bool => {
:must_not => [
{
:range => {:"reservations.start_date" => {:gte => "2017-02-10", :lte => "2017-02-12"}}
},
{
:range => {:"reservations.end_date" => {:gte => "2017-02-10", :lte => "2017-02-12"}}
}
]
}
}
}
}
which means no start date beetween the range you want, no end date beetween the range you want.
This is the query I came up with which is supposed to take into account all conditions, namely:
either there are no reservations (1st top-level bool/should)
or there are at least one reservation and the reservation start and end dates do not overlap with the requested dates.
Here, we're asking for free apartments between 2017-02-10 and 2017-02-12
{
"bool": {
"minimum_should_match": 1,
"should": [
{
"nested": {
"path": "reservations",
"query": {
"bool": {
"must_not": {
"exists": {
"field": "reservations.start_date"
}
}
}
}
}
},
{
"bool": {
"must": [
{
"nested": {
"path": "reservations",
"query": {
"bool": {
"minimum_should_match": 1,
"should": [
{
"range": {
"reservations.start_date": {
"gt": "2017-02-10"
}
}
},
{
"range": {
"reservations.end_date": {
"lt": "2017-02-10"
}
}
}
]
}
}
}
},
{
"nested": {
"path": "reservations",
"query": {
"bool": {
"minimum_should_match": 1,
"should": [
{
"range": {
"reservations.start_date": {
"gt": "2017-02-12"
}
}
},
{
"range": {
"reservations.end_date": {
"lt": "2017-02-12"
}
}
}
]
}
}
}
}
]
}
}
]
}
}