I'm storing an arbitrary nested object as a flattened field "_meta" which contains various information related to a product.
Here is the mapping for that field:
"mappings": {
"dynamic": "strict",
"properties": {
"_meta": {
"type": "flattened"
},
...
So when trying to search for:
{
"query": {
"exists": {
"field": "_meta.user"
}
}
}
I'm expecting to retrieve all documents that have that field populated. I get zero hits, although if I search for a particular document, I can see that at least one document has that field populated:
"user": {
"origin_title": "some title",
"origin_title_en": "some other title",
"address": "some address",
"performed_orders_count": 0,
"phone": "some phone",
"name": "some name",
"tariff": null,
"proposal_image_background_color": null
},
So how exactly does searching through a flattened data field work?
Why I'm not getting any results?
Tldr;
It is because of the way flattened fields work.
In your case:
{
"_meta":{
"user": {
"name": "some name"
}
}
}
Elasticsearch available representation are:
{
"_meta": ["some name"],
"_meta.user.name": "some name"
}
To reproduce
For the set up:
PUT /74025685/
{
"mappings": {
"dynamic": "strict",
"properties": {
"_meta":{
"type": "flattened"
}
}
}
}
POST /_bulk
{"index":{"_index":"74025685"}}
{"_meta":{"user": "some user"}}
{"index":{"_index":"74025685"}}
{"_meta":{"user": null, "age": 10}}
{"index":{"_index":"74025685"}}
{"_meta":{"user": ""}}
{"index":{"_index":"74025685"}}
{"_meta":{"user": {"username": "some user"}}}
This query is going to find 2 records:
GET 74025685/_search
{
"query": {
"term": {
"_meta": {
"value": "some user"
}
}
}
}
This one, is only going to match the first documents:
GET 74025685/_search
{
"query": {
"term": {
"_meta.user": {
"value": "some user"
}
}
}
}
And so for the exist query:
This one will only return the last doc.
GET 74025685/_search
{
"query": {
"exists": {
"field": "_meta.user.username"
}
}
}
Whereas this one os going to return the 1st and 3rd:
GET 74025685/_search
{
"query": {
"exists": {
"field": "_meta.user"
}
}
}
Related
i like to get a script field if a specific value exist in a array .
Here is my mapping -
"mappings": {
"properties": {
"field1": {
"type": "text",
},
"field2": {
"type": "text",
},
}
}
Here is my example documents -
{
"field1": "hello 1",
"field2": ["id1","id3"]
}
{
"field1": "hello 2",
"field2": ["id2","id3"]
}
{
"field1": "hello 3",
}
{
"field1": "mobile 1",
"field2": ["id1","id4"]
}
Here is my search query -
{
"query": {
"match" : {
"title": "hello"
}
},
"script_fields":{
"testField2":{
"script": {
"source":"doc['field2'].value == params.id ? true : false",
"params": {
"id": "id1"
}
}
}
}
}
what i want to achieve is to get all those document which has "hello" in their title and add a custom field (testField2) in every doc . the value of that custom field would be True if "field2" contain a specific id ("id1") else False
If field2 is only mapped as text, you won't be able to use its field values (via doc['field2'].value). You can only use its _source:
POST your-index/_search
{
"query": {
"match": {
"field1": "hello"
}
},
"script_fields": {
"testField2": {
"script": {
"source": "params._source.containsKey('field2') ? params._source.field2.contains(params.id) : false",
"params": {
"id": "id1"
}
}
}
}
}
Note that accessing _source is not as efficient as using the doc values -- so in short, you were using the right strategy but with the wrong mapping.
In order to be able to access the doc values, either use the keyword data type or set the fielddata parameter on your existing text field to true. I'm going to go with the keyword here:
DELETE your-index
PUT your-index
{
"mappings": {
"properties": {
"field1": {
"type": "text"
},
"field2": {
"type": "keyword" <--
}
}
}
}
After reindexing (which is required after a "mapping breaking change") you can then call:
POST your-index/_search
{
"query": {
"match": {
"field1": "hello"
}
},
"script_fields": {
"testField2": {
"script": {
"source": "doc['field2'].size() != 0 ? doc['field2'].asList().contains(params.id) : false",
"params": {
"id": "id1"
}
}
}
}
}
Finally, notice the .asList() method. It's needed to convert the doc values which are an instance of the class ScriptDocValues.Strings to a consumable ArrayList on which you can then call the conventional .contains() method.
How can I define mapping in Elasticsearch 7 to index a document with a field value from another index? For example, if I have a users index which has a mapping for name, email and account_number but the account_number value is actually in another index called accounts in field number.
I've tried something like this without much success (I only see "name", "email" and "account_id" in the results):
PUT users/_mapping
{
"properties": {
"name": {
"type": "text"
},
"email": {
"type": "text"
},
"account_id": {
"type": "integer"
},
"accounts": {
"properties": {
"number": {
"type": "text"
}
}
}
}
}
The accounts index has the following mapping:
{
"properties": {
"name": {
"type": "text"
},
"number": {
"type": "text"
}
}
}
As I understand it, you want to implement field joining as is usually done in relational databases. In elasticsearch, this is possible only if the documents are in the same index. (Link to doc). But it seems to me that in your case you need to work differently, I think your Account object needs to be nested for User.
PUT /users/_mapping
{
"mappings": {
"properties": {
"account": {
"type": "nested"
}
}
}
}
You can further search as if it were a separate document.
GET /users/_search
{
"query": {
"nested": {
"path": "account",
"query": {
"bool": {
"must": [
{ "match": { "account.number": 1 } }
]
}
}
}
}
}
I would like to perform an AND operation in ElasticSearch using the URI Search (q=). How do I do it?
If I have document like:
[{ "name":"Test 1", "pub":"2"}, { "name":"Test 2", "pub":"1"}, { "name":"A", "pub":"1"}]
And I would like to query for documents containing with a name containing "Test" AND where pub equals "1". How do I do that?
Thanks!
Assuming your document looks like this:
{
"my_field": [
{ "name":"Test 1", "pub":"2"},
{ "name":"Test 2", "pub":"1"},
{ "name":"A", "pub":"1"}
]
}
And the mapping of my_field is of type nested similar to this:
{
"mappings": {
"doc_type": {
"properties": {
"my_field": {
"type": "nested",
"properties": {
"name": { "type": "string" },
"pub": {"type": "integer" }
}
}
}
}
}
}
Then you can query your index and get the expected documents with the following nested query:
POST /_search
{
"query": {
"nested": {
"path": "my_field",
"query": {
"bool": {
"filter": [
{
"match": {
"name": "Test"
}
},
{
"match": {
"pub": 1
}
}
]
}
}
}
}
}
Actually you'd need nested fields. The following is a good resource.
https://www.elastic.co/guide/en/elasticsearch/guide/current/nested-objects.html
Getting started with elasticsearch, not sure if this is possible with one query along with pagination. I have a index with two types: user & blog. Example mapping:
"mappings": {
"user": {
"properties": {
"name" : { "type": "string" }
}
},
"blog": {
"properties": {
"title" : { "type": "string" },
"author_name" : { "type": "string" }
}
}
}
}
sample data
user:
[
{"name": "jemmy"},
{"name": "Tom"}
]
blog:
[
{"title": "foo bar", "author": "jemmy"},
{"title": "magic foo", "author": "Tom"},
{"title": "bigdata for dummies", "author": "Tom"},
{"title": "elasticsearch", "author": "Tom"},
{"title": "JS cookbook", "author": "jemmy"},
]
I'd like to query on the index such a way that when I search for blog it should do subquery on on each match. For example:
POST /test_index/blog/_search
{
"query": {
"match": {
"_all": "foo"
}
}
}
Expected (pseudo) results:
[
{
title: "foo bar",
author_name: "Jemmy",
author_post_count: 2
},
{
title: "magic foo",
author_name: "Tom",
author_post_count: 3
}
]
Here author_post_count is blog post count that the user has authored. If it could return those blog posts instead of count that would be great too. Is this possible? Perhaps the term i'm using not right, but I hope my question is clear.
Try something like this:
POST /test_index/blog/_search
{
"query": {
"match": {
"_all": "foo"
}
},
"aggs": {
"counting_posts": {
"global": {},
"aggs": {
"authors": {
"terms": {
"field": "author",
"size": 10
}
}
}
}
}
}
Be careful though with terms aggregation because it is considering the actual tokenized list of terms from the index, not what you actually index (lowercase/uppercase, tokenized in a way or another).
Here's the mapping:
PUT books-index
{
"mappings": {
"books": {
"properties": {
"tags": {
"type": "nested",
"fields": {
"name": {
"type": "string"
},
"weight": {
"type": "float"
}
}
}
}
}
}
}
Then doing a nested query using a field_value_factor fails with an error
GET books-index/books/_search
{
"query": {
"nested": {
"path": "tags",
"score_mode": "sum",
"query": {
"function_score": {
"query": {
"match": {
"tags.name": "world"
}
},
"field_value_factor": {
"field": "weight"
}
}
}
}
}
}
The error: "nested: ElasticsearchException[Unable to find a field mapper for field [weight]]"
Interestingly, if there's one book in the index with tags - there's no error and the query works well.
Why is this happening? how can I prevent the error when there are no books with tags in the index?
Any ideas?
Thank you!
P.S. There's also an issue on github for this: https://github.com/elastic/elasticsearch/issues/12871
it looks like your mapping is incorrect.
After PUTing the mapping you provided, try executing GET books-index/_mapping, It will show these results:
"books-index": {
"mappings": {
"books": {
"properties": {
"tags": {
"type": "nested"
}
}
}
}
}
It's missing name and weight! The problem with the mapping is that you used either you used fields instead of properties, or you forget to put in a second properties key.
I modified your mapping to reflect that you were looking for a nested name and type within tags, as it looks like that is what your query wants.
PUT books-index
{
"mappings": {
"books": {
"properties": {
"tags": {
"type": "nested",
"properties": { // <--- HERE!
"name": {
"type": "string"
},
"weight": {
"type": "float"
}
}
}
}
}
}
}