Searching for nested documents on a specific parent - elasticsearch

In my Elasticsearch index, I have a book type that has many pages. Using nested documents, I have successfully mapped the relationship and I can index documents like this:
PUT /my_index/book/1
{
"title": "Harry Potter and the Chamber of Secrets",
"pages": [
{
"id": 1,
"text": "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam a metus est. Duis ut est et mi feugiat bibendum feugiat eu tortor. Pellentesque accumsan, eros nec commodo euismod, odio dolor lobortis diam, in pulvinar lacus turpis sed justo. Ut placerat ut nulla sed blandit. Aenean vel turpis erat. Phasellus vehicula laoreet ex, nec dapibus leo tempus vitae. Nulla gravida efficitur metus, in euismod justo placerat sit amet. Maecenas tristique est mauris, sagittis scelerisque turpis suscipit vel. Nullam ultricies sapien sit amet neque aliquam hendrerit sed non nibh."
},
{
"id": 2,
"text": "Pellentesque facilisis turpis in diam maximus luctus. Mauris leo diam, pellentesque a malesuada vitae, scelerisque at ipsum. Fusce tincidunt neque dui. Nullam ac ex luctus, convallis leo eget, feugiat augue. Cras condimentum, purus eu scelerisque sodales, diam est commodo lectus, at finibus orci turpis nec lectus. Mauris in lectus ut diam finibus pellentesque quis tincidunt urna. Curabitur tristique luctus metus at interdum. Curabitur imperdiet ex vel enim pretium, a convallis velit tempor. Nullam odio eros, tincidunt ut consectetur non, scelerisque eget urna. Fusce placerat dui et odio tempus rutrum. Integer non dui eu ante interdum volutpat. Mauris quis ante sed lacus euismod mattis."
}
]
}
I want to query and obtain all pages from a book which contain a certain word in the text field. I can do this like this:
POST /my_index/book/_search
{
"_source": false,
"query": {
"nested": {
"path": "pages",
"query": {
"query_string": {
"query": "Mundo",
"fields": ["pages.text"]
}
},
"inner_hits": {}
}
}
}
My problem however, is that I cannot find a way to specify which book I want to search in. Since I already know which book I'll be searching all pages from. I was trying to do this with a ids query, but I cannot find a way to use it with the nested query together.
How can I specify the book I want to search in (pages from this book), and make the response return a list of all the id values of each page? This id value is a reference to the page's page number in the book.

You can combine two conditions under Bool query like so:
{
"_source": false,
"query": {
"bool": {
"must": [
{
"term": {
"title": "Harry Potter and the Chamber of Secrets"
}
},
{
"nested": {
"path": "pages",
"query": {
"query_string": {
"query": "Mundo",
"fields": [
"pages.text"
]
}
},
"inner_hits": {
"_source": [
"pages.id"
]
}
}
}
]
}
}
}
As can be seen, I have encapsulate your nested query under bool-must (AND condition) and added term which defines the title to filter by.
Second thing, as you described you want only the pages-ids. So, under inner_hits I added "_source": ["id"] which gives you the ids only (it just the same trick
you did when you set the source off at the head of the query.
Pay attention, that, default size for inner_hits is 10, so you may also want to add "size": 1000 under inner_hits.

I will have another data model for your problem.
I will create two types:
A type book
A type Page
You will be able to create a parent/child relationship (The book being the parent and the page the child).
You will be able to make simple query on the page type then retrieve easily the parent of the page.

Related

How to import a document with specific format to Elasticsearch?

I am learning and follow tutorial at https://www.udemy.com/course/elasticsearch-complete-guide/ .
I am using elasticsearch-7.12.0, Kibana 7.12.0, Logstash 7.12.0 on Windows 10 x64, I have document
{
"index": {
"_id": 1
}
}
{
"name": "Wine - Maipo Valle Cabernet",
"price": 152,
"in_stock": 38,
"sold": 47,
"tags": [
"Alcohol",
"Wine"
],
"description": "Aliquam augue quam, sollicitudin vitae, consectetuer eget, rutrum at, lorem. Integer tincidunt ante vel ipsum. Praesent blandit lacinia erat. Vestibulum sed magna at nunc commodo placerat. Praesent blandit. Nam nulla. Integer pede justo, lacinia eget, tincidunt eget, tempus vel, pede. Morbi porttitor lorem id ligula.",
"is_active": true,
"created": "2004\/05\/13"
}
{
"index": {
"_id": 2
}
}
{
"name": "Tart Shells - Savory",
"price": 99,
"in_stock": 10,
"sold": 430,
"tags": [],
"description": "Pellentesque at nulla. Suspendisse potenti. Cras in purus eu magna vulputate luctus. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Vivamus vestibulum sagittis sapien. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Etiam vel augue. Vestibulum rutrum rutrum neque. Aenean auctor gravida sem.",
"is_active": true,
"created": "2007\/10\/14"
}
Full file: https://gist.github.com/donhuvy/759d48d2ec96f70531d77d173cd9657e
I try
PUT _bulk
{
"index": {
"_id": 1
}
}
{
"name": "Wine - Maipo Valle Cabernet",
"price": 152,
"in_stock": 38,
"sold": 47,
"tags": [
"Alcohol",
"Wine"
],
"description": "Aliquam augue quam, sollicitudin vitae, consectetuer eget, rutrum at, lorem. Integer tincidunt ante vel ipsum. Praesent blandit lacinia erat. Vestibulum sed magna at nunc commodo placerat. Praesent blandit. Nam nulla. Integer pede justo, lacinia eget, tincidunt eget, tempus vel, pede. Morbi porttitor lorem id ligula.",
"is_active": true,
"created": "2004\/05\/13"
}
{
"index": {
"_id": 2
}
}
{
"name": "Tart Shells - Savory",
"price": 99,
"in_stock": 10,
"sold": 430,
"tags": [],
"description": "Pellentesque at nulla. Suspendisse potenti. Cras in purus eu magna vulputate luctus. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Vivamus vestibulum sagittis sapien. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Etiam vel augue. Vestibulum rutrum rutrum neque. Aenean auctor gravida sem.",
"is_active": true,
"created": "2007\/10\/14"
}
but not success.
My error
How to import the above document with specific format to Elasticsearch?
I did
PUT /vy
POST /vy/_bulk
{"index":{"_id":1}}
{"purchased_at":"2016-07-10T16:52:43Z","lines":[{"product_id":6,"amount":71.32,"quantity":1},{"product_id":3,"amount":58.96,"quantity":3},{"product_id":1,"amount":29.8,"quantity":3}],"total_amount":160.08,"salesman":{"id":11,"name":"Matthus Mitkcov"},"sales_channel":"store","status":"processed"}
{"index":{"_id":2}}
{"purchased_at":"2016-03-08T16:18:37Z","lines":[{"product_id":7,"amount":31.44,"quantity":2},{"product_id":4,"amount":74.82,"quantity":2}],"total_amount":106.26,"salesman":{"id":79,"name":"Ulberto Woodruff"},"sales_channel":"store","status":"completed"}
{"index":{"_id":3}}
{"purchased_at":"2016-01-04T02:46:54Z","lines":[{"product_id":9,"amount":38.89,"quantity":1}],"total_amount":38.89,"salesman":{"id":68,"name":"Thea Chidgey"},"sales_channel":"store","status":"completed"}
{"index":{"_id":4}}
{"purchased_at":"2016-04-14T11:44:29Z","lines":[{"product_id":4,"amount":26.18,"quantity":2},{"product_id":8,"amount":76.45,"quantity":1},{"product_id":1,"amount":65.59,"quantity":3}],"total_amount":168.22,"salesman":{"id":81,"name":"Melany Cerith"},"sales_channel":"store","status":"cancelled"}
{"index":{"_id":5}}
{"purchased_at":"2016-02-18T17:11:21Z","lines":[{"product_id":2,"amount":56.06,"quantity":3},{"product_id":5,"amount":79.75,"quantity":3}],"total_amount":135.81,"salesman":{"id":23,"name":"Flynn Dome"},"sales_channel":"web","status":"completed"}
{"index":{"_id":6}}
{"purchased_at":"2016-03-26T07:28:22Z","lines":[{"product_id":7,"amount":86.88,"quantity":1},{"product_id":5,"amount":27.34,"quantity":3},{"product_id":5,"amount":88.52,"quantity":2}],"total_amount":202.74,"salesman":{"id":16,"name":"Mirilla Penkman"},"sales_channel":"web","status":"completed"}
{"index":{"_id":7}}
{"purchased_at":"2016-05-06T18:37:33Z","lines":[{"product_id":3,"amount":25.27,"quantity":2}],"total_amount":25.27,"salesman":{"id":5,"name":"Modestia McCreadie"},"sales_channel":"app","status":"confirmed"}
{"index":{"_id":8}}
{"purchased_at":"2016-11-26T11:24:36Z","lines":[{"product_id":4,"amount":51.26,"quantity":1},{"product_id":8,"amount":72.41,"quantity":1}],"total_amount":123.67,"salesman":{"id":55,"name":"Grange Smiths"},"sales_channel":"phone","status":"cancelled"}
{"index":{"_id":9}}
{"purchased_at":"2016-12-26T08:29:47Z","lines":[{"product_id":2,"amount":98.87,"quantity":3}],"total_amount":98.87,"salesman":{"id":5,"name":"Carlyn Stegers"},"sales_channel":"web","status":"cancelled"}
{"index":{"_id":10}}
{"purchased_at":"2016-12-21T12:47:43Z","lines":[{"product_id":7,"amount":21.97,"quantity":3},{"product_id":10,"amount":19.26,"quantity":3}],"total_amount":41.23,"salesman":{"id":83,"name":"Kennie Gilks"},"sales_channel":"app","status":"cancelled"}
{"index":{"_id":11}}
{"purchased_at":"2016-04-02T04:42:31Z","lines":[{"product_id":6,"amount":86.89,"quantity":1},{"product_id":4,"amount":18.04,"quantity":3},{"product_id":10,"amount":18.61,"quantity":3}],"total_amount":123.54,"salesman":{"id":39,"name":"Ashely Rawnsley"},"sales_channel":"phone","status":"cancelled"}
{"index":{"_id":12}}
{"purchased_at":"2016-09-08T16:24:12Z","lines":[{"product_id":1,"amount":11.75,"quantity":1},{"product_id":5,"amount":77.33,"quantity":2},{"product_id":2,"amount":86.76,"quantity":1}],"total_amount":175.84,"salesman":{"id":40,"name":"Raynor Dennis"},"sales_channel":"web","status":"pending"}
{"index":{"_id":13}}
{"purchased_at":"2016-01-12T06:20:27Z","lines":[{"product_id":6,"amount":93.69,"quantity":2}],"total_amount":93.69,"salesman":{"id":62,"name":"Byran Gallihaulk"},"sales_channel":"app","status":"processed"}
{"index":{"_id":14}}
{"purchased_at":"2016-01-31T13:06:58Z","lines":[{"product_id":10,"amount":80.33,"quantity":2}],"total_amount":80.33,"salesman":{"id":43,"name":"Eveleen Ickovici"},"sales_channel":"store","status":"processed"}
{"index":{"_id":15}}
{"purchased_at":"2016-02-20T23:02:31Z","lines":[{"product_id":5,"amount":20.51,"quantity":1},{"product_id":10,"amount":93.35,"quantity":1}],"total_amount":113.86,"salesman":{"id":78,"name":"Erda Boyet"},"sales_channel":"store","status":"confirmed"}
{"index":{"_id":16}}
{"purchased_at":"2016-08-17T09:52:50Z","lines":[{"product_id":9,"amount":61.95,"quantity":1}],"total_amount":61.95,"salesman":{"id":91,"name":"Gherardo MattiCCI"},"sales_channel":"web","status":"confirmed"}
{"index":{"_id":17}}
{"purchased_at":"2016-06-05T18:59:23Z","lines":[{"product_id":8,"amount":32.46,"quantity":3},{"product_id":6,"amount":13.7,"quantity":2},{"product_id":8,"amount":28.74,"quantity":2}],"total_amount":74.9,"salesman":{"id":82,"name":"Shanon Eilhertsen"},"sales_channel":"phone","status":"confirmed"}
{"index":{"_id":18}}
{"purchased_at":"2016-07-26T12:10:15Z","lines":[{"product_id":3,"amount":23.16,"quantity":1},{"product_id":4,"amount":18.94,"quantity":1},{"product_id":10,"amount":38.65,"quantity":2}],"total_amount":80.75,"salesman":{"id":58,"name":"Thekla De Beauchemp"},"sales_channel":"phone","status":"processed"}
{"index":{"_id":19}}
{"purchased_at":"2016-05-08T19:54:44Z","lines":[{"product_id":9,"amount":29.8,"quantity":2},{"product_id":6,"amount":97.61,"quantity":3}],"total_amount":127.41,"salesman":{"id":96,"name":"Susannah Grigoryev"},"sales_channel":"phone","status":"cancelled"}
{"index":{"_id":20}}
{"purchased_at":"2016-06-19T21:34:51Z","lines":[{"product_id":2,"amount":23.3,"quantity":2},{"product_id":5,"amount":30.44,"quantity":2},{"product_id":6,"amount":22.15,"quantity":1}],"total_amount":75.89,"salesman":{"id":66,"name":"Janeta Crutchfield"},"sales_channel":"phone","status":"completed"}
{"index":{"_id":21}}
{"purchased_at":"2016-12-09T17:24:57Z","lines":[{"product_id":1,"amount":90.25,"quantity":1}],"total_amount":90.25,"salesman":{"id":20,"name":"Saxon Whitebread"},"sales_channel":"web","status":"cancelled"}
{"index":{"_id":22}}
{"purchased_at":"2016-09-18T12:26:52Z","lines":[{"product_id":7,"amount":48.03,"quantity":3},{"product_id":1,"amount":18.25,"quantity":1}],"total_amount":66.28,"salesman":{"id":96,"name":"Annissa Cleever"},"sales_channel":"web","status":"completed"}
{"index":{"_id":23}}
{"purchased_at":"2016-09-28T04:29:57Z","lines":[{"product_id":1,"amount":27.74,"quantity":3},{"product_id":8,"amount":82.88,"quantity":1}],"total_amount":110.62,"salesman":{"id":15,"name":"Catherina Tombleson"},"sales_channel":"phone","status":"pending"}
{"index":{"_id":24}}
{"purchased_at":"2016-08-05T23:08:23Z","lines":[{"product_id":5,"amount":90.7,"quantity":1}],"total_amount":90.7,"salesman":{"id":4,"name":"Saloma Raith"},"sales_channel":"app","status":"cancelled"}
{"index":{"_id":25}}
{"purchased_at":"2016-09-02T21:14:18Z","lines":[{"product_id":10,"amount":59.98,"quantity":1},{"product_id":4,"amount":34.72,"quantity":1}],"total_amount":94.7,"salesman":{"id":85,"name":"Raffaello Larrad"},"sales_channel":"web","status":"cancelled"}
{"index":{"_id":26}}
{"purchased_at":"2016-11-12T00:57:06Z","lines":[{"product_id":4,"amount":39.05,"quantity":3},{"product_id":8,"amount":39.29,"quantity":3}],"total_amount":78.34,"salesman":{"id":40,"name":"Maryanne O'Neill"},"sales_channel":"store","status":"confirmed"}
{"index":{"_id":27}}
{"purchased_at":"2016-12-30T07:37:52Z","lines":[{"product_id":7,"amount":12.41,"quantity":3},{"product_id":5,"amount":55.88,"quantity":3},{"product_id":6,"amount":33.89,"quantity":1}],"total_amount":102.18,"salesman":{"id":25,"name":"Myles Sedgemore"},"sales_channel":"web","status":"cancelled"}
{"purchased_at":"2016-10-03T15:57:59Z","lines":[{"product_id":9,"amount":12.15,"quantity":3},{"product_id":4,"amount":54.07,"quantity":1},{"product_id":10,"amount":83.01,"quantity":3}],"total_amount":149.23,"salesman":{"id":58,"name":"Leodora Mithon"},"sales_channel":"phone","status":"pending"}
GET /vy/_search
{
"query": {
"match_all": {}
}
}

Laravel API Customizing associated columns

I have an api source that I combined in related tables. I have customized the objects, but I cannot customize the associated data in it. How do I do this?
Previous Output:
{
"id": 1,
"user_id": 3,
"image": "https://localhost/800/400/cats/varem/?20215",
"description": "Omnis eius corporis incidunt illo dolor et est odio. Et fuga hic id. Velit voluptas tempore quas voluptas iusto. Quia rerum quibusdam consectetur distinctio vero. Nihil et sed reiciendis sequi. Possimus facere ducimus provident est. Qui ut rerum ab ullam sequi est. In placeat nesciunt odio numquam iure.",
"created_at": "2017-09-18T10:51:17.000000Z",
"updated_at": "2020-12-25T08:55:40.000000Z",
"likes": [
{
"id": 7,
"user_id": 10,
"post_id": 1,
"created_at": "1970-03-31T14:17:28.000000Z",
"updated_at": "2020-12-25T08:55:40.000000Z"
}
]
}
Function:
public function like_show($id) {
return Posts::selectRaw('id as post_id, user_id, image as post_image, description as post_description, created_at')
->with('likes')
->where('user_id', '=', $id)
->get();
}
Next Output:
{
"post_id": 1,
"user_id": 3,
"post_image": "https://localhost/800/400/cats/varem/?20215",
"post_description": "Omnis eius corporis incidunt illo dolor et est odio. Et fuga hic id. Velit voluptas tempore quas voluptas iusto. Quia rerum quibusdam consectetur distinctio vero. Nihil et sed reiciendis sequi. Possimus facere ducimus provident est. Qui ut rerum ab ullam sequi est. In placeat nesciunt odio numquam iure.",
"created_at": "2017-09-18T10:51:17.000000Z",
"likes": [
]
}
Output I want:
{
"post_id": 1,
"user_id": 3,
"post_image": "https://localhost/800/400/cats/varem/?20215",
"post_description": "Omnis eius corporis incidunt illo dolor et est odio. Et fuga hic id. Velit voluptas tempore quas voluptas iusto. Quia rerum quibusdam consectetur distinctio vero. Nihil et sed reiciendis sequi. Possimus facere ducimus provident est. Qui ut rerum ab ullam sequi est. In placeat nesciunt odio numquam iure.",
"created_at": "2017-09-18T10:51:17.000000Z",
"likes": [
"like_id": 3,
"user_id": 10,
"post_id": 1,
]
}
alihan.dev, You need to create Likes colletion and Likes api resoure and post reationship with Likes then following code will work for you :).
class Posts extends JsonResource {
//likes is relationship
public function toArray($request) {
return [
'group_id' => $this->group_id,
'u_t_id' => $this->u_t_id,
'group_by' => $this->group_by,
'likes' =>Likes::collection($this->likes),
];
}
}

How do I get the correct subnode with elasticsearch

I am trying to get specific parts of the data back from a specific document. The document contains product information. This is just a small snippet of the document to explain it better.
"_source": {
"product": {
"code": "AM109",
"stuff": "98989",
"languages": [
{
"language": "en",
"labels": {
"name": "Fusce nulla augue",
"description": "Cras ultricies diam vel orci feugiat ornare. Nulla convallis ligula quis fringilla suscipit. Proin malesuada ligula enim, sit amet semper lorem imperdiet in."
}
},
{
"language": "es",
"labels": {
"name": "Aliquam ut odio quam",
"description": "Nulla pulvinar tortor eu ex vestibulum, nec dictum diam tristique. Pellentesque vitae nibh diam. In id purus at sapien eleifend pulvinar non at velit."
}
}
}
}
The document contains different information for different countries, and labels for different languages. To explain I use the labels.
I need to get the information of the correct product based on the code (product.code), and for the languages it should already only return the labels in the language as requested. E.g. if language es is requested:
"_source": {
"product": {
"code": "AM109",
"stuff": "98989",
"languages": [
{
"language": "es",
"labels": {
"name": "Aliquam ut odio quam",
"description": "Nulla pulvinar tortor eu ex vestibulum, nec dictum diam tristique. Pellentesque vitae nibh diam. In id purus at sapien eleifend pulvinar non at velit."
}
}
}
}
I have tried this with nested options and filters, but I cannot get the correct data returned.
This is an existing system so I cannot make changes to the structure, but I can change the mapping if needed and repopulate the index.
Who can help me with this?
As all documents come with information about all languages and you only search in one language at a given time I suggest the following approach:
Store your documents in language specific indices (advantage: same field names)
You can send your documents always to the same index, but define an ingest pipeline which appends a language-specific suffix such as "_en" to your _index-field, to get your document routed/stored in the language-specific index
Adjust your query to only query the index containing the language
If the last step is not possible, you could define an alias, pointing to all language-specific indices and in your query filter on a field, containing the language-code

elasticsearch aggs not working on some fields

Mapping
{
"products":{
"mappings":{
"product":{
"dynamic":"strict",
"properties":{
"CompanyId":{
"type":"integer"
},
"CompanyName":{
"type":"text",
"fields":{
"raw":{
"type":"keyword"
}
}
}
}
}
}
}
}
This works
{
"query":{
"match_all":{}
},
"size":20,
"aggs":{
"CompanyId":{
"terms":{
"field":"CompanyId",
"size":20,
"order":{
"_count":"desc"
}
}
}
}
}
But this does not (buckets array is empty)
{
"query":{
"match_all":{}
},
"size":20,
"aggs":{
"CompanyName.raw":{
"terms":{
"field":"CompanyName.raw",
"size":20,
"order":{
"_count":"desc"
}
}
}
}
}
Why?
This is to get rid of the "mostly code" error.
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse nec elementum sapien, at sollicitudin lacus. Vestibulum sit amet lectus odio. Pellentesque eu tempor elit. Aenean tempor consequat sem, nec rutrum orci euismod ut. Donec blandit malesuada sagittis. Nam pharetra porttitor diam. Mauris lobortis metus a auctor ornare. Suspendisse nec bibendum mauris. Sed efficitur purus mauris, vitae euismod purus volutpat id. Vivamus eget turpis placerat, hendrerit odio non, euismod leo. Curabitur ornare consectetur imperdiet. Proin auctor sagittis libero sit amet euismod. Nam euismod vitae est ut mollis. Fusce eget est orci. Mauris in urna sem. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae;
Since the mapping was changed the index needs to be rebuilt.
POST products/_update_by_query?conflicts=proceed

RestKit Custom Data Structure

I need to consume a web service with non-key-differentiated responses like this:
"data": [
{
"type": "sometype",
"id": "1",
"attributes": {
"att1": "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nam ac dapibus nibh. Nam neque neque, volutpat nec sem ut, hendrerit porta felis.",
"att2": null,
"att3": null,
"att4": null,
"att5": null
}
},
{
"type": "sometype",
"id": "2",
"attributes": {
"att1": "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nam ac dapibus nibh. Nam neque neque, volutpat nec sem ut, hendrerit porta felis.",
"att2": null,
"att3": null,
"att4": null,
"att5": null
}
},
{
"type": "sometype",
"id": "3",
"attributes": {
"att1": "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nam ac dapibus nibh. Nam neque neque, volutpat nec sem ut, hendrerit porta felis.",
"att2": null,
"att3": null,
"att4": null,
"att5": null
}
},
]
Where SomeType would be an NSObject or NSManagedObject with an id property, as well as the other 5 att1-5 properties.
Is it possible to consume custom responses like this using RestKit where the response of every call is basically the exact same structure?

Resources