I am fairly new to the elastic search, and I am using the Elastic App Search.
So I am trying to update data in elastic app-search through MongoDB Realm App which basically provide triggers on CRUD operations.
I am able to add documents or update existing fields.
But the problem is I am unable to add elements to the array field. I want to add or delete elements from array, after some research I found out that it can be done using some scripts:
"script": {
"source": "ctx._source.fieldToUpdate.add(elementToAdd);",
"lang": "painless"
}
But it's just not working. I am using REST APIs to add or update data in elastic app search. And I am using elastic cloud managed service.
UPDATE - 1
I was using ES App Search, and I created and named the engine as "articles", when I tried to run queries using kibana, I had to use some weird name ".ent-search-engine-documents-article".
So I tried using the same name in Elastic Search REST API
POST /.ent-search-engine-documents-article/_update/docid
And it worked perfectly fine, but I want to perform the same work using REST API of APP Search only.
To perform CRUD operations on your data stored through AppSearch, you should use the Documents API.
AppSearch does not handle nested objects and only provides 4 field types: text, number, date and geolocation. if you are posting objects, it will flatten and stringify them as you described in your comment.
It's also the case for arrays, so you can't just add elements to a field that holds an array as it's just a text field, you need to re-write the whole field (though it does detect them as arrays and handles each element separately if you use that field as a facet for instance).
as for how to patch with the AppSearch REST API, here's a small example inspired from the official documentation:
curl -X PATCH 'https://[instance id].ent-search.[region].[provider].cloud.es.io:443/api/as/v1/engines/articles/documents' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer private-xxxxxxxxxxxxxxxxxxxx' \
-d '[
{ "id": "your_article_id",
"source": "your article source",
"lang": "painless" }
]'
There are also clients for several programming languages that you may find helpful or more intuitive to use.
The weird names you see for your engines, like ".ent-search-engine-documents-article" are the underlying indices on ElasticSearch, and you normally would not like to manipulate them directly.
Related
I am exploring deepset haystack and found it very interesting for multiple use cases like a chatbot, search engine, document search, etc
But have not found any reference where I can create multiple indexes for different documents and search based on indexes. I thought of using meta tags for conditional search(on a particular area) by tagging the documents first and then using the params parameter of query API but the same doesn't seem to work and throws an error(I used its vanilla docker-compose based setup)
You can use multiple indices in the same document store if you want to support multiple use cases, indeed. The write_documents method of the document store has a parameter index so that you can store documents for your different use cases in different indices. In the same way, you can pass an index parameter to the query method.
As you expected, there is an alternative solution that uses the meta field of documents. However, the format needs to be slightly different. Your query needs to have the following format:
{"query": "What's the capital town?", "params": {"filters": {"name": "75_Algeria75.txt"}}}
and your documents need to have the following format:
{'text': 'Algeria is...', 'meta':{'name': "75_Algeria75.txt"}}
In SOLR I can use In-Place-Update to update the value of any fields. Here the value of popularity field will be incremented every time by 20 and add it to the the current value of popularity without considering any other things.
{
"id":"mydoc",
"price":{"set":99},
"popularity":{"inc":20}
}
For Elasticsearch, I can also use the _update api using script to In-Place-update.
POST /website/blog/1/_update
{
"script" : "ctx._source.popularity+=20"
}
But my problem is I want to use the _bulk api using python to in-place-update multiple documents at one with some incremental values. Here I've seen the documentation on how to use the _bulk endpoint to set different values with update action payload. I'm just having some difficulties how can I make the same POST JSON datasets for _bulk with python elasticsearch client for the script update way.
I am taking some courses on udemy on elastic search and trying to set up an elastic search project. I have gotten the bulk api to work and can successfully send batch data into an index on elastic search. But I am having trouble sending data without the bulk api. Because I have read on the elastic search docs that there is a 'false' analogy that a type is like a table in a database and an index is like a database. I decided for this project to create an index for each of the entities that I want to persist which is users and statistics. Therefore I have indices which are called statisitics and users. When I make the following request from postman
headers: Content-Type application/json
POST http://localhost:9200/users
with body:
{"id": 1, "name":"myname"}
I get an error
{"error":"Incorrect HTTP method for uri [/users] and method [POST],
allowed: [PUT, DELETE, GET, HEAD]","status":405}
How can I allow this http method?
Hello I had the same issue using PHP and JAVA, this happen simply because you must add the _type after the index name (described here) :
https://www.elastic.co/guide/en/elasticsearch/guide/current/index-doc.html
Try curl -XPOST http://localhost:9200/users/_admin or something similar ;)
See https://www.elastic.co/guide/en/elasticsearch/guide/current/mapping.html
Since version 6.2, we need to use _doc for the type.
https://discuss.elastic.co/t/cant-use-doc-as-type-despite-it-being-declared-the-preferred-method/113837
https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html#_schedule_for_removal_of_mapping_types
In fact, when you say there is "no types" allowed in ElasticSearch, it is false (at least for versions up to 8.x). What is true is that there is only 1 type allowed for each index.
I'm using ElasticSearch for an application in order to store and search for data.
Because it's also important to search for relationships in my particular case, I recently changed the structure of my data and I am using the _parent field now. (http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-parent-field.html)
I already designed the search query, which works perfectly fine. However, I now have a problem when inserting a new child entry in my database.
It should work this way:
Without the _parent field, when I wanted to achieve this
$ curl -XPUT 'http://localhost:9200/data/child/1' -d '
I inserted the data this way using the JEST API:
client.execute(new Index.Builder(payload).index("data").type("child").build());
When I want to achieve this:
$ curl -XPOST localhost:9200/data/child?parent=154121-d'
How am I supposed to implement this using the JEST-API?
It's possible to provide a parameter for the request. The correct line of code to achieve this:
$ curl -XPOST localhost:9200/data/child?parent=154121-d'
would be achieved with this:
client.execute(new Index.Builder(payload).index("data").type("child").setParameter("parent", "154121").build());
Is it possible to specify a script be executed when inserting a document into ElasticSearch using its Index API? This functionality exists when updating an existing document with new information using its Update API, by passing in a script attribute in the HTTP request body. I think it would be useful too in the Index API because perhaps there are some fields the user wants to be auto-calculated and populated during insertion, without having to send an additional Update request after the insertion to have the script be executed.
Elasticsearch 1.3
If you just need to search/filter on the fields that you'd like to add, the mapping transform capabilities that were added into 1.3.0 could possibly work for you:
The document can be transformed before it is indexed by registering a
script in the transform element of the mapping. The result of the
transform is indexed but the original source is stored in the _source
field.
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-transform.html
You can also have the same transformation run when you get a document as well by adding the _source_transform url parameter to the request:
The get endpoint will retransform the source if the _source_transform
parameter is set.The transform is performed before any source
filtering but it is mostly designed to make it easy to see what was
passed to the index for debugging.
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/_get_transformed.html
However, I don't think the _search endpoint accepts the _source_transform url parameter so I don't think you can apply the transformation to search results. That would be a nice feature request.
Elasticsearch 1.4
Elasticsearch 1.4 added a couple features which makes all this much nicer. As you mentioned, the update API allows you to specify a script to be executed. The update API in 1.4 can also accept a default document to be used in the case of an upsert. From the 1.4 docs:
curl -XPOST 'localhost:9200/test/type1/1/_update' -d '{
"script" : "ctx._source.counter += count",
"params" : {
"count" : 4
},
"upsert" : {
"counter" : 1
}
}'
In the example above, if the document doesn't exist it uses the contents of the upsert key to initialize the document. So in the case above the counter key in the newly created document will have a value of 1.
Now, if we set scripted_upsert to true (scripted_upsert is another new option in 1.4), our script will run against the newly initialized document:
curl -XPOST 'localhost:9200/test/type1/2/_update' -d '{
"script": "ctx._source.counter += count",
"params": {
"count": 4
},
"upsert": {
"counter": 1
},
"scripted_upsert": true
}'
In this example, if the document didn't exist the counter key would have a value of 5.
Full documentation from Elasticsearch site.