I am periodically training an anomaly detection model and I am saving it in elasticsearch index of the form 'anomaly_detection_model-' + date_model_trained. This means that I am ending up with indices of the form: anomaly_detection_model-31.08.2022, anomaly_detection_model-29.08.2022, anomaly_detection_model-27.08.2022, etc. I need to be able to get access to the latest model at real-time in order to make the predictions without knowing what is the date that this model was trained. Do you have any ideas of how this could be possible?
if you need to access to the last created index try this:
GET /_cat/indices?pretty&s=creation.date:desc
Related
Is there a way to dynamically append a driving distance of the CONSUMER in ES?
I am trying to create an app and sort PROVIDERS by driving distance.
One thing I could think of is get all the PROVIDERS within a range then put them in an array. Iterate to add driving distance then sort array. But this is not efficient.
Any suggestions?
thank you!
You can use runtime fields to achieve what you want, A runtime field is a field that is evaluated at query time. Runtime fields enable you to:
Add fields to existing documents without reindexing your data Start
working with your data without understanding how it’s structured
Override the value returned from an indexed field at query time
Define fields for a specific use without modifying the underlying
schema
For more information you can check Elasticsearch blog post here, and for runtime fields here.
I am using olivere/elastic library for elasticsearch in my go app . I have list of values for a particular field (say fieldA) of elasticsearch document. I want to update a particular field of all document by searching on field fieldA .
This : Updating a record in ElasticSearch using olivere/elastic in google go
explains the update part. But in my case in don't have Id of documents to be updated . So, either i can make search call to retrieve document ids and then update them , or is there another way am missing? Thanks in Advance.
If you need to update a list of documents, you can use the Update By Query API. The unit tests give you a hint about how the syntax looks like. However, if you have individual values for individual documents, I guess there's no other way than updating them one by one. The fastest way to achieve that is by using the Bulk API.
I'm quite sure that I want to be notified with the inserted document by each insertion in the couch db.
something like this:
http://localhost:5058/db-name/_chnages/_view/inserted-document
And I like the response to be something like the following:
{
"id":"0552065465",
"name":"james"
.
.
.
}
Reconnecting to the database for giving the actual document by each notification can cause performance issues.
Can I define a view that return the actual document by each change?
There are 3 possible way to define if a document was just added:
You add a status field to your document with a specific status for new documents.
If the revision starts with a 1- but it's not 100% accurate according to this if you do replication.
In the changes response, check if the number of revision of the document is equal to one. If so, it means it was just added(best solution IMO)
If you want to query the _changes endpoint and directly get the newly inserted documents, you can use the approach #1 and use a filter function that only returns documents with status="new".
Otherwise, you should go with approach #3 and filter the _changes responses locally. Eg: your application would receive all changes and only handle documents with revisions array count equal to 1.
And as you mentioned, you want to receive the document, not only the _id and the _rev. To do so, you can simply add the query parameter: include_docs=true
I have a type Product, which has multiple Prices, but the returned model can only ever have one price.
I need to have multiple prices in elastic, to be able to vary on time without having to reindex. I also need to be able to sort products based on price.
I have tried both with nested and child properties, but I don't seem to be able to query it correctly.
So is it possible to achieve this using elastic? If not, how should I structure my index instead?
You can set the field data type to array. Then sort by for example the max value using the mode option.
See for examples:
https://www.elastic.co/guide/en/elasticsearch/reference/current/array.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html
What ended up doing is to get the current price via inner hits https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-inner-hits.html
And then add the inner hit to the model after it was returned from elastic.
I need to add a new field to ALL documents in an index without pulling down the document and pushing it back up (this will take about a day). Is it possible to use the _BULK api to achieve this?
I have also researched the update_by_query plugin, and it seems to would take just as long as pulling them down and pushing them back myself.
Yes, the bulk API supports updates which can add a new field using a partial document or script. To iterate through your document ids do a scan and scroll with the fields parameter set to an empty array.