I'm new to Elasticsearch operations. I had to do a job of renaming the field names in an existing Elasticsearch index.
I updated the existing ingest pipeline by resubmitting the ingest pipeline configuration with additional rename processor.
I did not update the index template because the new pipeline name is still same.
Then I posted the request for re-indexing without prior creating the new index.
I noticed that the new index have got created on the master node.
I did not add any new alias name for the new index yet as I was thinking to see if the operation completes fine then I'll add those to reflect in my kibana search.
Also, I had not specified any field type and I see one of the date field was getting the string data and while conversion of field name, errors started coming in elasticsearch logs.
Now, I see the data for the index is not coming in the kibana search at all and there are error message in the logs
[o.e.x.i.IndexLifecycleRunner] [leader-node-02] policy [logs-for-dev-policy] for index new-index-v0] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [2]
I'm thinking to delete the newly created index new-index-v0 as the old index index-for-dev is still there.
I tried reading the documentation but could not find a clue to know if deleting the new index will create any problem to the older index .
Any suggestions please ?
I am using Elasticsearch version 7.2 .
Related
recently we came across a requirement where we need to add 3 new fields to our Existing Index. We pull this data from our source database using logstash. We have 1000's of documents stored in the current Index already. In the past, it was being told that whenever a change has happened to an existing index (such as adding a new field) we need to reindex with complete data reload again. Since we want the previous documents to have these new fields with data populated in them.
Is there any other way we can achieve this by not dropping the existing index or deleting any documents and reloading? I was hoping we can have a better way of doing this with the latest 7. X version.
No need to drop the index and recreate one. After you have updated the mapping of the index, you just upsert the documents in the index again, with the new fields. Each document will be overwritten by the new one.
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html
I am working on a task of reindexing my Elastic search indexes in case any change happens. There are 2 ways that I can find to implement this but they look same to me unless I am missing something.
I am getting data to my Elastic search service from Postgres of service B, which has a paginated endpoint.
Approach 1:
Create alias which will point to our existing index.
When reindex is triggered, create a new index and once the reindexing is complete, point the alias, which was pointing to old index, to the newly created index.
Delete the old index.
Approach 2:
Create a new Index.
Use the reindex API to copy the data from old index to new index, which will apply the new changes to the old documents.
To me, both of these look same. Disadvantage of using approach 2 seems that it will create a new index name, hence we will have to change the index names while querying.
Also, considering my reindexing operation would not be a frequent task, I am reading the data from a paginated endpoint and then creating indexes again, Approach 1 seems to make more sense to me.
In approach1, you are using alias. In approach 2, you are not using alias.
Both would be same if you add alias to approach2 as step3 and step4 - delete the old index.
Refer As you need to do little often.
I'm using Clevercloud to host my application and I want to store logs in an Elasticsearch index. I read the documentation here and I tried to create a drain as explained. I was able to create the drain but no data has been added in my Elasticsearch.
Somebody has an idea ?
I found the solution : I couldn't see datas because I was looking at the wrong ES index. Even if you specified an index in your URL, logs are in logstash format so by default it will create a new index per day named logstash-YYYY-MM-DD. The datas was in those indexes.
I'm new to elastic search however had to work with it. I have successfully set it up using logstash to connect it to my oracle database(one particular table). Now if new records are added to one of the tables in my oracle database(which I built the index on), what should be done?
I have thought of two solutions,
Re-build the indices by running the logstash conf file.
On insert into the table, also POST to elastic search.
The first solution is not working like it should. I mean that if 'users' is the table that I have updated with new records, then on re-building indices(for the 'users' table) in elastic search, the new records also should be reflected in the logstash get query.
The first should would help as a POC.
So, Any help is appreciated.
Thank you Val for pointing me in the right direction.
However, for the first brute-force solution it was about changing the document type in the logstash conf file.
{"document_type":"same_type"}
This must be consistent with the previously mentioned type. I had run it with different type, first time(Same_type). After adding new records, I used same_type. So, the elastic search as thrown an exception for multiple mapping rejection.
For further clarification, it looked up here.
Thank you guys.
I've already got my index (response_summary) created using logstash, which puts data into the index from a MySQL database.
My concern here is, how will I be able to update the index manually whenever a new set of records are being added to the database without deleting and recreating the index yet again.
Or is there a way that it can be done automatically, whenever a db change is done?
Any help could be appreciated.
No way with ES. There were the rivers in ES, but they were removed in ES 2.0. The alternative is the Logstash JDBC input plugin to automatically pickup changes based on a defined schedule.
For doing the same with files, you have the LS file input plugin which is tailing the files to pick up the new changes and, also, to keep track of where it left off in case LS is restarted.