I'm running an ELK stack and would like a strategy for automatically retiring logs older than a certain age... I have tried using elasticsearch-curator but it requires python 2.7 and I have python 2.6.6, and am anxious to upgrade python in case I break other packages.
Is there a similar product or does the elasticsearch api cater for such a requirement?
The older versions of curator will work with older versions of ElasticSearch ( I'm using curator version 3.1.0 with a 1.7.1 ES cluster.)
We started out using the Elasticsearch S3 archiving plugin but soon discovered certain limitations when wanting to restore data. We also experienced performance issues with the plugin, which tended to slow down the entire cluster. Since then, we have migrated to a new system in which we archive the data for us and our customers using our own code before indexing the data to Elasticsearch in a clear text format. This gives us all the flexibility we and our customers require.
You may be mistaking the dependency on a given version of the elasticsearch-py module for a version of Elasticsearch. Curator version 3.5.1 requires es-py 2.3, but works with any version of elasticsearch greater than 1.0.
Related
I have an elastic search cluster running 5.6. I plan to upgrade my cluster but i plan to do it by basically running a ES cluster 8.3 running in parallel and then moving data over to it.
The preferred way i think is to do snapshot and restore https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html
But I am confused about what exactly Snapshot version compatibility mean :
In the above does it mean that if i take a snapshot in elasticsearch 5.6, I cannot restore directly in version 8.3 ?? (which mean I have to first move to 6.x then to 7.x and finally to 8.x ) ??
The below index compatibility matrix, however says that a version in 5.x will work in 8.x ?
Am i missing something ? or can someone help me elaborate this?
So, the underlying problem is that data written in Lucene version N, can only be read with N+1. For Elasticsearch 5 to 8 the Lucene version was always 1 greater than the ES version (so 6 to 9).
That means, both for an upgrade or a restored snapshot: If your data was written with 5.x, you can only read / restore it with 6.x. For 7.x or 8.x you'll need to reindex the data. I would do a remote reindex straight from 5.x to 8.latest if possible: https://www.elastic.co/guide/en/elasticsearch/reference/current/reindex-upgrade-remote.html
There are some small caveats but they will probably not apply to you:
This doesn't apply to source only snapshots, but those always need a reindex, so that's not going to add any benefit for you.
8.3 added a feature to still read snapshots from 5.0 on but it is slower, doesn't support all features, and it is a commercial feature (platinum license if I'm not mistaken).
Depending on what kind of data it is: If it's aging out (like logs or metrics), maybe you don't have to migrate it to the new cluster?
I am getting security warnings because client 7.13.0 has issues. However, I use Bonsai with Heroku which is server 7.10.2 and that is incompatible with the 7.15.0 client.
I tried the following
heroku addons:create bonsai -a <app> --version=7.14.2
But it is still showing 7.10.2
Can I get Heroku to support the newer version of elastic search server?
Bonsai support here. The current list of our supported Elasticsearch versions is here. This list is the same whether you have a cluster with us directly or through Heroku. Trying to provision a version of Elasticsearch that's not on this list will get you the closest available version to what was requested.
To shade in some context about why we're still running 7.10.2, it's important to know that in January 2021, Elastic announced that version 7.11 and on would be licensed under the Server Side Public License (SSPL) and not Apache 2.0.
This change has pretty severe implications for hosted Elasticsearch providers like Bonsai, and has limited our ability to release new versions. We're working on some long term solutions, but the process has been slower than we'd like.
In any case, Elasticsearch 7.10.2 is the latest Elasticsearch version we offer as of December 2021 (check the first link for updates). We also support OpenSearch, which is an Apache 2-licensed fork of Elasticsearch, maintained by AWS.
As always, please don't hesitate to reach out to us at support#bonsai.io with questions or issues.
Not at the moment, at least via Bonsai:
Protip: Bonsai supports a command line flag for specifying which version of Elasticsearch you want to use. Note that we only support specific versions of Elasticsearch, so you can’t provision an arbitrary one. We have a list of available versions documented here.
That link leads to a page that lists support for 5.6.16, 6.5.4, and 7.10.2 for multi-tenant classes and 1.7.5 to 7.10.2 for single-tenant.
Note that this page is on the docs.bonsai.io domain, so it is Bonsai-specific, not Heroku-specific.
The official Elasticsearch addon appears to support newer versions, but it costs a minimum of $67 USD per month:
We support the two most recent major versions at all times
Searchbox might do what you want, and it has a free tier:
For specific version of Elasticsearch please use CLI command with version information as;
heroku addons:add searchbox:starter --es_version=2
Available versions are 0,1,2,5,6 and 7
It isn't clear what minor versions are used.
I've been reading the documentation for 2.4 and can see that ElasticSearch is a required module to upgrade - we'd been using a third party search module which we will continue to use for 2.4 too however you can't physically upgrade to 2.4 unless you have an active connection to ElasticSearch.
My question is: is there a way to upgrade to 2.4 without installing ElasticSearch on the server? I find it hard to believe that Adobe have decided to scrap the MYSQL search and replace it with one you have to pay and subscribe to.
Any advice is greatly appreciated!
Circumstances:
For 3 years there is an application, that uses tarantool (now it is 1.10.3), and once upon a time we've decided to move it to Kubernetes and replace old and ugly dockerfile, based on Jessie to the official image tarantool/tarantool:2.3.1. I don't know will it be all fine with the data.
In this case, I have two questions, and I will be really happy to read the correct answers from you:
Does tarantool 2.3.1-2-g92750c828 support recovering from snapshots, made by tarantool 1.10.3?
How can I surely load the snapshot data by the new version? It would be great to make it without restarting tarantool, because I have PVC like "emptyDir" that can't handle any data when pod restarts)
Yes, Tarantool 2.3 is compatible with Tarantool 1.10 in terms of binary protocol and snapshot format. If you can't simply run Tarantool 2.3 based on snapshot of 1.10, please file an issue - https://github.com/tarantool/tarantool/issues
I want to note several things: after upgrade to 2.3 it could be impossible to run it back on 1.10 (some system spaces/records incompatible will be created).
Sorry, I can't answer for this question for Kubernetes. But you can't simply stop and run again single instance. But upgrade without downtime is available for replication cluster - https://www.tarantool.io/en/doc/2.4/book/admin/upgrades/#upgrading-tarantool-in-a-replication-cluster
I want to upgrade elastic search version with updated mapping( i have renamed/added/deleted some field)
how can I migrate data , any reference apprecited
Thanks
There are multiple ways to achieve that. It depends on the amount of data the cluster holds, whether maintenance downtime is acceptable or not, and do you have backward compatibility issues with the fields names/types.
A simple (but a bit costly) way to upgrade would be to spin up a new cluster with version 6.0 and reindex the data from the old cluster to the new. another method can be to upgrade to latest 5.X version (5.6) and from there do a rolling restart. from https://www.elastic.co/guide/en/elastic-stack/6.0/upgrading-elastic-stack.html : "Rolling upgrades are supported when upgrading from Elasticsearch 5.6 to 6.0.1"