Goodmorning everyone. I am trying to Flink connector Elasticsearch 5.2.1 and I have problems with the authentication XPACK
Using a different transport clients is currently (March 2017, Flink 1.2) not supported in Flink.
However, I've filed a JIRA to add the feature: FLINK-6065
Make TransportClient for ES5 pluggable
Until this has been implemented into Flink, I recommend overriding the ElasticsearchSink and using a different call bridge calling the PreBuiltXPackTransportClient.
Related
In my flink job, I am trying to use elasticsearch7 connector. I could not find a way to communicate over SSL when using Elasticsearch7SinkBuilder Am I missing something?
Also there is an deprecated class ElasticsearchSink.Builder where it has RestClientFactory which I can use to setup https connection but since this class is deprecated thus wondering if same is possible with Elasticsearch7SinkBuilder.
Also since there is no connector for elasticsearch 8 yet, Is it possible to use any of the above client with elastic 8?
Note: Earlier it used to be elastic connector compatibility matrix in Flink doc but I could not find it any more in lasted version of doc.
SSL connections to Elasticsearch7 are currently not supported and are tracked under https://issues.apache.org/jira/browse/FLINK-27054
An Elasticsearch8 connector is being tracked under https://issues.apache.org/jira/browse/FLINK-26088. You can't use the ES7 connector to connect to an ES8 cluster.
Small question regarding Spring Boot and Cassandra please.
I am currently using Spring Boot 2.5.1, with its associated Cassandra connector: spring-boot-starter-data-cassandra also at 2.5.1, which I think in turn utilize the driver 4.11.1
c.d.o.d.i.core.DefaultMavenCoordinates : DataStax Java driver for Apache Cassandra(R) (com.datastax.oss:java-driver-core) version 4.11.1
Upon application start, I am observing a strange log which I do not understand:
c.d.o.d.i.core.session.DefaultSession : [s0] Negotiated protocol version V4 for the initial contact point, but cluster seems to support V5, keeping the negotiated version
It seems the cluster supports V5 (whatever that is) but my app is "doing negotiation" with V4.
May I ask how I can configure my application, to leverage this cassandra V5 please?
Thank you
The native protocol defines the format of the messages between the driver the Cassandra cluster over TCP connections. Java driver 4 supports protocol versions v3 (C* 2.1), v4 (C* 2.2, 3.x) and v5 (C* 4.0).
Since Cassandra 4.0 is not released yet, native protocol v5 support is still in beta so the Java driver automatically negotiates to protocol v4 (recommended).
You can set ProtocolVersion in CassandraClusterFactoryBean but it isn't something you need to worry about since the log entry you posted is informational only. Cheers!
We have an application that uses spring data Elasticsearch (version 3.1.20.RELEASE) with transport client to connect to elastic search version 6.1.2.
Now we have to use the same application to connect to elastic search version 6.0.1 which is AWS-managed ES. The problem is, it doesn't seem to expose transport ports (9300) for clients.
If we move to a higher version of spring data Elasticsearch, it doesn't seem to support the elastic search cluster 6.0.x, and the current and lower versions of spring data Elasticsearch don't seem to support the REST clients.
We cannot upgrade our ES cluster version. So, either we have to find a way to connect to AWS with the transport client, or we have to make our application compatible with rest client. How can we solve this?
AWS does not expose the transport port / protocol. So you must use the REST protocol. And for Spring Data Elasticsearch that means version 3.2 at least. But this needs Elasticsearch 6.8. So the only way to use Spring Data Elasticsearch is to upgrade your ES cluster.
The other solution is to implement access the the cluster with Elasticsearch's RestClient and RestHighlevelClient and not using Spring data Elasticsearch
I am curious to know which client should I use for elasticSearch using java API. There are multiples clients like Jest, Transport, ElasticSearch clients. Also I have to perform CRUD operations on ES.
You should use the java high level rest client as its an official Elasticsearch java client(that supports all documents CRUD operations).
JEST is not official client and not available for latest ES version(not even 7.0 while 7.8 is released so not keeping pace with ES versions).
Transport client is used by high level client and is getting deprecated mention in this official doc.
Please read this thread for more info on all these clients and how they work internally.
What is the best practice using ElasticSearch from Java?
For instance one can easily find documentation and examples for delete-by-query functionality using REST API.
This is not the case for Java Transport Client.
Where can I find usage examples for Java Transport Client ?
Does Java Transport Client cover whole ElasticSearch functionality like HTTP client via REST API?
The best practice of using Elasticsearch from Java: Follow This
Nexts:
You can follow the library : JEST
Yes, in maximum time, Java Transport Client cover whole ElasticSearch functionality like HTTP client via REST API
To complete #sunkuet02 answer:
As mentioned in documentation, the TransportClient is the preferred way if you're using java (performance, serialization, ...).
But the jar is quite heavy (with dependencies) and it requires that you use the same versions between client and server to work.
If you want a very lightweight client, there is a new Java REST client in Elastic 5.x.
I don't know the differences with Jest, but it's compatible with last Elastic 5.x versions (but not with previous versions) and it's developed by the Elastic team.
So it might be a good option to consider, according to your needs.