Apache Camel Integration with Elasticsearch - elasticsearch

I'm working on a project using Apache Camel and Elasticsearch and I was wondering which version of Elasticsearch does Camel support?
My pom.xml looks like this:
<dependencies>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-core</artifactId>
<version>2.18.2</version>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-elasticsearch</artifactId>
<version>2.18.2</version>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-xmljson</artifactId>
<version>2.18.2</version>
</dependency>
<dependency>
<groupId>xom</groupId>
<artifactId>xom</artifactId>
<version>1.2.5</version>
</dependency>
But when I want to route a file to elasticsearch, I've got the following error:
java.lang.IllegalStateException: Received message from unsupported version: [2.0.0] minimal compatible version is: [5.0.0]
I found that this exception is due to a node or a TransportClient using an old version. So I try to add the elasticsearch dependency :
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>5.1.2</version>
</dependency>
But it gives me a new error:
Exception in thread "main" java.lang.NoClassDefFoundError: org/elasticsearch/action/WriteConsistencyLevel
So I'm wondering.. Which version of ES can I use with Apache Camel?
The code for trying to send data to elasticsearch:
XmlJsonDataFormat xmlJsonFormat = new XmlJsonDataFormat();
from("file://C:/Projects/?fileName=data.xml&charset=utf-8")
.marshal(xmlJsonFormat)
.to("elasticsearch://clusterES?transportAddresses=127.0.0.1:9300&operation=BULK_INDEX&indexName=xml&indexType=account");

I don't think you need to add any other pom except the camel-elasticsearch. It seems more likely that you have a TransportClient running on an older version. You need to find it and upgrade the TransportClient.
https://www.elastic.co/guide/en/elasticsearch/guide/current/_transport_client_versus_node_client.html
https://discuss.elastic.co/t/received-message-from-unsupported-version-2-0-0-minimal-compatible-version-is-5-0-0/64708

Related

java.lang.ClassNotFoundException: io.netty.handler.logging.ByteBufFormat

I'm on integrating a third-party library into our application. For that I have added all the dependencies, however facing below error stack on application run.
Error stack-trace:
`
Caused by: java.lang.ClassNotFoundException: io.netty.handler.logging.ByteBufFormat
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 31 common frames omitted
`
Maven dependencies:
`
<dependency>
<groupId>com.affinda.api</groupId>
<artifactId>affinda-api-client</artifactId>
<version>0.4.2</version>
</dependency>
<dependency>
<groupId>com.microsoft.rest</groupId>
<artifactId>client-runtime</artifactId>
<version>1.7.14</version>
</dependency>
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>azure-client-runtime</artifactId>
<version>1.7.14</version>
</dependency>
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>azure-client-authentication</artifactId>
<version>1.7.14</version>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-identity</artifactId>
<version>1.5.0</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
<version>4.1.84.Final</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-handler</artifactId>
<version>4.1.84.Final</version>
</dependency>
`
I did check that ByteBufFormat is in netty-handler library from the docs and did check the dependency tree but haven't got any clue.
It might be too late, however I had the same problem.
The cause is that another dependency is overriding the netty version in your final pom. In my case it was spring boot dependencies (2.2.0-RELEASE) which was overriding netty to version 4.1.42Final instead of needed 4.1.86Final.
If you want to check who is responsible, you can run maven with goal help:effective-pom and search for the effective netty version management owner.
In order to solve the issue you can just specify the netty version among maven properties:
<properties>
.
.
<netty.version>4.1.86.Final</netty.version>
</properties>
In this way you will put back the expected netty version and everything will work like a charm.
Be careful to check that the other library is still working as well.

What are springboot dependencies needed for ElasticSearch 5.6.7?

Please share me the dependencies information about using ES 5.6.7 with springboot ?
As i have ES 2.4 and it is working now, but after upgrade failing to work.
You can use :
<!-- https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch -->
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>5.6.6</version>
</dependency>
You can check here for further more dependency

Elasticsearch 5 connector in Apache Flink 1.3

By reading the documentation I understood that with Apache Flink 1.3 I should be able to use Elasticsearch 5.x.
However, in my pom.xml :
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-elasticsearch5_2.10</artifactId>
<version>1.3.0</version>
</dependency>
I got this :
Dependency
"org.apache.flink:flink-connector-elasticsearch5_2.10:1.3.0" not found
Any idea why this dependency is unfound ?
This was a bug in the 1.3.0 release and is being fixed for 1.3.1 (which is due very soon). See the mailing list for more details.
Use the following in the pom.xml
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-elasticsearch5_2.10</artifactId>
<version>1.3.1</version>
</dependency>

java.lang.NoClassDefFoundError: com/google/common/util/concurrent/FutureFallback

I'm seeing the following error :
java.lang.NoClassDefFoundError: com/google/common/util/concurrent/FutureFallback
when the cluster.connect() is called :
String hosts = CassandraClientUtil.getHost();
String localDC = CassandraClientUtil.getLocalDC();
Cluster cluster = null;
if (StringUtils.isNotEmpty(localDC))
{
cluster = Cluster.builder().addContactPoints(hosts.split(","))
.withCredentials(CassandraCopsComponentLogger.USER_NAME, CassandraCopsComponentLogger.AUTH_CODE)
.withQueryOptions(new QueryOptions().setConsistencyLevel(ConsistencyLevel.LOCAL_ONE))
.withLoadBalancingPolicy(new TokenAwarePolicy(DCAwareRoundRobinPolicy.builder().withLocalDc(localDC).build())).build();
}
else
{
cluster = Cluster.builder().addContactPoints(hosts.split(","))
.withCredentials(CassandraCopsComponentLogger.USER_NAME, CassandraCopsComponentLogger.AUTH_CODE)
.withQueryOptions(new QueryOptions().setConsistencyLevel(ConsistencyLevel.LOCAL_ONE)).build();
}
Session session = cluster.connect();
CassandraCopsComponentLogger.mappingManager = new MappingManager(session);
The pom.xml has the following dependencies :
<dependencies>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>16.0.1</version>
</dependency>
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>2.1.9</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
<version>3.9.0.Final</version>
</dependency>
<dependency>
<groupId>com.codahale.metrics</groupId>
<artifactId>metrics-core</artifactId>
<version>3.0.2</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.5</version>
</dependency>
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-mapping</artifactId>
<version>2.1.9</version>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.3.1</version>
</dependency>
</dependencies>
I saw a post on stackoverflow here
where they recommended to upgrade the guava version to 16.0.1 but that did not help me solve my problem. Some directions from here will be really helpful as I'm new to cassandra. To add more background this thing works as a standalone project, when I include this project as a maven dependency to some other project it raises this runtime error.
com.google.common.util.concurrent.FutureFallback is deprecated in Guava 19.0 and removed since Guava 20.0.
Use Guava 19.0 and do not use Guava 20.0 or greater, until you upgrade the Cassandra driver.
I updated the Cassandra driver version to latest available and it should fix the issue.
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>3.5.0</version>
</dependency>
Don't add external guava version . whatever datastax-cassandra-core using only you can put that version . otherwise don't need of that .
If anyone like me didn't know, that there is a new version (4.x) out there with a new and different group id, take a look at the quickstart. This new version still uses Guava however it's shaded.
The driver now requires Java 8. It does not depend on Guava anymore (we still use it internally but it's shaded).
More information can be found in the upgrade guide.

Maven: 1.7.4 openejb dependancy fails

Wanted to make use of openejb on top of tomcat v7 using maven instead of installing tomee. Referring to Apache documentation, 3 dependencies have to be added to the maven project.
<dependency>
<groupId>org.apache.openejb</groupId>
<artifactId>javaee-api</artifactId>
<version>6.0-6</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.openejb</groupId>
<artifactId>openejb-core</artifactId>
<version>4.7.4</version>
</dependency>
<dependency>
<groupId>org.apache.openejb</groupId>
<artifactId>tomee</artifactId>
<version>1.7.4</version>
</dependency>
but the last depency generates following error: Missing artifact org.apache.openejb:tomee:jar:1.7.4
The correct artifactId being visible in:
http://mvnrepository.com/artifact/org.apache.openejb/apache-tomee/1.7.4
is apache-tomee and not tomee
so, replace the last depency by the following one and the problem will be solved:
<dependency>
<groupId>org.apache.openejb</groupId>
<artifactId>apache-tomee</artifactId>
<version>1.7.4</version>
</dependency>

Resources