elasticsearch and camel integration - elasticsearch

Question

I am trying to integrate camel with elasticsearch.
In applicationContext.xml added the following
<route id="timer-to-console">
<from uri="timer://foo?fixedRate=true&period=10s"/>
<transform>
<simple>Hello Web Application, how are you?</simple>
</transform>
<to uri="stream:out"/>
<to uri="elasticsearch://local"/>
</route>
Then when I run
mvn jetty:run
I am getting the following
veryCounter=0, firedTime=Mon Apr 21 13:14:43 PDT 2014}
BodyType String
Body Hello Web Application, how are you?
]
Stacktrace
----------------------------------------------------------------------------------------
java.lang.IllegalArgumentException: operation is missing
at org.apache.camel.component.elasticsearch.ElasticsearchProducer.process(ElasticsearchProducer.java:54)
at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
My elasticsearch is running locally, I am using ES 1.1.1.
what do I need to specify for
elasticsearch://clusterName[?options]
Thanks,

Answer 1

From a quick glance at the Apache Camel Elasticsearch Component page they show the following example:
elasticsearch://local?operation=INDEX&indexName=twitter&indexType=tweet
This would INDEX (add) into an index named twitter with a type of tweet. You can use whatever values you want for the indexName and indexType.
Update: Looking at the Elasticsearch Camel Component documentation again... I think in order to use local as the server name in the elasticsearch connection, you would need to be running your local Elasticsearch instance with a clustername of local. By default Elasticsearch configuration (elasticsearch.yml) is setup to run with a clustername of elasticsearch

Answer 2

You have to use the cluster name specified in elasticsearch.yml, i.e. if it says cluster.name: elasticsearch123 then you have to route for example as follows:
from("file:data_json?noop=true&moveFailed=data_json/.error")
.convertBodyTo(byte[].class)
.to("elasticsearch://elasticsearch123?operation=INDEX&indexName=myindexname&indexType=mytypename");
Using local simply doesn't work for me. Unfortunately it doesn't even throw any error which makes it difficult to debug.
Also notice the conversion .convertBodyTo(byte[].class) - that's also essential, otherwise Camel converts the JSON files to the class Properties (it looks for a conversion from GenericFile to Map and it finds a fallback type converter to Properties). Of course, you could also convert to the other types supported by the component.
It's also worth mentioning that the camel-elasticsearch component of version 2.14.1 depends on org.elasticsearch:elasticsearch:1.0.0 which is quite an old version given how fast Elasticsearch moves (the current version is 1.4.4) and how often it likes to break compatibility. That said, the component seems to work with Elasticsearch 1.4.3.
Update: the current master branch on GitHub has Elasticsearch upgraded to version 1.4.2: https://github.com/apache/camel/blob/2470023e25b8814279cbadb3ebc8002bed6d8fc8/parent/pom.xml#L144
The parameter/clusterName local in fact means that it will look for an Elasticsearch instance started in the same JVM, from Elasticsearch's JavaDoc (NodeBuilder.local(boolean)):
A local node is a node that uses a local (JVM level) discovery and
transport. Other (local) nodes started within the same JVM (actually, class-loader) will be
discovered and communicated with. Nodes outside of the JVM will not be discovered.