Elasticsearch 5 connector in Apache Flink 1.4 - maven

Question

I'm a newbie to Flink, I'm trying to play with ElasticSearch connector but without success.
I installed ElasticSearch 5.5, Kibana, and Flink 1.4 on Docker.
Here is my code:
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients_2.11</artifactId>
<version>1.4.0</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java</artifactId>
<version>1.4.0</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_2.11</artifactId>
<version>1.4.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-rabbitmq_2.11</artifactId>
<version>1.4.0</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-elasticsearch5_2.11</artifactId>
<version>1.4.0</version>
</dependency>
</dependencies>
while java code is:
DataStream<String> consumer = .....
consumer.print();
Map<String, String> config = new HashMap<>();
config.put("cluster.name", "docker-cluster");
config.put("bulk.flush.max.actions", "1");
List<InetSocketAddress> transports = new ArrayList<>();
transports.add(new InetSocketAddress("127.0.0.1", 9300));
consumer.addSink(new ElasticsearchSink<>(config, transports, new ElasticsearchSinkFunction<String>() {
public IndexRequest createIndexRequest(String element) {
Map<String, String> json = new HashMap<>();
json.put("id", "2");
return Requests.indexRequest()
.index("signinindex").type("signin").source(json);
}
public void process(String element, RuntimeContext ctx, RequestIndexer indexer) {
indexer.add(createIndexRequest(element));
}
}));
env.execute();
But when I try to submit a new job in IDE, it seems to stay waiting, so I'm wondering if there are some problems with ElasticSearch 5.5 which conflicts with Flink.
Thanks a lot.
[UPDATE]: I made some tests with ES 5.1.2 and now when submitting my jar I receive the following exception (I tried to upgrade Flink to 1.4.1 with all connectors too):
java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: Could not run the jar.
at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleJsonRequest$0(JarRunHandler.java:90)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flink.util.FlinkException: Could not run the jar.
... 9 more
Caused by: java.lang.RuntimeException: Could not look up the main(String[]) method from the class ProtoSignInMap: org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSinkFunction
at org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:495)
at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:216)
at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:147)
at org.apache.flink.runtime.webmonitor.handlers.JarActionHandler.getJobGraphAndClassLoader(JarActionHandler.java:83)
at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleJsonRequest$0(JarRunHandler.java:69)
... 8 more
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSinkFunction
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:489)
... 12 more
Caused by: java.lang.ClassNotFoundException: org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkFunction
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 18 more
Obviously the main method exists.

Answer 1

Flink currently supports Elasticsearch up to version 5.1.x. For more info see: https://issues.apache.org/jira/browse/FLINK-7386