java.lang.NoSuchMethodError: com.google.common.collect.Sets$SetView.iterator()Lcom/google/common/collect/UnmodifiableIterator; - elasticsearch

Question

I've been trying to connect kafka to elasticsearch using kafka-connect api.Kafka version is 0.11.0.0.These are the steps I followed:
1.Buiding Elasticsearch Connector:
https://github.com/confluentinc/kafka-connect-elasticsearch.git
2.Build the connector
$ cd kafka-connect-elasticsearch
$ mvn clean package
3.Finally running the script:
$ bin/connect-standalone.sh config/connect-standalone.properties config/elasticsearch-connect.properties
It throws the following exception:
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.collect.Sets$SetView.iterator()Lcom/google/common/collect/UnmodifiableIterator;
at org.reflections.Reflections.expandSuperTypes(Reflections.java:380)
at org.reflections.Reflections.<init>(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:221)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:198)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:159)
at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:47)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:68)
Can't understand what's going wrong.

Answer 1

From experience, this error means that you are bringing an older version of guava earlier in your classpath. Connect worker requires guava >= 20 for org.reflections to work correctly.
kafka-connect-elasticsearch or any other connector that brings with it guava 18.0 or older will prohibit the worker from starting up. This error message means that the older guava jar was encountered first in the classpath.
Two solutions:
Indeed, as Hans Jespersen mentions, using classloading isolation by setting your plugin.path in Connect worker's configuration, will allow the connector to work as-is without interfering with the Connect framework.
If adding the connector to the CLASSPATH is your only option, make sure it's added after Kafka Connect's dependencies, so that the most recent guava will be picked up.

Answer 2

This appears to describe the answer for your problem https://github.com/confluentinc/kafka-connect-elasticsearch/issues/104
It's a little confusing, but after you build the connector there are a
number of things in the target directory. The
kafka-connect-elasticsearch-.jar is only the JAR file with
the connector code, but that doesn't include all the libraries. One of
those directories in the target directory, namely
target/kafka-connect-elasticsearch-*-development/share/java/kafka-connect-elasticsearch/,
does contain all the libraries. Add this directory to the Kafka
Connect worker's classpath, or copy all of those JAR files into a
directory that is already on the classpath.

Answer 3

For future readers, I have a similar issue w/ no usage for kafka. I spent a lot of hours to try to figure out the issue, and finally, I have a pattern.
In my project, I used ro.isdc.wro4j:wro4j-extensions:1.8.0, and org.reflections:reflections:0.9.11
<dependency>
<groupId>ro.isdc.wro4j</groupId>
<artifactId>wro4j-extensions</artifactId>
<version>1.8.0</version>
</dependency>
<dependency>
<groupId>org.reflections</groupId>
<artifactId>reflections</artifactId>
<version>0.9.11</version>
</dependency>
The conflict happened becuase ro.isdc.wro4j:wro4j-extensions:1.8.0 used com.google.javascript:closure-compiler:jar:v20160315, I solved it by exclude com.google.javascript:closure-compiler:jar:v20160315 from the pom.
<dependency>
<groupId>ro.isdc.wro4j</groupId>
<artifactId>wro4j-extensions</artifactId>
<version>1.8.0</version>
<exclusions>
<exclusion>
<groupId>com.google.javascript</groupId>
<artifactId>closure-compiler</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.reflections</groupId>
<artifactId>reflections</artifactId>
<version>0.9.11</version>
</dependency>
However, the problem is a build-dependent which we don't know which jar may cause the conflict. In this case, you need to know the location of the jar by running something like the following:
CodeSource source = com.google.common.collect.Sets.class.getProtectionDomain().getCodeSource();
if(source != null)
logger.warn(source.getLocation().toString());
and see what is the output. In my case, the output is
file:/tmp/jetty-0.0.0.0-8087-ROOT.war-_-any-1541284168668078443.dir/webapp/WEB-INF/lib/closure-compiler-v20160315.jar
Hopefully, the answer will help you to find a way to solve the issue,
Mughrabi

Answer 4

I ran into this error recently (Jan 2020) on my localhost (macOS Catalina) and was able to resolve this by updating my .zprofile file (zsh shell). If you have bash shell, you can make changes to .bash_profile or .bashrc file.
• Error was caused because my jvm.classpath was including dependencies of guava verison < 20. My .zprofile file had exports to hadoop-3.1.1 and apache-hive-3.1.1 which was the issue for error.
• I am using kafka_2.12-2.0.0. So, I just commented the hadoop-3.1.1 & apache-hive-3.1.1in my .zprofile and ran my twitter kafka connector successfully.
• I was able to trace this error by following the comments in this thread by #Konstantine Karantasis.
Hope this is helpful.