Apache Beam run with Flink throws NoSuchMethodError - protocol-buffers

Question

I get a beam program jar by maven,and i want to run it with flink local.
when i run like this ,it is ok:
mvn exec:java -Dexec.mainClass=GroupbyTest -Dexec.args="--runner=FlinkRunner
--flinkMaster=localhost:6123
--filesToStage=target/beamTest-1.0-SNAPSHOT.jar"
but when i use flink run,there are something wrong with protobuf:
./bin/flink run /home/maqy/Documents/beam_samples/beamTest/target/beamTest-1.0-SNAPSHOT.jar --runner=FlinkRunner
and there are logs:
Using the result of 'hadoop classpath' to augment the Hadoop classpath: /usr/local/hadoop-2.7.5/etc/hadoop:/usr/local/hadoop-2.7.5/share/hadoop/common/lib/*:/usr/local/hadoop-2.7.5/share/hadoop/common/*:/usr/local/hadoop-2.7.5/share/hadoop/hdfs:/usr/local/hadoop-2.7.5/share/hadoop/hdfs/lib/*:/usr/local/hadoop-2.7.5/share/hadoop/hdfs/*:/usr/local/hadoop-2.7.5/share/hadoop/yarn/lib/*:/usr/local/hadoop-2.7.5/share/hadoop/yarn/*:/usr/local/hadoop-2.7.5/share/hadoop/mapreduce/lib/*:/usr/local/hadoop-2.7.5/share/hadoop/mapreduce/*:/usr/local/hadoop-2.7.5/contrib/capacity-scheduler/*.jar
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/maqy/%e4%b8%8b%e8%bd%bd/flink-1.4.0/lib/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.7.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Cluster configuration: Standalone cluster with JobManager at localhost/127.0.0.1:6123
Using address localhost:6123 to connect to JobManager.
JobManager web interface address http://localhost:8081
Starting execution of program
------------------------------------------------------------
The program finished with the following exception:
java.lang.NoSuchMethodError: com.google.protobuf.Descriptors$Descriptor.getOneofs()Ljava/util/List;
at com.google.protobuf.GeneratedMessageV3$FieldAccessorTable.<init>(GeneratedMessageV3.java:1707)
at com.google.protobuf.AnyProto.<clinit>(AnyProto.java:52)
at org.apache.beam.model.pipeline.v1.RunnerApi.<clinit>(RunnerApi.java:53271)
at org.apache.beam.model.pipeline.v1.RunnerApi$Components$TransformsDefaultEntryHolder.<clinit>(RunnerApi.java:448)
at org.apache.beam.model.pipeline.v1.RunnerApi$Components$Builder.internalGetTransforms(RunnerApi.java:1339)
at org.apache.beam.model.pipeline.v1.RunnerApi$Components$Builder.getTransformsOrDefault(RunnerApi.java:1404)
at org.apache.beam.runners.core.construction.SdkComponents.registerPTransform(SdkComponents.java:81)
at org.apache.beam.runners.core.construction.PipelineTranslation$1.visitPrimitiveTransform(PipelineTranslation.java:87)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:670)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:662)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:662)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311)
at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245)
at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458)
at org.apache.beam.runners.core.construction.PipelineTranslation.toProto(PipelineTranslation.java:59)
at org.apache.beam.runners.core.construction.PipelineTranslation.toProto(PipelineTranslation.java:53)
at org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.translate(FlinkPipelineExecutionEnvironment.java:91)
at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:110)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297)
at GroupbyTest.main(GroupbyTest.java:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:525)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:417)
at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:396)
at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:802)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:282)
at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1054)
at org.apache.flink.client.CliFrontend$1.call(CliFrontend.java:1101)
at org.apache.flink.client.CliFrontend$1.call(CliFrontend.java:1098)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1098)
I want to know how to fix it , thanks.

Answer 1

A NoSuchMethodError typically indicates a version problem. A class with the right package and name is found in the classpath (otherwise you'd see a ClassNotFoundException) but it lacks a certain method.
This usually happens if the classpath contains the wrong version of a dependency, in your case that's probably Google ProtoBuf.

Answer 2

I have solved this problem by using Beam 2.6.