When running my Beam pipeline locally it all works as expected but when trying to run it on the DataflowRunner I suddenly get the error below. Honestly I don't even know where to start evaluating this because the DataflowRunner seems to be a black box.
Jan 14, 2019 11:26:51 AM org.apache.beam.runners.dataflow.DataflowRunner fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 165 files. Enable logging at DEBUG level to see which files will be staged.
Exception in thread "main" java.lang.IncompatibleClassChangeError: Class org.apache.beam.model.pipeline.v1.RunnerApi$StandardPTransforms$Primitives does not implement the requested interface com.google.protobuf.ProtocolMessageEnum
at org.apache.beam.runners.core.construction.BeamUrns.getUrn(BeamUrns.java:27)
at org.apache.beam.runners.core.construction.PTransformTranslation.<clinit>(PTransformTranslation.java:58)
at org.apache.beam.runners.core.construction.UnconsumedReads$1.visitValue(UnconsumedReads.java:49)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:666)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311)
at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245)
at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458)
at org.apache.beam.runners.core.construction.UnconsumedReads.ensureAllReadsConsumed(UnconsumedReads.java:40)
at org.apache.beam.runners.dataflow.DataflowRunner.replaceTransforms(DataflowRunner.java:868)
at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:660)
at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:173)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
at (my code: pipe.run().waitUntilFinish();)
check the versions of beam etc and upgrade your dependencies where possible.
I had the same error and after seeing you get this error, I thought it must be a dependency conflict as it didn't exist before.
I'm using scio to deploy to dataflow and just referenced what they're using. https://github.com/spotify/scio/blob/v0.7.1/build.sbt
I updated guava and protobuf also.
I know you're using java, but try updating beam to 2.9.0 and maybe guava, protobuf...
Related
I find similar error on https://issues.apache.org/jira/browse/KYLIN-2511
env:
hadoop-2.7.1
hbase-1.3.2
apache-hive-2.1.1-bin
apache-kylin-1.6.0-hbase1.x-bin
I've tried copy all the hive libs to kylin, but get another ERROR.
org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoClassDefFoundError: org/apache/hadoop/hive/serde2/typeinfo/TypeInfo
The missing class should be in hive-exec-.jar; Check and debug the "bin/find-hive-dependency.sh" to see why it wasn't able to locate this jar from your server. You can manually add it to the "hive_exec_path" variable.
BTW, Kylin 1.6 is quite old, try to upgrade to a 2.x version.
Why you just try the method mentioned in https://issues.apache.org/jira/browse/KYLIN-2511. You'd better prepare the env according to the document of v16. It is better for using the latest version of Kylin. It has more feature and fixes some bugs.
So I'm trying to install the forge api (Minecraft 1.10.2) for developing a mod, and I'm having an issue with using ForgeGradle. Every time I run it with "bash gradlew setupDecompWorkspace", I get this error:
FAILURE: Build failed with an exception.
What went wrong:
Could not resolve all dependencies for configuration ':forgeGradleMcpData'.
Could not resolve de.oceanlabs.mcp:mcp:1.10.2.
Required by:
com.yourname.modid:forge-1:1.0
Could not resolve de.oceanlabs.mcp:mcp:1.10.2.
Could not get resource '(had to remove this link because too many links)'.
Could not GET '(had to remove this link because too many links)'.
org.apache.http.client.ClientProtocolException (no error message)
Could not resolve de.oceanlabs.mcp:mcp:1.10.2.
Could not get resource 'https://libraries.minecraft.net/de/oceanlabs/mcp/mcp/1.10.2/mcp-1.10.2.pom'.
Could not GET 'https://libraries.minecraft.net/de/oceanlabs/mcp/mcp/1.10.2/mcp-1.10.2.pom'. Received status code 403 from server: Forbidden
Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
I've found several people with the same issue, but they were posted back in (roughly) Minecraft 1.6-1.8. Does anyone have any idea about what I should do? I'm stumped and could use some help.
You need to set the mappings property in the minecraft closure in your build.gradle file so ForgeGradle knows which version of the MCP mappings to use.
For example, this will use the latest mappings (as of this posting):
minecraft {
mappings = "snapshot_20161002"
}
The snapshot mappings are built daily and the versions are in the form of snapshot_YYYYMMDD, you can see all the mappings versions here.
I had a similar issue with forge for 1.11. I couldn't use the latest mappings file for some reason. Keeping the mappings file at snapshot_29160518 worked for me.
See this commit for a 1.10.2 plugin build.gradle file:
https://github.com/ljsimin/MinecraftJoypadSplitscreenMod/commit/76b7c83bc35cbe11d6516b31f8e5e4f7ec74b99c
Following the official guide of Titan DB here, and trying to run the command:
graph = TitanFactory.open('conf/titan-cassandra-es.properties')
I got this error:
Backend shorthand unknown: conf/titan-cassandra-es.properties
Obviously, the reason is the incorrect path to the
titan-cassandra-es.properties
file. So I changed it to:
graph = TitanFactory.open('../conf/titan-cassandra-es.properties')
and got this error:
Encountered unregistered class ID: 141.
The error happens in the following version:
titan-0.5.4-hadoop2
On titan-1.0.0-hadoop2 instead of this error message I get this one:
Invalid import definition: 'com.thinkaurelius.titan.hadoop.MapReduceIndexManagement'; reason: startup failed: script14747941661821834264593.groovy: 1: unable to resolve class com.thinkaurelius.titan.hadoop.MapReduceIndexManagement # line 1, column 1. import com.thinkaurelius.titan.hadoop.MapReduceIndexManagement ^
1 error
And on titan-1.0.0-hadoop2 I get this one:
The input line is too long.
The syntax of the command is incorrect.
Does anyone know how to handle this issue?
It seems like you have not even managed to get Titan 1 to start up yet.
I do not believe Titan 1 has been deployed to support Windows out of the box. I.e. the downloadable package will not just work with windows.
Saying that I have managed to get Titan DB 1 to work on windows. To do so, all you have to do is install Cassandra 2.x on Windows. This guide may help you out. Start cassandra and enable thrift connections.
With that done you should be able to get Titan doing basic operations on windows. From there you may find dealing with you current errors easier.
Side Note: Windows support for Titan 0.5.x may be more substantial. So you could look into that as well.
I'm trying to start karaf 3.0.1 in a solaris box (without internet) but getting the following error:
karaf: Ignoring predefined value for KARAF_HOME
Could not resolve mvn:org.eclipse/org.eclipse.osgi/3.8.2.v20130124-134944
and in karaf.log:
Jun 30, 2014 12:21:09 PM org.apache.karaf.main.Main main
SEVERE: Could not launch framework
java.lang.RuntimeException: Could not resolve mvn:org.eclipse/org.eclipse.osgi/3.8.2.v20130124-134944
at org.apache.karaf.main.util.SimpleMavenResolver.resolve(SimpleMavenResolver.java:59)
at org.apache.karaf.main.Main.createClassLoader(Main.java:315)
at org.apache.karaf.main.Main.launch(Main.java:234)
at org.apache.karaf.main.Main.main(Main.java:171)
the bundles are well in place (system folder) and the org.ops4j.pax.url.mvn.cfg file states:
org.ops4j.pax.url.mvn.repositories=\
file:${karaf.home}/${karaf.default.repository}#id=system.repository, \
file:${karaf.data}/kar#id=kar.repository#multi\
http://repo1.maven.org/maven2#id=central,\
http://repository.springsource.com/maven/bundles/release#id=spring.ebr.release,\
http://repository.springsource.com/maven/bundles/external#id=spring.ebr.external
I've tried running the framework using the three methods (server, service, client) but nothing seems to be working.
my environment is:
KARAF_BASE=/export/home/mehdi/bin/karaf
KARAF_HOME=/export/home/mehdi/bin/karaf
KARAF_ETC=/export/home/mehdi/bin/karaf/etc
KARAF_DATA=/export/home/mehdi/bin/karaf/data
JAVA_HOME=/opt/temp/jre1.7.0_13
I googled a bit and found a workaround which says to add -h 127.0.0.1 to the client script, but still nothing.
Does anyone know about HPCC/HDFS connector.we are using both HPCC and HADOOP.There is one utility(HPCC/HDFS connector) developed by HPCC which allows HPCC cluster to acess HDFS data
i have installed the connector but when i run the program to acess data from hdfs it gives error as libhdfs.so.0 doesn't exist.
I tried to build libhdfs.so using command
ant compile-libhdfs -Dlibhdfs=1
its giving me error as
target "compile-libhdfs" does not exist in the project "hadoop"
i used one more command
ant compile-c++-libhdfs -Dlibhdfs=1
its giving error as
ivy-download:
[get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
[get] To: /home/hadoop/hadoop-0.20.203.0/ivy/ivy-2.1.0.jar
[get] Error getting http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
to /home/hadoop/hadoop-0.20.203.0/ivy/ivy-2.1.0.jar
BUILD FAILED java.net.ConnectException: Connection timed out
any suggestion will be a great help
Chhaya, you might not need to build libhdfs.so, depending on how you installed hadoop, you might already have it.
Check in HADOOP_LOCATION/c++/Linux-<arch>/lib/libhdfs.so, where HADOOP_LOCATION is your hadoop install location, and arch is the machine’s architecture (i386-32 or amd64-64).
Once you locate the lib, make sure the H2H connector is configured correctly (see page 4 here).
It's just a matter of updating the HADOOP_LOCATION var in the config file:
/opt/HPCCSystems/hdfsconnector.conf
good luck.