I try to Configuring Spring XD to use Hadoop (horthonworks) but when I excute this line In a terminal "./xd-singlenode --hadoopDistro hadoop11" ,I get error 'hadoop11' is not a valid value for option --hadoopDistro Possible values are [cdh5, hdp22, phd21, hadoop27, phd30]
The error message is pretty clear, the Spring Xd version your are trying to use is wrong. For horthonworks configuration you should use hdp22.
Regards
Related
I've been trying to read from a .csv file on many ways, utilizing SparkContext object. I found it possible through scala.io.Source.fromFile function, but I want to use spark object. Everytime I run function textfile for org.apache.spark.SparkContext I get the same error:
scala> sparkSession.read.csv("file://C:\\Users\\184229\\Desktop\\bigdata.csv")
21/12/29 16:47:32 WARN streaming.FileStreamSink: Error while looking for metadata directory.
java.lang.UnsupportedOperationException: Not implemented by the DistributedFileSystem FileSystem implementation
.....
As it's mentioned in the title I run the code on Windows in IntelliJ
[Edit]
In build.sbt have no redundant or overlapped dependencies. I use hadoop-tools, spark-sql and hadoop-xz.
Have you tried to run your spark-shell using local mode?
spark-shell --master=local
Also pay attention to not use both Hadoop-code and Hadoop-commons as a dependencies since you may have conflicting jars issues.
I've found the solution, precisely one of my colleague did that.
In dependencies build.sbt I changed hadoop-tools to hadoop-commons and it worked out.
I'm using confluent so I've installed dibezium connectors according to confluent docs using confluent-hub in connect.properties I do have entry
plugin.path=/usr/share/java,/opt/confluent-6.0.0/share/confluent-hub-components
I need to use io.debezium.transforms.ContentBasedRouter https://debezium.io/documentation/reference/1.3/configuration/content-based-routing.html
so according to debezium doc I've downloaded debezium-scripting-1.3.1.Final.jar
and put it into
/opt/confluent-6.0.0/share/confluent-hub-components/ and copied it into
/opt/confluent-6.0.0/share/confluent-hub-components/debezium-debezium-connector-sqlserver/lib directories
here the entries in my mysql_src.json connector
"transforms": "unwrap,route",
"transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
"transforms.unwrap.add.fields": "source.snapshot",
"transforms.route.type": "io.debezium.transforms.ContentBasedRouter",
"transforms.route.language": "jsr223.groovy",
"transforms.route.topic.expression": "value.__source_snapshot == 'false' ? 'test'"
when I'm trying to configure/load this connector I'm getting following error message
[2020-12-15 22:18:45,351] ERROR [Worker clientId=connect-1, groupId=connect-cluster] Failed to reconfigure connector's tasks, retrying after backoff: (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1369)
java.lang.NoClassDefFoundError: io/debezium/DebeziumException
Any suggestions how to fix this problem ?
According the docs, you need to additionally obtain a JSR-223 script engine implementation and add its contents to the Debezium plug-in directories of your Kafka Connect environment, since:
Debezium does not come with any implementations of the JSR 223 API. To use an expression language with Debezium, you must download the JSR 223 script engine implementation for the language, and add to your Debezium connector plug-in directories, along any other JAR files used by the language implementation.
I am not sure that configuration is correct but I passed first configuration problem (I hope) I'm facing another problem now which I will describe in different question.
I am not sure what was wrong, I did following
Clean up zookeeper directories
Clean up kafka directories
Run kafka in distributed mode using command line start/stop scripts (not using confluent cli)
this solved java.lang.NoClassDefFoundError: io/debezium/DebeziumException
error
In my old setup for Quarkus 1.3.2.Final
I have this in my property file
quarkus.vault.url=${vault_path}
quarkus.vault.tls.use-kubernetes-ca-cert=true
quarkus.vault.authentication.kubernetes.role=${someVaultRole}
quarkus.vault.tls.ca-cert=${someTlsCertLocation}
quarkus.vault.kv-secret-engine-mount-path=${someSecretEngingPath}
when i tried to access the secrets using VaultKVSecretEngine.readSecret(path), it gives me the secrets for that path
But when I try to migrate to Quarkus 1.4.2.Final, I got this error
"io.quarkus.vault.runtime.client.VaultClientException code=403 body={\"errors\":[\"1 error occurred:\\n\\t* permission denied\\n\\n\"]}\
when i tried to call VaultKVSecretEngine.readSecret(path) with my old setup.
Question is, what are the configs in my application.properties that i missed for quarkus 1.4.2.final vault access?
since by default, quarkus 1.4.2.Final uses kv-secret-engine-version 2, it is needed to specify the kv-secret-engine-version to version 1 when migrating from quarkus 1.3.2.Final to 1.4.2.Final since 1.3.2.Final uses kv-secret-engine-version 1
I find similar error on https://issues.apache.org/jira/browse/KYLIN-2511
env:
hadoop-2.7.1
hbase-1.3.2
apache-hive-2.1.1-bin
apache-kylin-1.6.0-hbase1.x-bin
I've tried copy all the hive libs to kylin, but get another ERROR.
org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoClassDefFoundError: org/apache/hadoop/hive/serde2/typeinfo/TypeInfo
The missing class should be in hive-exec-.jar; Check and debug the "bin/find-hive-dependency.sh" to see why it wasn't able to locate this jar from your server. You can manually add it to the "hive_exec_path" variable.
BTW, Kylin 1.6 is quite old, try to upgrade to a 2.x version.
Why you just try the method mentioned in https://issues.apache.org/jira/browse/KYLIN-2511. You'd better prepare the env according to the document of v16. It is better for using the latest version of Kylin. It has more feature and fixes some bugs.
I am follow the guide for setting up spring boot with the following link.
http://docs.spring.io/spring-boot/docs/1.4.1.RELEASE/reference/htmlsingle/#getting-started-installing-the-cli
section 10.2.2
when I type $ spring --version
I receive the error below.
/cygdrive/c/Users/Jesse/Documents/.sdkman/candidates/springboot/current/bin/spring: line 83: [: C:\Program: binary operator expected
Error: Could not find or load main class org.springframework.boot.loader.JarLauncher
You need to set the SPRING_HOME variable.
After setting, SPRING_HOME was not resolving correctly for me even though it was set in windows as a user and a system variable, and was also visible when running export via Git Bash. I ended up replacing the last line in my spring.sh file, essentialy forcing the classpath for the java command:
"${JAVA_HOME}/bin/java" ${JAVA_OPTS} -cp "/drive_letter/dir/to/spring/spring-x.x.x.RELEASE/lib/spring-boot-cli-x.x.x.RELEASE.jar" org.springframework.boot.loader.JarLauncher "$#"