I have installed SPRING-XD version 1.1.0 on a Centos machine. Using xd-singlenode I want to connect it to a SQL Server database via jdbc source and put the data into file.
I created some streams as follows:
1)xd:>stream create connectiontest --definition "jdbc --url=jdbc:sqlserver://sqlserverhost:1433/SampleDatabase --username=sample --password=***** --query= 'SELECT * FROM schema.tablename' |file" --deploy
2)xd:>stream create connectiontest --definition "jdbc --connectionProperties=jdbc:sqlserver://sqlserverhost:1433/SampleDatabase --username=sample --password=***** --initSQL= 'SELECT * FROM schema.tablename' |file" --deploy
Everytime I deploy the stream it gives the following error:
Command failed org.springframework.xd.rest.client.impl.SpringXDException: Multiple top level module resources found :file [/opt/pivotal/spring-xd-1.1.0.RELEASE/xd/config/jms-hornetq.properties],file [/opt/pivotal/spring-xd-1.1.0.RELEASE/xd/config/hadoop.properties],file [/opt/pivotal/spring-xd-1.1.0.RELEASE/xd/config/xd-admin-logger.properties],file [/opt/pivotal/spring-xd-1.1.0.RELEASE/xd/config/xd-singlenode-logger.properties],file [/opt/pivotal/spring-xd-1.1.0.RELEASE/xd/config/xd-container-logger.properties],file [/opt/pivotal/spring-xd-1.1.0.RELEASE/xd/config/jms-activemq.properties],file [/opt/pivotal/spring-xd-1.1.0.RELEASE/xd/config/httpSSL.properties]
Earlier I set springxd_home pointing to my springxd directory. After removing the path it is working fine now.
Thanks for the support.
Related
Similar to https://answers.sap.com/questions/12675326/sap-dbtech-jdbc-708-receive-of-connect-fail.html
Connection to SAP Hana Cloud fails with the error "RTEException: -708 Receive of connect failed."
Steps to reproduce:
java -version
OpenJDK Runtime Environment (build 11.0.15+10-Ubuntu-0ubuntu0.20.04.1)
java -jar ngdbc.jar -V
package package com.sap.db.jdbc, Java Platform API Specification, version 1.4, SAP HANA JDBC Driver, SAP SE, 1.120.05-8c23c50e159e9883edab0e2ebdd4e02c5919cd08
java -jar ngdbc.jar -u DBADMIN,PASSWORD -n BIG-IDENTIFIER.hana.trial-us10.hanacloud.ondemand.com:443 -d test -o encrypt=true -o validatecertificate=false
(the certificate is imported, and the error occurs even without the parameters encrypt and validatecertificate)
Contents of trace log:
ClassLoader: jdk.internal.loader.ClassLoaders$AppClassLoader#55054057
Process-ID: 320850
package package com.sap.db.jdbc, Java Platform API Specification, version 1.4, SAP HANA JDBC Driver, SAP SE, 1.120.05-8c23c50e159e9883edab0e2ebdd4e02c5919cd08 on Java 11.0.15
---- Thread 1eb44e46 main Timestamp: 2022-09-29 09:50:03.162
new Connection 'jdbc:sap://BIG-IDENTIFIER.hana.trial-us10.hanacloud.ondemand.com:443'
password=****
databaseName=test
host=BIG-IDENTIFIER....
options=
cmd=Select top 1 42 as "connect test" fro...
user=DBADMIN
HOSTLIST: [BIG-IDENTIFIER.hana.trial-us10.hanacloud.ondemand.com:443,]
new RTEException: -708 Receive of connect failed.
whereAmIjava.lang.Throwable
at com.sap.db.util.Tracer.whereAmI(Tracer.java:280)
at com.sap.db.rte.comm.RTEException.(RTEException.java:51)
at com.sap.db.rte.comm.BasicSocketComm.receiveInfoRequest(BasicSocketComm.java:587)
at com.sap.db.rte.comm.BasicSocketComm.doInfoRequest(BasicSocketComm.java:84)
at com.sap.db.rte.comm.BasicSocketComm.connectDB(BasicSocketComm.java:187)
at com.sap.db.rte.comm.SocketComm$1.open(SocketComm.java:47)
at com.sap.db.jdbc.topology.Topology.getSession(Topology.java:88)
at com.sap.db.jdbc.Driver.openByURL(Driver.java:1216)
at com.sap.db.jdbc.Driver.connect(Driver.java:313)
at com.sap.db.jdbc.Driver.main(Driver.java:858)
using null
=> FAILED
any thoughts on the why and how to solve it?
Since this is HANA Cloud Trial, I don't think, that you should use -d to specify a tenant database name. Please try to remove this parameter as you may be trying to connect to a non-existing tenant database. You can also refer to this tutorial.
Second thing to consider is the allowlist for IP addresses. Per default connections are blocked if you have not added your client ip to the allowlist. More details can be found in this blog.
Last but not least, please check if your are using the latest version of ngdb.jar. It can be obtained on this site.
I want to create hive table on top of phoenix table in emr.
I am facing a NoClassDefFoundError: org.apache.hadoop.hbase.security.SecurityInfo
What I have done so far:
I followed the instructions from https://phoenix.apache.org/hive_storage_handler.html and added phoenix-hive-5.0.0-HBase-2.0.jar to hive-env.sh as well as in hive-site.xml .
Restarted the hive service systemctl restart hive-server2.service
Restarted the metastore systemctl restart hive-hcatalog-server.service
Executed create table command from hue:
create external table ext_table (
i1 int,
s1 string,
f1 float,
d1 decimal
)
STORED BY 'org.apache.phoenix.hive.PhoenixStorageHandler'
TBLPROPERTIES (
"phoenix.table.name" = "ext_table",
"phoenix.zookeeper.quorum" = "localhost",
"phoenix.zookeeper.znode.parent" = "/hbase",
"phoenix.zookeeper.client.port" = "2181",
"phoenix.rowkeys" = "i1",
"phoenix.column.mapping" = "i1:i1, s1:s1, f1:f1, d1:d1"
);
Got an exception: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.security.SecurityInfo)
I am using emr-6.1.0
HBase 2.2.5
Phoenix 5.0.0
Hive 3.1.2
Anybody has an idea what can be the issue?
Update
I followed the advice from #leftjoin and used ADD JAR from hue to add phoenix-hive jar to classpath. Then I faced jar compatibility issue caused by phoenix hive connector that I use:
phoenix-hive-5.0.0-HBase-2.0.jar.
The newer versions of phoenix connectors are not archived into single bundle that could be downloaded from phoenix website . Instead
the connectors are located now in github repo.
I built the new phoenix-hive connector (versions: Phoenix->5.1.0, Hive->3.1.2, Hbase->2.2) and used it to create the Hive table.
As a result I got another exception, which I am not able to fix:
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org/apache/phoenix/compat/hbase/CompatSteppingSplitPolicy
I think it is still somehow connected to dependency issues. But no clue what is exactly.
As a workaround put jar into hdfs and execute ADD JAR command before create table and query:
ADD JAR hdfs://path/to/your/jar/phoenix-hive-5.0.0-HBase-2.0.jar;
connect-standalone.properties
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
bootstrap.servers=10.33.62.20:9092,10.33.62.110:9092,10.33.62.200:9092
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
offset.storage.file.filename=/tmp/connect.offsets
offset.flush.interval.ms=10000
plugin.path=/grid/1/mukul/confluent-5.0.0/share/java
source-sqlite.properties
name=test-source-sqlite-jdbc-autoincrement
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=5
connection.url=jdbc:mysql://10.32.177.178:3306/test&user=xxxx&password=xxxxx
table.whitelist=banner_hourly_statistics_v2
group.id=test-mysql-kafka
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
config.storage.topic=demo-1-distributed-config
offset.storage.topic=demo-1-distributed-offset
status.storage.topic=demo-1-distributed-status
bootstrap.servers=10.33.62.20:9092,10.33.62.110:9092,10.33.62.200:9092
mode=bulk
#incrementing.column.name=id
topic.prefix=test-sqlite-jdbc-
CMD: connect-standalone /grid/1/mukul/confluent-5.0.0/etc/kafka/connect-standalone.properties /grid/1/mukul/confluent-5.0.0/etc/kafka-connect-jdbc/source-quickstart-sqlite.properties
In the startup logs, it clearly shows loading JDBC Connectors:
[2018-08-09 06:59:30,072] INFO Loading plugin from: /grid/1/mukul/confluent-5.0.0/share/java/kafka-connect-jdbc (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:218)
[2018-08-09 06:59:30,133] INFO Registered loader: PluginClassLoader{pluginLocation=file:/grid/1/mukul/confluent-5.0.0/share/java/kafka-connect-jdbc/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:241)
[2018-08-09 06:59:30,133] INFO Added plugin 'io.confluent.connect.jdbc.JdbcSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:170)
[2018-08-09 06:59:30,133] INFO Added plugin 'io.confluent.connect.jdbc.JdbcSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:170)
But it fails with following exception:
Invalid value java.sql.SQLException: No suitable driver found for jdbc:mysql://10.32.177.178:3306/test&user=xxxx&password=xxxx for configuration Couldn't open connection to jdbc:mysql://10.32.177.178:3306/test&user=xxxx&password=xxx
Invalid value java.sql.SQLException: No suitable driver found for jdbc:mysql://10.32.177.178:3306/test&user=xxxx&password=xxxx for configuration Couldn't open connection to jdbc:mysql://10.32.177.178:3306/test&user=xxxx&password=xxxx
You can also find the above list of errors at the endpoint `/{connectorType}/config/validate`
at org.apache.kafka.connect.util.ConvertingFutureCallback.result(ConvertingFutureCallback.java:79)
at org.apache.kafka.connect.util.ConvertingFutureCallback.get(ConvertingFutureCallback.java:66)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:110)
Tried changing the plugin directories too but it didn't work. Tried moving the confluent share/* to /usr/share/java too but it also didn't work.
Download the JAR from the URL: https://dev.mysql.com/downloads/connector/j/5.1.html
Place inside the Plugin dir
Run the connect
It will take start pulling data from MySql.
May be a little late. I had the same issue of "No Driver found.." when I connect DB2 using kafka jdbc connector.
1st Possible Solution:
I resolved it by placing the DB2 driver at the exact location where jdbc-connector is.
With in Kafka connect:
find / -name kafka-connect-jdbc\*.jar
Once you found the location from the above command, copy DB2 jar at that location:
cp {your DB2 jar location}/db2.jar {copy the location from 'find' command}
Example
cp /Download/db2.jar /Users/share/java/kafka-connect-java/
Restart kafka-connect, it will pick up the DB2 drivers
2nd Possible Solution:
Download the jt400 jar (jdk-8) and put it next to the other jdbc drivers (DB2, SQL etc)
Happy coding :)
I am trying to run the kafka-connect-elasticsearch plugin from Confluent in order to stream topics from Kafka (V0.11.0.1) directly into Elasticsearch (without putting Logstash in between).
I build the connector using Maven -
$ cd kafka-connect-elasticsearch
$ mvn clean package
I then created the require configuration file -
name=es-cluster-lab
connector.class=io.confluent.connect.elasticsearch.ElasticsearchSinkConnector
tasks.max=1
topics=filebeats-test
topic.index.map=filebeats-test:kafka_test_index
key.ignore=true
schema-ignore=true
connection.url=http://elastic:9200
type.name=log
As per the new Kafka Classpath Isolation spec, I also added the following line to my connect-standalone.properties file -
plugin.path=/home/kafka/kafka-connect-elasticsearch-3.3.0/target/kafka-connect-elasticsearch-3.3.0-development/share/java/kafka-connect-elasticsearch/
I go to run the script ...
bin/connect-standalone.sh config/connect-standalone.properties config/elasticsearch-connect.properties
... and receive the below error.
[2017-09-14 16:08:26,599] INFO Loading plugin from: /home/kafka/kafka-connect-elasticsearch-3.3.0/target/kafka-connect-elasticsearch-3.3.0-development/share/java/kafka-connect-elasticsearch/slf4j-api-1.7.25.jar (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:176)
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.collect.Sets$SetView.iterator()Lcom/google/common/collect/UnmodifiableIterator;
at org.reflections.Reflections.expandSuperTypes(Reflections.java:380)
at org.reflections.Reflections.<init>(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:221)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:198)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:190)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:150)
at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:47)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:68)
I also tried to move the JAR files into the /app/kafka/libs directory (default CLASSPATH) and even tried to create a subdirectory /app/kafka/libs/connect_libs and add that manually to my CLASSPATH environment variable.
Not sure what my next step is besides putting Logstash between Kafka and Elastic.
try to change the guava version to 20 before you build it
I think you are missing the star '*' at the end of the path of the plugin path.
plugin.path=/home/kafka/kafka-connect-elasticsearch-3.3.0/target/kafka-connect-elasticsearch-3.3.0-development/share/java/kafka-connect-elasticsearch/*
I install spring-xd-1.2.1.RELEASE and start in Spring XD in xd-signle mode, when I type the following command
xd:>stream create --definition "time | log" --name ticktock --deploy
I get the following result:
Command failed org.springframework.xd.rest.client.impl.SpringXDException: Could not find module with name 'log' and type 'sink'
When I type the following command:
xd:> module list
I get the following resul:
Source Processor Sink Job
gemfire gemfire-json-server filejdbc
gemfire-cq gemfire-server hdfsjdbc
jdbc jdbc jdbchdfs
kafka rabbit sqoop
rabbit redis
twittersearch
twitterstream
Some default modules are missed ? What happens ? Is there any other configuration to set before starting spring xd ?
Check XD_HOME/modules/sink/log - Is this folder exist?