I'm facing issue while connecting Zetezza database from confluent source jdbc connector.
source-jdbc connector properties:
name=source-netezza
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=1
connection.url=jdbc:netezza://10.245.*.*:5480/DATABASE=mydb,user=admin,password=password
table.whitelist=netezza_str
mode=bulk
topic.prefix=netezza_str_
After executing the command I'm getting below errors
Invalid value java.sql.SQLException: No suitable driver found for jdbc:netezza://10.245.*.*:5480/mydb,user=admin,password=password for configuration Couldn't open connection to jdbc:netezza://10.245.*.*:5480/mydb,user=admin,password=password
I've tried with netezza-3.2.2 jar and nzjdbc jar.
Related
I am trying to configure Kafka Connect with Elastic search.
But when I try to start it I am getting following ERRORs:
ERROR StatusLogger Log4j2 could not find a logging implementation.
ERROR Failed to create client to verify connection. (io.confluent.connect.elasticsearch.Validator:120)
ElasticsearchException[java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused]; nested: ExecutionException[java.net.ConnectException: Connection refused]; nested: ConnectException[Connection refused]
I am trying to start it with following command in terminal:
connect-standalone.sh config/connect-standalone.properties config/elasticsearch.properties
This is connect-standalone.properties:
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
offset.flush.interval.ms=10000
# EDIT BELOW IF NEEDED
bootstrap.servers=localhost:9092
offset.storage.file.filename=/tmp/connect.offsets
plugin.path=/home/stjepan/kafka_2.13-3.2.3/connectors
This is elasticsearch.properties:
name=elasticsearch-sink
connector.class=io.confluent.connect.elasticsearch.ElasticsearchSinkConnector
tasks.max=1
topics=wikimedia.recentchange
key.ignore=true
connection.url=localhost:9200
# connection.url=https://kafka-course-5842482143.eu-west-1.bonsaisearch.net
# connection.username=he6de7ka5o
# connection.password=yozz8ryqmg
type.name=kafka-connect
# necessary for this connector
schema.ignore=true
behavior.on.malformed.documents=IGNORE
# OVERRIDE
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
What am I doing wrong? Wikimedia Connector works fine...
I didn't try anything else because I don't find any idea what can I change except ports, but it didn't helped...
You're probably referring to the course material of https://www.conduktor.io/apache-kafka-for-beginners (or the related course on Packt Pub)
I worked around this issue by using an OpenSearch Kafka Sink Connector:
https://github.com/aiven/opensearch-connector-for-apache-kafka
under https://github.com/aiven/opensearch-connector-for-apache-kafka/blob/main/config/quickstart-elasticsearch.properties there is a Connector Config that you can merge into your existing config.
Hope that helps
Based on the below error is see , Presto does not seem to be a valid dataSource that is compatible with Spring JPA
Description:
Failed to bind properties under 'spring.jpa.database' to org.springframework.orm.jpa.vendor.Database:
Property: spring.jpa.database
Value: presto
Origin: class path resource [application.yml]:101:22
Reason: failed to convert java.lang.String to org.springframework.orm.jpa.vendor.Database
Action: Update your application's configuration. The following values are valid: DB2 DEFAULT DERBY H2 HANA HSQL INFORMIX MYSQL ORACLE POSTGRESQL SQL_SERVER SYBASE
Thoughts or advise please ?
I am using JDBC source Kafka connector to dump data from the SQL Server to Kafka in standalone mode. I have downloaded the confluentinc-kafka-connect-jdbc-5.5.0.zip for the JDBC connector jars & copied all the jars at plugin.path. I am using the below worker & connector configuration.
worker.peroperties
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter=org.apache.kafka.connect.json.JsonConverter
bootstrap.servers=localhost:9092
plugin.path=/home/kafka_2.12-2.5.0/connector/plugin/
offset.storage.file.filename=file
connector1.properties
name=test12_connector
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
connection.url=jdbc:sqlserver://host;databaseName=db;
connection.user=user
connection.password=password
query=SELECT TOP 100 * FROM table;
topic.prefix=test-
mode=bulk
I am running connector in standalone mode using below command-
sh bin/connect-standalone.sh -daemon connector/worker.properties connector/connector1.properties
I am using the latest SQL Server JDBC driver & copied at the same plugin path.
Also, added same plugin.path in CLASSPATH.
But the issue I am facing that it is getting stuck at below lines in logs & not producing any data in the topic from the table (expected topic: test-table)
-logs
[2020-06-29 18:41:31,402] DEBUG Dialect SqlServerDatabaseDialect scored 100 against JDBC subprotocol 'sqlserver' and source 'jdbc:sqlserver://host:1433;databaseName=db;' (io.confluent.connect.jdbc.dialect.DatabaseDialects:127)
[2020-06-29 18:41:31,402] DEBUG Dialect SqliteDatabaseDialect scored 0 against JDBC subprotocol 'sqlserver' and source 'jdbc:sqlserver://host:1433;databaseName=db;' (io.confluent.connect.jdbc.dialect.DatabaseDialects:127)
[2020-06-29 18:41:31,402] DEBUG Dialect SybaseDatabaseDialect scored 100 against JDBC subprotocol 'sqlserver' and source 'jdbc:sqlserver://host:1433;databaseName=db;' (io.confluent.connect.jdbc.dialect.DatabaseDialects:127)
[2020-06-29 18:41:31,403] DEBUG Dialect VerticaDatabaseDialect scored 0 against JDBC subprotocol 'sqlserver' and source 'jdbc:sqlserver://host:1433;databaseName=db;' (io.confluent.connect.jdbc.dialect.DatabaseDialects:127)
[2020-06-29 18:41:31,403] DEBUG Using dialect SqlServerDatabaseDialect with score 100 against JDBC subprotocol 'sqlserver' and source 'jdbc:sqlserver://host:1433;databaseName=db;' (io.confluent.connect.jdbc.dialect.DatabaseDialects:133)
It is not even showing any error logs. I have changed the log level to TRACE but still, the connector is not showing any error & only shows the above logs.
Please help me out if I am missing anything.
I'd try to play with connector properties probably adding:
"tasks.max": "1",
"internal.key.converter.schemas.enable": "false",
"reporter.admin.bootstrap.servers": "localhost:9092",
"internal.value.converter.schemas.enable": "false",
It also seems that it hangs not getting connection to source SQL server, so either connection string is not right or it can't find the driver JAR file for this dialect.
Check the logs if JDBC SQL Server dialect driver is loaded.
I am working on streaming the data from postgreSQL to HDFS. I had setup confluent environment on HDP 2.6 sandbox. My jdbc source configs for postgreSQL are
name=jdbc_1
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=1
connection.url=jdbc:postgresql://host:port/db?currentSchema=schema&user=user&password=password
mode=timestamp
timestamp.column.name=col1
validate.non.null=false
topic.prefix=psql-
All other properties for connection are also fine and i am running it by
./bin/connect-standalone ./etc/kafka/connect-standalone.properties ./etc/kafka-connect-jdbc/source.properties
Its working fine and creating topics based on the number of tables in the database as
psql-table1
psql-table2
Now i want to run HDFS sinks on all the topics to create separate dir for every table in the postgreSQL database.
But when i run HDFS sink with command
./bin/connect-standalone ./etc/kafka/connect-standalone.properties ./etc/kafka-connect-hdfs/hdfs-postGres.properties
by running the source i am getting error
ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:113)
org.apache.kafka.connect.errors.ConnectException: Unable to start REST server
at org.apache.kafka.connect.runtime.rest.RestServer.start(RestServer.java:214)
at org.apache.kafka.connect.runtime.Connect.start(Connect.java:53)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:95)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:331)
at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:299)
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:235)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.server.Server.doStart(Server.java:398)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.apache.kafka.connect.runtime.rest.RestServer.start(RestServer.java:212)
... 2 more
and if i stop the source connection and start the sink it works fine.
Anyone can help me that how i can setup multiple sink connectors.
Kafka Connect starts a rest server on port 8083.
If you run more that one standalone connector on a single machine, you need to change it with the rest.port property
Or you can run connect-distributed, then POST your source and sink configurations individually as JSON payloads running on a single Connect server, then you wouldn't have this Address already in use issue.
I am creating a jdbc connection to hive using javax.sql.DataSource and passing zookeeper service discovery (obtained from Ambari) string to Hive .
Zookeeper Hive URL : jdbc:hive2://localhost:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;transportMode=http;httpPath=cliservice
If I make direct jdbc connection with HiveServer host and port then connection work properly but it fails with zookeeper string.
After that I tested zookeeper string with beeline and I worked fine.
Below is exception when connection is made.
Caused by: java.sql.SQLException: Could not open client transport for any of the Server URI's in ZooKeeper: Unable to read HiveServer2 uri from ZooKeeper
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:205)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:163)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:307)
at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:200)
at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:710)
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:644)
at org.apache.tomcat.jdbc.pool.ConnectionPool.init(ConnectionPool.java:466)
at org.apache.tomcat.jdbc.pool.ConnectionPool.<init>(ConnectionPool.java:143)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.pCreatePool(DataSourceProxy.java:115)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.createPool(DataSourceProxy.java:102)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:126)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:85)
at com.thinkbiganalytics.kerberos.KerberosUtil.getConnectionWithOrWithoutKerberos(KerberosUtil.java:60)
at com.thinkbiganalytics.hive.service.RefreshableDataSource.getConnectionForValidation(RefreshableDataSource.java:113)
at com.thinkbiganalytics.hive.service.RefreshableDataSource.testAndRefreshIfInvalid(RefreshableDataSource.java:133)
at com.thinkbiganalytics.hive.service.RefreshableDataSource.getConnection(RefreshableDataSource.java:145)
at com.thinkbiganalytics.kerberos.KerberosUtil.getConnectionWithOrWithoutKerberos(KerberosUtil.java:60)
at com.thinkbiganalytics.schema.DBSchemaParser.listCatalogs(DBSchemaParser.java:80)
... 118 more
Caused by: org.apache.hive.jdbc.ZooKeeperHiveClientException: Unable to read HiveServer2 uri from ZooKeeper
at org.apache.hive.jdbc.ZooKeeperHiveClientHelper.getNextServerUriFromZooKeeper(ZooKeeperHiveClientHelper.java:86)
at org.apache.hive.jdbc.Utils.updateConnParamsFromZooKeeper(Utils.java:506)
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:203)
... 136 more
Caused by: org.apache.hive.jdbc.ZooKeeperHiveClientException: Tried all existing HiveServer2 uris from ZooKeeper.
at org.apache.hive.jdbc.ZooKeeperHiveClientHelper.getNextServerUriFromZooKeeper(ZooKeeperHiveClientHelper.java:73)
... 138 more
Did anyone encounter this ?
After spending my 2 days , i figured out problem . I have hive 0.14 dependency in my code where this problem is occurring. To fix i updated below two hive maven dependencies..
Hive Services - https://mvnrepository.com/artifact/org.apache.hive/hive-service/1.2.1000.2.4.2.10-1
Hive JDBC - https://mvnrepository.com/artifact/org.apache.hive/hive-jdbc/1.2.1000.2.4.2.10-1