I have successfully used Spring Cloud Stream Kafka Binder (org.springframework.cloud:spring-cloud-starter-stream-kafka:3.0.1.RELEASE) and Spring Cloud Azure Event Hubs (com.microsoft.azure:spring-cloud-starter-azure-eventhubs:1.2.3) with Spring Boot 2.2.6 to Publish and Consume messages from Azure Event Hub (with Kafka API enabled).
However, when i try to integrate the same versions of the Spring Cloud libraries with Spring Boot 1.5.22, I am facing the issue java.lang.NoClassDefFoundError: org/springframework/integration/support/converter/ConfigurableCompositeMessageConverter
When i used spring-cloud-starter-stream-kafka:1.3.4.RELEASE and com.microsoft.azure:spring-cloud-starter-azure-eventhubs:1.1.0, I am getting zookeeper connection issues probably due to a different set of properties needed for configuration
2020-04-29 17:01:43.104 INFO 81976 --- [localhost:2181)] [org.apache.zookeeper.ClientCnxn ] [-] [-] : Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2020-04-29 17:01:43.107 WARN 81976 --- [localhost:2181)] [org.apache.zookeeper.ClientCnxn ] [-] [-] : Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
My application.yaml looks like below
spring:
cloud:
azure:
auto-create-resources: true
credential-file-path: my.azureauth
eventhub:
checkpoint-storage-account: azuremigrationv2
namespace: somenamespace
region: Central US
resource-group: some-rg
stream:
bindings:
consumer:
destination: event-hub-1
group: testconsumergroup
content-type: application/json
nativeEncoding: true
consumer:
concurrency: 1
valueSerde: JsonSerde
requeue-rejected: true
I would like to know which versions of the libraries i am using are compatible with Spring Boot >= 1.5.20
My understanding is that config properties within spring.cloud.azure are for the dependency azure-spring-cloud-starter-eventhubs and the config properties within spring.cloud.stream are for use with azure-spring-cloud-stream-binder-eventhubs . I could be wrong about this but I think it is correct. There is no good documentation on it that I could find and it makes things quite confusing if you don't know the difference. It caused me to waste a day or two doing POCs until I started understanding it.
NOTE: When configuring binders, spring.cloud.stream contains spring.cloud.azure subkeys. Link
Related
I have a Spring project in which I am using Apache Camel and ActiveMQ as message broker. When I started my Java application which receives from the topic and when the broker is not started it gives an error (which is valid), but even after starting the broker JMS connection is not refreshed.
I tried the same case in the Spring Boot app and it automatically refreshed the JMS connection. Something needs to be handled manually in Spring but not sure of that thing.
This is the error from my Spring app:
org.apache.camel.component.jms.DefaultJmsMessageListenerContainer: Could not refresh JMS Connection for destination 'test.events.topic' - retrying using FixedBackOff{interval=5000, currentAttempts=1, maxAttempts=unlimited}. Cause: Could not connect to broker URL: tcp://activemq:61616. Reason: java.net.UnknownHostException: activemq
This is what the Spring Boot application reports:
Successfully refreshed JMS Connection
Below is the broker configuration:
broker:
host: localhost
port: 61616
protocol: tcp
endpoint: ${broker.protocol}://${broker.host}:${broker.port}
url: failover:(${broker.endpoint}?wireFormat.maxInactivityDurationInitalDelay=30000)?timeout=3000&jms.useCompression=true&startupMaxReconnectAttempts=0&jms.redeliveryPolicy.maximumRedeliveries=${maxConsumers}
I have a spring boot application written using jhipster that was handed to our project not long ago. It's been running under WebLogic and I'm in the process of moving it to Tomcat9, getting ready for cloud migration. I found that when I access actuator's "health", the application returned {"status" : "DOWN"} and it was attempting to connect to the database.
I don't want it to attempt the connection. I tried looking through the application but found no setting that seems to be causing this. I'm sure it's there somewhere, though.
Does anyone have any idea where I should look?
The following exception was thrown:
2020-04-13 07:41:32.003 ERROR 16156 --- [nio-8080-exec-9] o.a.tomcat.jdbc.pool.ConnectionPool : Unable to create initial connections of pool.
java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:743)
at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:666)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:566)
at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:310)
at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:203)
at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:732)
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:664)
at org.apache.tomcat.jdbc.pool.ConnectionPool.init(ConnectionPool.java:479)
at org.apache.tomcat.jdbc.pool.ConnectionPool.<init>(ConnectionPool.java:154)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.pCreatePool(DataSourceProxy.java:118)
.
.
.
DB health check is enabled when you add actuator framework.
You can disable the db health check with this property:
management.health.db.enabled=false
Read more about all the actuator properties here:
https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#actuator-properties
I have setup a Spring Boot application to receive Kafka messages from an existing and working Kafka producer. The setup is standard, and based on the following: https://www.codenotfound.com/spring-kafka-consumer-producer-example.html
Messages are not received, and the following is continually displayed in the console:
WARN org.apache.clients.NetworkClient :Bootstrap broker <hostname>:9092 disconnected
In addition, the following debug message is logged:
org.apache.common.errors.Timeout: Failed to update metadata after 60000 ms.
The console message is discussed in the following link:
https://community.hortonworks.com/content/supportkb/150148/errorwarn-bootstrap-broker-6668-disconnected-orgap.html
The logged message is discussed here:
https://community.cloudera.com/t5/Data-Ingestion-Integration/Error-when-sending-message-to-topic-in-Kafka/td-p/41440
Very likely, the timeout will not happen when the first issue is resolved.
The solution to the console message which is given is to explicitly pass --security-protocol SSL as an argument to the producer or consumer command.
Given that I am listening on an existing Kafka broker and topic, no settings can be changed there. Any changes must be on the Spring Boot side.
Is it possible to configure application.yml so that --security-protocol SSL is passed an an argument to the consumer? Also, has anyone experienced this before, and is there another way to resolve the issue using the configuration options available in Spring Boot and Spring Kafka?
Thanks
See the documentation.
Scroll down to Kafka. Arbitrary Kafka properties can be set using
spring:
kafka:
properties:
security.protocol: SSL
applies to consumer and producer (and admin in 2.0).
In the upcoming 2.0 release (currently RC1), there is also
spring:
kafka:
properties:
consumer:
some.property: foo
for properties that only apply to consumers (and similarly for producers and admins).
My application uses the frameworks and application server below:
Spring Boot(1.5.4.RELEASE) with hibernate.
Websphere Application Server Liberty for Java on Bluemix
The application works fine however after the running for hours connections time out. See following exception:
2017-09-01T11:40:40.57+0900 [APP/PROC/WEB/0] OUT 2017-09-01 02:40:40,563 [http-nio-8080-exec-2] [5baba2cb-5bfd-4846-b8e0-8782aa729639] [] WARN o.h.e.jdbc.spi.SqlExceptionHelper [SqlExceptionHelper.java:127] - SQL Error: -4499, SQLState: 08001
2017-09-01T11:40:40.58+0900 [APP/PROC/WEB/0] OUT or socket output stream. Error location: Reply.fill() - socketInputStream.read (-1). Message: Connection timed out (Write failed). ERRORCODE=-4499, SQLSTATE=08001
I know Spring boot needs to be set configuration properties in application.properties.
spring.datasource.testOnBorrow=true
spring.datasource.testWhileIdle=true
spring.datasource.timeBetweenEvictionRunsMillis=60000
spring.datasource.numTestsPerEvictionRun=3
spring.datasource.minEvictableIdleTimeMillis=600000
spring.datasource.validationQuery=SELECT 1
However, I found out with Spring boot 1.3+, we must use the implementation-specific settings using their respective prefix (spring.datasource.tomcat., spring.datasource.hikari., and spring.datasource.dbcp2.*), and (apparently) DOES NOT SUPPORT Websphere Application Server Liberty Profile.
My question is how should I configure to keep connections alive (or renew them)?
Thank you in advance!
What datasource are you using? In our case we are using tomcat which is the default option. So we have
# DATASOURCE (DataSourceAutoConfiguration & DataSourceProperties)
spring.datasource.tomcat.test-on-borrow=true
spring.datasource.tomcat.test-while-idle=true
spring.datasource.tomcat.time-between-eviction-runs-millis=300000
spring.datasource.tomcat.validation-query=SELECT 1
I'm using Kafka for Asyng calls between microservices, and i'm using Spring Sleuth for logging. The logging is ok, but when there is a message from Microservice1 to Microservice2, the logging's messages have different Trace-ID. Don't they have to have the same trace-Id but a different SpanId? is there any special configuration?
Message headers by default will not be transported by Spring Cloud Kafka binder, you have to set it via spring.cloud.stream.kafka.binder.headers manually as described in the Spring Cloud Stream Reference Guide. And then check if those tracing related headers been sent properly.
You can set Zipkin headers as following in your application.yml:
spring:
cloud:
stream:
kafka:
binder:
headers:
- X-B3-TraceId
- X-B3-SpanId
- X-B3-Sampled
- X-B3-ParentSpanId
- X-Span-Name
- X-Span-Export
Or in your application.properties:
spring.cloud.stream.kafka.binder.headers[0]=X-B3-TraceId
spring.cloud.stream.kafka.binder.headers[1]=X-B3-SpanId
spring.cloud.stream.kafka.binder.headers[2]=B3-Sampled
spring.cloud.stream.kafka.binder.headers[3]=X-B3-ParentSpanId
spring.cloud.stream.kafka.binder.headers[4]=X-Span-Name
spring.cloud.stream.kafka.binder.headers[5]=X-Span-Export
Or in a comma-separated list:
spring.cloud.stream.kafka.binder.headers=X-B3-TraceId,X-B3-SpanId,B3-Sampled,\
X-B3-ParentSpanId,X-Span-Name,X-Span-Export