I have mircroservice environment based on spring-boot, where i am using zipkin server and discovery-server(eureka) and config-server. Now i have a rest-microservice which sends logs to zipkin server and this microservice is required to resolve where is zipkin server using discovery-server.
following is zipkin configuration i have in my rest-microservice's application.properties(pulled from config-server).
spring.zipkin.baseUrl=http://MTD-ZIPKIN-SERVER/
spring.zipkin.locator.discovery.enabled=true
spring.zipkin.enabled=true
...
here MTD-ZIPKIN-SERVER is zipkin-server name in discovery-server.
discovery-server dashboard.
but it does not try to resolve zipkin from discovery-server, instead it tries to connect directly using spring.zipkin.baseUrl, and i get below exception.
Dropped 1 spans due to ResourceAccessException(I/O error on POST request for "http://MTD-ZIPKIN-SERVER/api/v1/spans":
MTD-ZIPKIN-SERVER; nested exception is java.net.UnknownHostException:
MTD-ZIPKIN-SERVER)
org.springframework.web.client.ResourceAccessException: I/O error on
POST request for "http://MTD-ZIPKIN-SERVER/api/v1/spans":
MTD-ZIPKIN-SERVER; nested exception is java.net.UnknownHostException:
MTD-ZIPKIN-SERVER at
org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:666)
at
org.springframework.web.client.RestTemplate.execute(RestTemplate.java:628)
at
org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:590)
at
org.springframework.cloud.sleuth.zipkin.RestTemplateSender.post(RestTemplateSender.java:73)
at
org.springframework.cloud.sleuth.zipkin.RestTemplateSender.sendSpans(RestTemplateSender.java:46)
at
zipkin.reporter.AsyncReporter$BoundedAsyncReporter.flush(AsyncReporter.java:245)
at
zipkin.reporter.AsyncReporter$Builder.lambda$build$0(AsyncReporter.java:166)
at zipkin.reporter.AsyncReporter$Builder$$Lambda$1.run(Unknown
Source) at java.lang.Thread.run(Thread.java:745) Caused by:
java.net.UnknownHostException: MTD-ZIPKIN-SERVER at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
if i provide exact zipkin url in property spring.zipkin.baseUrl like below
spring.zipkin.baseUrl=http://localhost:5555/
then my rest-microservice is able to connect to zipkin-server.
My goal here is to read zipkin-server location from discovery-srever. What am i doing wrong? Do i need to add some zipkin enabling annotation on my spring-boot rest-microservice?
This feature is available in edgware release train. That corresponds to version 1.3.x of sleuth
Related
I'm having an issue that has gone from "annoyance" to "flip the desk".
I have a very basic Spring Boot MVC service with the following Hikari Connection Pool setup:
spring.datasource.username=ME
spring.datasource.password=mypassword
#spring.datasource.url=jdbc:oracle:thin:#SERVICE
spring.datasource.url=jdbc:oracle:thin:#localhost:1521/SERVICE
# need the duplicate because hikari is "special", but it's the spring default
#spring.datasource.jdbc-url=jdbc:oracle:thin:#SERVICE
spring.datasource.jdbc-url=jdbc:oracle:thin:#localhost:1521/SERVICE
# HikariCP settings
spring.datasource.hikari.minimumIdle=5
spring.datasource.hikari.maximumPoolSize=5
spring.datasource.hikari.idleTimeout=30000
spring.datasource.hikari.maxLifetime=2000000
spring.datasource.hikari.connectionTimeout=30000
spring.datasource.hikari.poolName=HikariPool
# leak detection - 30 seconds/30k ms
spring.datasource.hikari.leak-detection-threshold=30000
It seems that a connection is being held. I've tried restarting my Oracle Service and Listener with no luck, and the only solution seems to be to reboot.
Can anyone assist?
Update: I removed Hikari and used this:
spring.datasource.type=org.springframework.jdbc.datasource.SimpleDriverDataSource
I no longer think this is directly related to Hikari.
Edit 2:
Here's the actual "caused by" root exception:
Caused by: oracle.net.ns.NetException: Listener refused the connection with the following error:
ORA-12514, TNS:listener does not currently know of service requested in connect descriptor
(CONNECTION_ID=Z0U0SUlgTNuHsXgisq9YSA==)
at oracle.net.ns.NSProtocolNIO.createRefusePacketException(NSProtocolNIO.java:816) ~[ojdbc8-21.5.0.0.jar:21.5.0.0.0]
at oracle.net.ns.NSProtocolNIO.handleConnectPacketResponse(NSProtocolNIO.java:396) ~[ojdbc8-21.5.0.0.jar:21.5.0.0.0]
at oracle.net.ns.NSProtocolNIO.negotiateConnection(NSProtocolNIO.java:207) ~[ojdbc8-21.5.0.0.jar:21.5.0.0.0]
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:350) ~[ojdbc8-21.5.0.0.jar:21.5.0.0.0]
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:2372) ~[ojdbc8-21.5.0.0.jar:21.5.0.0.0]
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:657) ~[ojdbc8-21.5.0.0.jar:21.5.0.0.0]
... 85 common frames omitted
That particular CONNECTION_ID is randomized and different each time.
there are many posts on this line , but none with accepted answer (or) answer that works
My Configuration/setup:
spring.flyway.enabled=false
spring.sql.init.continue-on-error=true
spring.datasource.initialization-mode=always
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
spring.datasource.initialization-mode=always
spring.jpa.hibernate.ddl-auto=validate
**Only in local**
spring.jpa.database-platform=org.hibernate.dialect.MySQL5Dialect
(to overcome the error - Access to DialectResolutionInfo cannot be null when 'hibernate.dialect' not set)
Added
#EnableAutoConfiguration(exclude=FlywayAutoConfiguration.class)
on spring boot startup class
Still getting the error:
Caused by: com.mysql.cj.exceptions.CJCommunicationsException:
Communications link failure
The last packet sent successfully to the server was 0 milliseconds
ago. The driver has not received any packets from the server. at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
The app runs on GCP APp engine.
I want to start the application even when the DB is down. What configuration will fix the issue ?
I have a strange issue that took me several days to narrow down. Basically, I have a Jhipster project based on Spring boot Version 2.1.10.RELEASE, which contains 4 microservices. We are interested here in 2 of them: Gateway and Corehub.
In the gateway, I have an angular app that performs a POST to /services/corehub/api/someendpoint which used to be working and that is failing now with different error messages, but the one I have more regularly is
{
"type": "https://www.jhipster.tech/problem/problem-with-message",
"title": "Method Not Allowed",
"status": 405,
"detail": "Request method 'POST' not supported",
"path": "/services/ambientcorehub/api/trips",
"message": "error.http.405"
}
I ended up looking at the traces of the Registry that keeps track of the microservices for internal communication and I found out that when this error occurs, I cannot find the corehub in the traces anymore. So it looks like the corehub micro service is not registered.
An other GIT branch of this service does not have this problem, so I performed a diff between these two branches and I removed the changes until I could narrow down the problem.
So, in the corehub, I have a JMS listener based on this mq-jms-spring implementation. The maven dependency is as follows:
<dependency>
<groupId>com.ibm.mq</groupId>
<artifactId>mq-jms-spring-boot-starter</artifactId>
<version>2.2.7</version>
</dependency>
I commented out my JmsListener class and its associated JmsContext (to get access to a topic), and kept only the configuration properties defining access to the server, along with the port, channel, topic name, etc.
If I comment the above maven dependency, my service works again.
If I keep the maven dependency, with the configuration only, my corehub microservice is not registered in the Registry and becomes not accessible anymore from the gateway and thus the Angular UI.
What is important to note, is that I have currently some network issue which prevents me from accessing the JMS Server.
So I believe the exception that is raised by this IBM library because the JMS server is not reachable, breaks the registration of the microservice towards the spring boot registry.
Here are the traces that come over and over in the corehub console:
2020-07-22 08:21:45.316 WARN 7964 --- [nfoReplicator-0] o.s.boot.actuate.jms.JmsHealthIndicator : JMS health check failed
com.ibm.msg.client.jms.DetailedIllegalStateException: JMSWMQ0018: Failed to connect to queue manager '' with connection mode 'Client' and host name '172.31.14.1(9010)'.
at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:489)
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:215)
at com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:448)
at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createV7ProviderConnection(WMQConnectionFactory.java:8475)
at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createProviderConnection(WMQConnectionFactory.java:7815)
at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl._createConnection(JmsConnectionFactoryImpl.java:303)
at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl.createConnection(JmsConnectionFactoryImpl.java:236)
at com.ibm.mq.jms.MQConnectionFactory.createCommonConnection(MQConnectionFactory.java:6005)
at com.ibm.mq.jms.MQConnectionFactory.createConnection(MQConnectionFactory.java:6030)
at org.springframework.jms.connection.SingleConnectionFactory.doCreateConnection(SingleConnectionFactory.java:409)
at org.springframework.jms.connection.SingleConnectionFactory.initConnection(SingleConnectionFactory.java:349)
at org.springframework.jms.connection.SingleConnectionFactory.getConnection(SingleConnectionFactory.java:327)
at org.springframework.jms.connection.SingleConnectionFactory.createConnection(SingleConnectionFactory.java:242)
at org.springframework.boot.actuate.jms.JmsHealthIndicator.doHealthCheck(JmsHealthIndicator.java:52)
at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:82)
at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:95)
at org.springframework.cloud.netflix.eureka.EurekaHealthCheckHandler.getHealthStatus(EurekaHealthCheckHandler.java:110)
at org.springframework.cloud.netflix.eureka.EurekaHealthCheckHandler.getStatus(EurekaHealthCheckHandler.java:106)
at com.netflix.discovery.DiscoveryClient.refreshInstanceInfo(DiscoveryClient.java:1406)
at com.netflix.discovery.InstanceInfoReplicator.run(InstanceInfoReplicator.java:117)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.ibm.mq.MQException: JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2538' ('MQRC_HOST_NOT_AVAILABLE').
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:203)
... 24 common frames omitted
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2538;AMQ9204: Connection to host '172.31.14.1(9010)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2538;AMQ9204: Connection to host '/172.31.14.1:9010' rejected. [1=java.net.ConnectException[Connection timed out: connect],3=/172.31.14.1:9010,4=TCP,5=Socket.connect]],3=172.31.14.1(9010),5=RemoteTCPConnection.bindAndConnectSocket]
at com.ibm.mq.jmqi.remote.api.RemoteFAP$Connector.jmqiConnect(RemoteFAP.java:13558)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1426)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1385)
at com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.jmqiConnect(InterceptedJmqiImpl.java:377)
at com.ibm.mq.ese.jmqi.ESEJMQI.jmqiConnect(ESEJMQI.java:562)
at com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:381)
... 23 common frames omitted
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2538;AMQ9204: Connection to host '/172.31.14.1:9010' rejected. [1=java.net.ConnectException[Connection timed out: connect],3=/172.31.14.1:9010,4=TCP,5=Socket.connect]
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.bindAndConnectSocket(RemoteTCPConnection.java:901)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.protocolConnect(RemoteTCPConnection.java:1381)
at com.ibm.mq.jmqi.remote.impl.RemoteConnection.connect(RemoteConnection.java:976)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getNewConnection(RemoteConnectionSpecification.java:553)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSessionFromNewConnection(RemoteConnectionSpecification.java:233)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSession(RemoteConnectionSpecification.java:141)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionPool.getSession(RemoteConnectionPool.java:127)
at com.ibm.mq.jmqi.remote.api.RemoteFAP$Connector.jmqiConnect(RemoteFAP.java:13302)
... 28 common frames omitted
Caused by: java.net.ConnectException: Connection timed out: connect
at java.base/java.net.PlainSocketImpl.connect0(Native Method)
at java.base/java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:101)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
at java.base/java.net.Socket.connect(Socket.java:609)
at java.base/java.net.Socket.connect(Socket.java:558)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection$4.run(RemoteTCPConnection.java:1022)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection$4.run(RemoteTCPConnection.java:1014)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.connectSocket(RemoteTCPConnection.java:1014)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.bindAndConnectSocket(RemoteTCPConnection.java:805)
... 35 common frames omitted
Here are some version numbers:
jhipster-dependencies.version: 3.0.7
Spring boot version: 2.1.10.RELEASE
ibmmq-jms-spring version(s) that are affected by this issue: Version 2.2.7
Java version (including vendor and platform): AdoptOpenJDK\jdk-11.0.6.10-hotspot
A small code sample that demonstrates the issue.
Here is my configuration in application.yml:
spring:
jms:
# Used for JMS Message reception.
isPubSubDomain: false
application:
oag:
# This can be a queue or a topic (if subdomain is defined)
# In case of a topic, sub domain must be set to public.
queueName: "BRIDGE.XXX.TO.YYY.TST"
isTopic: false
ibm:
mq:
queueManager:
channel: XXX_GWT11.BT1
connName: 172.31.14.1(9010)
user: xxxx
password:
Would it be possible to catch this exception to avoid breaking regular Spring Boot registration mechanism?
I cannot afford having my cluster down because I cannot access the JMS server.
Beside this, I opened this message on IBM MQ side here and a person suggested me to stop the JMS health indicator. So I set the following property to no avail:
management:
endpoint:
jms:
# Prevent Unreachable JMS Server from unregistering corehub from the registry, leading to unreachable microservice from the Gateway
enabled: false
Corresponding documentation is here
Any help would be greatly appreciated.
Thank you
Christophe
If you do not want your application to be considered unhealthy when JMS is down, disabling the JMS health indicator is what I would recommend. It hasn't worked for you as you have used management.endpoint.jms.enabled. The correct property to use is management.health.jms.enabled:
management:
health:
jms:
enabled: false
I have two Spring boot services in Kubernetes (Google Cloud).
Service1 executes http request using org.springframework.web.client.RestTemplate to Service2 using Service name. In Both cases Service1 receives the following error:
org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://service1/tcp:/10.11.252.122:8080": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)
when 10.11.252.122 is Service2 cluster IP.
It happens only when Service1 and Service2 are in the same cluster, in other cases code works properly.
What can cause Spring to add tcp:/Cluser IP to url?
Thanks,
I am getting the following exception while hitting my backend Spring boot application which is deployed on a Kubernetes container:
java.lang.NumberFormatException: For input string: "443,80"
All my services are registered with eureka:
#Eureka
spring.application.name=app-name
eureka.client.registerWithEureka=true
eureka.client.fetchRegistry=true
eureka.client.serviceUrl.defaultZone=http://app-eureka-dev/eureka
eureka.instance.preferIpAddress=true
eureka.instance.non-secure-port-enabled=true
And all my requests are routed through ingress/zuul services.
spring.application.name=app-gateway
eureka.client.registerWithEureka=false
eureka.client.fetchRegistry=true
eureka.client.serviceUrl.defaultZone=http://app-eureka-dev/eureka
When we try hitting backend services from the Swagger API, I am getting below exception.
java.lang.NumberFormatException: For input string: "443,80"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at springfox.documentation.swagger2.web.HostNameProvider.componentsFrom(HostNameProvider.java:72)
at springfox.documentation.swagger2.web.Swagger2Controller.getDocumentation(Swagger2Controller.java:84)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod
I am connecting to the eureka service through the container name, even though I am getting above exception. Is there any other configuration required, as we do ssl offloading in ingress, rest should be plain http or unsecure calls within the container services.
This should have been fixed with Springfox 2.7.0, as can be seen in this GitHub issue and the release notes of this version.
Before Springfox 2.7.0, the following code was used to determine the port number in HostNameProvider:
String port = request.getHeader("X-Forwarded-Port");
if (hasText(port)) {
builder.port(Integer.parseInt(port));
}
So basically it used the X-Forwarded-Port header to determine the port number. However, in your case it seems like it passes both the HTTP and HTTPS ports (443,80), which is obviously not a valid integer.
Upgrading your springfox-swagger2 dependency to 2.7.0 (or higher) should do the trick.
I experienced the same problem in SWAGGER when my DTO properties annotated with #APIModelProerty. Majorly, the problem occurred when Long type properties are present in the DTO object and failed to convert the empty string ("") value into 0.
So, it solved when I added example property in #APIModelProperty(example = "1", required = "false").