I have two Spring boot services in Kubernetes (Google Cloud).
Service1 executes http request using org.springframework.web.client.RestTemplate to Service2 using Service name. In Both cases Service1 receives the following error:
org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://service1/tcp:/10.11.252.122:8080": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)
when 10.11.252.122 is Service2 cluster IP.
It happens only when Service1 and Service2 are in the same cluster, in other cases code works properly.
What can cause Spring to add tcp:/Cluser IP to url?
Thanks,
Related
I come to you today to ask you about this problem that I am having with my Docker container.
Indeed I protected my rest API resource with OAuth2 as indicated by the pom.xml,
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-oauth2-resource-server</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-oauth2-jose</artifactId>
</dependency>
this is application properties config
spring.security.oauth2.resourceserver.jwt.issuer-uri=https://authServer.com/auth/realms/api-dev
My authorisation server is keycloak and is deployed in a Docker container
when I start my resource without docker everything is correct and I can authenticate clients accesstoken and access the resource protect without problem.
But when I containerize my server resource I receive a 401 error as a return on my postman client and the docker container log is as follows
org.springframework.security.authentication.AuthenticationServiceException: An error occurred while attempting to decode the Jwt: Couldn't retrieve remote JWK set: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://authServer.com/auth/realms/api-dev/protocol/openid-connect/certs": Connection timed out (Connection timed out); nested exception is java.net.ConnectException: Connection timed out (Connection timed out)
Caused by: org.springframework.security.oauth2.jwt.JwtException: An error occurred while attempting to decode the Jwt: Couldn't retrieve remote JWK set: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://authServer.com/auth/realms/api-dev/protocol/openid-connect/certs": Connection timed out (Connection timed out); nested exception is java.net.ConnectException: Connection timed out (Connection timed out)
at org.springframework.security.oauth2.jwt.NimbusJwtDecoder.createJwt(NimbusJwtDecoder.java:169) ~[spring-security-oauth2-jose-5.7.4.jar!/:5.7.4]
at org.springframework.security.oauth2.jwt.NimbusJwtDecoder.decode(NimbusJwtDecoder.java:137) ~[spring-security-oauth2-jose-5.7.4.jar!/:5.7.4]
at org.springframework.security.oauth2.server.resource.authentication.JwtAuthenticationProvider.getJwt(JwtAuthenticationProvider.java:97) ~[spring-security-oauth2-resource-server-5.7.4.jar!/:5.7.4]
... 55 common frames omitted
Caused by: com.nimbusds.jose.RemoteKeySourceException: Couldn't retrieve remote JWK set: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://authServer.com/auth/realms/api-dev/protocol/openid-connect/certs": Connection timed out (Connection timed out); nested exception is java.net.ConnectException: Connection timed out (Connection timed out)
Indeed I was trying to protect my resource and give authorisation only to those who have a valid access token
In order to check that the signature of JWT is valid, Spring Security needs to acquire the certificate defined by the configuration option
spring.security.oauth2.resourceserver.jwt.issuer
through the cert OIDC endpoint defined by the Keycloak's realm configuration.
When you use different docker containers for Keycloak and Resource Server, you should ensure that one container can interact with the other on the network level. You can find many tutorials that describe how to implement network availability between containers in details just pick the one where you can use names instead of IP addresses.
We are trying to connect to IBM MQ from our Nifi instance(nifi-1.11.4) using the processor 'consumeJMS ' and the controller service 'JMSConnectionFactoryProvider'.
The JMSConnectionFactoryProvider is configured as shown below:
The consumeJMS processor is configured as shown below:
The Nifi instance is able to connect to the MQ channel and receive real-time messages when the MQ is configured without any security i.e without SSL. However, on configuring the MQ with security i.e with TLS V1.2 and also configuring the 'StandardSSLContextService' controller service , we get the error as shown below:
JMSWMQ0018: Failed to connect to queue manager 'XXXXXX' with connection mode 'Client' and host name 'XXXXXX.YY.com(1414)'.
CC=2;RC=2397;AMQ9204: Connection to host 'XXXXXX.YY.com(1414)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2397;AMQ9641: Remote CipherSpec error for channel 'ZZZZZZ.XXXXXX.04' to host ''. [3=ZZZZZZ.XXXXXX.04]],3=XXXXXX.YY.com(1414),5=RemoteConnection.analyseErrorSegment]
mqiException: CC=2;RC=2397;AMQ9641: Remote CipherSpec error for channel 'ZZZZZZ.XXXXXX.04' to host ''. [3=ZZZZZZ.XXXXXX.04]
2020-10-28 20:18:24,300 ERROR [Timer-Driven Process Thread-9] o.apache.nifi.jms.processors.ConsumeJMS ConsumeJMS[id=c26ce2ba-] Error while trying to process JMS message: org.springframework.jms.UncategorizedJmsException: Uncategorized exception occurred during JMS processing; nested exception is com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0018: Failed to connect to queue manager 'XXXXXX' with connection mode 'Client' and host name 'XXXXXX.YY..com(1414)'.
Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.; nested exception is com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2397' ('MQRC_JSSE_ERROR').
org.springframework.jms.UncategorizedJmsException: Uncategorized exception occurred during JMS processing; nested exception is com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0018: Failed to connect to queue manager 'XXXXXX' with connection mode 'Client' and host name 'XXXXXX.YY..com(1414)'.
Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.; nested exception is com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2397' ('MQRC_JSSE_ERROR').
CC=2;RC=2397;AMQ9204: Connection to host 'XXXXXX.YY..com(1414)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2397;AMQ9641: Remote CipherSpec error for channel 'ZZZZZZ.XXXXXX.04' to host ''. [3=ZZZZZZ.XXXXXX.04]],3=XXXXXX.YY..com(1414),5=RemoteConnection.analyseErrorSegment].
As discussed with our IBM Sever admin, they mentioned they have different ciphersuite setup on their side. They advised us to configure same Ciphersuite on NIFI instance in order to overcome this issue.
Server Side Error:
===================
Process(12232.3231) User(abc) Program(mqmgr123)
Host(VMServer) Installation(Installation1)
VRMF(0.0.0.01) QMgr(YYYYYY)
AMQ9639: Remote channel 'XXXXXX.YYYYYY.04' did not specify a CipherSpec.
EXPLANATION:
Remote channel 'XXXXXX.YYYYYY.04' did not specify a CipherSpec when the local
channel expected one to be specified.
The remote host is 'vmclient (00.001.00.05)'.
The channel did not start.
ACTION:
Change the remote channel 'XXXXXX.YYYYYY.04' on host 'vmclient
(00.001.00.05)' to specify a CipherSpec so that both ends of the channel have
matching CipherSpecs.
I have a strange issue that took me several days to narrow down. Basically, I have a Jhipster project based on Spring boot Version 2.1.10.RELEASE, which contains 4 microservices. We are interested here in 2 of them: Gateway and Corehub.
In the gateway, I have an angular app that performs a POST to /services/corehub/api/someendpoint which used to be working and that is failing now with different error messages, but the one I have more regularly is
{
"type": "https://www.jhipster.tech/problem/problem-with-message",
"title": "Method Not Allowed",
"status": 405,
"detail": "Request method 'POST' not supported",
"path": "/services/ambientcorehub/api/trips",
"message": "error.http.405"
}
I ended up looking at the traces of the Registry that keeps track of the microservices for internal communication and I found out that when this error occurs, I cannot find the corehub in the traces anymore. So it looks like the corehub micro service is not registered.
An other GIT branch of this service does not have this problem, so I performed a diff between these two branches and I removed the changes until I could narrow down the problem.
So, in the corehub, I have a JMS listener based on this mq-jms-spring implementation. The maven dependency is as follows:
<dependency>
<groupId>com.ibm.mq</groupId>
<artifactId>mq-jms-spring-boot-starter</artifactId>
<version>2.2.7</version>
</dependency>
I commented out my JmsListener class and its associated JmsContext (to get access to a topic), and kept only the configuration properties defining access to the server, along with the port, channel, topic name, etc.
If I comment the above maven dependency, my service works again.
If I keep the maven dependency, with the configuration only, my corehub microservice is not registered in the Registry and becomes not accessible anymore from the gateway and thus the Angular UI.
What is important to note, is that I have currently some network issue which prevents me from accessing the JMS Server.
So I believe the exception that is raised by this IBM library because the JMS server is not reachable, breaks the registration of the microservice towards the spring boot registry.
Here are the traces that come over and over in the corehub console:
2020-07-22 08:21:45.316 WARN 7964 --- [nfoReplicator-0] o.s.boot.actuate.jms.JmsHealthIndicator : JMS health check failed
com.ibm.msg.client.jms.DetailedIllegalStateException: JMSWMQ0018: Failed to connect to queue manager '' with connection mode 'Client' and host name '172.31.14.1(9010)'.
at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:489)
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:215)
at com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:448)
at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createV7ProviderConnection(WMQConnectionFactory.java:8475)
at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createProviderConnection(WMQConnectionFactory.java:7815)
at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl._createConnection(JmsConnectionFactoryImpl.java:303)
at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl.createConnection(JmsConnectionFactoryImpl.java:236)
at com.ibm.mq.jms.MQConnectionFactory.createCommonConnection(MQConnectionFactory.java:6005)
at com.ibm.mq.jms.MQConnectionFactory.createConnection(MQConnectionFactory.java:6030)
at org.springframework.jms.connection.SingleConnectionFactory.doCreateConnection(SingleConnectionFactory.java:409)
at org.springframework.jms.connection.SingleConnectionFactory.initConnection(SingleConnectionFactory.java:349)
at org.springframework.jms.connection.SingleConnectionFactory.getConnection(SingleConnectionFactory.java:327)
at org.springframework.jms.connection.SingleConnectionFactory.createConnection(SingleConnectionFactory.java:242)
at org.springframework.boot.actuate.jms.JmsHealthIndicator.doHealthCheck(JmsHealthIndicator.java:52)
at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:82)
at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:95)
at org.springframework.cloud.netflix.eureka.EurekaHealthCheckHandler.getHealthStatus(EurekaHealthCheckHandler.java:110)
at org.springframework.cloud.netflix.eureka.EurekaHealthCheckHandler.getStatus(EurekaHealthCheckHandler.java:106)
at com.netflix.discovery.DiscoveryClient.refreshInstanceInfo(DiscoveryClient.java:1406)
at com.netflix.discovery.InstanceInfoReplicator.run(InstanceInfoReplicator.java:117)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.ibm.mq.MQException: JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2538' ('MQRC_HOST_NOT_AVAILABLE').
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:203)
... 24 common frames omitted
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2538;AMQ9204: Connection to host '172.31.14.1(9010)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2538;AMQ9204: Connection to host '/172.31.14.1:9010' rejected. [1=java.net.ConnectException[Connection timed out: connect],3=/172.31.14.1:9010,4=TCP,5=Socket.connect]],3=172.31.14.1(9010),5=RemoteTCPConnection.bindAndConnectSocket]
at com.ibm.mq.jmqi.remote.api.RemoteFAP$Connector.jmqiConnect(RemoteFAP.java:13558)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1426)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1385)
at com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.jmqiConnect(InterceptedJmqiImpl.java:377)
at com.ibm.mq.ese.jmqi.ESEJMQI.jmqiConnect(ESEJMQI.java:562)
at com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:381)
... 23 common frames omitted
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2538;AMQ9204: Connection to host '/172.31.14.1:9010' rejected. [1=java.net.ConnectException[Connection timed out: connect],3=/172.31.14.1:9010,4=TCP,5=Socket.connect]
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.bindAndConnectSocket(RemoteTCPConnection.java:901)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.protocolConnect(RemoteTCPConnection.java:1381)
at com.ibm.mq.jmqi.remote.impl.RemoteConnection.connect(RemoteConnection.java:976)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getNewConnection(RemoteConnectionSpecification.java:553)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSessionFromNewConnection(RemoteConnectionSpecification.java:233)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSession(RemoteConnectionSpecification.java:141)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionPool.getSession(RemoteConnectionPool.java:127)
at com.ibm.mq.jmqi.remote.api.RemoteFAP$Connector.jmqiConnect(RemoteFAP.java:13302)
... 28 common frames omitted
Caused by: java.net.ConnectException: Connection timed out: connect
at java.base/java.net.PlainSocketImpl.connect0(Native Method)
at java.base/java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:101)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
at java.base/java.net.Socket.connect(Socket.java:609)
at java.base/java.net.Socket.connect(Socket.java:558)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection$4.run(RemoteTCPConnection.java:1022)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection$4.run(RemoteTCPConnection.java:1014)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.connectSocket(RemoteTCPConnection.java:1014)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.bindAndConnectSocket(RemoteTCPConnection.java:805)
... 35 common frames omitted
Here are some version numbers:
jhipster-dependencies.version: 3.0.7
Spring boot version: 2.1.10.RELEASE
ibmmq-jms-spring version(s) that are affected by this issue: Version 2.2.7
Java version (including vendor and platform): AdoptOpenJDK\jdk-11.0.6.10-hotspot
A small code sample that demonstrates the issue.
Here is my configuration in application.yml:
spring:
jms:
# Used for JMS Message reception.
isPubSubDomain: false
application:
oag:
# This can be a queue or a topic (if subdomain is defined)
# In case of a topic, sub domain must be set to public.
queueName: "BRIDGE.XXX.TO.YYY.TST"
isTopic: false
ibm:
mq:
queueManager:
channel: XXX_GWT11.BT1
connName: 172.31.14.1(9010)
user: xxxx
password:
Would it be possible to catch this exception to avoid breaking regular Spring Boot registration mechanism?
I cannot afford having my cluster down because I cannot access the JMS server.
Beside this, I opened this message on IBM MQ side here and a person suggested me to stop the JMS health indicator. So I set the following property to no avail:
management:
endpoint:
jms:
# Prevent Unreachable JMS Server from unregistering corehub from the registry, leading to unreachable microservice from the Gateway
enabled: false
Corresponding documentation is here
Any help would be greatly appreciated.
Thank you
Christophe
If you do not want your application to be considered unhealthy when JMS is down, disabling the JMS health indicator is what I would recommend. It hasn't worked for you as you have used management.endpoint.jms.enabled. The correct property to use is management.health.jms.enabled:
management:
health:
jms:
enabled: false
Running Consul and spring-cloud-consul I have five micro services all registered and showing as healthy. Each of these services is running on a different EC2 instance.
Using the name of the micro service that is registered in consul I can't get a connection through to any of them using a RestTemplate
If Service A was to call Service B with the following code
private final static String SERVICE = "http://instance-service";
#LoadBalanced
#Autowired
private RestTemplate restTemplate;
#GetMapping("/instances")
public ResponseEntity<String> instances() {
ParameterizedTypeReference<String> reference = new ParameterizedTypeReference<String>() {
};
return restTemplate.exchange(SERVICE, HttpMethod.GET, null, reference);
}
Service A throws
2017-12-19 17:30:36.000 ERROR 1 --- [nio-8443-exec-7] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://instance-service": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)] with root cause
and even with debugging set to TRACE I can see nothing coming in at all on Service B.
However if I take my identical setup and run it on a single EC2 instance everything works as expected.
All services are running in docker, but the consul agent is running on the host of each EC2 instance.
Each micro service has the following bootstrap.yml set
spring:
cloud:
consul:
config:
enabled: true
discovery:
instance-id: ${spring.application.name}
prefer-ip-address: true
If I give the RestTemplate a random endpoint it returns Unknown host so it knows the micro service is there, it's just being blocked somewhere before it hits the codebase.
How can I debug this further to find the real reason the services are getting rejected when running on different hosts?
I have tried opening all ports on the security config just to rule that out but it made no difference.
Edit: Looking at the Consul UI - I see my services are registered under the a private IP (172.18.0.2).
Should this be a public facing address?
172.18.0.2 is an ip in a a docker bridged network, you definitely want the ip address that is registered in consul to be accessible by clients that use the service discovery..
I don't know your network setup or needs but you can (some ideas, there are many) -
if you expose ports to the docker host (-p), make sure the ip of the docker host is registered for each service instance (I'm not familiar with bootstrap.yml or spring cloud)
run docker containers with --net=host
register the services in consul in different methods and outside of spring cloud's scope (API call to the consul agent, service definition file, https://github.com/gliderlabs/registrator, ...)
I have mircroservice environment based on spring-boot, where i am using zipkin server and discovery-server(eureka) and config-server. Now i have a rest-microservice which sends logs to zipkin server and this microservice is required to resolve where is zipkin server using discovery-server.
following is zipkin configuration i have in my rest-microservice's application.properties(pulled from config-server).
spring.zipkin.baseUrl=http://MTD-ZIPKIN-SERVER/
spring.zipkin.locator.discovery.enabled=true
spring.zipkin.enabled=true
...
here MTD-ZIPKIN-SERVER is zipkin-server name in discovery-server.
discovery-server dashboard.
but it does not try to resolve zipkin from discovery-server, instead it tries to connect directly using spring.zipkin.baseUrl, and i get below exception.
Dropped 1 spans due to ResourceAccessException(I/O error on POST request for "http://MTD-ZIPKIN-SERVER/api/v1/spans":
MTD-ZIPKIN-SERVER; nested exception is java.net.UnknownHostException:
MTD-ZIPKIN-SERVER)
org.springframework.web.client.ResourceAccessException: I/O error on
POST request for "http://MTD-ZIPKIN-SERVER/api/v1/spans":
MTD-ZIPKIN-SERVER; nested exception is java.net.UnknownHostException:
MTD-ZIPKIN-SERVER at
org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:666)
at
org.springframework.web.client.RestTemplate.execute(RestTemplate.java:628)
at
org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:590)
at
org.springframework.cloud.sleuth.zipkin.RestTemplateSender.post(RestTemplateSender.java:73)
at
org.springframework.cloud.sleuth.zipkin.RestTemplateSender.sendSpans(RestTemplateSender.java:46)
at
zipkin.reporter.AsyncReporter$BoundedAsyncReporter.flush(AsyncReporter.java:245)
at
zipkin.reporter.AsyncReporter$Builder.lambda$build$0(AsyncReporter.java:166)
at zipkin.reporter.AsyncReporter$Builder$$Lambda$1.run(Unknown
Source) at java.lang.Thread.run(Thread.java:745) Caused by:
java.net.UnknownHostException: MTD-ZIPKIN-SERVER at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
if i provide exact zipkin url in property spring.zipkin.baseUrl like below
spring.zipkin.baseUrl=http://localhost:5555/
then my rest-microservice is able to connect to zipkin-server.
My goal here is to read zipkin-server location from discovery-srever. What am i doing wrong? Do i need to add some zipkin enabling annotation on my spring-boot rest-microservice?
This feature is available in edgware release train. That corresponds to version 1.3.x of sleuth