A reactive Spring Boot application is used to generate Server Sent Events for clients subscribed by a key, e.g:
#GetMapping(path = "/sse", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<String> sse(#RequestParam String key) {
return Flux.interval(Duration.ofSeconds(2))
.map((i) -> key + " Fixed delay task - " + System.currentTimeMillis() / 1000);
}
Up to 10K concurrent Event Source connections can be established when run without Spring Cloud Gateway in front. With more connections the following exception is thrown:
2022-09-07 12:59:02.123 WARN 90517 --- [ctor-http-nio-1] io.netty.channel.DefaultChannelPipeline : An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.io.IOException: Too many open files
at java.base/sun.nio.ch.Net.accept(Native Method) ~[na:na]
at java.base/sun.nio.ch.ServerSocketChannelImpl.implAccept(ServerSocketChannelImpl.java:425) ~[na:na]
The problem occurs when Spring Cloud Gateway is in front of the service. The connection limit drops down to 2K connections. When exceeded, the service throws:
2022-09-07 13:09:37.590 WARN 90707 --- [ctor-http-nio-9] reactor.netty.channel.FluxReceive : [559c58ed-1, L:/127.0.0.1:9999 - R:/127.0.0.1:54567] An exception has been observed post termination, use DEBUG level to see the full stack: java.net.SocketException: Connection reset
And the gateway throws:
2022-09-07 12:59:55.696 ERROR 90264 --- [ctor-http-nio-2] a.w.r.e.AbstractErrorWebExceptionHandler : [7772ec3b-4438] 500 Server Error for HTTP GET "/sse?key=2682"
java.net.SocketException: Connection reset
at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[na:na]
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
*__checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ HTTP GET "/sse?key=2682" [ExceptionHandlingWebHandler]
Original Stack Trace:
at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[na:na]
at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[na:na]
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:258) ~[netty-buffer-4.1.79.Final.jar:4.1.79.Final]
The question is, why does Spring Cloud Gateway have such an impact on the number of concurrent connections?
The source code of the service app, the gateway, and a client JS app, used for receiving SSEs through multiple connections, is available on the git repo.
Related
I have a java (java-11) gRPC client streamer trying to talk with a golang gRPC server, with tls and envoy proxy in the middle.
And I'm getting
UNAVAILABLE: Failed ALPN negotiation: Unable to find compatible
protocol
error from the ServerResponseObserver.
Full stacktrace:
12/05/22 18:29:29 (798) grpc-default-executor-1 SEVERE *** ERROR *** server respond with onError:
io.grpc.StatusRuntimeException: UNAVAILABLE: Failed ALPN negotiation: Unable to find compatible protocol
Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
at io.grpc.Status.asRuntimeException(Status.java:535)
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:468)
at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:432)
at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:465)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
The calls do not reach the gRPC server and seem like they won't pass the envoy proxy.
envoy trace logs show a few ssl errors such as "ssl error occurred while read: WANT_READ"
configuration of the java gRPC client streamer:
ChannelCredentials credentials;
credentials = TlsChannelCredentials.newBuilder()
.trustManager(new File("<path-to-file>/cacert.pem"))
.build();
originChannel = Grpc.newChannelBuilderForAddress(host, port, credentials).build();
ClientInterceptor interceptor = new HeaderClientInterceptor();
Channel channel = ClientInterceptors.intercept(originChannel, interceptor);
stub = LogStreamerGrpc.newStub(channel);
Apparently, when using a proxy one need to ensure the proxy is configured correctly with h2 enabled for ALPN
from envoy documantation alpn_protocols:
alpn_protocols:
(repeated string) Supplies the list of ALPN protocols that the listener should expose. In practice this is likely to be set to one of two values (see the codec_type parameter in the HTTP connection manager for more information):
“h2,http/1.1” If the listener is going to support both HTTP/2 and HTTP/1.1.
“http/1.1” If the listener is only going to support HTTP/1.1.
There is no default for this parameter. If empty, Envoy will not expose ALPN.
For me, supporting h2 was enough.
Here is the configuration I added to envoy.yaml running on go under in order to make it work:
transport_socket:
common_tls_context:
alpn_protocols: h2
I added the following filter to my spring boot application.
#Component
public class RequestFilter implements Filter {
#Override
public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
HttpServletRequest req = (HttpServletRequest) request;
HttpServletResponse res = (HttpServletResponse) response;
String authorizationHeader = res.getHeader("Authorization");
System.out.println("auth :" + authorizationHeader);
chain.doFilter(req, res);
}
}
When I try to start the application locally it fails to start with this error message:
10-May-2022 17:09:43.268 SEVERE [RMI TCP Connection(2)-127.0.0.1] org.apache.catalina.core.StandardContext.startInternal One or more Filters failed to start. Full details will be found in the appropriate container log file
10-May-2022 17:09:43.269 SEVERE [RMI TCP Connection(2)-127.0.0.1] org.apache.catalina.core.StandardContext.startInternal Context [] startup failed due to previous errors
10-May-2022 17:09:43.369 SEVERE [RMI TCP Connection(2)-127.0.0.1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#8811a2]) and a value of type [org.springframework.security.core.context.SecurityContextImpl] (value [SecurityContextImpl [Null authentication]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
10-May-2022 17:09:43.369 SEVERE [RMI TCP Connection(2)-127.0.0.1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ROOT] created a ThreadLocal with key of type [com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier$1] (value [com.hazelcast.client.impl.spi.impl.ClientResponseHandlerSupplier$1#429304b7]) and a value of type [com.hazelcast.internal.util.MutableInteger] (value [com.hazelcast.internal.util.MutableInteger#378734f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
[2022-05-10 05:09:43,390] Artifact *****.war exploded: Error during artifact deployment. See server log for details.
10-May-2022 17:09:45.941 INFO [hz.client_1.internal-2] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [com.hazelcast.client.impl.protocol.codec.ClientPingCodec]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [com.hazelcast.client.impl.protocol.codec.ClientPingCodec]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1440)
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1428)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1267)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1228)
at com.hazelcast.client.impl.connection.tcp.HeartbeatManager.checkConnection(HeartbeatManager.java:94)
at com.hazelcast.client.impl.connection.tcp.HeartbeatManager.run(HeartbeatManager.java:78)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset$$$capture(FutureTask.java:308)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
The error suggests to find more details in the container logs.
When check the tomcat error logs on my local machine, there are no errors logged to it.
Any idea what might be causing this error? The problem is there is no relevant information in the tomcat log to debug this.
I have a strange issue that took me several days to narrow down. Basically, I have a Jhipster project based on Spring boot Version 2.1.10.RELEASE, which contains 4 microservices. We are interested here in 2 of them: Gateway and Corehub.
In the gateway, I have an angular app that performs a POST to /services/corehub/api/someendpoint which used to be working and that is failing now with different error messages, but the one I have more regularly is
{
"type": "https://www.jhipster.tech/problem/problem-with-message",
"title": "Method Not Allowed",
"status": 405,
"detail": "Request method 'POST' not supported",
"path": "/services/ambientcorehub/api/trips",
"message": "error.http.405"
}
I ended up looking at the traces of the Registry that keeps track of the microservices for internal communication and I found out that when this error occurs, I cannot find the corehub in the traces anymore. So it looks like the corehub micro service is not registered.
An other GIT branch of this service does not have this problem, so I performed a diff between these two branches and I removed the changes until I could narrow down the problem.
So, in the corehub, I have a JMS listener based on this mq-jms-spring implementation. The maven dependency is as follows:
<dependency>
<groupId>com.ibm.mq</groupId>
<artifactId>mq-jms-spring-boot-starter</artifactId>
<version>2.2.7</version>
</dependency>
I commented out my JmsListener class and its associated JmsContext (to get access to a topic), and kept only the configuration properties defining access to the server, along with the port, channel, topic name, etc.
If I comment the above maven dependency, my service works again.
If I keep the maven dependency, with the configuration only, my corehub microservice is not registered in the Registry and becomes not accessible anymore from the gateway and thus the Angular UI.
What is important to note, is that I have currently some network issue which prevents me from accessing the JMS Server.
So I believe the exception that is raised by this IBM library because the JMS server is not reachable, breaks the registration of the microservice towards the spring boot registry.
Here are the traces that come over and over in the corehub console:
2020-07-22 08:21:45.316 WARN 7964 --- [nfoReplicator-0] o.s.boot.actuate.jms.JmsHealthIndicator : JMS health check failed
com.ibm.msg.client.jms.DetailedIllegalStateException: JMSWMQ0018: Failed to connect to queue manager '' with connection mode 'Client' and host name '172.31.14.1(9010)'.
at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:489)
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:215)
at com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:448)
at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createV7ProviderConnection(WMQConnectionFactory.java:8475)
at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createProviderConnection(WMQConnectionFactory.java:7815)
at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl._createConnection(JmsConnectionFactoryImpl.java:303)
at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl.createConnection(JmsConnectionFactoryImpl.java:236)
at com.ibm.mq.jms.MQConnectionFactory.createCommonConnection(MQConnectionFactory.java:6005)
at com.ibm.mq.jms.MQConnectionFactory.createConnection(MQConnectionFactory.java:6030)
at org.springframework.jms.connection.SingleConnectionFactory.doCreateConnection(SingleConnectionFactory.java:409)
at org.springframework.jms.connection.SingleConnectionFactory.initConnection(SingleConnectionFactory.java:349)
at org.springframework.jms.connection.SingleConnectionFactory.getConnection(SingleConnectionFactory.java:327)
at org.springframework.jms.connection.SingleConnectionFactory.createConnection(SingleConnectionFactory.java:242)
at org.springframework.boot.actuate.jms.JmsHealthIndicator.doHealthCheck(JmsHealthIndicator.java:52)
at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:82)
at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:95)
at org.springframework.cloud.netflix.eureka.EurekaHealthCheckHandler.getHealthStatus(EurekaHealthCheckHandler.java:110)
at org.springframework.cloud.netflix.eureka.EurekaHealthCheckHandler.getStatus(EurekaHealthCheckHandler.java:106)
at com.netflix.discovery.DiscoveryClient.refreshInstanceInfo(DiscoveryClient.java:1406)
at com.netflix.discovery.InstanceInfoReplicator.run(InstanceInfoReplicator.java:117)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.ibm.mq.MQException: JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2538' ('MQRC_HOST_NOT_AVAILABLE').
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:203)
... 24 common frames omitted
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2538;AMQ9204: Connection to host '172.31.14.1(9010)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2538;AMQ9204: Connection to host '/172.31.14.1:9010' rejected. [1=java.net.ConnectException[Connection timed out: connect],3=/172.31.14.1:9010,4=TCP,5=Socket.connect]],3=172.31.14.1(9010),5=RemoteTCPConnection.bindAndConnectSocket]
at com.ibm.mq.jmqi.remote.api.RemoteFAP$Connector.jmqiConnect(RemoteFAP.java:13558)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1426)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1385)
at com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.jmqiConnect(InterceptedJmqiImpl.java:377)
at com.ibm.mq.ese.jmqi.ESEJMQI.jmqiConnect(ESEJMQI.java:562)
at com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:381)
... 23 common frames omitted
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2538;AMQ9204: Connection to host '/172.31.14.1:9010' rejected. [1=java.net.ConnectException[Connection timed out: connect],3=/172.31.14.1:9010,4=TCP,5=Socket.connect]
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.bindAndConnectSocket(RemoteTCPConnection.java:901)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.protocolConnect(RemoteTCPConnection.java:1381)
at com.ibm.mq.jmqi.remote.impl.RemoteConnection.connect(RemoteConnection.java:976)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getNewConnection(RemoteConnectionSpecification.java:553)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSessionFromNewConnection(RemoteConnectionSpecification.java:233)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSession(RemoteConnectionSpecification.java:141)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionPool.getSession(RemoteConnectionPool.java:127)
at com.ibm.mq.jmqi.remote.api.RemoteFAP$Connector.jmqiConnect(RemoteFAP.java:13302)
... 28 common frames omitted
Caused by: java.net.ConnectException: Connection timed out: connect
at java.base/java.net.PlainSocketImpl.connect0(Native Method)
at java.base/java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:101)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
at java.base/java.net.Socket.connect(Socket.java:609)
at java.base/java.net.Socket.connect(Socket.java:558)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection$4.run(RemoteTCPConnection.java:1022)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection$4.run(RemoteTCPConnection.java:1014)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.connectSocket(RemoteTCPConnection.java:1014)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.bindAndConnectSocket(RemoteTCPConnection.java:805)
... 35 common frames omitted
Here are some version numbers:
jhipster-dependencies.version: 3.0.7
Spring boot version: 2.1.10.RELEASE
ibmmq-jms-spring version(s) that are affected by this issue: Version 2.2.7
Java version (including vendor and platform): AdoptOpenJDK\jdk-11.0.6.10-hotspot
A small code sample that demonstrates the issue.
Here is my configuration in application.yml:
spring:
jms:
# Used for JMS Message reception.
isPubSubDomain: false
application:
oag:
# This can be a queue or a topic (if subdomain is defined)
# In case of a topic, sub domain must be set to public.
queueName: "BRIDGE.XXX.TO.YYY.TST"
isTopic: false
ibm:
mq:
queueManager:
channel: XXX_GWT11.BT1
connName: 172.31.14.1(9010)
user: xxxx
password:
Would it be possible to catch this exception to avoid breaking regular Spring Boot registration mechanism?
I cannot afford having my cluster down because I cannot access the JMS server.
Beside this, I opened this message on IBM MQ side here and a person suggested me to stop the JMS health indicator. So I set the following property to no avail:
management:
endpoint:
jms:
# Prevent Unreachable JMS Server from unregistering corehub from the registry, leading to unreachable microservice from the Gateway
enabled: false
Corresponding documentation is here
Any help would be greatly appreciated.
Thank you
Christophe
If you do not want your application to be considered unhealthy when JMS is down, disabling the JMS health indicator is what I would recommend. It hasn't worked for you as you have used management.endpoint.jms.enabled. The correct property to use is management.health.jms.enabled:
management:
health:
jms:
enabled: false
We have an activemq listener MDB with EJB, listening to a broker over SSL using activemq-rar.rar connector that has the broker, SSL cert, credential configurations. The listener is running well in JBOSS EAP 6.4. However, the need is to convert this MDB into Spring Boot ActiveMQ listener. I could see many articles examples about Spring Boot ActiveMQ with simple tcp://localhost:61616. Is there a complete reference to working example of producing message to and listening message from external broker over SSL using Spring Boot.
I have tried to create produce and consume messages using tcp per the instructions at
https://www.devglan.com/spring-boot/spring-boot-jms-activemq-example
and
http://activemq.apache.org/how-do-i-use-ssl.html
spring.activemq.broker-url=ssl://localhost:61616
spring.activemq.user=admin
spring.activemq.password=admin
spring.activemq.keyStore=broker.ks
spring.activemq.keyStorePassword=password
javax.net.ssl.keyStore=client.ks
javax.net.ssl.keyStorePassword=password
javax.net.ssl.trustStore=client.ts
I am seeing below exception. Need help.
2019-01-14 00:54:33.209 INFO 13964 --- [ main] com.rime.springboot.amc.Application : Started Application in 5.783 seconds (JVM running for 7.081)
2019-01-14 00:54:38.203 INFO 13964 --- [enerContainer-1] o.s.j.l.DefaultMessageListenerContainer : JMS message listener invoker needs to establish shared Connection
2019-01-14 00:54:38.244 ERROR 13964 --- [enerContainer-1] o.s.j.l.DefaultMessageListenerContainer : Could not refresh JMS Connection for destination 'inbound.queue' - retrying using FixedBackOff{interval=5000, currentAttempts=0, maxAttempts=unlimited}. Cause: Could not connect to broker URL: ssl://localhost:61616. Reason: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
2019-01-14 00:54:43.341 ERROR 13964 --- [enerContainer-1] o.s.j.l.DefaultMessageListenerContainer : Could not refresh JMS Connection for destination 'inbound.queue' - retrying using FixedBackOff{interval=5000, currentAttempts=1, maxAttempts=unlimited}. Cause: Could not connect to broker URL: ssl://localhost:61616. Reason: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
It appears that auto-configuration via properties is not currently supported. See: https://github.com/spring-projects/spring-boot/issues/17365 and https://github.com/spring-projects/spring-boot/issues/17589.
In the meantime, it is possible to override the default connection with your own ActiveMQSslConnectionFactory bean, providing the SSL configuration properties you need:
#Configuration
public class ActiveMQConfiguration {
#Bean
public ActiveMQSslConnectionFactory activeMQSslConnectionFactory(
#Value("${spring.activemq.broker-url}") String brokerUrl,
#Value("${spring.activemq.ssl.trustStorePath}") String trustStorePath,
#Value("${spring.activemq.ssl.trustStorePass}") String trustStorePass,
#Value("${spring.activemq.ssl.keyStorePath}") String keyStorePath,
#Value("${spring.activemq.ssl.keyStorePass}") String keyStorePass) throws Exception {
ActiveMQSslConnectionFactory factory = new ActiveMQSslConnectionFactory(brokerUrl);
factory.setTrustStore(trustStorePath);
factory.setTrustStorePassword(trustStorePass);
factory.setKeyStore(keyStorePath);
factory.setKeyStorePassword(keyStorePass);
return factory;
}
}
The spring.activemq.ssl.* properties aren't based on any existing or documented properties, so you don't need to use those specifically.
I have mircroservice environment based on spring-boot, where i am using zipkin server and discovery-server(eureka) and config-server. Now i have a rest-microservice which sends logs to zipkin server and this microservice is required to resolve where is zipkin server using discovery-server.
following is zipkin configuration i have in my rest-microservice's application.properties(pulled from config-server).
spring.zipkin.baseUrl=http://MTD-ZIPKIN-SERVER/
spring.zipkin.locator.discovery.enabled=true
spring.zipkin.enabled=true
...
here MTD-ZIPKIN-SERVER is zipkin-server name in discovery-server.
discovery-server dashboard.
but it does not try to resolve zipkin from discovery-server, instead it tries to connect directly using spring.zipkin.baseUrl, and i get below exception.
Dropped 1 spans due to ResourceAccessException(I/O error on POST request for "http://MTD-ZIPKIN-SERVER/api/v1/spans":
MTD-ZIPKIN-SERVER; nested exception is java.net.UnknownHostException:
MTD-ZIPKIN-SERVER)
org.springframework.web.client.ResourceAccessException: I/O error on
POST request for "http://MTD-ZIPKIN-SERVER/api/v1/spans":
MTD-ZIPKIN-SERVER; nested exception is java.net.UnknownHostException:
MTD-ZIPKIN-SERVER at
org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:666)
at
org.springframework.web.client.RestTemplate.execute(RestTemplate.java:628)
at
org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:590)
at
org.springframework.cloud.sleuth.zipkin.RestTemplateSender.post(RestTemplateSender.java:73)
at
org.springframework.cloud.sleuth.zipkin.RestTemplateSender.sendSpans(RestTemplateSender.java:46)
at
zipkin.reporter.AsyncReporter$BoundedAsyncReporter.flush(AsyncReporter.java:245)
at
zipkin.reporter.AsyncReporter$Builder.lambda$build$0(AsyncReporter.java:166)
at zipkin.reporter.AsyncReporter$Builder$$Lambda$1.run(Unknown
Source) at java.lang.Thread.run(Thread.java:745) Caused by:
java.net.UnknownHostException: MTD-ZIPKIN-SERVER at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
if i provide exact zipkin url in property spring.zipkin.baseUrl like below
spring.zipkin.baseUrl=http://localhost:5555/
then my rest-microservice is able to connect to zipkin-server.
My goal here is to read zipkin-server location from discovery-srever. What am i doing wrong? Do i need to add some zipkin enabling annotation on my spring-boot rest-microservice?
This feature is available in edgware release train. That corresponds to version 1.3.x of sleuth