MQ Exception: MQRC_SSL_PEER_NAME_MISMATCH - ibm-mq

I am connecting to MQ message broker via SSL and I sometimes get the exception below. The strange thing is that I am not always having this error and restarting my client process often resolves the issue.
I have already set the SSLPeername when creating the MQConnectionfactory to a value of CN=* but this doesn't seems to resolve the issue. It is also strange to me that it doesn't occur always but only from time to time. Can someone assist me on this ?
thx and kr,
Frederik.
com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0018: Failed to connect to queue manager 'UM01' with connection mode '1' and host name 'xxx.xx.net(1449)'.
at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:585) ~[com.ibm.mqjms.jar:7.5.0.1 - p750-001-130308]
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:221) ~[com.ibm.mqjms.jar:7.5.0.1 - p750-001-130308]
at com.ibm.msg.client.wmq.internal.WMQSession.connect(WMQSession.java:392) ~[com.ibm.mqjms.jar:7.5.0.1 - p750-001-130308]
at com.ibm.msg.client.wmq.internal.WMQSession.<init>(WMQSession.java:301) ~[com.ibm.mqjms.jar:7.5.0.1 - p750-001-130308]
at com.ibm.msg.client.wmq.internal.WMQConnection.createSession(WMQConnection.java:884) ~[com.ibm.mqjms.jar:7.5.0.1 - p750-001-130308]
at com.ibm.msg.client.jms.internal.JmsConnectionImpl.createSession(JmsConnectionImpl.java:594) ~[com.ibm.mqjms.jar:7.5.0.1 - p750-001-130308]
at com.ibm.msg.client.jms.internal.JmsQueueConnectionImpl.createQueueSession(JmsQueueConnectionImpl.java:148) ~[com.ibm.mqjms.jar:7.5.0.1 - p750-001-130308]
at com.ibm.mq.jms.MQQueueConnection.createQueueSession(MQQueueConnection.java:140) ~[com.ibm.mqjms.jar:7.5.0.1 - p750-001-130308]
at com.ibm.mq.jms.MQQueueConnection.createSession(MQQueueConnection.java:162) ~[com.ibm.mqjms.jar:7.5.0.1 - p750-001-130308]
at xxxx.server.comm.jms.spring.SingleConnectionFactory.createSession(SingleConnectionFactory.java:521) ~[xxxx.server.comm.jms.jar:na]
at xxxx.server.comm.jms.spring.CachingConnectionFactory.getSession(CachingConnectionFactory.java:252) ~[xxxx.server.comm.jms.jar:na]
at xxxx.server.comm.jms.spring.SingleConnectionFactory$SharedConnectionInvocationHandler.invoke(SingleConnectionFactory.java:731) ~[xxxx.server.comm.jms.jar:na]
at com.sun.proxy.$Proxy23.createSession(Unknown Source) ~[na:na]
at xxxx.server.comm.jms.JMSSenderImpl.send(JMSSenderImpl.java:93) ~[xxxx.server.comm.jms.jar:na]
at java.util.TimerThread.mainLoop(Unknown Source) [na:1.8.0_102]
at java.util.TimerThread.run(Unknown Source) [na:1.8.0_102]
Caused by: com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2398' ('MQRC_SSL_PEER_NAME_MISMATCH').
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:209) ~[com.ibm.mqjms.jar:7.5.0.1 - p750-001-130308]
... 20 common frames omitted
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2398;AMQ9204: Connection to host 'xxx.xxx.net(1449)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2398;AMQ9636: SSL distinguished name does not match peer name, channel '?'. [4=CN=xxxxx]],3=xxx.xxx.net(1449),5=RemoteTCPConnection.protocolConnect]
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:2050) ~[com.ibm.mq.jmqi.jar:7.5.0.1 - p750-001-130308]
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1223) ~[com.ibm.mq.jmqi.jar:7.5.0.1 - p750-001-130308]
at com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.jmqiConnect(InterceptedJmqiImpl.java:311) ~[com.ibm.mq.jmqi.jar:7.5.0.1 - p750-001-130308]
at com.ibm.mq.ese.jmqi.ESEJMQI.jmqiConnect(ESEJMQI.java:337) ~[com.ibm.mq.jmqi.jar:7.5.0.1 - p750-001-130308]
at com.ibm.msg.client.wmq.internal.WMQSession.connect(WMQSession.java:348) ~[com.ibm.mqjms.jar:7.5.0.1 - p750-001-130308]
... 19 common frames omitted
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2398;AMQ9636: SSL distinguished name does not match peer name, channel '?'. [4=CN=xxx]
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.protocolConnect(RemoteTCPConnection.java:1219) ~[com.ibm.mq.jmqi.jar:7.5.0.1 - p750-001-130308]
at com.ibm.mq.jmqi.remote.impl.RemoteConnection.connect(RemoteConnection.java:710) ~[com.ibm.mq.jmqi.jar:7.5.0.1 - p750-001-130308]
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSessionFromNewConnection(RemoteConnectionSpecification.java:400) ~[com.ibm.mq.jmqi.jar:7.5.0.1 - p750-001-130308]
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSession(RemoteConnectionSpecification.java:299) ~[com.ibm.mq.jmqi.jar:7.5.0.1 - p750-001-130308]
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionPool.getSession(RemoteConnectionPool.java:164) ~[com.ibm.mq.jmqi.jar:7.5.0.1 - p750-001-130308]
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiConnect(RemoteFAP.java:1595) ~[com.ibm.mq.jmqi.jar:7.5.0.1 - p750-001-130308]
... 23 common frames omitted
Here is the code I use to make a connection:
final MQQueueConnectionFactory factory = new MQQueueConnectionFactory();
factory.setChannel(this.jmsConfig.getChannel());
factory.setHostName(this.jmsConfig.getHostname());
factory.setPort(this.jmsConfig.getPort());
factory.setQueueManager(this.jmsConfig.getQueuemanager());
factory.setTransportType(JMSC.MQJMS_TP_CLIENT_MQ_TCPIP);
String sslPeerName = this.jmsConfig.getSslPeerName();
if (StringUtils.isNotBlank(sslPeerName))
{
logger.info("Setting SSL Peername: {} ", sslPeerName);
factory.setSSLPeerName(sslPeerName);
}
String sslCipherSuite = this.jmsConfig.getSslCipherSuite();
if (StringUtils.isNotBlank(sslCipherSuite))
{
logger.info("Setting SSL Cipher Suite: {} ", sslCipherSuite);
factory.setSSLCipherSuite(sslCipherSuite);
}
connection = factory.createConnection();
The exception happens during a createSession
con.createSession(transacted, ackMode);

Related

RedisConnectionFailureException: Redis connection failed; nested exception is io.lettuce.core.RedisConnectionException: Unable to connect to [redis://

I have two businesses connected to the same redis-cluster, and the other business does not report any error, but this business will report the following error. During the error reporting period, I manually use redis-cli to connect to the cluster, and I can set/get and view the cluster information at the same time. Node There is no problem with information, slots, etc., and there is no problem with node memory/cpu。Does anyone know what the problem might be, thank you
org.springframework.data.redis.RedisConnectionFailureException: Unable to connect to Redis; nested exception is org.springframework.data.redis.connection.PoolException: Could not get a resource from the pool; nested exception is org.springframework.data.redis.RedisConnectionFailureException: Redis connection failed; nested exception is io.lettuce.core.RedisConnectionException: Unable to connect to [redis://********************#xxxx:6388?timeout=1s,]
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$ExceptionTranslatingConnectionProvider.translateException(LettuceConnectionFactory.java:1553)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$ExceptionTranslatingConnectionProvider.getConnection(LettuceConnectionFactory.java:1461)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getNativeConnection(LettuceConnectionFactory.java:1247)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$SharedConnection.getConnection(LettuceConnectionFactory.java:1230)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getClusterConnection(LettuceConnectionFactory.java:378)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getConnection(LettuceConnectionFactory.java:355)
at org.springframework.data.redis.core.RedisConnectionUtils.fetchConnection(RedisConnectionUtils.java:193)
at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:144)
at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:105)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:209)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:189)
at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:96)
at org.springframework.data.redis.core.DefaultValueOperations.set(DefaultValueOperations.java:236)
at com.jazz.neptune.biz.util.RedisUtil.set(RedisUtil.java:139)
at com.jazz.neptune.webscoket.goodway.WebSocketUtilManager.updateTableDetail(WebSocketUtilManager.java:471)
at com.jazz.neptune.webscoket.goodway.WebSocketUtilManager.handleMsg(WebSocketUtilManager.java:375)
at com.jazz.neptune.webscoket.goodway.WebSocketUtilManager.onMessage(WebSocketUtilManager.java:153)
at com.jazz.neptune.webscoket.goodway.websocket.dispatcher.MainThreadResponseDelivery.onMessage(MainThreadResponseDelivery.java:150)
at com.jazz.neptune.webscoket.goodway.websocket.dispatcher.DefaultResponseDispatcher.onMessage(DefaultResponseDispatcher.java:32)
at com.jazz.neptune.webscoket.goodway.websocket.response.TextResponse.onResponse(TextResponse.java:32)
at com.jazz.neptune.webscoket.goodway.websocket.dispatcher.EngineThread.run(EngineThread.java:41)
Caused by: org.springframework.data.redis.connection.PoolException: Could not get a resource from the pool; nested exception is org.springframework.data.redis.RedisConnectionFailureException: Redis connection failed; nested exception is io.lettuce.core.RedisConnectionException: Unable to connect to [redis://********************#xxxxxx:6388?timeout=1s, ]
at org.springframework.data.redis.connection.lettuce.LettucePoolingConnectionProvider.getConnection(LettucePoolingConnectionProvider.java:109)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory$ExceptionTranslatingConnectionProvider.getConnection(LettuceConnectionFactory.java:1459)
... 19 common frames omitted
Caused by: org.springframework.data.redis.RedisConnectionFailureException: Redis connection failed; nested exception is io.lettuce.core.RedisConnectionException: Unable to connect to [redis://********************#xxxxx:6388?timeout=1s,]
at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:66)
at org.springframework.data.redis.connection.lettuce.LettuceFutureUtils.join(LettuceFutureUtils.java:74)
at org.springframework.data.redis.connection.lettuce.LettuceConnectionProvider.getConnection(LettuceConnectionProvider.java:53)
at org.springframework.data.redis.connection.lettuce.LettucePoolingConnectionProvider.lambda$null$0(LettucePoolingConnectionProvider.java:97)
at io.lettuce.core.support.ConnectionPoolSupport$RedisPooledObjectFactory.create(ConnectionPoolSupport.java:211)
at io.lettuce.core.support.ConnectionPoolSupport$RedisPooledObjectFactory.create(ConnectionPoolSupport.java:201)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:58)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:889)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:424)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:349)
at io.lettuce.core.support.ConnectionPoolSupport$1.borrowObject(ConnectionPoolSupport.java:122)
at io.lettuce.core.support.ConnectionPoolSupport$1.borrowObject(ConnectionPoolSupport.java:117)
at org.springframework.data.redis.connection.lettuce.LettucePoolingConnectionProvider.getConnection(LettucePoolingConnectionProvider.java:103)
... 20 common frames omitted
Caused by: io.lettuce.core.RedisConnectionException: Unable to connect to [redis://********************#xxxxx:6388?timeout=1s, ]
at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:78)
at io.lettuce.core.cluster.RedisClusterClient.lambda$transformAsyncConnectionException$40(RedisClusterClient.java:1157)
at io.lettuce.core.DefaultConnectionFuture.lambda$thenCompose$1(DefaultConnectionFuture.java:253)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)
at io.lettuce.core.DefaultConnectionFuture.thenCompose(DefaultConnectionFuture.java:250)
at io.lettuce.core.cluster.RedisClusterClient.transformAsyncConnectionException(RedisClusterClient.java:1154)
at io.lettuce.core.cluster.RedisClusterClient.connectAsync(RedisClusterClient.java:417)
at org.springframework.data.redis.connection.lettuce.ClusterConnectionProvider.getConnectionAsync(ClusterConnectionProvider.java:108)
at org.springframework.data.redis.connection.lettuce.ClusterConnectionProvider.getConnectionAsync(ClusterConnectionProvider.java:40)
... 31 common frames omitted
Caused by: java.lang.IllegalStateException: Cannot connect, Event executor group is terminated.
at io.lettuce.core.AbstractRedisClient.initializeChannelAsync(AbstractRedisClient.java:360)
at io.lettuce.core.cluster.RedisClusterClient.connectStatefulAsync(RedisClusterClient.java:752)
at io.lettuce.core.cluster.RedisClusterClient.connect(RedisClusterClient.java:658)
at io.lettuce.core.cluster.RedisClusterClient.lambda$connectClusterAsync$7(RedisClusterClient.java:639)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:100)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:100)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:100)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:100)
at reactor.core.publisher.Operators.error(Operators.java:198)
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:48)
at reactor.core.publisher.Mono.subscribe(Mono.java:4361)
at reactor.core.publisher.Mono.subscribeWith(Mono.java:4476)
at reactor.core.publisher.Mono.toFuture(Mono.java:4881)
at io.lettuce.core.cluster.RedisClusterClient.connectClusterAsync(RedisClusterClient.java:652)
... 34 common frames omitted
Suppressed: java.lang.IllegalStateException: Cannot connect, Event executor group is terminated.
at io.lettuce.core.AbstractRedisClient.initializeChannelAsync(AbstractRedisClient.java:360)
at io.lettuce.core.cluster.RedisClusterClient.connectStatefulAsync(RedisClusterClient.java:752)
at io.lettuce.core.cluster.RedisClusterClient.connect(RedisClusterClient.java:658)
at io.lettuce.core.cluster.RedisClusterClient.lambda$connectClusterAsync$7(RedisClusterClient.java:639)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
... 43 common frames omitted
Suppressed: java.lang.IllegalStateException: Cannot connect, Event executor group is terminated.
at io.lettuce.core.AbstractRedisClient.initializeChannelAsync(AbstractRedisClient.java:360)
at io.lettuce.core.cluster.RedisClusterClient.connectStatefulAsync(RedisClusterClient.java:752)
at io.lettuce.core.cluster.RedisClusterClient.connect(RedisClusterClient.java:658)
at io.lettuce.core.cluster.RedisClusterClient.lambda$connectClusterAsync$7(RedisClusterClient.java:639)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
... 42 common frames omitted
Suppressed: java.lang.IllegalStateException: Cannot connect, Event executor group is terminated.
at io.lettuce.core.AbstractRedisClient.initializeChannelAsync(AbstractRedisClient.java:360)
at io.lettuce.core.cluster.RedisClusterClient.connectStatefulAsync(RedisClusterClient.java:752)
at io.lettuce.core.cluster.RedisClusterClient.connect(RedisClusterClient.java:658)
at io.lettuce.core.cluster.RedisClusterClient.lambda$connectClusterAsync$7(RedisClusterClient.java:639)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
... 41 common frames omitted
Suppressed: java.lang.IllegalStateException: Cannot connect, Event executor group is terminated.
at io.lettuce.core.AbstractRedisClient.initializeChannelAsync(AbstractRedisClient.java:360)
at io.lettuce.core.cluster.RedisClusterClient.connectStatefulAsync(RedisClusterClient.java:752)
at io.lettuce.core.cluster.RedisClusterClient.connect(RedisClusterClient.java:658)
at io.lettuce.core.cluster.RedisClusterClient.lambda$connectClusterAsync$7(RedisClusterClient.java:639)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
... 40 common frames omitted
Suppressed: java.lang.IllegalStateException: Cannot connect, Event executor group is terminated.
at io.lettuce.core.AbstractRedisClient.initializeChannelAsync(AbstractRedisClient.java:360)
at io.lettuce.core.cluster.RedisClusterClient.connectStatefulAsync(RedisClusterClient.java:752)
at io.lettuce.core.cluster.RedisClusterClient.connect(RedisClusterClient.java:658)
at io.lettuce.core.cluster.RedisClusterClient.lambda$connectClusterAsync$6(RedisClusterClient.java:635)
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:44)
... 38 common frames omitted
restart redis-cluster

Cant connect to MQ Spring boot SSL

I am trying to establish a connection to the MQ service
For SSL connection I use the commands
-Djavax.net.ssl.trustStore=/opt/app/key.jks"
-Djavax.net.ssl.trustStorePassword=111111
Appliction.properties config
ibm.mq.connName=10.20.31.25(1414)
ibm.mq.channel=OIV.CHAN
ibm.mq.queueManager=OIV
ibm.mq.user=TEST
ibm.mq.password=passw0rd
ibm.mq.ssl-cipher-spec=TLS_RSA_WITH_AES_256_CBC_SHA
When the application starts, everything is ok
INFO IbmJmsConfiguration - Initializing SSL context:
protocol=TLSv1.2, keyStore=null, trustStore=/opt/app/key.jks
INFO IbmJmsConfiguration - SSL context initialized:
keyManagers item(s) = 0, trustManagers item(s) = 1
But when making a request to MQ, I get an error
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2393;AMQ9771: SSL handshake failed. [1=java.lang.IllegalArgumentException[Unsupported ciphersuite SSL_RSA_WITH_AES_256_CBC_SHA256]
text error
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2393;AMQ9771: SSL handshake
failed. [1=java.lang.IllegalArgumentException[Unsupported ciphersuite
SSL_RSA_WITH_AES_256_CBC_SHA],3=10.90.51.15/10.90.50.15:1414
(10.96.51.15),4=SSLSocket.createSocket,5=default]
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.makeSocketSecure(RemoteTCPConnection.java:2360)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.bindAndConnectSocket(RemoteTCPConnection.java:816)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.protocolConnect(RemoteTCPConnection.java:1381)
at com.ibm.mq.jmqi.remote.impl.RemoteConnection.connect(RemoteConnection.java:976)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getNewConnection(RemoteConnectionSpecification.java:553)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSessionFromNewConnection(RemoteConnectionSpecification.java:233)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionSpecification.getSession(RemoteConnectionSpecification.java:141)
at com.ibm.mq.jmqi.remote.impl.RemoteConnectionPool.getSession(RemoteConnectionPool.java:127)
at com.ibm.mq.jmqi.remote.api.RemoteFAP$Connector.jmqiConnect(RemoteFAP.java:13302)
... 74 common frames omitted
Caused by: java.lang.IllegalArgumentException: Unsupported ciphersuite SSL_RSA_WITH_AES_256_CBC_SHA
at sun.security.ssl.CipherSuite.valueOf(CipherSuite.java:228)
at sun.security.ssl.CipherSuiteList.<init>(CipherSuiteList.java:79)
at sun.security.ssl.SSLSocketImpl.setEnabledCipherSuites(SSLSocketImpl.java:2491)
at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.makeSocketSecure(RemoteTCPConnection.java:2351)
... 82 common frames omitted

Spring boot Show Error with tomcat 8 connection pool Too many open files

Anybody may be help?
I have app that I bilt In Spring Boot whith externa Tomcat 8 using connection pool in context.xml.
But when I call with threads show me error after few minutes.
Look my code and configuration:
TKS 4ALL!!!
context.xml
<Resource name="jdbc/ocp-api" auth="Container"
type="javax.sql.DataSource"
maxTotal="150"
max-active= "100"
max-idle= "80"
min-idle="8"
maxWaitMillis="2000"
username="postgres"
password="postgres"
driverClassName="org.postgresql.Driver"
url="jdbc:postgresql://<myIpServer>:5432/db_ocp_ago2017"/>
application.properties
spring.datasource.jndi-name=jdbc/ocp-api
spring.jpa.show-sql=true
spring.application.name=ocp-api
server.contextPath=/ocp-api
spring.jackson.date-format=yyyy-MM-dd HH:mm:ss.SSSZ
spring.jackson.joda-date-time-format=yyyy-MM-dd' 'HH:mm:ss.SSSZ
logging.level.org.springframework.web=ERROR
logging.level.com.mkyong=DEBUG
# Logging pattern for the console
logging.pattern.console= "%d{yyyy-MM-dd HH:mm:ss} - %msg%n"
# Logging pattern for file
logging.pattern.file= "%d{yyyy-MM-dd HH:mm:ss} [%thread] %-5level %logger{36} - %msg%n"
logging.file=/var/supportcomm/ocp/log/ocp-api.log
flyway.baselineOnMigrate=true
26-Jul-2018 16:40:29.049 WARNING
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.deploy.NamingResourcesImpl.cleanUp Failed to
retrieve JNDI naming context for container
[StandardEngine[Catalina].StandardHost[localhost].StandardContext[/ocp-api]]
so no cleanup was performed for that container
javax.naming.NameNotFoundException: Name [comp/env] is not bound in
this Context. Unable to find [comp].
at org.apache.naming.NamingContext.lookup(NamingContext.java:824)
at org.apache.naming.NamingContext.lookup(NamingContext.java:172)
at org.apache.catalina.deploy.NamingResourcesImpl.cleanUp(NamingResourcesImpl.java:993)
at org.apache.catalina.deploy.NamingResourcesImpl.stopInternal(NamingResourcesImpl.java:975)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:221)
at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5551)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:221)
at org.apache.catalina.util.LifecycleBase.destroy(LifecycleBase.java:259)
at org.apache.catalina.core.ContainerBase.removeChild(ContainerBase.java:832)
at org.apache.catalina.startup.HostConfig.undeploy(HostConfig.java:1395)
at org.apache.catalina.startup.HostConfig.checkResources(HostConfig.java:1303)
at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1581)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:284)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:95)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1140)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1376)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1380)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1348)
at java.lang.Thread.run(Thread.java:745)
"2018-07-26 16:51:27 - Cannot forward to error page for request
[/occupation] as the response has already been committed. As a result,
the response may have the wrong status code. If your application is
running on WebSphere Application Server you may be able to resolve
this problem by setting
com.ibm.ws.webcontainer.invokeFlushAfterService to false
"org.apache.catalina.connector.ClientAbortException:
java.io.IOException: Broken pipe
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:396)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:426)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:345)
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:320)
at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:110)
at com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1054)
at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:607)
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:286)
at org.springframework.http.converter.AbstractGenericHttpMessageConverter.write(AbstractGenericHttpMessageConverter.java:106)
at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:231)
at org.springframework.web.servlet.mvc.method.annotation.HttpEntityMethodProcessor.handleReturnValue(HttpEntityMethodProcessor.java:203)
at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:81)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:113)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:124)
at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)
at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:172)
at org.apache.coyote.http11.InternalNioOutputBuffer.writeToSocket(InternalNioOutputBuffer.java:139)
at org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(InternalNioOutputBuffer.java:197)
at org.apache.coyote.http11.InternalNioOutputBuffer.access$000(InternalNioOutputBuffer.java:41)
at org.apache.coyote.http11.InternalNioOutputBuffer$SocketOutputBuffer.doWrite(InternalNioOutputBuffer.java:320)
at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:118)
at org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:256)
at org.apache.coyote.Response.doWrite(Response.java:491)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:391)
... 66 common frames omitted
26-Jul-2018 16:52:20.989 SEVERE [http-nio-8280-Acceptor-0]
org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept
failed java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:693)
at java.lang.Thread.run(Thread.java:745)
[pool-24-thread-14] WARN
org.hibernate.engine.jdbc.spi.SqlExceptionHelper - SQL Error: 0,
SQLState: 08001"2018-07-26 16:52:32 - SQL Error: 0, SQLState: null
2018-07-26 16:52:32 - SQL Error: 0, SQLState: null 2018-07-26 16:52:32
- SQL Error: 0, SQLState: null 2018-07-26 16:52:32 - SQL Error: 0, SQLState: null 2018-07-26 16:52:32 - SQL Error: 0, SQLState: null
2018-07-26 16:52:32 - SQL Error: 0, SQLState: null 2018-07-26 16:52:33
- java.io.FileNotFoundException: /vol/app/apache-tomcat-8.0.38/webapps/ocp-api/WEB-INF/lib/jackson-core-2.8.10.jar
(Too many open files) 2018-07-26 16:52:34 -
java.io.FileNotFoundException:
/vol/app/apache-tomcat-8.0.38/webapps/ocp-api/WEB-INF/lib/jackson-core-2.8.10.jar
(Too many open files) 2018-07-26 16:52:34 -
java.io.FileNotFoundException:
/vol/app/apache-tomcat-8.0.38/webapps/ocp-api/WEB-INF/lib/jackson-core-2.8.10.jar
(Too many open files)

After upgrading from SonarQube 5.3 to 5.6.2, Backround Task is throwing IllegalStateException: Fail to read ISSUES.LOCATIONS

After upgrading from SonarQube 5.3 to Sonaeqube 5.6.2, I am unable to successfully run a scan.
Note that I am using a very slightly modified version of 5.6.2 to workaround an issue described here: IllegalStateException when migrating databases from SonarQube 5.3 to 5.6.2 (and 5.5), but I'm fairly confident that has nothing to do with this issue. Completely different part of the code.
I am using version 2.1 of the gradle plugin. I have the following sonar plugins installed:
sonar-android-plugin-1.1.jar
sonar-build-breaker-plugin-2.1.jar
sonar-cobertura-plugin-1.7.jar
sonar-findbugs-plugin-3.4.4.jar
sonar-generic-coverage-plugin-1.2.jar
sonar-groovy-plugin-1.4.jar
sonar-java-plugin-4.2.jar
sonar-javascript-plugin-2.16.0.2922.jar
sonar-pitest-plugin-0.6.jar
sonar-piwik-plugin-1.0.jar
sonar-python-plugin-1.6.jar
sonar-timeline-plugin-1.5.jar
sonar-xml-plugin-1.4.1.jar
Gradle is throwing "IllegalStateException: Report processing did not complete successfully: FAILED". and the background task on the server is throwing "IllegalStateException: Fail to read ISSUES.LOCATIONS [KEE=AVYt-LEmlODHqmsVS6V8]". The full stack trace can be found below.
I have examined the data in MySQL and can confirm that the row does, in fact, exist:
# id, kee, rule_id, severity, manual_severity, message, line, gap, status, resolution, checksum, reporter, assignee, author_login, action_plan_key, issue_attributes, effort, created_at, updated_at, issue_creation_date, issue_update_date, issue_close_date, tags, component_uuid, project_uuid, locations, issue_type
'1357', 'AVYt-LEmlODHqmsVS6V8', '1223', 'MINOR', '0', 'Define a constant instead of duplicating this literal \"Item \" 4 times.', '258', '4.00000000000000000000', 'OPEN', NULL, '1296deed4ac6511ff2f9516a1f25f984', NULL, NULL, NULL, NULL, '', '10', '1469650091279', '1474887211270', '1469650055000', '1469650055000', NULL, 'design', 'AVYt-KA3lODHqmsVS6VG', 'AVYt-GJ8lODHqmsVS6U5', ?, '1'
and the locations column contains a BLOB with the following (as text):
ÂÂ$ +
ÂÂ$ +Duplication
ÂÂ$ +Duplication
½½& -Duplication
ÃÃ* 1Duplication
If anyone could shed some light on the situation, I would be very appreciative.
Cheers!
Robert
2016.09.26 22:41:32 INFO [o.s.s.c.s.ComputationStepExecutor] Compute test measures | time=6ms
2016.09.26 22:41:32 INFO [o.s.s.c.s.ComputationStepExecutor] Compute complexity measures | time=52ms
2016.09.26 22:41:32 INFO [o.s.s.c.s.ComputationStepExecutor] Load measure computers | time=2ms
2016.09.26 22:41:32 ERROR [o.s.s.c.t.CeWorkerCallableImpl] Failed to execute task AVdoqQI3n6cQT_03NFYE
org.sonar.server.computation.component.VisitException: Visit of Component {key=:md049gateway:src/main/java/com/nike/caps/md049gateway/application/impl/JobServiceImpl.java,type=FILE} failed
at org.sonar.server.computation.component.VisitException.rethrowOrWrap(VisitException.java:44) ~[sonar-server-5.6.3-SNAPSHOT.jar:na]
at org.sonar.server.computation.component.VisitorsCrawler.visit(VisitorsCrawler.java:64) ~[sonar-server-5.6.3-SNAPSHOT.jar:na]
at org.sonar.server.computation.component.VisitorsCrawler.visitChildren(VisitorsCrawler.java:100) ~[sonar-server-5.6.3-SNAPSHOT.jar:na]
at org.sonar.server.computation.component.VisitorsCrawler.visitImpl(VisitorsCrawler.java:87) ~[sonar-server-5.6.3-SNAPSHOT.jar:na]
at org.sonar.server.computation.component.VisitorsCrawler.visit(VisitorsCrawler.java:62) ~[sonar-server-5.6.3-SNAPSHOT.jar:na]
at org.sonar.server.computation.component.VisitorsCrawler.visitChildren(VisitorsCrawler.java:100) ~[sonar-server-5.6.3-SNAPSHOT.jar:na]
at org.sonar.server.computation.component.VisitorsCrawler.visitImpl(VisitorsCrawler.java:87) ~[sonar-server-5.6.3-SNAPSHOT.jar:na]
at org.sonar.server.computation.component.VisitorsCrawler.visit(VisitorsCrawler.java:62) ~[sonar-server-5.6.3-SNAPSHOT.jar:na]
at org.sonar.server.computation.step.ExecuteVisitorsStep.execute(ExecuteVisitorsStep.java:50) ~[sonar-server-5.6.3-SNAPSHOT.jar:na]
at org.sonar.server.computation.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:64) ~[sonar-server-5.6.3-SNAPSHOT.jar:na]
at org.sonar.server.computation.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:52) ~[sonar-server-5.6.3-SNAPSHOT.jar:na]
at org.sonar.server.computation.taskprocessor.report.ReportTaskProcessor.process(ReportTaskProcessor.java:75) ~[sonar-server-5.6.3-SNAPSHOT.jar:na]
at org.sonar.server.computation.taskprocessor.CeWorkerCallableImpl.executeTask(CeWorkerCallableImpl.java:81) [sonar-server-5.6.3-SNAPSHOT.jar:na]
at org.sonar.server.computation.taskprocessor.CeWorkerCallableImpl.call(CeWorkerCallableImpl.java:56) [sonar-server-5.6.3-SNAPSHOT.jar:na]
at org.sonar.server.computation.taskprocessor.CeWorkerCallableImpl.call(CeWorkerCallableImpl.java:35) [sonar-server-5.6.3-SNAPSHOT.jar:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_101]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_101]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_101]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_101]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
Caused by: java.lang.IllegalStateException: Fail to process issues of component ':md049gateway:src/main/java/com/nike/caps/md049gateway/application/impl/JobServiceImpl.java'
at org.sonar.server.computation.issue.IntegrateIssuesVisitor.processIssues(IntegrateIssuesVisitor.java:74) ~[sonar-server-5.6.3-SNAPSHOT.jar:na]
at org.sonar.server.computation.issue.IntegrateIssuesVisitor.visitAny(IntegrateIssuesVisitor.java:59) ~[sonar-server-5.6.3-SNAPSHOT.jar:na]
at org.sonar.server.computation.component.TypeAwareVisitorWrapper.visitAny(TypeAwareVisitorWrapper.java:82) ~[sonar-server-5.6.3-SNAPSHOT.jar:na]
at org.sonar.server.computation.component.VisitorsCrawler.visitNode(VisitorsCrawler.java:107) ~[sonar-server-5.6.3-SNAPSHOT.jar:na]
at org.sonar.server.computation.component.VisitorsCrawler.visitImpl(VisitorsCrawler.java:90) ~[sonar-server-5.6.3-SNAPSHOT.jar:na]
at org.sonar.server.computation.component.VisitorsCrawler.visit(VisitorsCrawler.java:62) ~[sonar-server-5.6.3-SNAPSHOT.jar:na]
... 21 common frames omitted
Caused by: org.apache.ibatis.exceptions.PersistenceException:
### Error querying database. Cause: java.lang.IllegalStateException: Fail to read ISSUES.LOCATIONS [KEE=AVYt-LEmlODHqmsVS6V8]
### The error may exist in org.sonar.db.issue.IssueMapper
### The error may involve org.sonar.db.issue.IssueMapper.selectNonClosedByComponentUuid-Inline
### The error occurred while setting parameters
### SQL: select i.id, i.kee as kee, i.rule_id as ruleId, i.severity as severity, i.manual_severity as manualSeverity, i.message as message, i.line as line, i.locations as locations, i.gap as gap, i.effort as effort, i.status as status, i.resolution as resolution, i.checksum as checksum, i.assignee as assignee, i.author_login as authorLogin, i.tags as tagsString, i.issue_attributes as issueAttributes, i.issue_creation_date as issueCreationTime, i.issue_update_date as issueUpdateTime, i.issue_close_date as issueCloseTime, i.created_at as createdAt, i.updated_at as updatedAt, r.plugin_rule_key as ruleKey, r.plugin_name as ruleRepo, r.language as language, p.kee as componentKey, i.component_uuid as componentUuid, p.module_uuid as moduleUuid, p.module_uuid_path as moduleUuidPath, p.path as filePath, root.kee as projectKey, i.project_uuid as projectUuid, i.issue_type as type from issues i inner join rules r on r.id=i.rule_id inner join projects p on p.uuid=i.component_uuid inner join projects root on root.uuid=i.project_uuid where i.component_uuid=? and i.status <> 'CLOSED'
### Cause: java.lang.IllegalStateException: Fail to read ISSUES.LOCATIONS [KEE=AVYt-LEmlODHqmsVS6V8]
at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:26) ~[mybatis-3.2.7.jar:3.2.7]
at org.apache.ibatis.session.defaults.DefaultSqlSession.select(DefaultSqlSession.java:130) ~[mybatis-3.2.7.jar:3.2.7]
at org.apache.ibatis.session.defaults.DefaultSqlSession.select(DefaultSqlSession.java:118) ~[mybatis-3.2.7.jar:3.2.7]
at org.apache.ibatis.binding.MapperMethod.executeWithResultHandler(MapperMethod.java:108) ~[mybatis-3.2.7.jar:3.2.7]
at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:60) ~[mybatis-3.2.7.jar:3.2.7]
at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:52) ~[mybatis-3.2.7.jar:3.2.7]
at com.sun.proxy.$Proxy52.selectNonClosedByComponentUuid(Unknown Source) ~[na:na]
at org.sonar.server.computation.issue.BaseIssuesLoader.loadForComponentUuid(BaseIssuesLoader.java:61) ~[sonar-server-5.6.3-SNAPSHOT.jar:na]
at org.sonar.server.computation.issue.TrackerBaseInputFactory$BaseLazyInput.loadIssues(TrackerBaseInputFactory.java:78) ~[sonar-server-5.6.3-SNAPSHOT.jar:na]
at org.sonar.core.issue.tracking.LazyInput.getIssues(LazyInput.java:50) ~[sonar-core-5.6.2.jar:na]
at org.sonar.core.issue.tracking.Tracking.<init>(Tracking.java:58) ~[sonar-core-5.6.2.jar:na]
at org.sonar.core.issue.tracking.Tracker.track(Tracker.java:37) ~[sonar-core-5.6.2.jar:na]
at org.sonar.server.computation.issue.TrackerExecution.track(TrackerExecution.java:41) ~[sonar-server-5.6.3-SNAPSHOT.jar:na]
at org.sonar.server.computation.issue.IntegrateIssuesVisitor.processIssues(IntegrateIssuesVisitor.java:67) ~[sonar-server-5.6.3-SNAPSHOT.jar:na]
... 26 common frames omitted
Caused by: java.lang.IllegalStateException: Fail to read ISSUES.LOCATIONS [KEE=AVYt-LEmlODHqmsVS6V8]
at org.sonar.db.issue.IssueDto.parseLocations(IssueDto.java:655) ~[sonar-db-5.6.2.jar:na]
at org.sonar.db.issue.IssueDto.toDefaultIssue(IssueDto.java:724) ~[sonar-db-5.6.2.jar:na]
at org.sonar.server.computation.issue.BaseIssuesLoader$1.handleResult(BaseIssuesLoader.java:64) ~[sonar-server-5.6.3-SNAPSHOT.jar:na]
at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.callResultHandler(DefaultResultSetHandler.java:306) ~[mybatis-3.2.7.jar:3.2.7]
at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.storeObject(DefaultResultSetHandler.java:300) ~[mybatis-3.2.7.jar:3.2.7]
at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleRowValuesForSimpleResultMap(DefaultResultSetHandler.java:292) ~[mybatis-3.2.7.jar:3.2.7]
at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleRowValues(DefaultResultSetHandler.java:266) ~[mybatis-3.2.7.jar:3.2.7]
at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSet(DefaultResultSetHandler.java:239) ~[mybatis-3.2.7.jar:3.2.7]
at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSets(DefaultResultSetHandler.java:150) ~[mybatis-3.2.7.jar:3.2.7]
at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:60) ~[mybatis-3.2.7.jar:3.2.7]
at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:73) ~[mybatis-3.2.7.jar:3.2.7]
at org.apache.ibatis.executor.ReuseExecutor.doQuery(ReuseExecutor.java:57) ~[mybatis-3.2.7.jar:3.2.7]
at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:267) ~[mybatis-3.2.7.jar:3.2.7]
at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:137) ~[mybatis-3.2.7.jar:3.2.7]
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:96) ~[mybatis-3.2.7.jar:3.2.7]
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:77) ~[mybatis-3.2.7.jar:3.2.7]
at org.apache.ibatis.session.defaults.DefaultSqlSession.select(DefaultSqlSession.java:128) ~[mybatis-3.2.7.jar:3.2.7]
... 38 common frames omitted
Caused by: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either that the input has been truncated or that an embedded message misreported its own length.
at com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:70) ~[protobuf-java-3.0.0-beta-2.jar:na]
at com.google.protobuf.CodedInputStream.readRawBytesSlowPath(CodedInputStream.java:1179) ~[protobuf-java-3.0.0-beta-2.jar:na]
at com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:517) ~[protobuf-java-3.0.0-beta-2.jar:na]
at org.sonar.db.protobuf.DbIssues$Location.<init>(DbIssues.java:1674) ~[sonar-db-5.6.2.jar:na]
at org.sonar.db.protobuf.DbIssues$Location.<init>(DbIssues.java:1614) ~[sonar-db-5.6.2.jar:na]
at org.sonar.db.protobuf.DbIssues$Location$1.parsePartialFrom(DbIssues.java:2407) ~[sonar-db-5.6.2.jar:na]
at org.sonar.db.protobuf.DbIssues$Location$1.parsePartialFrom(DbIssues.java:2401) ~[sonar-db-5.6.2.jar:na]
at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:495) ~[protobuf-java-3.0.0-beta-2.jar:na]
at org.sonar.db.protobuf.DbIssues$Flow.<init>(DbIssues.java:954) ~[sonar-db-5.6.2.jar:na]
at org.sonar.db.protobuf.DbIssues$Flow.<init>(DbIssues.java:910) ~[sonar-db-5.6.2.jar:na]
at org.sonar.db.protobuf.DbIssues$Flow$1.parsePartialFrom(DbIssues.java:1527) ~[sonar-db-5.6.2.jar:na]
at org.sonar.db.protobuf.DbIssues$Flow$1.parsePartialFrom(DbIssues.java:1521) ~[sonar-db-5.6.2.jar:na]
at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:495) ~[protobuf-java-3.0.0-beta-2.jar:na]
at org.sonar.db.protobuf.DbIssues$Locations.<init>(DbIssues.java:112) ~[sonar-db-5.6.2.jar:na]
at org.sonar.db.protobuf.DbIssues$Locations.<init>(DbIssues.java:55) ~[sonar-db-5.6.2.jar:na]
at org.sonar.db.protobuf.DbIssues$Locations$1.parsePartialFrom(DbIssues.java:852) ~[sonar-db-5.6.2.jar:na]
at org.sonar.db.protobuf.DbIssues$Locations$1.parsePartialFrom(DbIssues.java:846) ~[sonar-db-5.6.2.jar:na]
at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:137) ~[protobuf-java-3.0.0-beta-2.jar:na]
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:169) ~[protobuf-java-3.0.0-beta-2.jar:na]
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:180) ~[protobuf-java-3.0.0-beta-2.jar:na]
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:185) ~[protobuf-java-3.0.0-beta-2.jar:na]
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49) ~[protobuf-java-3.0.0-beta-2.jar:na]
at org.sonar.db.protobuf.DbIssues$Locations.parseFrom(DbIssues.java:253) ~[sonar-db-5.6.2.jar:na]
at org.sonar.db.issue.IssueDto.parseLocations(IssueDto.java:653) ~[sonar-db-5.6.2.jar:na]
... 54 common frames omitted
2016.09.26 22:41:32 ERROR [o.s.s.c.t.CeWorkerCallableImpl] Executed task | project=:md049gateway | type=REPORT | id=AVdoqQI3n6cQT_03NFYE | time=1642ms
This appears to be due to some sort of corruption in our databases. Hopefully, the changes being made in 5.6.3 (see linked question) will allow us to track down the issue.

Spring-XD Curator Connection Timeout

In Spring-XD the Curator Connection times out:
WARN ConnectionStateManager-0 curator.ConnectionState - Connection
attempt unsuccessful after 63021 (greater than max timeout of 60000).
Resetting connection and trying again with a new connection.
Curator tries to re-establish the connection, but fails. Please check the logs below. Has anyone faced similar issue? Please let me know if you know of any ways to resolve the issue or if you know of any workarounds.
Also the default Curator connection time out is 60000. Is there a way to increase it? Does spring-xd expose a property which can be set?
2014-12-10 01:24:41,003 WARN ConnectionStateManager-0
server.ContainerRegistrar - >>> disconnected container:
1c8a234d-4b8d-4d65-b374-xxxxe8619 2014-12-10 01:24:41,004 INFO
DeploymentsPathChildrenCache-0 server.ContainerRegistrar - Path cache
event: null, type: CONNECTION_SUSPENDED 2014-12-10 01:24:41,005 INFO
ConnectionStateManager-0 server.ContainerRegistrar - Undeploying
module [ModuleDescriptor#350920b1 moduleName = 'rabbit', moduleLabel =
'rabbit', group = 'xxx-ingestion-2', sourceChannelName = [null],
sinkChannelName = [null], sinkChannelName = [null], index = 0, type =
source, parameters = map['vhost' -> 'xxx_virtual_host', 'requeue' ->
'false', 'outputType' -> 'text/plain', 'queues' -> 'xx.xxx.queue',
'addresses' -> 'xxxmq.xx.xxxx.com'], children = list[[empty]]]
2014-12-10 01:24:46,022 ERROR pool-22-thread-1
connection.CachingConnectionFactory - Channel shutdown: clean
connection shutdown; protocol method:
method<connection.close>(reply-code=200, reply-text=OK, class-id=0, method-id=0)
2014-12-10 01:24:56,007 **ERROR CuratorFramework-0
curator.ConnectionState - Connection timed out for connection string
(514.xx.93.xxx:2181,504.58.xxx.xx:2181) and timeout (15000) / elapsed**
(15004) org.apache.curator.CuratorConnectionLossException:
KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:198)
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88)
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:793)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:779)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$400(CuratorFrameworkImpl.java:58)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:265)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744) 2014-12-10 01:24:56,161
ERROR main-EventThread curator.ConnectionState - Connection timed out
for connection string (514.xx.93.xxx:2181,504.58.xxx.xx:2181) and
timeout (15000) / elapsed (15159)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode =
ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:198)
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88)
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:474)
at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:302)
at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:291)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:287)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
at org.springframework.xd.dirt.server.ContainerRegistrar$StreamModuleWatcher.process(ContainerRegistrar.java:744)
at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
2014-12-10 01:25:03,014 ERROR CuratorFramework-0 imps.CuratorFrameworkImpl - **Background retry gave up**
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode =
ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:198)
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88)
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:793)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:779)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$400(CuratorFrameworkImpl.java:58)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:265)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Is this reproducible? Are you running in clustered or single node mode?
The Curator connection timeout (in milliseconds) can be set via system property curator-default-connection-timeout.

Resources