i have a server which has an public ip and private ip on which elasticsearch service is running the private ip is 10.x.x.x and public ip is 159.x.x.x
but its throwing java.net.NoRouteToHostException: No route to host exception here is the code
val setting=ImmutableSettings.settingsBuilder();
//setting.put("network.host",host)
setting.put("path.home", ".")
setting.put("discovery.zen.ping.unicast.hosts",host);
setting.put("http.port",http_port)
setting.put("transport.tcp.port",tcp_port)
val node= nodeBuilder().clusterName(clusterName).client(true).settings(setting).node();
val client=node.client()
es.http.port="9200-9300"
es.tcp.port="9300-9400"
es.host="10.x.x.x" //giving the private ip
here is stacktrace
expire [null], clean_interval [1m]
03:00:04.753 9024 [main] cache DEBUG - [Maynard Tiboldt] using size [-1] [-1b], expire [null]
03:00:04.772 9043 [main] meta DEBUG - [Maynard Tiboldt] using gateway.local.auto_import_dangled [YES], gateway.local.delete_timeout
[30s], with gateway.local.dangling_timeout [2h]
03:00:04.778 9049 [main] udp DEBUG - [Maynard Tiboldt] using enabled [false], host [null], port [9700-9800], bulk_actions [1000], b
ulk_size [5mb], flush_interval [5s], concurrent_requests [4]
03:00:04.784 9055 [main] node INFO - [Maynard Tiboldt] initialized
03:00:04.784 9055 [main] node INFO - [Maynard Tiboldt] starting ...
03:00:04.809 9080 [main] SelectorUtil DEBUG - Using select timeout of 500
03:00:04.810 9081 [main] SelectorUtil DEBUG - Epoll-bug workaround enabled = false
03:00:04.834 9105 [main] netty DEBUG - [Maynard Tiboldt] using profile[default], worker_count[4], port[9300-9400], bind_host[null], publish_host[null], compress[false], connect_timeout[30s], connections_per_node[2/3/6/1/1], receive_predictor[512kb->512kb]
03:00:04.879 9150 [main] netty DEBUG - [Maynard Tiboldt] Bound profile [default] to address [/0:0:0:0:0:0:0:0:9300]
03:00:04.886 9157 [main] transport INFO - [Maynard Tiboldt] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.17.0.8:9300]}
03:00:04.922 9193 [main] discovery INFO - [Maynard Tiboldt] myCluster/pYsg3lDZTcyxvDl671T04A
03:00:04.923 9194 [elasticsearch[Maynard Tiboldt][clusterService#updateTask][T#1]] service DEBUG - [Maynard Tiboldt] processing [initial_join]: execute
03:00:04.930 9201 [elasticsearch[Maynard Tiboldt][clusterService#updateTask][T#1]] service DEBUG - [Maynard Tiboldt] processing [initial_join]: took 6ms no change in cluster_state
03:00:05.141 9412 [elasticsearch[Maynard Tiboldt][[unicast_connect]][T#1]] netty DEBUG - [Maynard Tiboldt] connected to node [[#zen_unicast_1#][elasticsearch1.myserver.com][inet[/159.x.x.x:9300]]]
03:00:06.486 10757 [elasticsearch[Maynard Tiboldt][[unicast_connect]][T#2]] netty DEBUG - [Maynard Tiboldt] connected to node [[Maynard Tiboldt][pYsg3lDZTcyxvDl671T04A][elasticsearch1.myserver.com][inet[/10.17.0.8:9300]]{data=false, client=true}]
03:00:09.461 13732 [elasticsearch[Maynard Tiboldt][transport_client_boss][T#1]{New I/O boss #5}] netty WARN - [Maynard Tiboldt] exception caught on transport layer [[id: 0x4dfc0f6a]], closing connection
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_66]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_66]
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152) [hazelserver-assembly-1.0.jar:1.0]
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105) [hazelserver-assembly-1.0.jar:1.0]
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79) [hazelserver-assembly-1.0.jar:1.0]
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [hazelserver-assembly-1.0.jar:1.0]
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) [hazelserver-assembly-1.0.jar:1.0]
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [hazelserver-assembly-1.0.jar:1.0]
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [hazelserver-assembly-1.0.jar:1.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_66]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_66]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]
03:00:09.463 13734 [elasticsearch[Maynard Tiboldt][transport_client_boss][T#1]{New I/O boss #5}] netty WARN - [Maynard Tiboldt] exception caught on transport layer [[id: 0x470656c3]], closing connection
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_66]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_66]
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152) [hazelserver-assembly-1.0.jar:1.0]
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105) [hazelserver-assembly-1.0.jar:1.0]
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79) [hazelserver-assembly-1.0.jar:1.0]
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [hazelserver-assembly-1.0.jar:1.0]
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) [hazelserver-assembly-1.0.jar:1.0]
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [hazelserver-assembly-1.0.jar:1.0]
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [hazelserver-assembly-1.0.jar:1.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_66]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_66]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]
03:00:09.466 13737 [elasticsearch[Maynard Tiboldt][generic][T#2]] netty DEBUG - [Maynard Tiboldt] disconnecting from [[#zen_unicast_1#][elasticsearch1.myserver.com][inet[/159.x.x.x:9300]]] due to explicit disconnect call
03:00:09.467 13738 [elasticsearch[Maynard Tiboldt][generic][T#1]] zen DEBUG - [Maynard Tiboldt] filtered ping responses: (filter_client[true], filter_data[false]) {none}
03:00:09.476 13747 [elasticsearch[Maynard Tiboldt][[unicast_connect]][T#1]] netty DEBUG - [Maynard Tiboldt] connected to node [[#zen_unicast_1#][elasticsearch1.myserver.com][inet[/159.203.107.230:9300]]]
03:00:12.459 16730 [elasticsearch[Maynard Tiboldt][transport_client_boss][T#1]{New I/O boss #5}] netty WARN - [Maynard Tiboldt] exception caught on transport layer [[id: 0x665f3000]], closing connection
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_66]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_66]
at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152) [hazelserver-assembly-1.0.jar:1.0]
here i am unable to see the private ip in logs which 10.x.x.x i can only see this 10.17.0.8:9300 and i don't know this ip address also elasticsearch client code and elasticsarch service is running on same server , Can someone please point out the mistake i am doing
Related
I have been working on a banking project. I have a problem that is about RabbitMQ and Docker.
I have two microservices that are called query and command. Both of them can not connect to my RabbitMq container. I have been trying to solve this problem, but I could not.
I can succesfully connect and open http://localhost:15672/#/ after docker compose.
When I run docker compose up, account-cmd and account-query services not running on the browser. Here is my docker-compose.yml file.
version: "3.4"
services:
customerdb:
container_name: customerdb
image: postgres
environment:
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-postgres}
POSTGRES_USER: ${POSTGRES_USER:-postgres}
volumes:
- ./customer/postgres_init.sql:/docker-entrypoint-initdb.d/postgres_init.sql
ports:
- "5432:5432"
restart: unless-stopped
querydb:
container_name: querydb
image: postgres
environment:
POSTGRES_USER: ${POSTGRES_USER:-postgres}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-postgres}
volumes:
- ./account.query/postgres_init.sql:/docker-entrypoint-initdb.d/postgres_init.sql
ports:
- "5433:5432"
restart: unless-stopped
rabbitmq:
container_name: "bank_rabbitmq"
image: "rabbitmq:3.8-management"
hostname: "rabbitmq"
environment:
RABBITMQ_DEFAULT_USER: "guest"
RABBITMQ_DEFAULT_PASS: "guest"
RABBITMQ_DEFAULT_VHOST: "/"
ports:
- "15672:15672"
- "5672:5672"
cmddb:
container_name: "cmddb"
image: mongo
restart: always
ports:
- "27017:27017"
customer-service:
image: bank/customer-service-api
container_name: customer-service
build:
context: ./customer
dockerfile: Dockerfile
ports:
- "5000:5000"
depends_on:
- customerdb
environment:
- SPRING_DATASOURCE_URL=jdbc:postgresql://customerdb:5432/customerdb
- SPRING_DATASOURCE_USERNAME=postgres
- SPRING_DATASOURCE_PASSWORD=postgres
account-cmd:
image: bank/account-cmd-service-api
container_name: account-cmd-service
build:
context: ./account.cmd
dockerfile: Dockerfile
ports:
- "5002:5002"
depends_on:
- cmddb
- rabbitmq
environment:
- SPRING_DATA_MONGODB_HOST=cmddb
- SPRING_DATA_MONGODB_PORT=27017
- SPRING_DATA_MONGODB_DATABASE=accountcmdb
- SPRING_RABBITMQ_HOST=rabbitmq
- SPRING_RABBITMQ_PORT=5672
- SPRING_RABBITMQ_USERNAME=guest
- SPRING_RABBITMQ_PASSWORD=guest
account-query:
image: bank/account-query-service-api
container_name: account-query-service
build:
context: ./account.query
dockerfile: Dockerfile
ports:
- "5003:5003"
depends_on:
- querydb
- rabbitmq
environment:
- SPRING_DATASOURCE_URL=jdbc:postgresql://querydb:5433/accountingdb
- SPRING_DATASOURCE_USERNAME=postgres
- SPRING_DATASOURCE_PASSWORD=postgres
- SPRING_RABBITMQ_HOST=rabbitmq
- SPRING_RABBITMQ_PORT=5672
- SPRING_RABBITMQ_USERNAME=guest
- SPRING_RABBITMQ_PASSWORD=guest
volumes:
customerdb:
cmddb:
querydb:
Here is my RabbitMQConfig on spring boot.
package com.bankingsolution.account.query.configuration;
import com.bankingsolution.common.constants.Contants;
import org.springframework.amqp.core.Binding;
import org.springframework.amqp.core.BindingBuilder;
import org.springframework.amqp.core.Queue;
import org.springframework.amqp.core.TopicExchange;
import org.springframework.amqp.rabbit.annotation.RabbitListenerConfigurer;
import org.springframework.amqp.rabbit.config.SimpleRabbitListenerContainerFactory;
import org.springframework.amqp.rabbit.connection.CachingConnectionFactory;
import org.springframework.amqp.rabbit.connection.ConnectionFactory;
import org.springframework.amqp.rabbit.listener.RabbitListenerEndpointRegistrar;
import org.springframework.amqp.support.converter.Jackson2JsonMessageConverter;
import org.springframework.amqp.support.converter.MessageConverter;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.messaging.converter.MappingJackson2MessageConverter;
import org.springframework.messaging.handler.annotation.support.DefaultMessageHandlerMethodFactory;
import java.util.Arrays;
import java.util.List;
#Configuration
public class RabbitMQConfig implements RabbitListenerConfigurer {
#Value("${spring.rabbitmq.host}")
private String hostName;
#Value("${spring.rabbitmq.username}")
private String userName;
#Value("${spring.rabbitmq.password}")
private String password;
#Value("${spring.rabbitmq.port}")
private int port;
#Bean
public TopicExchange exchange() {
return new TopicExchange(Contants.TOPIC_KEY);
}
#Bean
Queue accountOpenedQueue() {
return new Queue(Contants.AccountOpenedTopic);
}
#Bean
Queue fundsDepositedQueue() {
return new Queue(Contants.FundsDepositedTopic);
}
#Bean
Queue fundsWithdrawnQueue() {
return new Queue(Contants.FundsWithDrawnTopic);
}
#Bean
Queue transactionCreatedQueue() {
return new Queue(Contants.TransactionCreatedTopic);
}
#Bean
Queue transactionFailedQueue() {
return new Queue(Contants.TransactionFailedTopic);
}
#Bean
public List<Binding> binding() {
return Arrays.asList(
BindingBuilder.bind(accountOpenedQueue()).to(exchange()).with(Contants.AccountOpenedTopic),
BindingBuilder.bind(fundsDepositedQueue()).to(exchange()).with(Contants.FundsDepositedTopic),
BindingBuilder.bind(fundsWithdrawnQueue()).to(exchange()).with(Contants.FundsWithDrawnTopic),
BindingBuilder.bind(transactionCreatedQueue()).to(exchange()).with(Contants.TransactionCreatedTopic),
BindingBuilder.bind(transactionFailedQueue()).to(exchange()).with(Contants.TransactionFailedTopic));
}
#Bean
public MessageConverter jsonMessageConverter() {
return new Jackson2JsonMessageConverter();
}
#Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory() {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(connectionFactory());
return factory;
}
#Bean
public ConnectionFactory connectionFactory() {
CachingConnectionFactory connectionFactory = new CachingConnectionFactory();
connectionFactory.setHost(this.hostName);
connectionFactory.setVirtualHost("/");
connectionFactory.setUsername(this.userName);
connectionFactory.setPassword(this.password);
connectionFactory.setPort(this.port);
return connectionFactory;
}
#Override
public void configureRabbitListeners(RabbitListenerEndpointRegistrar registrar) {
registrar.setMessageHandlerMethodFactory(myHandlerMethodFactory());
}
#Bean
public DefaultMessageHandlerMethodFactory myHandlerMethodFactory() {
DefaultMessageHandlerMethodFactory factory = new DefaultMessageHandlerMethodFactory();
factory.setMessageConverter(new MappingJackson2MessageConverter());
return factory;
}
}
Here is sample error on Docker for a microservice.
at org.springframework.amqp.rabbit.core.RabbitAdmin.getQueueInfo(RabbitAdmin.java:463) ~[spring-rabbit-2.4.2.jar!/:2.4.2]
at org.springframework.amqp.rabbit.core.RabbitAdmin.getQueueProperties(RabbitAdmin.java:447) ~[spring-rabbit-2.4.2.jar!/:2.4.2]
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.attemptDeclarations(AbstractMessageListenerContainer.java:1925) ~[spring-rabbit-2.4.2.jar!/:2.4.2]
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.redeclareElementsIfNecessary(AbstractMessageListenerContainer.java:1906) ~[spring-rabbit-2.4.2.jar!/:2.4.2]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.initialize(SimpleMessageListenerContainer.java:1349) ~[spring-rabbit-2.4.2.jar!/:2.4.2]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1195) ~[spring-rabbit-2.4.2.jar!/:2.4.2]
at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:na]
at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) ~[na:na]
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) ~[na:na]
at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) ~[na:na]
at java.base/java.net.SocksSocketImpl.connect(Unknown Source) ~[na:na]
at java.base/java.net.Socket.connect(Unknown Source) ~[na:na]
at com.rabbitmq.client.impl.SocketFrameHandlerFactory.create(SocketFrameHandlerFactory.java:60) ~[amqp-client-5.13.1.jar!/:5.13.1]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1223) ~[amqp-client-5.13.1.jar!/:5.13.1]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1173) ~[amqp-client-5.13.1.jar!/:5.13.1]
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.connectAddresses(AbstractConnectionFactory.java:640) ~[spring-rabbit-2.4.2.jar!/:2.4.2]
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.connect(AbstractConnectionFactory.java:615) ~[spring-rabbit-2.4.2.jar!/:2.4.2]
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:565) ~[spring-rabbit-2.4.2.jar!/:2.4.2]
... 12 common frames omitted
2022-03-22 01:07:40.006 INFO 1 --- [ntContainer#4-3] o.s.a.r.c.CachingConnectionFactory : Attempting to connect to: [rabbitmq:5672]
2022-03-22 01:07:40.006 INFO 1 --- [ntContainer#4-2] o.s.a.r.c.CachingConnectionFactory : Attempting to connect to: [rabbitmq:5672]
2022-03-22 01:07:40.007 ERROR 1 --- [ntContainer#4-2] o.s.a.r.l.SimpleMessageListenerContainer : Failed to check/redeclare auto-delete queue(s).
org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)
at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:61) ~[spring-rabbit-2.4.2.jar!/:2.4.2]
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:602) ~[spring-rabbit-2.4.2.jar!/:2.4.2]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:724) ~[spring-rabbit-2.4.2.jar!/:2.4.2]
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.createConnection(ConnectionFactoryUtils.java:252) ~[spring-rabbit-2.4.2.jar!/:2.4.2]
at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:2175) ~[spring-rabbit-2.4.2.jar!/:2.4.2]
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:2148) ~[spring-rabbit-2.4.2.jar!/:2.4.2]
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:2128) ~[spring-rabbit-2.4.2.jar!/:2.4.2]
at org.springframework.amqp.rabbit.core.RabbitAdmin.getQueueInfo(RabbitAdmin.java:463) ~[spring-rabbit-2.4.2.jar!/:2.4.2]
at org.springframework.amqp.rabbit.core.RabbitAdmin.getQueueProperties(RabbitAdmin.java:447) ~[spring-rabbit-2.4.2.jar!/:2.4.2]
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.attemptDeclarations(AbstractMessageListenerContainer.java:1925) ~[spring-rabbit-2.4.2.jar!/:2.4.2]
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.redeclareElementsIfNecessary(AbstractMessageListenerContainer.java:1906) ~[spring-rabbit-2.4.2.jar!/:2.4.2]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.initialize(SimpleMessageListenerContainer.java:1349) ~[spring-rabbit-2.4.2.jar!/:2.4.2]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1195) ~[spring-rabbit-2.4.2.jar!/:2.4.2]
at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:na]
at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) ~[na:na]
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) ~[na:na]
at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source) ~[na:na]
at java.base/java.net.SocksSocketImpl.connect(Unknown Source) ~[na:na]
at java.base/java.net.Socket.connect(Unknown Source) ~[na:na]
at com.rabbitmq.client.impl.SocketFrameHandlerFactory.create(SocketFrameHandlerFactory.java:60) ~[amqp-client-5.13.1.jar!/:5.13.1]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1223) ~[amqp-client-5.13.1.jar!/:5.13.1]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1173) ~[amqp-client-5.13.1.jar!/:5.13.1]
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.connectAddresses(AbstractConnectionFactory.java:640) ~[spring-rabbit-2.4.2.jar!/:2.4.2]
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.connect(AbstractConnectionFactory.java:615) ~[spring-rabbit-2.4.2.jar!/:2.4.2]
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:565) ~[spring-rabbit-2.4.2.jar!/:2.4.2]
... 12 common frames omitted
2022-03-22 01:07:40.008 INFO 1 --- [ntContainer#4-2] o.s.a.r.c.CachingConnectionFactory : Attempting to connect to: [rabbitmq:5672]
2022-03-22 01:07:40.023 INFO 1 --- [ main] c.b.account.query.QueryApplication : Started QueryApplication in 8.535 seconds (JVM running for 9.399)
2022-03-22 01:07:44.965 WARN 1 --- [ntContainer#0-2] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)
2022-03-22 01:07:44.965 WARN 1 --- [ntContainer#0-1] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)
2022-03-22 01:07:44.966 WARN 1 --- [ntContainer#0-3] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)
2022-03-22 01:07:44.966 WARN 1 --- [ntContainer#0-4] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)
2022-03-22 01:07:44.969 INFO 1 --- [ntContainer#0-3] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer#27755487: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2022-03-22 01:07:44.969 INFO 1 --- [ntContainer#0-2] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer#2e5b7fba: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2022-03-22 01:07:44.969 INFO 1 --- [ntContainer#0-4] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer#4f0cab0a: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2022-03-22 01:07:44.969 INFO 1 --- [ntContainer#0-1] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer#60d40ff4: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2022-03-22 01:07:44.970 INFO 1 --- [ntContainer#0-5] o.s.a.r.c.CachingConnectionFactory : Attempting to connect to: [rabbitmq:5672]
2022-03-22 01:07:44.971 WARN 1 --- [ntContainer#1-1] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)
2022-03-22 01:07:44.971 INFO 1 --- [ntContainer#1-1] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer#6fc6deb7: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2022-03-22 01:07:44.973 WARN 1 --- [ntContainer#1-4] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)
2022-03-22 01:07:44.973 INFO 1 --- [ntContainer#1-4] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer#7da39774: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2022-03-22 01:07:44.976 WARN 1 --- [ntContainer#1-3] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)
2022-03-22 01:07:44.979 INFO 1 --- [ntContainer#1-3] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer#367f0121: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2022-03-22 01:07:44.979 WARN 1 --- [ntContainer#1-2] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)
2022-03-22 01:07:44.979 INFO 1 --- [ntContainer#1-2] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer#441b8382: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2022-03-22 01:07:44.982 WARN 1 --- [ntContainer#2-1] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)
2022-03-22 01:07:44.982 INFO 1 --- [ntContainer#2-1] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer#39bbd9e0: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2022-03-22 01:07:44.984 WARN 1 --- [ntContainer#2-4] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)
2022-03-22 01:07:44.984 INFO 1 --- [ntContainer#2-4] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer#4397a639: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2022-03-22 01:07:44.986 WARN 1 --- [ntContainer#2-3] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)
2022-03-22 01:07:44.986 INFO 1 --- [ntContainer#2-3] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer#27fe9713: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2022-03-22 01:07:44.988 WARN 1 --- [ntContainer#2-2] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)
2022-03-22 01:07:44.988 INFO 1 --- [ntContainer#2-2] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer#11c3ff67: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2022-03-22 01:07:44.994 WARN 1 --- [ntContainer#3-1] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)
2022-03-22 01:07:44.999 WARN 1 --- [ntContainer#3-3] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)
2022-03-22 01:07:44.999 WARN 1 --- [ntContainer#3-4] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)
2022-03-22 01:07:44.999 INFO 1 --- [ntContainer#3-3] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer#3e4e4c1: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2022-03-22 01:07:44.999 INFO 1 --- [ntContainer#3-4] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer#46a488c2: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2022-03-22 01:07:44.999 INFO 1 --- [ntContainer#3-1] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer#3ae126d1: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2022-03-22 01:07:45.000 WARN 1 --- [ntContainer#3-2] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)
2022-03-22 01:07:45.000 INFO 1 --- [ntContainer#3-2] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer#7e7f3cfd: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2022-03-22 01:07:45.004 WARN 1 --- [ntContainer#4-1] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)
2022-03-22 01:07:45.005 INFO 1 --- [ntContainer#4-1] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer#49d831c2: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2022-03-22 01:07:45.008 WARN 1 --- [ntContainer#4-4] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)
2022-03-22 01:07:45.009 INFO 1 --- [ntContainer#4-4] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer#28bdbe88: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2022-03-22 01:07:45.010 WARN 1 --- [ntContainer#4-3] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)
2022-03-22 01:07:45.010 INFO 1 --- [ntContainer#4-3] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer#43fda8d9: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2022-03-22 01:07:45.011 WARN 1 --- [ntContainer#4-2] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused (Connection refused)
2022-03-22 01:07:45.011 INFO 1 --- [ntContainer#4-2] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer#53a7a60c: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2022-03-22 01:07:45.038 INFO 1 --- [ntContainer#0-5] o.s.a.r.c.CachingConnectionFactory : Created new connection: rabbitConnectionFactory#51abf713:45/SimpleConnection#427bde5e [delegate=amqp://rabbitmq#172.23.0.5:5672/, localPort= 54968]
I would be very happy if someone could help.
Here is my github repository.
https://github.com/dogaanismail/bank-solution
Description:
The Tomcat connector configured to listen on port 8080 failed to start. The port may already be in use or the connector may be misconfigured.
you see the controller class
#Controller
public class HomeController implements ErrorController {
#RequestMapping("/v1")
public String home()
{
return "index";
}
private static final String PATH = "/error";
#RequestMapping(value = PATH)
public String error() {
return "Error handling kova";
}
#Override
public String getErrorPath() {
return PATH;
}
}//CLASS
I have added the following attribute into the file.
application.properties add
server.port=8087
and the error output looks like this:
I am having problems changing my port number.
and error port problem
org.apache.catalina.LifecycleException: Failed to start component [Connector[HTTP/1.1-8080]]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
at org.apache.catalina.core.StandardService.addConnector(StandardService.java:225) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.addPreviouslyRemovedConnectors(TomcatWebServer.java:256) [spring-boot-2.1.0.BUILD-SNAPSHOT.jar:2.1.0.BUILD-SNAPSHOT]
at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.start(TomcatWebServer.java:198) [spring-boot-2.1.0.BUILD-SNAPSHOT.jar:2.1.0.BUILD-SNAPSHOT]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.startWebServer(ServletWebServerApplicationContext.java:300) [spring-boot-2.1.0.BUILD-SNAPSHOT.jar:2.1.0.BUILD-SNAPSHOT]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.finishRefresh(ServletWebServerApplicationContext.java:162) [spring-boot-2.1.0.BUILD-SNAPSHOT.jar:2.1.0.BUILD-SNAPSHOT]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:553) [spring-context-5.0.6.RELEASE.jar:5.0.6.RELEASE]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140) [spring-boot-2.1.0.BUILD-SNAPSHOT.jar:2.1.0.BUILD-SNAPSHOT]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759) [spring-boot-2.1.0.BUILD-SNAPSHOT.jar:2.1.0.BUILD-SNAPSHOT]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:395) [spring-boot-2.1.0.BUILD-SNAPSHOT.jar:2.1.0.BUILD-SNAPSHOT]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:327) [spring-boot-2.1.0.BUILD-SNAPSHOT.jar:2.1.0.BUILD-SNAPSHOT]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1255) [spring-boot-2.1.0.BUILD-SNAPSHOT.jar:2.1.0.BUILD-SNAPSHOT]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1243) [spring-boot-2.1.0.BUILD-SNAPSHOT.jar:2.1.0.BUILD-SNAPSHOT]
at com.userfront.UserFront1Application.main(UserFront1Application.java:10) [classes/:na]
Caused by: org.apache.catalina.LifecycleException: Protocol handler start failed
at org.apache.catalina.connector.Connector.startInternal(Connector.java:1020) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
... 13 common frames omitted
Caused by: java.net.BindException: Address already in use: bind
at sun.nio.ch.Net.bind0(Native Method) ~[na:1.8.0_171]
at sun.nio.ch.Net.bind(Unknown Source) ~[na:1.8.0_171]
at sun.nio.ch.Net.bind(Unknown Source) ~[na:1.8.0_171]
at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source) ~[na:1.8.0_171]
at sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source) ~[na:1.8.0_171]
at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:210) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
at org.apache.tomcat.util.net.AbstractEndpoint.start(AbstractEndpoint.java:1150) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
at org.apache.coyote.AbstractProtocol.start(AbstractProtocol.java:591) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
at org.apache.catalina.connector.Connector.startInternal(Connector.java:1018) ~[tomcat-embed-core-8.5.31.jar:8.5.31]
... 14 common frames omitted
2018-05-26 23:19:10.439 INFO 1460 --- [ main] o.apache.catalina.core.StandardService : Stopping service [Tomcat]
2018-05-26 23:19:10.460 INFO 1460 --- [ main] ConditionEvaluationReportLoggingListener :
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2018-05-26 23:19:10.468 ERROR 1460 --- [ main] o.s.b.d.LoggingFailureAnalysisReporter :
***************************
APPLICATION FAILED TO START
***************************
Description:
The Tomcat connector configured to listen on port 8080 failed to start. The port may already be in use or the connector may be misconfigured.
Action:
Verify the connector's configuration, identify and stop any process that's listening on port 8080, or configure this application to listen on another port.
2018-05-26 23:19:10.470 INFO 1460 --- [ main] ConfigServletWebServerApplicationContext : Closing org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext#1761df8: startup date [Sat May 26 23:19:05 EEST 2018]; root of context hierarchy
2018-05-26 23:19:10.473 INFO 1460 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Unregistering JMX-exposed beans on shutdown
Open Command Prompt and run netstat -aon | findstr 0.0:80 to find the UID using the port 8080.
Then open Task Manager, select Details, find the port showed in CMD and end task.
There is a red icon to stop the process.
Just hit that and then hit the play or run icon. This will kill the port and start again
Using Spring JdbcTemplate with HikariCP and JTDS to connect to MS SQL Server database in legacy code that I'm maintaining. I'm trying to handle the case where the application has started and later for whatever reason the database becomes unavailable and a connection exception is thrown.
I already have a custom SQL error code translator (below) that captures many other exceptions thrown by jdbcTemplate. But the connection refused exception is not being caught even though it is being encapsulated as a java.sql.SQLException (see the stack trace).
Why is the translator not capturing this exception?
public class CustomSqlErrorCodeTranslator extends
SQLErrorCodeSQLExceptionTranslator {
private final Log log = LogFactory.getLog(getClass());
#Override
protected DataAccessException customTranslate
(String task, String sql, SQLException sqlException) {
/*
* We want to capture all codes.
*/
StringBuilder logMsg = new StringBuilder("*********************SQL Exception:");
logMsg.append(", errorCode: " + sqlException.getErrorCode())
.append(", message: " + sqlException.getMessage())
;
log.error(logMsg.toString());
return null;
}
}
public class Dao {
//HikariDataSource dataSource;
DataSource dataSource;
protected JdbcTemplate jdbcTemplate;
private final Log log = LogFactory.getLog(getClass());
private boolean connected = false;
private CustomSqlErrorCodeTranslator exceptionHandler;
public void setDataSource(DataSource ds) {
//this.jdbcTemplate = new JdbcTemplate(ds);
try {
this.jdbcTemplate = new JdbcTemplate(ds);
exceptionHandler = new CustomSqlErrorCodeTranslator();
this.jdbcTemplate.setExceptionTranslator(exceptionHandler);
connected = true;
} catch (Exception e) {
if (e.getCause() != null && e.getCause() instanceof SQLException) {
log.error("******Cannot connect to database.***************************");
log.debug(e.getMessage());
}
}
}
}
16:59:33,542 [main] WARN PoolBase - springHikariCP - Connection net.sourceforge.jtds.jdbc.JtdsConnection#42206bce failed alive test with exception I/O Error: Connection reset by peer: socket write error
16:59:33,542 [Hikari connection closer (pool springHikariCP)] DEBUG PoolBase - springHikariCP - Closing connection net.sourceforge.jtds.jdbc.JtdsConnection#42206bce: (connection evicted or dead)
16:59:34,530 [main] DEBUG HikariPool - Timeout failure pool springHikariCP stats (total=0, active=0, idle=0, waiting=0)
16:59:34,544 [Hikari connection adder (pool springHikariCP)] DEBUG HikariPool - springHikariCP - Cannot acquire connection from data source
java.sql.SQLException: Network error IOException: Connection refused: connect
at net.sourceforge.jtds.jdbc.JtdsConnection.<init>(JtdsConnection.java:436)
at net.sourceforge.jtds.jdbc.Driver.connect(Driver.java:184)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:314)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:171)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:436)
at com.zaxxer.hikari.pool.HikariPool.access$500(HikariPool.java:65)
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:567)
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:560)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at net.sourceforge.jtds.jdbc.SharedSocket.createSocketForJDBC3(SharedSocket.java:288)
at net.sourceforge.jtds.jdbc.SharedSocket.<init>(SharedSocket.java:251)
at net.sourceforge.jtds.jdbc.JtdsConnection.<init>(JtdsConnection.java:331)
... 13 more
16:59:35,746 [Hikari connection adder (pool springHikariCP)] DEBUG HikariPool - springHikariCP - Cannot acquire connection from data source
java.sql.SQLException: Network error IOException: Connection refused: connect
at net.sourceforge.jtds.jdbc.JtdsConnection.<init>(JtdsConnection.java:436)
at net.sourceforge.jtds.jdbc.Driver.connect(Driver.java:184)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:314)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:171)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:436)
at com.zaxxer.hikari.pool.HikariPool.access$500(HikariPool.java:65)
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:567)
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:560)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at net.sourceforge.jtds.jdbc.SharedSocket.createSocketForJDBC3(SharedSocket.java:288)
at net.sourceforge.jtds.jdbc.SharedSocket.<init>(SharedSocket.java:251)
at net.sourceforge.jtds.jdbc.JtdsConnection.<init>(JtdsConnection.java:331)
... 13 more
16:59:37,010 [Hikari connection adder (pool springHikariCP)] DEBUG HikariPool - springHikariCP - Cannot acquire connection from data source
Sorry that it took me this long to realize I needed to add to the stack trace.
Here is the stack trace when I stop the database. Now it is clear that JDBC Template is involved. And if JdbcTemplate is involved it seems to me the custom exception handler should pick it up. Has anyone else had any experience capturing the database connection exception using the custom error code translator?
5:06:05,076 [ActiveMQ Session Task-1] DEBUG HikariPool - Timeout failure pool HikariPool-0 stats (total=1, active=1, idle=0, waiting=2)
org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLTransientConnectionException: HikariPool-0 - Connection is not available, request timed out after 10000ms.
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:615)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:866)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:927)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:937)
at com.pevco.persist.resources.AvailabilityDaoImpl.create(AvailabilityDaoImpl.java:51)
at com.pevco.nexgen.health.handlers.SystemModelListener.onMessage(SystemModelListener.java:135)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:746)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:684)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:651)
at org.springframework.jms.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:622)
at org.springframework.jms.listener.SimpleMessageListenerContainer.processMessage(SimpleMessageListenerContainer.java:330)
at org.springframework.jms.listener.SimpleMessageListenerContainer$2.onMessage(SimpleMessageListenerContainer.java:306)
at org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.java:1321)
at org.apache.activemq.ActiveMQSessionExecutor.dispatch(ActiveMQSessionExecutor.java:131)
at org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java:202)
at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:129)
at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:47)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLTransientConnectionException: HikariPool-0 - Connection is not available, request timed out after 10000ms.
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:195)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:147)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:83)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
... 20 more
Caused by: java.sql.SQLException: Network error IOException: Connection refused: connect
at net.sourceforge.jtds.jdbc.JtdsConnection.<init>(JtdsConnection.java:436)
at net.sourceforge.jtds.jdbc.Driver.connect(Driver.java:184)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:314)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:171)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:436)
at com.zaxxer.hikari.pool.HikariPool.access$500(HikariPool.java:65)
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:567)
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:560)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at net.sourceforge.jtds.jdbc.SharedSocket.createSocketForJDBC3(SharedSocket.java:288)
at net.sourceforge.jtds.jdbc.SharedSocket.<init>(SharedSocket.java:251)
at net.sourceforge.jtds.jdbc.JtdsConnection.<init>(JtdsConnection.java:331)
... 12 more
15:06:06,217 [ActiveMQ Session Task-1] DEBUG HikariPool - Timeout failure pool HikariPool-0 stats (total=1, active=1, idle=0, waiting=2)
15:06:06,217 [ActiveMQ Session Task-1] WARN SimpleMessageListenerContainer - Execution of JMS message listener failed, and no ErrorHandler has been set.
(I'll award bounty before time expires, but would like to get the accurate answer.)
As "reos" pointed out, there is nothing related to JdbcTemplate in the stack trace. That means you tried to open the connection outside the JdbcTemplate.
Even if you used the JdbcTemplate, the exception would not get translated. Look at the execute method in the source code. They don't use the custom exception translator if the data source fails to obtain the connection.
The problem is because the connection is being asked by Hikari component. Probably Hikari lost the connection and it is trying to reconnect.
In the log there is not any part related to the JdbcTemplate. That is because the problem is cause by a reconnection try and not because a new transaction.
If the transaction is not passing by the template the error translator is never called. The event that throws the exception is no fired by the business logic. It is fired by the reconnect or create connection functionality of Hikari
A MapReduce job implemented in an Apache Crunch pipeline is failing with the error message Error injecting constructor, javax.xml.bind.JAXBException: property "retainReferenceToInfo" is not supported.
The Crunch pipeline is very similar to other functioning pipelines; does anyone have any theories or intuitions about the errors seen below?
Thank you for any help.
An excerpt of the application log can be found below:
2016-02-16 13:34:31,925 INFO [main] org.mortbay.log: Extract jar:file:/data7/hadoop/yarn/local/filecache/96/mapreduce.tar.gz/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.7.1.2.3.4.0-3485.jar!/webapps/mapreduce to /data6/hadoop/yarn/local/usercache/anonymizedusername/appcache/application_1455519038503_6094/container_e26_1455519038503_6094_02_000001/tmp/Jetty_0_0_0_0_52016_mapreduce____.kjcl3z/webapp
2016-02-16 13:34:32,627 WARN [main] org.mortbay.log: failed guice: com.google.inject.ProvisionException: Guice provision errors:
1) Error injecting constructor, javax.xml.bind.JAXBException: property "retainReferenceToInfo" is not supported
at org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver.<init>(JAXBContextResolver.java:70)
at org.apache.hadoop.mapreduce.v2.app.webapp.AMWebApp.setup(AMWebApp.java:33)
while locating org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver
1 error
2016-02-16 13:34:32,628 ERROR [main] org.mortbay.log: Failed startup of context org.mortbay.jetty.webapp.WebAppContext#1981d861{/,jar:file:/data7/hadoop/yarn/local/filecache/96/mapreduce.tar.gz/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.7.1.2.3.4.0-3485.jar!/webapps/mapreduce}
com.google.inject.ProvisionException: Guice provision errors:
1) Error injecting constructor, javax.xml.bind.JAXBException: property "retainReferenceToInfo" is not supported
at org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver.<init>(JAXBContextResolver.java:70)
at org.apache.hadoop.mapreduce.v2.app.webapp.AMWebApp.setup(AMWebApp.java:33)
while locating org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver
1 error
at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:987)
at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1013)
at com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory$GuiceInstantiatedComponentProvider.getInstance(GuiceComponentProviderFactory.java:332)
at com.sun.jersey.core.spi.component.ioc.IoCProviderFactory$ManagedSingleton.<init>(IoCProviderFactory.java:179)
at com.sun.jersey.core.spi.component.ioc.IoCProviderFactory.wrap(IoCProviderFactory.java:100)
at com.sun.jersey.core.spi.component.ioc.IoCProviderFactory._getComponentProvider(IoCProviderFactory.java:93)
at com.sun.jersey.core.spi.component.ProviderFactory.getComponentProvider(ProviderFactory.java:153)
at com.sun.jersey.core.spi.component.ProviderServices.getComponent(ProviderServices.java:251)
at com.sun.jersey.core.spi.component.ProviderServices.getProviders(ProviderServices.java:148)
at com.sun.jersey.core.spi.factory.ContextResolverFactory.init(ContextResolverFactory.java:83)
at com.sun.jersey.server.impl.application.WebApplicationImpl._initiate(WebApplicationImpl.java:1271)
at com.sun.jersey.server.impl.application.WebApplicationImpl.access$700(WebApplicationImpl.java:169)
at com.sun.jersey.server.impl.application.WebApplicationImpl$13.f(WebApplicationImpl.java:775)
at com.sun.jersey.server.impl.application.WebApplicationImpl$13.f(WebApplicationImpl.java:771)
at com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193)
at com.sun.jersey.server.impl.application.WebApplicationImpl.initiate(WebApplicationImpl.java:771)
at com.sun.jersey.guice.spi.container.servlet.GuiceContainer.initiate(GuiceContainer.java:121)
at com.sun.jersey.spi.container.servlet.ServletContainer$InternalWebComponent.initiate(ServletContainer.java:318)
at com.sun.jersey.spi.container.servlet.WebComponent.load(WebComponent.java:609)
at com.sun.jersey.spi.container.servlet.WebComponent.init(WebComponent.java:210)
at com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:373)
at com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:710)
at com.google.inject.servlet.FilterDefinition.init(FilterDefinition.java:114)
at com.google.inject.servlet.ManagedFilterPipeline.initPipeline(ManagedFilterPipeline.java:98)
at com.google.inject.servlet.GuiceFilter.init(GuiceFilter.java:172)
at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:713)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:857)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:306)
at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.serviceStart(MRClientService.java:142)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1148)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1557)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1553)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1486)
Caused by: javax.xml.bind.JAXBException: property "retainReferenceToInfo" is not supported
at com.sun.xml.bind.v2.ContextFactory.createContext(ContextFactory.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:247)
at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:234)
at javax.xml.bind.ContextFinder.find(ContextFinder.java:441)
at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:641)
at com.sun.jersey.api.json.JSONJAXBContext.<init>(JSONJAXBContext.java:246)
at org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver.<init>(JAXBContextResolver.java:72)
at org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver$$FastClassByGuice$$d18e5b53.newInstance(<generated>)
at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.Scopes$1$1.get(Scopes.java:65)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974)
... 50 more
2016-02-16 13:34:32,641 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup#0.0.0.0:52016
2016-02-16 13:34:32,642 INFO [main] org.mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup#0.0.0.0:52016
2016-02-16 13:34:32,642 WARN [2076864428#qtp-1188623367-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup#0.0.0.0:52016] org.apache.hadoop.http.HttpServer2: HttpServer Acceptor: isRunning is false. Rechecking.
2016-02-16 13:34:32,643 WARN [2076864428#qtp-1188623367-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup#0.0.0.0:52016] org.apache.hadoop.http.HttpServer2: HttpServer Acceptor: isRunning is false
2016-02-16 13:34:32,746 ERROR [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Webapps failed to start. Ignoring for now:
org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:310)
at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.serviceStart(MRClientService.java:142)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1148)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1557)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1553)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1486)
Caused by: java.io.IOException: Unable to initialize WebAppContext
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:879)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:306)
... 10 more
Caused by: com.google.inject.ProvisionException: Guice provision errors:
1) Error injecting constructor, javax.xml.bind.JAXBException: property "retainReferenceToInfo" is not supported
at org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver.<init>(JAXBContextResolver.java:70)
at org.apache.hadoop.mapreduce.v2.app.webapp.AMWebApp.setup(AMWebApp.java:33)
while locating org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver
1 error
at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:987)
at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1013)
at com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory$GuiceInstantiatedComponentProvider.getInstance(GuiceComponentProviderFactory.java:332)
at com.sun.jersey.core.spi.component.ioc.IoCProviderFactory$ManagedSingleton.<init>(IoCProviderFactory.java:179)
at com.sun.jersey.core.spi.component.ioc.IoCProviderFactory.wrap(IoCProviderFactory.java:100)
at com.sun.jersey.core.spi.component.ioc.IoCProviderFactory._getComponentProvider(IoCProviderFactory.java:93)
at com.sun.jersey.core.spi.component.ProviderFactory.getComponentProvider(ProviderFactory.java:153)
at com.sun.jersey.core.spi.component.ProviderServices.getComponent(ProviderServices.java:251)
at com.sun.jersey.core.spi.component.ProviderServices.getProviders(ProviderServices.java:148)
at com.sun.jersey.core.spi.factory.ContextResolverFactory.init(ContextResolverFactory.java:83)
at com.sun.jersey.server.impl.application.WebApplicationImpl._initiate(WebApplicationImpl.java:1271)
at com.sun.jersey.server.impl.application.WebApplicationImpl.access$700(WebApplicationImpl.java:169)
at com.sun.jersey.server.impl.application.WebApplicationImpl$13.f(WebApplicationImpl.java:775)
at com.sun.jersey.server.impl.application.WebApplicationImpl$13.f(WebApplicationImpl.java:771)
at com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193)
at com.sun.jersey.server.impl.application.WebApplicationImpl.initiate(WebApplicationImpl.java:771)
at com.sun.jersey.guice.spi.container.servlet.GuiceContainer.initiate(GuiceContainer.java:121)
at com.sun.jersey.spi.container.servlet.ServletContainer$InternalWebComponent.initiate(ServletContainer.java:318)
at com.sun.jersey.spi.container.servlet.WebComponent.load(WebComponent.java:609)
at com.sun.jersey.spi.container.servlet.WebComponent.init(WebComponent.java:210)
at com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:373)
at com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:710)
at com.google.inject.servlet.FilterDefinition.init(FilterDefinition.java:114)
at com.google.inject.servlet.ManagedFilterPipeline.initPipeline(ManagedFilterPipeline.java:98)
at com.google.inject.servlet.GuiceFilter.init(GuiceFilter.java:172)
at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:713)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:857)
... 11 more
Caused by: javax.xml.bind.JAXBException: property "retainReferenceToInfo" is not supported
at com.sun.xml.bind.v2.ContextFactory.createContext(ContextFactory.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:247)
at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:234)
at javax.xml.bind.ContextFinder.find(ContextFinder.java:441)
at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:641)
at com.sun.jersey.api.json.JSONJAXBContext.<init>(JSONJAXBContext.java:246)
at org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver.<init>(JAXBContextResolver.java:72)
at org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver$$FastClassByGuice$$d18e5b53.newInstance(<generated>)
at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.Scopes$1$1.get(Scopes.java:65)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974)
... 50 more
2016-02-16 13:34:32,755 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
2016-02-16 13:34:32,756 INFO [Socket Reader #1 for port 57690] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 57690
2016-02-16 13:34:32,762 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2016-02-16 13:34:32,762 INFO [IPC Server listener on 57690] org.apache.hadoop.ipc.Server: IPC Server listener on 57690: starting
2016-02-16 13:34:32,790 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
2016-02-16 13:34:32,791 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
2016-02-16 13:34:32,791 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
2016-02-16 13:34:32,893 ERROR [main] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Exception while registering
java.lang.NullPointerException
at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.getHttpPort(MRClientService.java:174)
at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:156)
at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:121)
at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:250)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:881)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1151)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1557)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1553)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1486)
2016-02-16 13:34:32,893 INFO [main] org.apache.hadoop.service.AbstractService: Service RMCommunicator failed in state STARTED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPointerException
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPointerException
at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:177)
at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:121)
at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:250)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:881)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1151)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1557)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1553)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1486)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.getHttpPort(MRClientService.java:174)
at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:156)
... 14 more
2016-02-16 13:34:32,895 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
2016-02-16 13:34:32,895 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter failed in state STARTED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPointerException
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPointerException
at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:177)
at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:121)
at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:250)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:881)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1151)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1557)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1553)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1486)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.getHttpPort(MRClientService.java:174)
at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:156)
... 14 more
2016-02-16 13:34:32,896 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state STARTED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPointerException
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPointerException
at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:177)
at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:121)
at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:250)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:881)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1151)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1557)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1553)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1486)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.getHttpPort(MRClientService.java:174)
at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:156)
... 14 more
It was a jetty dependency collision in the org.apache.avro:avro-mapred artifact.
I seem to be running into an issue bumping up against the 8k size limit that embedded tomcat has for stomp websocket messages.
When sending a message from server to client I am getting the following error. Based on the documentation I've read, it seems that tomcat has a limit of 8k for messages going across websockets, but ive also read that Stomp can send partial messages and have the client reassemble them which doesnt seem to be happening.
The message is never getting to the client side handler so I'm pretty confident the issue is in my WebSocketConfig but it seems no matter what parameter I've tried I can not get over the A) size limit of 8k messages and/or B) if its over the buffer limit, send it in partial message chunks.
Both sides have the following error code
[code=1009, reason=The decoded text message was too big for the output buffer and the endpoint does not support partial messages]
I pretty sure I'm missing something easy but can't seem to put my finger on it. Any extra eyes would be appreciated. Thanks!
Server Side Stomp WebSocket Config
#Configuration
#EnableWebSocketMessageBroker
public class WebSocketConfig extends AbstractWebSocketMessageBrokerConfigurer {
private static final Logger logger = LoggerFactory.getLogger(WebSocketConfig.class);
#Override
public void configureMessageBroker(MessageBrokerRegistry config) {
config.enableSimpleBroker(
"/resp",
"/not",
"/sub"
);
config.setApplicationDestinationPrefixes("/admin");
config.setUserDestinationPrefix("/admin");
}
#Override
public void registerStompEndpoints(StompEndpointRegistry registry) {
RequestUpgradeStrategy upgradeStrategy = new TomcatRequestUpgradeStrategy();
registry.addEndpoint("/cmd",
"/connect")
.setHandshakeHandler(new DefaultHandshakeHandler(upgradeStrategy))
.setAllowedOrigins("*")
.withSockJS();
}
#Override
public boolean configureMessageConverters(List<MessageConverter> messageConverters) {
messageConverters.add(new org.springframework.messaging.converter.StringMessageConverter());
return true;
}
#Override
public void configureWebSocketTransport(WebSocketTransportRegistration registration) {
registration.setSendTimeLimit(15*1000);
registration.setMessageSizeLimit(16*1024);
registration.setSendBufferSizeLimit(16*1024);
}
#Bean
public ServletServerContainerFactoryBean createServletServerContainerFactoryBean() {
ServletServerContainerFactoryBean container = new ServletServerContainerFactoryBean();
container.setMaxTextMessageBufferSize(16*1024);
container.setMaxBinaryMessageBufferSize(16*1024);
logger.info("Websocket factory returned");
ContainerProvider.getWebSocketContainer().setDefaultMaxTextMessageBufferSize(20*1024);
return container;
}
}
Server Side Trace Messages
2016-01-27 18:20:54.342 DEBUG 3358 --- [clientInboundChannel-13] o.s.m.s.b.SimpleBrokerMessageHandler : Processing MESSAGE destination=/sub/user-user5e1cc9e1c97a450180ed6ad41d575a33 session=5e1cc9e1c97a450180ed6ad41d575a33 user=core payload={"headers":{"deviceSessionId":"67669nys","message-id":"bf8Pcpx"},"payload":"{\"t...(truncated)
2016-01-27 18:20:54.342 DEBUG 3358 --- [clientInboundChannel-13] o.s.m.s.b.SimpleBrokerMessageHandler : Broadcasting to 1 sessions.
2016-01-27 18:20:54.342 INFO 3358 --- [clientInboundChannel-13] c.l.a.c.AdminProfileController : Response sent: /channelList
2016-01-27 18:20:54.342 TRACE 3358 --- [clientInboundChannel-13] o.s.m.h.i.InvocableHandlerMethod : Method [processObject] returned [null]
2016-01-27 18:20:54.342 TRACE 3358 --- [clientOutboundChannel-3] o.s.messaging.simp.stomp.StompEncoder : Encoding STOMP MESSAGE, headers={apiVersion=[5], message-id=[bf8Pcpx], destination=[/admin/sub/user], content-type=[text/plain;charset=UTF-8], subscription=[0]}
2016-01-27 18:20:54.343 TRACE 3358 --- [clientOutboundChannel-3] s.w.s.s.t.s.WebSocketServerSockJsSession : Cancelling heartbeat in session 5e1cc9e1c97a450180ed6ad41d575a33
2016-01-27 18:20:54.346 TRACE 3358 --- [clientOutboundChannel-3] s.w.s.s.t.s.WebSocketServerSockJsSession : Preparing to write SockJsFrame content='a["MESSAGE\napiVersion:5\nmessage-id:bf8Pcpx\ndestination:/admin/sub/user\nconte...(truncated)'
2016-01-27 18:20:54.347 TRACE 3358 --- [clientOutboundChannel-3] s.w.s.s.t.s.WebSocketServerSockJsSession : Writing SockJsFrame content='a["MESSAGE\napiVersion:5\nmessage-id:bf8Pcpx\ndestination:/admin/sub/user\nconte...(truncated)'
2016-01-27 18:20:54.347 TRACE 3358 --- [clientOutboundChannel-3] o.s.w.s.adapter.NativeWebSocketSession : Sending TextMessage payload=[a["MESSAGE..], byteCount=20097, last=true], StandardWebSocketSession[id=0, uri=/admin/connect/401/5e1cc9e1c97a450180ed6ad41d575a33/websocket]
2016-01-27 18:20:54.373 TRACE 3358 --- [clientOutboundChannel-3] s.w.s.s.t.s.WebSocketServerSockJsSession : Scheduled heartbeat in session 5e1cc9e1c97a450180ed6ad41d575a33
2016-01-27 18:20:54.373 TRACE 3358 --- [http-nio-8084-exec-6] s.w.s.s.t.s.WebSocketServerSockJsSession : Cancelling heartbeat in session 5e1cc9e1c97a450180ed6ad41d575a33
2016-01-27 18:20:54.373 DEBUG 3358 --- [http-nio-8084-exec-6] s.w.s.h.LoggingWebSocketHandlerDecorator : WebSocketServerSockJsSession[id=5e1cc9e1c97a450180ed6ad41d575a33] closed with CloseStatus[code=1009, reason=The decoded text message was too big for the output buffer and the endpoint does not support partial messages]
2016-01-27 18:20:54.373 DEBUG 3358 --- [http-nio-8084-exec-6] o.s.w.s.m.SubProtocolWebSocketHandler : Clearing session 5e1cc9e1c97a450180ed6ad41d575a33
2016-01-27 18:20:54.374 TRACE 3358 --- [http-nio-8084-exec-6] ationConfigEmbeddedWebApplicationContext : Publishing event in org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext#648c94da: SessionDisconnectEvent[sessionId=5e1cc9e1c97a450180ed6ad41d575a33, CloseStatus[code=1009, reason=The decoded text message was too big for the output buffer and the endpoint does not support partial messages]]
2016-01-27 18:20:54.374 DEBUG 3358 --- [http-nio-8084-exec-6] o.s.b.f.s.DefaultListableBeanFactory : Returning cached instance of singleton bean
Client Side Error Stack
2016-01-28 09:33:14.342 ERROR 810 --- [lient-AsyncIO-1] o.s.w.s.s.c.WebSocketClientSockJsSession : Transport error in WebSocketClientSockJsSession[id='c5c680b9a75e488ba7bb129f90b700e6, url=ws://localhost:8084/admin/connect]
java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: Unable to write the complete message as the WebSocket connection has been closed
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.startMessageBlock(WsRemoteEndpointImplBase.java:282) ~[tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.WsSession.sendCloseMessage(WsSession.java:584) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.WsSession.doClose(WsSession.java:488) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.WsSession.close(WsSession.java:455) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.WsFrameClient.close(WsFrameClient.java:94) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.WsFrameClient.access$100(WsFrameClient.java:31) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.WsFrameClient$WsFrameClientCompletionHandler.completed(WsFrameClient.java:134) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.WsFrameClient$WsFrameClientCompletionHandler.completed(WsFrameClient.java:108) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) [na:1.8.0_45]
at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157) [na:1.8.0_45]
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553) [na:1.8.0_45]
at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276) [na:1.8.0_45]
at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297) [na:1.8.0_45]
at java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420) [na:1.8.0_45]
at org.apache.tomcat.websocket.AsyncChannelWrapperNonSecure.read(AsyncChannelWrapperNonSecure.java:52) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.WsFrameClient.processSocketRead(WsFrameClient.java:79) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.WsFrameClient.access$300(WsFrameClient.java:31) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.WsFrameClient$WsFrameClientCompletionHandler.completed(WsFrameClient.java:125) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.WsFrameClient$WsFrameClientCompletionHandler.completed(WsFrameClient.java:108) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) [na:1.8.0_45]
at sun.nio.ch.Invoker$2.run(Invoker.java:218) [na:1.8.0_45]
at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) [na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Unable to write the complete message as the WebSocket connection has been closed
at org.apache.tomcat.websocket.FutureToSendHandler.get(FutureToSendHandler.java:102) ~[tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.startMessageBlock(WsRemoteEndpointImplBase.java:275) ~[tomcat-embed-websocket-8.0.30.jar:8.0.30]
... 24 common frames omitted
Caused by: java.io.IOException: Unable to write the complete message as the WebSocket connection has been closed
at org.apache.tomcat.websocket.WsSession.registerFuture(WsSession.java:658) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.FutureToSendHandler.get(FutureToSendHandler.java:92) ~[tomcat-embed-websocket-8.0.30.jar:8.0.30]
... 25 common frames omitted
2016-01-28 09:33:14.343 WARN 810 --- [lient-AsyncIO-1] l.e.c.w.CustomDefaultStompSessionHandler : handleTransportError
java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: Unable to write the complete message as the WebSocket connection has been closed
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.startMessageBlock(WsRemoteEndpointImplBase.java:282) ~[tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.WsSession.sendCloseMessage(WsSession.java:584) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.WsSession.doClose(WsSession.java:488) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.WsSession.close(WsSession.java:455) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.WsFrameClient.close(WsFrameClient.java:94) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.WsFrameClient.access$100(WsFrameClient.java:31) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.WsFrameClient$WsFrameClientCompletionHandler.completed(WsFrameClient.java:134) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.WsFrameClient$WsFrameClientCompletionHandler.completed(WsFrameClient.java:108) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) [na:1.8.0_45]
at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157) [na:1.8.0_45]
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553) [na:1.8.0_45]
at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276) [na:1.8.0_45]
at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297) [na:1.8.0_45]
at java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420) [na:1.8.0_45]
at org.apache.tomcat.websocket.AsyncChannelWrapperNonSecure.read(AsyncChannelWrapperNonSecure.java:52) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.WsFrameClient.processSocketRead(WsFrameClient.java:79) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.WsFrameClient.access$300(WsFrameClient.java:31) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.WsFrameClient$WsFrameClientCompletionHandler.completed(WsFrameClient.java:125) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.WsFrameClient$WsFrameClientCompletionHandler.completed(WsFrameClient.java:108) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) [na:1.8.0_45]
at sun.nio.ch.Invoker$2.run(Invoker.java:218) [na:1.8.0_45]
at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) [na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Unable to write the complete message as the WebSocket connection has been closed
at org.apache.tomcat.websocket.FutureToSendHandler.get(FutureToSendHandler.java:102) ~[tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.startMessageBlock(WsRemoteEndpointImplBase.java:275) ~[tomcat-embed-websocket-8.0.30.jar:8.0.30]
... 24 common frames omitted
Caused by: java.io.IOException: Unable to write the complete message as the WebSocket connection has been closed
at org.apache.tomcat.websocket.WsSession.registerFuture(WsSession.java:658) [tomcat-embed-websocket-8.0.30.jar:8.0.30]
at org.apache.tomcat.websocket.FutureToSendHandler.get(FutureToSendHandler.java:92) ~[tomcat-embed-websocket-8.0.30.jar:8.0.30]
... 25 common frames omitted
java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: Unable to write the complete message as the WebSocket connection has been closed
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.startMessageBlock(WsRemoteEndpointImplBase.java:282)
at org.apache.tomcat.websocket.WsSession.sendCloseMessage(WsSession.java:584)
at org.apache.tomcat.websocket.WsSession.doClose(WsSession.java:488)
at org.apache.tomcat.websocket.WsSession.close(WsSession.java:455)
at org.apache.tomcat.websocket.WsFrameClient.close(WsFrameClient.java:94)
**2016-01-28 09:33:14.345 DEBUG 810 --- [lient-AsyncIO-1] o.s.w.s.s.c.WebSocketClientSockJsSession : Transport closed with CloseStatus[code=1009, reason=The decoded text message was too big for the output buffer and the endpoint does not support partial messages] in WebSocketClientSockJsSession[id='c5c680b9a75e488ba7bb129f90b700e6, url=ws://localhost.local:8084/admin/connect]**
2016-01-28 09:33:14.345 DEBUG 810 --- [lient-AsyncIO-1] o.s.m.simp.stomp.DefaultStompSession : Connection closed session id=31d8ea85-ff24-ce54-c7cc-72352b88b493
2016-01-28 09:33:14.345 WARN 810 --- [lient-AsyncIO-1] l.e.c.w.CustomDefaultStompSessionHandler : Server disconnect
null
at org.apache.tomcat.websocket.WsFrameClient.access$100(WsFrameClient.java:31)
at org.apache.tomcat.websocket.WsFrameClient$WsFrameClientCompletionHandler.completed(WsFrameClient.java:134)
at org.apache.tomcat.websocket.WsFrameClient$WsFrameClientCompletionHandler.completed(WsFrameClient.java:108)
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
at java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420)
at org.apache.tomcat.websocket.AsyncChannelWrapperNonSecure.read(AsyncChannelWrapperNonSecure.java:52)
at org.apache.tomcat.websocket.WsFrameClient.processSocketRead(WsFrameClient.java:79)
at org.apache.tomcat.websocket.WsFrameClient.access$300(WsFrameClient.java:31)
at org.apache.tomcat.websocket.WsFrameClient$WsFrameClientCompletionHandler.completed(WsFrameClient.java:125)
at org.apache.tomcat.websocket.WsFrameClient$WsFrameClientCompletionHandler.completed(WsFrameClient.java:108)
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
at sun.nio.ch.Invoker$2.run(Invoker.java:218)
at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Unable to write the complete message as the WebSocket connection has been closed
at org.apache.tomcat.websocket.FutureToSendHandler.get(FutureToSendHandler.java:102)
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.startMessageBlock(WsRemoteEndpointImplBase.java:275)
... 24 more
Caused by: java.io.IOException: Unable to write the complete message as the WebSocket connection has been closed
at org.apache.tomcat.websocket.WsSession.registerFuture(WsSession.java:658)
at org.apache.tomcat.websocket.FutureToSendHandler.get(FutureToSendHandler.java:92)
... 25 more
org.springframework.messaging.simp.stomp.ConnectionLostException: Connection closed
at org.springframework.messaging.simp.stomp.DefaultStompSession.afterConnectionClosed(DefaultStompSession.java:459)
at org.springframework.web.socket.messaging.WebSocketStompClient$WebSocketTcpConnectionHandlerAdapter.afterConnectionClosed(WebSocketStompClient.java:353)
at org.springframework.web.socket.sockjs.client.AbstractClientSockJsSession.afterTransportClosed(AbstractClientSockJsSession.java:321)
at org.springframework.web.socket.sockjs.client.WebSocketTransport$ClientSockJsWebSocketHandler.afterConnectionClosed(WebSocketTransport.java:172)
at org.springframework.web.socket.adapter.standard.StandardWebSocketHandlerAdapter.onClose(StandardWebSocketHandlerAdapter.java:141)
at org.apache.tomcat.websocket.WsSession.fireEndpointOnClose(WsSession.java:538)
at org.apache.tomcat.websocket.WsSession.doClose(WsSession.java:489)
at org.apache.tomcat.websocket.WsSession.close(WsSession.java:455)
at org.apache.tomcat.websocket.WsFrameClient.close(WsFrameClient.java:94)
at org.apache.tomcat.websocket.WsFrameClient.access$100(WsFrameClient.java:31)
at org.apache.tomcat.websocket.WsFrameClient$WsFrameClientCompletionHandler.completed(WsFrameClient.java:134)
at org.apache.tomcat.websocket.WsFrameClient$WsFrameClientCompletionHandler.completed(WsFrameClient.java:108)
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:553)
at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
at sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
at java.nio.channels.AsynchronousSocketChannel.read(AsynchronousSocketChannel.java:420)
at org.apache.tomcat.websocket.AsyncChannelWrapperNonSecure.read(AsyncChannelWrapperNonSecure.java:52)
at org.apache.tomcat.websocket.WsFrameClient.processSocketRead(WsFrameClient.java:79)
at org.apache.tomcat.websocket.WsFrameClient.access$300(WsFrameClient.java:31)
at org.apache.tomcat.websocket.WsFrameClient$WsFrameClientCompletionHandler.completed(WsFrameClient.java:125)
at org.apache.tomcat.websocket.WsFrameClient$WsFrameClientCompletionHandler.completed(WsFrameClient.java:108)
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
at sun.nio.ch.Invoker$2.run(Invoker.java:218)
at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-01-28 09:33:14.346 DEBUG 810 --- [lient-AsyncIO-1] o.s.w.s.s.c.WebSocketClientSockJsSession : Closing session with CloseStatus[code=1000, reason=null] in WebSocketClientSockJsSession[id='c5c680b9a75e488ba7bb129f90b700e6, url=ws://localhost.local:8084/admin/connect]
2016-01-28 09:33:14.346 DEBUG 810 --- [lient-AsyncIO-1] o.s.w.s.s.c.WebSocketClientSockJsSession : Ignoring close (already closing or closed), current state=CLOSED
You can change max text buffer size according to your needs:
WebSocketContainer container = ContainerProvider.getWebSocketContainer();
container.setDefaultMaxTextMessageBufferSize(MAX_TEXT_MESSAGE_BUFFER_SIZE);
WebSocketClient wsClient = new StandardWebSocketClient(container);
After banging my head against the wall for a week, I discovered the problem. The client I was using (internal spring java websocket stomp client) is not capable of partial messaging nor is it configurable with the size of buffer for messages. The answer(workaround) was to take the internal java stop client code and implement my own version of it that had a configurable message buffer size as well as implement a way to handle partial messages. Hope this anyone else who hits this problem.