Im trying to configure local logstash config with spring boot in yml format.
Here it is:
appenders:
logstash:
enabled: true
input-type: tcp
destination: 127.0.0.1:4560
and after running app I catch in console:
WARN in net.logstash.logback.appender.LogstashTcpSocketAppender[logstash] - Log destination 127.0.0.1:5010: connection failed. java.net.ConnectException: Connection refused (Connection refused)
at java.net.ConnectException: Connection refused (Connection refused)
I checked all listeners and the port is not in use. I seems to be easy to fix but I dk how.
I am unable to perform sudo apt-get update on by Ubuntu EC2 instance.
When I run -
sudo apt-get update
I get the below error:-
Err:1 http://eu-central-1.ec2.archive.ubuntu.com/ubuntu bionic InRelease
Could not connect to eu-central-1.ec2.archive.ubuntu.com:80 (52.59.228.109), connection timed out Could not connect to eu-central-1.ec2.archive.ubuntu.com:80 (52.59.244.233), connection timed out Could not connect to eu-central-1.ec2.archive.ubuntu.com:80 (18.196.1.133), connection timed out Could not connect to eu-central-1.ec2.archive.ubuntu.com:80 (35.158.129.174), connection timed out Could not connect to eu-central-1.ec2.archive.ubuntu.com:80 (35.159.12.228), connection timed out Could not connect to eu-central-1.ec2.archive.ubuntu.com:80 (52.59.220.169), connection timed out
Err:2 http://eu-central-1.ec2.archive.ubuntu.com/ubuntu bionic-updates InRelease
Unable to connect to eu-central-1.ec2.archive.ubuntu.com:http:
Err:3 http://eu-central-1.ec2.archive.ubuntu.com/ubuntu bionic-backports InRelease
Unable to connect to eu-central-1.ec2.archive.ubuntu.com:http:
Err:4 http://security.ubuntu.com/ubuntu bionic-security InRelease
Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1560:8001::11). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1560:8001::14). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1562::19). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1360:8001::17). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1360:8001::21). - connect (101: Network is unreachable) Could not connect to security.ubuntu.com:80 (91.189.88.161), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.162), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.23), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.26), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.149), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.152), connection timed out
Reading package lists... Done
W: Failed to fetch http://eu-central-1.ec2.archive.ubuntu.com/ubuntu/dists/bionic/InRelease Could not connect to eu-central-1.ec2.archive.ubuntu.com:80 (52.59.228.109), connection timed out Could not connect to eu-central-1.ec2.archive.ubuntu.com:80 (52.59.244.233), connection timed out Could not connect to eu-central-1.ec2.archive.ubuntu.com:80 (18.196.1.133), connection timed out Could not connect to eu-central-1.ec2.archive.ubuntu.com:80 (35.158.129.174), connection timed out Could not connect to eu-central-1.ec2.archive.ubuntu.com:80 (35.159.12.228), connection timed out Could not connect to eu-central-1.ec2.archive.ubuntu.com:80 (52.59.220.169), connection timed out
W: Failed to fetch http://eu-central-1.ec2.archive.ubuntu.com/ubuntu/dists/bionic-updates/InRelease Unable to connect to eu-central-1.ec2.archive.ubuntu.com:http:
W: Failed to fetch http://eu-central-1.ec2.archive.ubuntu.com/ubuntu/dists/bionic-backports/InRelease Unable to connect to eu-central-1.ec2.archive.ubuntu.com:http:
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/bionic-security/InRelease Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1560:8001::11). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1560:8001::14). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1562::19). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1360:8001::17). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1360:8001::21). - connect (101: Network is unreachable) Could not connect to security.ubuntu.com:80 (91.189.88.161), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.162), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.23), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.26), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.149), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.152), connection timed out
W: Some index files failed to download. They have been ignored, or old ones used instead.
To fix this issue I followed this and did the below -
sudo sed -i 's/eu-central-1\.ec2\.//g' /etc/apt/sources.list
The I rebooted my system and tried again. Now I get this error on performing sudo apt-get update -
Err:1 http://archive.ubuntu.com/ubuntu bionic InRelease
Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8001::21). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1560:8001::11). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1560:8001::14). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8001::17). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (91.189.88.161), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.88.162), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.91.23), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.88.149), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.88.152), connection timed out
Err:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8001::21). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1560:8001::11). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1560:8001::14). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8001::17). - connect (101: Network is unreachable)
Err:3 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8001::21). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1560:8001::11). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1560:8001::14). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8001::17). - connect (101: Network is unreachable)
Err:4 http://security.ubuntu.com/ubuntu bionic-security InRelease
Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1562::19). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1360:8001::17). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1360:8001::21). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1560:8001::11). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1560:8001::14). - connect (101: Network is unreachable) Could not connect to security.ubuntu.com:80 (91.189.88.162), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.23), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.26), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.149), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.152), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.161), connection timed out
Reading package lists... Done
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic/InRelease Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8001::21). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1560:8001::11). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1560:8001::14). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8001::17). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (91.189.88.161), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.88.162), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.91.23), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.88.149), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.88.152), connection timed out
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-updates/InRelease Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8001::21). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1560:8001::11). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1560:8001::14). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8001::17). - connect (101: Network is unreachable)
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-backports/InRelease Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8001::21). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1560:8001::11). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1560:8001::14). - connect (101: Network is unreachable) Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8001::17). - connect (101: Network is unreachable)
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/bionic-security/InRelease Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1562::19). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1360:8001::17). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1360:8001::21). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1560:8001::11). - connect (101: Network is unreachable) Cannot initiate the connection to security.ubuntu.com:80 (2001:67c:1560:8001::14). - connect (101: Network is unreachable) Could not connect to security.ubuntu.com:80 (91.189.88.162), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.23), connection timed out Could not connect to security.ubuntu.com:80 (91.189.91.26), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.149), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.152), connection timed out Could not connect to security.ubuntu.com:80 (91.189.88.161), connection timed out
W: Some index files failed to download. They have been ignored, or old ones used instead.
How do I fix this?
I am not sure if there is any security concern about it because I am not an AWS Expert but I had the same issue and To solve it I had to add another outbound rule to All traffic and set the destination to anywhere.
It sounds like your instance does not have internet access.
Either your instance needs to be in a public subnet, with a route table to an attached internet gateway:https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Internet_Gateway.html
Or your instance is in a private subnet, which needs a route table to a nat gateway: https://aws.amazon.com/premiumsupport/knowledge-center/ec2-internet-connectivity/
Edit: If you've confirmed the above to be in place then check that the outbound security group rules allow for outgoing internet access on the expected ports.
Oh got it your ec2 instace need vpc and subnet which you can check from your aws vpc mangement https://ap-south-1.console.aws.amazon.com/vpc/home?region=ap-south-1#subnets:
or if you need guidance use https://docs.aws.amazon.com/batch/latest/userguide/create-public-private-vpc.html
Assuming you have the ec2 in a public subnet with a nat gateway attached. The security group outbound http port 80 rule is required as per the error message. Hope this helps
if you already tried all the solution posted then change the region and it will work , I made the exact same steps in two different regions and it worked in one and the other didn't, this problem faced alot of people.
I have a similar issue. The EC2 indeed has Internet access but not able to connect the Ubuntu repos:
root#ip-10-0-8-82:~# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=111 time=1.91 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=111 time=1.94 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.911/1.928/1.945/0.017 ms
root#ip-10-0-8-82:~# host archive.ubuntu.com
archive.ubuntu.com has address 91.189.88.152
archive.ubuntu.com has address 91.189.88.142
archive.ubuntu.com has IPv6 address 2001:67c:1360:8001::23
archive.ubuntu.com has IPv6 address 2001:67c:1360:8001::24
root#ip-10-0-8-82:~# ping 91.189.88.152
PING 91.189.88.152 (91.189.88.152) 56(84) bytes of data.
^C
--- 91.189.88.152 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2032ms
root#ip-10-0-8-82:~# tracepath 91.189.88.152
1?: [LOCALHOST] pmtu 9001
1: 10.0.0.1 0.248ms pmtu 1500
1: no reply
2: no reply
3: no reply
4: no reply
5: no reply
6: 100.65.10.1 0.552ms asymm 7
7: 52.95.65.144 1.531ms asymm 13
8: 52.95.66.86 4.035ms asymm 13
9: 52.95.66.163 1.765ms asymm 12
10: 100.92.58.40 50.536ms asymm 19
11: 52.93.135.225 48.973ms asymm 18
12: 150.222.246.119 57.410ms asymm 17
13: 100.91.182.0 48.686ms asymm 16
14: 52.93.9.18 50.418ms asymm 19
15: 52.93.9.37 52.182ms asymm 17
16: no reply
17: no reply
18: no reply
19: no reply
20: no reply
21: no reply
22: no reply
23: no reply
24: no reply
25: no reply
26: no reply
27: no reply
28: no reply
29: no reply
30: no reply
Too many hops: pmtu 1500
Resume: pmtu 1500
I did ran the apt-get update before inside the same EC2 with the same IP address. No changes were made to the EC2 or rebooted. Suddenly it stopped working.
Looks a network/routing issue outside the EC2.
stop or remove anti-virus software from the local system and stop the firewall. reprocess installation. Now it works fine.
I was facing same issue. I have fixed it by changing security rules.
just change your setting to default security rules.
Like in inbound only 8080 port and 22 port are configured. delete the rest security laws.
After some JMS connectivity problems I've noticed in logs:
o.a.a.b.T.Transport:? - Transport Connection to: tcp://100.100.100.100:61141 failed: java.io.EOFException
o.a.a.b.T.Transport:? - Transport Connection to: tcp://100.100.100.100:61156 failed: java.io.EOFException
o.a.a.b.T.Transport:? - Transport Connection to: tcp://100.100.100.100:61148 failed: java.io.EOFException
o.a.a.b.T.Transport:? - Transport Connection to: tcp://100.100.100.100:61161 failed: java.io.EOFException
o.a.a.b.T.Transport:? - Transport Connection to: tcp://100.100.100.100:61192 failed: java.io.EOFException
o.a.a.b.T.Transport:? - Transport Connection to: tcp://100.100.100.100:61197 failed: java.io.EOFException
o.a.a.b.T.Transport:? - Transport Connection to: tcp://100.100.100.100:61226 failed: java.io.EOFException
o.a.a.b.T.Transport:? - Transport Connection to: tcp://100.100.100.100:61273 failed: java.io.EOFException
o.a.a.b.T.Transport:? - Transport Connection to: tcp://100.100.100.100:61241 failed: java.io.EOFException
Why JMS is retrying in such way? Does ActiveMQ client/broker technology have any port discovery/negotation protocol?
ActiveMQ has discovery using IP Multicast and failover mechanisms. Not clear why it reconnects to the same host, failover config may have the same host. Client configuration would be helpful to understand.
We're using Active MQ 5.6.0, and have configured a number of JMS message listeners (17 currently), each using a Spring 3.1.1 DefaultMessageListenerContainer, deploying to a Jetty 7.
First of all, message routing and handling works and performs fine. The problem is the time it takes to deploy the Spring configuration: each deployment takes longer than the one before it, so we're talking something like polynomial rather than linear time. That's going to be crippling if we ever get to 30 or so.
The example (trimmed) log file shows the 17 deployments appearing to fight one another for connection, until - after over 2.5 minutes - all are connected:
22:33:21,147 - (Def...r) - JMS message listener invoker needs to establish shared Connection
22:33:22,148 - (Def...r) - JMS message listener invoker needs to establish shared Connection
22:33:23,160 - (Def...r) - JMS message listener invoker needs to establish shared Connection
22:33:23,166 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerA' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:33:24,160 - (Def...r) - JMS message listener invoker needs to establish shared Connection
22:33:24,172 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerB' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:33:25,173 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerC' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:33:26,184 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerD' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:33:27,166 - (Def...r) - JMS message listener invoker needs to establish shared Connection
[*snip*]
22:35:25,135 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerI' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:26,146 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerH' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:27,167 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerD' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:28,178 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerE' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:29,179 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerF' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:31,191 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerG' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:32,192 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerH' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:33,203 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerI' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:34,214 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerJ' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:35,225 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerK' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:36,229 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerL' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:37,230 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerM' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:38,242 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerN' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:40,266 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerC' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:41,277 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerB' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:42,280 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerA' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:44,292 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerN' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:45,293 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerM' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:46,294 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerL' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:47,300 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerK' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:48,282 - (Def...r) - JMS message listener invoker needs to establish shared Connection
22:35:48,311 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerJ' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:49,322 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerI' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:50,322 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerH' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:51,323 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerG' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:53,345 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerF' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:54,366 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerE' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:55,367 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerD' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:56,378 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerH' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:57,389 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerI' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:58,394 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerJ' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:59,400 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerK' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:00,396 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerL' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:01,409 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerM' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:02,412 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerN' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:03,421 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerO' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:04,422 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerA' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:05,426 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerB' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:06,437 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerC' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:09,471 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerO' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:10,472 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerN' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:11,483 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerM' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:12,483 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerL' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:13,459 - (Def...r) - JMS message listener invoker needs to establish shared Connection
22:36:13,494 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerK' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:14,515 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerJ' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:15,516 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerI' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:16,518 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerH' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:17,530 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerD' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:18,541 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerE' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:19,542 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerF' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:21,546 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerG' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:22,557 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerH' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:23,569 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerI' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:24,580 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerJ' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:25,581 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerK' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:26,582 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerL' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:27,593 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerM' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:28,594 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerN' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:29,605 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerO' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:30,616 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerP' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:32,638 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerC' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:33,639 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerB' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:34,650 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerA' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:36,674 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerP' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:37,685 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerO' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:38,805 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerN' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:39,806 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerM' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:40,653 - (Def...r) - JMS message listener invoker needs to establish shared Connection
22:36:40,807 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerL' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:41,824 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerK' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:42,825 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerJ' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:43,837 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerI' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:44,838 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerH' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:45,851 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerG' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:47,863 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerF' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:48,874 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerE' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:49,875 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerD' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:50,876 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerH' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:51,891 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerI' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:52,902 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerJ' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:53,913 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerK' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:54,924 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerL' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:55,925 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerM' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:56,926 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerN' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:57,937 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerO' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:58,935 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerP' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:59,936 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerQ' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:37:00,520 - (Def...r) - Successfully refreshed JMS Connection
22:37:00,520 - (Def...r) - Successfully refreshed JMS Connection
22:37:00,520 - (Def...r) - Successfully refreshed JMS Connection
22:37:00,520 - (Def...r) - Successfully refreshed JMS Connection
22:37:00,520 - (Def...r) - Successfully refreshed JMS Connection
22:37:00,520 - (Def...r) - Successfully refreshed JMS Connection
22:37:00,520 - (Def...r) - Successfully refreshed JMS Connection
22:37:00,520 - (Def...r) - Successfully refreshed JMS Connection
22:37:00,520 - (Def...r) - Successfully refreshed JMS Connection
22:37:00,520 - (Def...r) - Successfully refreshed JMS Connection
22:37:00,520 - (Def...r) - Successfully refreshed JMS Connection
22:37:00,520 - (Def...r) - Successfully refreshed JMS Connection
For the sake of brevity, I won't post each Spring bean, but here are the important bits:
<jee:jndi-lookup id="srcConnectionFactory" jndi-name="${jndi.jms.connfactory}">
<jee:environment>
java.naming.factory.initial = ${jndi.jms.naming.factory.initial}
java.naming.provider.url = ${jndi.jms.naming.url}
</jee:environment>
</jee:jndi-lookup>
<bean id="connectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
<property name="targetConnectionFactory" ref="srcConnectionFactory"/>
<property name="cacheConsumers" value="false"/>
<property name="reconnectOnException" value="true"/>
<property name="sessionCacheSize" value="20"></property>
</bean>
<jee:jndi-lookup id="handlerA_Dest" jndi-name="handlerA">
<jee:environment>
java.naming.factory.initial = ${jndi.jms.naming.factory.initial}
java.naming.provider.url = ${jndi.jms.naming.url}
queue.handlerA = handlerA
</jee:environment>
</jee:jndi-lookup>
[x 17]
<bean id="handlerA_Container" class="org.springframework.jms.listener.DefaultMessageListenerContainer"
p:connectionFactory-ref="connectionFactory"
p:messageListener-ref="handlerA"
p:destination-ref="handlerA_Dest">
<property name="concurrentConsumers"><value>10</value></property>
<property name="maxConcurrentConsumers"><value>10</value></property>
<property name="idleConsumerLimit"><value>5</value></property>
</bean>
[x 17]
Any idea why these containers should be contending at all?
Unfortunately, the MessageSelector approach is not (any longer) an option for us. It's one Listener per type-of-Message.
Update It's true that, strictly speaking, the MQ broker is indeed down at the start of this deployment, but it's deployed into the same Jetty (this all happens in the Maven integration test phase), so even if there was a brief period where the broker started up second, I wouldn't expect it to be 2-3 minutes late.
What I'm seeing looks really looks like the broker deployment is being 'blocked' by this Spring deployment...
This is nothing to do with Spring or contention between clients.
The broker wasn't available from 22:33:23,166 until 22:37:00,520.
Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
Looks to me like they all connected within 1 millisecond as soon as the broker was available.