GeoServer Image Mosaics: Queries on PostGIS index tables exceed max connection limit - geoserver

I have succesfully created 2 Geoserver ImageMosaics for time series rasters, with the index being a PostGIS database table, defined in the datastore.properties file. The optional parameters of the properties config file as such as Connection\ timeout and max\ connections are all left on default.
Now, I am running into issues with the availability of the raster layers, as the number of open connections from (what I assume is) the GeoServer instance to the database exceeds PostgreSQL's limit of 100 connections at a time.
Below is a partial readout of PostgreSQL's pg_stat_activity; to show the active connections to the database (some names replaced).
My question is, whether this reflects normal behaviour:
According to the datastore.properties default values, I would expect only 10 max connections for each ImgMosaic (here: 13 open connections for image_mosaic_1). Also, all connections are idle, sometimes for hours.
I am aware of the option to use JNDI for GeoServer connection pooling. But I'm unsure if this would rectify the issue, or if this indicates a faulty configuration of the ImgMosaics (or wrong understanding on my part).
pg_stat_activity output (partial)
application_name
backend_start
query_start
state_change
wait_event_type
wait_event
state
query
backend_type
PostgreSQL JDBC Driver
2021-11-02 15:00:42
2021-11-02 15:00:48
2021-11-02 15:00:48
Client
ClientRead
idle
SELECT min("time_stamp") FROM "image_mosaic_schema"."img_mosaic_1" WHERE ("time_stamp" BETWEEN $1 AND $2 AND "time_stamp" > $3)
client backend
PostgreSQL JDBC Driver
2021-11-02 13:09:59
2021-11-02 13:34:58
2021-11-02 13:34:58
Client
ClientRead
idle
SELECT min("time_stamp") FROM "image_mosaic_schema"."img_mosaic_1" WHERE ("time_stamp" BETWEEN $1 AND $2 AND "time_stamp" > $3)
client backend
PostgreSQL JDBC Driver
2021-11-02 14:42:59
2021-11-02 14:43:00
2021-11-02 14:43:00
Client
ClientRead
idle
SELECT min("time_stamp") FROM "image_mosaic_schema"."img_mosaic_1" WHERE ("time_stamp" BETWEEN $1 AND $2 AND "time_stamp" > $3)
client backend
PostgreSQL JDBC Driver
2021-11-02 15:00:42
2021-11-02 15:00:49
2021-11-02 15:00:49
Client
ClientRead
idle
SELECT min("time_stamp") FROM "image_mosaic_schema"."img_mosaic_1" WHERE ("time_stamp" BETWEEN $1 AND $2 AND "time_stamp" > $3)
client backend
PostgreSQL JDBC Driver
2021-11-02 14:42:33
2021-11-02 14:43:01
2021-11-02 14:43:01
Client
ClientRead
idle
SELECT min("time_stamp") FROM "image_mosaic_schema"."img_mosaic_1" WHERE ("time_stamp" BETWEEN $1 AND $2 AND "time_stamp" > $3)
client backend
PostgreSQL JDBC Driver
2021-11-02 14:42:59
2021-11-02 14:43:01
2021-11-02 14:43:01
Client
ClientRead
idle
SELECT min("time_stamp") FROM "image_mosaic_schema"."img_mosaic_1" WHERE ("time_stamp" BETWEEN $1 AND $2 AND "time_stamp" > $3)
client backend
PostgreSQL JDBC Driver
2021-11-02 14:43:00
2021-11-02 14:43:00
2021-11-02 14:43:00
Client
ClientRead
idle
SELECT min("time_stamp") FROM "image_mosaic_schema"."img_mosaic_1" WHERE ("time_stamp" BETWEEN $1 AND $2 AND "time_stamp" > $3)
client backend
PostgreSQL JDBC Driver
2021-11-02 14:43:00
2021-11-02 14:43:00
2021-11-02 14:43:00
Client
ClientRead
idle
SELECT min("time_stamp") FROM "image_mosaic_schema"."img_mosaic_1" WHERE ("time_stamp" BETWEEN $1 AND $2 AND "time_stamp" > $3)
client backend
PostgreSQL JDBC Driver
2021-11-02 13:10:24
2021-11-02 13:34:56
2021-11-02 13:34:56
Client
ClientRead
idle
SELECT min("time_stamp") FROM "image_mosaic_schema"."img_mosaic_1" WHERE ("time_stamp" BETWEEN $1 AND $2 AND "time_stamp" > $3)
client backend
PostgreSQL JDBC Driver
2021-11-02 13:10:07
2021-11-02 13:34:58
2021-11-02 13:34:58
Client
ClientRead
idle
SELECT min("time_stamp") FROM "image_mosaic_schema"."img_mosaic_1" WHERE ("time_stamp" BETWEEN $1 AND $2 AND "time_stamp" > $3)
client backend
PostgreSQL JDBC Driver
2021-11-02 14:43:00
2021-11-02 14:43:01
2021-11-02 14:43:01
Client
ClientRead
idle
SELECT min("time_stamp") FROM "image_mosaic_schema"."img_mosaic_1" WHERE ("time_stamp" BETWEEN $1 AND $2 AND "time_stamp" > $3)
client backend
PostgreSQL JDBC Driver
2021-11-02 15:00:42
2021-11-02 15:00:48
2021-11-02 15:00:48
Client
ClientRead
idle
SELECT min("time_stamp") FROM "image_mosaic_schema"."img_mosaic_1" WHERE ("time_stamp" BETWEEN $1 AND $2 AND "time_stamp" > $3)
client backend
PostgreSQL JDBC Driver
2021-11-02 15:00:42
2021-11-02 15:00:42
2021-11-02 15:00:42
Client
ClientRead
idle
SELECT min("time_stamp") FROM "image_mosaic_schema"."img_mosaic_1" WHERE ("time_stamp" BETWEEN $1 AND $2 AND "time_stamp" > $3)
client backend
PostgreSQL JDBC Driver
2021-11-02 15:19:58
2021-11-02 15:19:59
2021-11-02 15:19:59
Client
ClientRead
idle
SELECT min("time_stamp") FROM "image_mosaic_schema"."img_mosaic_2" WHERE ("time_stamp" BETWEEN $1 AND $2 AND "time_stamp" > $3)
client backend
PostgreSQL JDBC Driver
2021-11-02 13:10:46
2021-11-02 13:54:10
2021-11-02 13:54:10
Client
ClientRead
idle
SELECT min("time_stamp") FROM "image_mosaic_schema"."img_mosaic_2" WHERE ("time_stamp" BETWEEN $1 AND $2 AND "time_stamp" > $3)
client backend
PostgreSQL JDBC Driver
2021-11-02 13:54:06
2021-11-02 13:54:10
2021-11-02 13:54:10
Client
ClientRead
idle
SELECT min("time_stamp") FROM "image_mosaic_schema"."img_mosaic_2" WHERE ("time_stamp" BETWEEN $1 AND $2 AND "time_stamp" > $3)
client backend
PostgreSQL JDBC Driver
2021-11-02 13:10:46
2021-11-02 13:54:10
2021-11-02 13:54:10
Client
ClientRead
idle
SELECT min("time_stamp") FROM "image_mosaic_schema"."img_mosaic_2" WHERE ("time_stamp" BETWEEN $1 AND $2 AND "time_stamp" > $3)
client backend
PostgreSQL JDBC Driver
2021-11-02 13:54:06
2021-11-02 13:54:31
2021-11-02 13:54:31
Client
ClientRead
idle
SELECT min("time_stamp") FROM "image_mosaic_schema"."img_mosaic_2" WHERE ("time_stamp" BETWEEN $1 AND $2 AND "time_stamp" > $3)
client backend
PostgreSQL JDBC Driver
2021-11-02 13:10:46
2021-11-02 13:28:16
2021-11-02 13:28:16
Client
ClientRead
idle
SELECT min("time_stamp") FROM "image_mosaic_schema"."img_mosaic_2" WHERE ("time_stamp" BETWEEN $1 AND $2 AND "time_stamp" > $3)
client backend
PostgreSQL JDBC Driver
2021-11-02 15:19:59
2021-11-02 15:20:01
2021-11-02 15:20:01
Client
ClientRead
idle
SELECT min("time_stamp") FROM "image_mosaic_schema"."img_mosaic_2" WHERE ("time_stamp" BETWEEN $1 AND $2 AND "time_stamp" > $3)
client backend
PostgreSQL JDBC Driver
2021-11-02 13:54:06
2021-11-02 13:54:10
2021-11-02 13:54:10
Client
ClientRead
idle
SELECT min("time_stamp") FROM "image_mosaic_schema"."img_mosaic_2" WHERE ("time_stamp" BETWEEN $1 AND $2 AND "time_stamp" > $3)
client backend
PostgreSQL JDBC Driver
2021-11-02 13:54:10
2021-11-02 13:54:34
2021-11-02 13:54:34
Client
ClientRead
idle
SELECT min("time_stamp") FROM "image_mosaic_schema"."img_mosaic_2" WHERE ("time_stamp" BETWEEN $1 AND $2 AND "time_stamp" > $3)
client backend
PostgreSQL JDBC Driver
2021-11-02 13:09:59
2021-11-02 13:34:58
2021-11-02 13:34:58
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 13:09:59
2021-11-02 13:34:58
2021-11-02 13:34:58
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 13:09:59
2021-11-02 13:34:58
2021-11-02 13:34:58
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 13:09:59
2021-11-02 13:34:50
2021-11-02 13:34:50
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 15:19:58
2021-11-02 15:20:01
2021-11-02 15:20:01
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 13:10:42
2021-11-02 13:28:16
2021-11-02 13:28:16
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 14:43:00
2021-11-02 14:43:01
2021-11-02 14:43:01
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 14:43:00
2021-11-02 14:43:01
2021-11-02 14:43:01
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 14:43:00
2021-11-02 14:43:01
2021-11-02 14:43:01
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 14:43:00
2021-11-02 14:43:00
2021-11-02 14:43:00
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 14:43:00
2021-11-02 14:43:00
2021-11-02 14:43:00
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 14:43:00
2021-11-02 14:43:00
2021-11-02 14:43:00
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 13:10:46
2021-11-02 13:28:16
2021-11-02 13:28:16
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 13:10:46
2021-11-02 13:28:16
2021-11-02 13:28:16
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 13:10:46
2021-11-02 13:54:34
2021-11-02 13:54:34
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 13:10:46
2021-11-02 13:28:16
2021-11-02 13:28:16
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 13:10:46
2021-11-02 13:54:10
2021-11-02 13:54:10
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 13:17:14
2021-11-02 13:34:58
2021-11-02 13:34:58
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 13:28:14
2021-11-02 13:28:15
2021-11-02 13:28:15
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 13:28:15
2021-11-02 13:28:15
2021-11-02 13:28:15
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 13:28:15
2021-11-02 13:28:16
2021-11-02 13:28:16
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 13:28:15
2021-11-02 13:28:15
2021-11-02 13:28:15
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 13:28:15
2021-11-02 13:28:15
2021-11-02 13:28:15
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 13:28:15
2021-11-02 13:28:15
2021-11-02 13:28:15
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 13:54:06
2021-11-02 13:54:34
2021-11-02 13:54:34
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 13:54:06
2021-11-02 13:54:33
2021-11-02 13:54:33
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 13:54:10
2021-11-02 13:54:34
2021-11-02 13:54:34
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 15:00:42
2021-11-02 15:00:49
2021-11-02 15:00:49
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 15:00:42
2021-11-02 15:00:48
2021-11-02 15:00:48
Client
ClientRead
idle
ROLLBACK
client backend
PostgreSQL JDBC Driver
2021-11-02 15:00:42
2021-11-02 15:00:48
2021-11-02 15:00:48
Client
ClientRead
idle
ROLLBACK
client backend


GeoServer uses a pool of Postgis connections so the number of open idle connections is normal.
If you have a slow Postgis server or other people are heavily using it, you can see time outs for requests, first make sure you have an index on your tile index table and that it is being used by the queries that GeoServer is making (these will appear in the log file if you turn logging up to GEOSERVER-DEVELOPER).
Failing that turn up the connection timeout from the default.

Related

Springboot Hikari Oracle RAC jdbc connection error

I'm getting error when deployingg project
INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting...
WARN o.s.b.a.j.DataSourceHealthIndicator - DataSource health check failed
java.lang.RuntimeException: Driver oracle.jdbc.OracleDriver claims to not accept jdbcUrl, jdbc-url:...
Connection String
jdbc:oracle:thin:
#(DESCRIPTION=(CONNECT_TIMEOUT=5)(TRANSPORT_CONNECT_TIMEOUT=3)(RETRY_COUNT=1)(ADDRESS=(PROTOCOL=TCP)(HOST=my2.org)(PORT=123))(ADDRESS=(PROTOCOL=TCP)(HOST=my1.org)(PORT=123))(CONNECT_DATA=(SERVICE_NAME=DG)))

Docker Desktop cannot access web from container

Tried almost every attempt found here in the community but didn't figure out how to access web through continers created in my Docker Desktop Environment.
This is my last attempt with Ubuntu (also tried Centos and Debian suites):
docker run -i -t --rm=true ubuntu /bin/bash
root#310f7a660efd:/# apt-get update -y
Err:1 http://archive.ubuntu.com/ubuntu bionic InRelease
Could not connect to archive.ubuntu.com:80 (91.189.88.24). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (91.189.88.173). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (91.189.88.174). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (91.189.88.162). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (91.189.88.31). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (91.189.88.149). - connect (111: Connection refused)
Err:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
Unable to connect to archive.ubuntu.com:http:
Err:3 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
Unable to connect to archive.ubuntu.com:http:
Err:4 http://security.ubuntu.com/ubuntu bionic-security InRelease
Could not connect to security.ubuntu.com:80 (91.189.91.14). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (91.189.88.24). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (91.189.88.162). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (91.189.88.31). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (91.189.88.174). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (91.189.91.26). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (91.189.91.24). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (91.189.91.23). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (91.189.88.173). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (91.189.88.149). - connect (111: Connection refused)
Reading package lists... Done
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic/InRelease Could not connect to archive.ubuntu.com:80 (91.189.88.24). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (91.189.88.173). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (91.189.88.174). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (91.189.88.162). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (91.189.88.31). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (91.189.88.149). - connect (111: Connection refused)
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-updates/InRelease Unable to connect to archive.ubuntu.com:http:
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/bionic-backports/InRelease Unable to connect to archive.ubuntu.com:http:
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/bionic-security/InRelease Could not connect to security.ubuntu.com:80 (91.189.91.14). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (91.189.88.24). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (91.189.88.162). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (91.189.88.31). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (91.189.88.174). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (91.189.91.26). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (91.189.91.24). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (91.189.91.23). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (91.189.88.173). - connect (111: Connection refused) Could not connect to security.ubuntu.com:80 (91.189.88.149). - connect (111: Connection refused)
W: Some index files failed to download. They have been ignored, or old ones used instead.
If i ping google.com from the server no problem but just can't access repos.
Seems a proxy problem but just because I set proxy in the Docker settings it allows me to download images from dockerhub.
Also tried DNS fixed to 8.8.8.8.
My OS is Windows 10 and this is Docker info:
Client:
Debug Mode: false
Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 3
Server Version: 19.03.2
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894--------------16b8817763c29fb
runc version: 425e105d5a03----------9eeede87f
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.184-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.952GiB
Name: docker-desktop
ID: UL2M:2GDU:GGIH:IO6C:EJWA:YUXM:5DFL:F6Q4:XTFE:K3MN:7474:TY4J
Docker Root Dir: /var/lib/docker
Debug Mode: true
File Descriptors: 38
Goroutines: 58
System Time: 2019-10-07T13:34:51.1886632Z
EventsListeners: 1
HTTP Proxy: xxxxxxxxxxxxxxxx:3128
HTTPS Proxy: xxxxxxxxxxxxxxxx:3128
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
This is quite sad situation since if a container can't access web it's very limited.
Also if I put unreachable path from those stack in my chrome they are perfectly available.
Hope for some advice to put me in the right direction.

Can not connect to ZooKeeper/Hive from host to Sandbox Hortonworks HDP VM

I downloaded HDP-Sandbox (in an Oracle VirtualBox VM) a while ago, never used it much, and I’m now trying to access data from the outside world using Hive HDBC.
I use hive-jdbc 1.2.2 from apache, which I got from mvnrepository, with all the dependencies in the classpath, or hortonworks JDBC got from their repository
http://mvnrepository.com/artifact/org.apache.hive/hive-jdbc/1.2.2
or
http://repo.hortonworks.com/content/repositories/releases/org/apache/hive/hive-jdbc/1.2.1000.2.6.4.0-91
I wrote this very basic program
import java.sql.Connection;
import java.sql.DriverManager;
public class Main {
public static void main(final String[] args) {
try {
final Class<?> driver = Class.forName("org.apache.hive.jdbc.HiveDriver");
DriverManager.setLoginTimeout(10);
try (final Connection connection = DriverManager.getConnection("jdbc:hive2://sandbox.hortonworks.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2")) {
//
}
} catch (final Exception e) {
e.printStackTrace();
}
}
}
Connection fails with a Null Pointer Exception or a timeout (see the output logs below, one for each JDBC driver I used), I guess it’s caused by a configuration error – even though I use my sandbox out of the… box.
My sandbox version
Sandbox information:
Created on: 25_10_2016_08_11_26 for
Hadoop stack version: Hadoop 2.7.3.2.5.0.0-1245
Ambari Version: 2.4.0.0-1225
Ambari Hash: 59175b7aa1ddb74b85551c632e3ce42fed8f0c85
Ambari build: Release : 1225
Java version: 1.8.0_111
OS Version: CentOS release 6.8 (Final)
My VM screen
HDP 2.5
http://hortonworks.com
To initiate your Hortonworks Sandbox session,
please open a browser and enter this address
in the browser's address field:
http://127.0.0.1:8888/
My hosts file :
…/…
127.0.0.1 sandbox.hortonworks.com
…/…
Hive Summary from the Ambari screen
Hive Metastore Started No alerts
HiveServer2 Started No alerts
WebHCat Server Started No alerts
Hive Client 1 Hive Client Installed
HiveServer2 JDBC URL jdbc:hive2://sandbox.hortonworks.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
Zookeeper Summary
ZooKeeper Server Started No alerts
ZooKeeper Client 1 ZooKeeper Client Installed
I’m stuck, any help would be greatly appreciated!
Thanks !
Program outputs:
With apache hive jdbc 1.2.2:
16:52:06,378 INFO hive.jdbc.Utils: Supplied authorities: sandbox.hortonworks.com:2181
16:52:06,500 INFO curator.framework.imps.CuratorFrameworkImpl: Starting
16:52:06,503 DEBUG curator.CuratorZookeeperClient: Starting
16:52:06,503 DEBUG curator.ConnectionState: Starting
16:52:06,503 DEBUG curator.ConnectionState: reset
16:52:06,515 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
16:52:06,515 INFO zookeeper.ZooKeeper: Client environment:java.version=9.0.1
16:52:06,515 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
16:52:06,515 INFO zookeeper.ZooKeeper: Client environment:java.home=D:\JDK\jdk9
16:52:06,516 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=sandbox.hortonworks.com:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState#24269709
16:52:06,521 DEBUG zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is false
16:52:06,631 INFO zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
16:52:06,633 INFO zookeeper.ClientCnxn: Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session
16:52:06,635 DEBUG zookeeper.ClientCnxn: Session establishment request sent on 127.0.0.1/127.0.0.1:2181
16:52:06,645 INFO zookeeper.ClientCnxn: Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x160ff8bf98b0008, negotiated timeout = 40000
16:52:06,650 INFO curator.framework.state.ConnectionStateManager: State change: CONNECTED
16:52:06,663 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x160ff8bf98b0008, packet:: clientPath:null serverPath:null finished:false header:: 1,12 replyHeader:: 1,1347,0 request:: '/hiveserver2,F response:: v{'serverUri=sandbox.hortonworks.com:10000;version=1.2.1000.2.5.0.0-1245;sequence=0000000007},s{467,467,1477381899524,1477381899524,0,15,0,0,10,1,1340}
16:52:06,669 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x160ff8bf98b0008, packet:: clientPath:null serverPath:null finished:false header:: 2,4 replyHeader:: 2,1347,0 request:: '/hiveserver2/serverUri=sandbox.hortonworks.com:10000;version=1.2.1000.2.5.0.0-1245;sequence=0000000007,F response:: #686976652e736572766572322e61757468656e7469636174696f6e3d4e4f4e453b686976652e736572766572322e7472616e73706f72742e6d6f64653d62696e6172793b686976652e736572766572322e7468726966742e7361736c2e716f703d617574683b686976652e736572766572322e7468726966742e62696e642e686f73743d73616e64626f782e686f72746f6e776f726b732e636f6d3b686976652e736572766572322e7468726966742e706f72743d31303030303b686976652e736572766572322e7573652e53534c3d66616c7365,s{1340,1340,1516116649382,1516116649382,0,0,0,99360168454324228,213,0,1340}
16:52:06,670 INFO hive.jdbc.ZooKeeperHiveClientHelper: Selected HiveServer2 instance with uri: hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=sandbox.hortonworks.com;hive.server2.thrift.port=10000;hive.server2.use.SSL=false
16:52:06,670 DEBUG curator.framework.imps.CuratorFrameworkImpl: Closing
16:52:06,671 DEBUG curator.CuratorZookeeperClient: Closing
16:52:06,671 DEBUG curator.ConnectionState: Closing
16:52:06,672 DEBUG zookeeper.ZooKeeper: Closing session: 0x160ff8bf98b0008
16:52:06,672 DEBUG zookeeper.ClientCnxn: Closing client for session: 0x160ff8bf98b0008
16:52:06,674 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x160ff8bf98b0008, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,1348,0 request:: null response:: null
16:52:06,674 DEBUG zookeeper.ClientCnxn: Disconnecting client for session: 0x160ff8bf98b0008
16:52:06,674 INFO zookeeper.ZooKeeper: Session: 0x160ff8bf98b0008 closed
16:52:06,674 INFO hive.jdbc.Utils: Resolved authority: hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=sandbox.hortonworks.com;hive.server2.thrift.port=10000;hive.server2.use.SSL=false
16:52:06,675 INFO zookeeper.ClientCnxn: EventThread shut down
16:52:06,731 INFO hive.jdbc.HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=sandbox.hortonworks.com;hive.server2.thrift.port=10000;hive.server2.use.SSL=false/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
16:52:06,733 DEBUG thrift.transport.TSaslTransport: opening transport org.apache.thrift.transport.TSaslClientTransport#49049a04
java.lang.NullPointerException
at org.apache.thrift.transport.TSocket.open(TSocket.java:170)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:206)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:178)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:678)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:252)
at com.mycompany.hadoop.hiveconnect.Main.main(Main.java:13)
With Hortonworks hive jdbc 1.2.1000.2.6.4.0-91
16:49:57,414 INFO hive.jdbc.Utils: Supplied authorities: sandbox.hortonworks.com:2181
16:49:57,544 INFO curator.framework.imps.CuratorFrameworkImpl: Starting
16:49:57,546 DEBUG curator.CuratorZookeeperClient: Starting
16:49:57,546 DEBUG curator.ConnectionState: Starting
16:49:57,546 DEBUG curator.ConnectionState: reset
16:49:57,555 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-91--1, built on 01/04/2018 10:14 GMT
16:49:57,555 INFO zookeeper.ZooKeeper: Client environment:java.version=9.0.1
16:49:57,555 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
16:49:57,555 INFO zookeeper.ZooKeeper: Client environment:java.home=D:\JDK\jdk9
16:49:57,556 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=sandbox.hortonworks.com:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState#1190200a
16:49:57,560 DEBUG zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is false
16:49:57,668 INFO zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
16:49:57,673 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /127.0.0.1:17993, server: 127.0.0.1/127.0.0.1:2181
16:49:57,674 DEBUG zookeeper.ClientCnxn: Session establishment request sent on 127.0.0.1/127.0.0.1:2181
16:49:57,787 INFO zookeeper.ClientCnxn: Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x160ff8bf98b0006, negotiated timeout = 40000
16:49:57,807 INFO curator.framework.state.ConnectionStateManager: State change: CONNECTED
16:49:58,378 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x160ff8bf98b0006, packet:: clientPath:null serverPath:null finished:false header:: 1,12 replyHeader:: 1,1343,0 request:: '/hiveserver2,F response:: v{'serverUri=sandbox.hortonworks.com:10000;version=1.2.1000.2.5.0.0-1245;sequence=0000000007},s{467,467,1477381899524,1477381899524,0,15,0,0,10,1,1340}
16:49:58,400 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x160ff8bf98b0006, packet:: clientPath:null serverPath:null finished:false header:: 2,4 replyHeader:: 2,1343,0 request:: '/hiveserver2/serverUri=sandbox.hortonworks.com:10000;version=1.2.1000.2.5.0.0-1245;sequence=0000000007,F response:: #686976652e736572766572322e61757468656e7469636174696f6e3d4e4f4e453b686976652e736572766572322e7472616e73706f72742e6d6f64653d62696e6172793b686976652e736572766572322e7468726966742e7361736c2e716f703d617574683b686976652e736572766572322e7468726966742e62696e642e686f73743d73616e64626f782e686f72746f6e776f726b732e636f6d3b686976652e736572766572322e7468726966742e706f72743d31303030303b686976652e736572766572322e7573652e53534c3d66616c7365,s{1340,1340,1516116649382,1516116649382,0,0,0,99360168454324228,213,0,1340}
16:49:58,402 DEBUG curator.framework.imps.CuratorFrameworkImpl: Closing
16:49:58,403 DEBUG curator.CuratorZookeeperClient: Closing
16:49:58,403 DEBUG curator.ConnectionState: Closing
16:49:58,405 DEBUG zookeeper.ZooKeeper: Closing session: 0x160ff8bf98b0006
16:49:58,405 DEBUG zookeeper.ClientCnxn: Closing client for session: 0x160ff8bf98b0006
16:49:58,466 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x160ff8bf98b0006, packet:: clientPath:null serverPath:null finished:false header:: 3,-11 replyHeader:: 3,1344,0 request:: null response:: null
16:49:58,466 DEBUG zookeeper.ClientCnxn: Disconnecting client for session: 0x160ff8bf98b0006
16:49:58,466 INFO zookeeper.ZooKeeper: Session: 0x160ff8bf98b0006 closed
16:49:58,467 INFO hive.jdbc.Utils: Resolved authority: sandbox.hortonworks.com:10000
16:49:58,467 INFO zookeeper.ClientCnxn: EventThread shut down
16:49:58,545 DEBUG thrift.transport.TSaslTransport: opening transport org.apache.thrift.transport.TSaslClientTransport#2794eab6
16:49:58,549 DEBUG thrift.transport.TSaslClientTransport: Sending mechanism name PLAIN and initial response of length 20
16:49:58,550 DEBUG thrift.transport.TSaslTransport: CLIENT: Writing message with status START and payload length 5
16:49:58,551 DEBUG thrift.transport.TSaslTransport: CLIENT: Writing message with status COMPLETE and payload length 20
16:49:58,551 DEBUG thrift.transport.TSaslTransport: CLIENT: Start message handled
16:49:58,551 DEBUG thrift.transport.TSaslTransport: CLIENT: Main negotiation loop complete
16:49:58,551 DEBUG thrift.transport.TSaslTransport: CLIENT: SASL Client receiving last message
16:50:08,552 WARN hive.jdbc.HiveConnection: Failed to connect to sandbox.hortonworks.com:10000
16:50:08,553 INFO curator.framework.imps.CuratorFrameworkImpl: Starting
16:50:08,554 DEBUG curator.CuratorZookeeperClient: Starting
16:50:08,554 DEBUG curator.ConnectionState: Starting
16:50:08,554 DEBUG curator.ConnectionState: reset
16:50:08,554 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=sandbox.hortonworks.com:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState#46cdf8bd
16:50:08,557 INFO zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
16:50:08,558 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /127.0.0.1:18018, server: 127.0.0.1/127.0.0.1:2181
16:50:08,558 DEBUG zookeeper.ClientCnxn: Session establishment request sent on 127.0.0.1/127.0.0.1:2181
16:50:08,592 INFO zookeeper.ClientCnxn: Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x160ff8bf98b0007, negotiated timeout = 40000
16:50:08,592 INFO curator.framework.state.ConnectionStateManager: State change: CONNECTED
16:50:08,594 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x160ff8bf98b0007, packet:: clientPath:null serverPath:null finished:false header:: 1,12 replyHeader:: 1,1345,0 request:: '/hiveserver2,F response:: v{'serverUri=sandbox.hortonworks.com:10000;version=1.2.1000.2.5.0.0-1245;sequence=0000000007},s{467,467,1477381899524,1477381899524,0,15,0,0,10,1,1340}
16:50:08,594 DEBUG curator.framework.imps.CuratorFrameworkImpl: Closing
16:50:08,594 DEBUG curator.CuratorZookeeperClient: Closing
16:50:08,594 DEBUG curator.ConnectionState: Closing
16:50:08,594 DEBUG zookeeper.ZooKeeper: Closing session: 0x160ff8bf98b0007
16:50:08,594 DEBUG zookeeper.ClientCnxn: Closing client for session: 0x160ff8bf98b0007
16:50:08,605 DEBUG zookeeper.ClientCnxn: Reading reply sessionid:0x160ff8bf98b0007, packet:: clientPath:null serverPath:null finished:false header:: 2,-11 replyHeader:: 2,1346,0 request:: null response:: null
16:50:08,605 DEBUG zookeeper.ClientCnxn: Disconnecting client for session: 0x160ff8bf98b0007
16:50:08,605 INFO zookeeper.ZooKeeper: Session: 0x160ff8bf98b0007 closed
16:50:08,605 INFO zookeeper.ClientCnxn: EventThread shut down
16:50:08,605 ERROR hive.jdbc.Utils: Unable to read HiveServer2 configs from ZooKeeper
java.sql.SQLException: Could not open client transport for any of the Server URI's in ZooKeeper: java.net.SocketTimeoutException: Read timed out
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:228)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:166)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:678)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:252)
at com.mycompany.hadoop.hiveconnect.Main.main(Main.java:13)
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:307)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204)
... 5 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.base/java.net.SocketInputStream.socketRead0(Native Method)
at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 10 more

Well, I could not solve this.
But later on, I downloaded the 2.6.4 sandbox and, magic, it worked like a charm!

Squirrel Client Connecting to Phoenix - Timeout Exception

I am trying to connect to Phoenix via Squirrel client. I am receiving the following logs in the Squirrel logs. The logs suggests that the ClientConnection to zooperkeeper is established however it fails when a SQLClient Connection is being established with a Timeout exception.
I have copied the phoenix client jar into the lib directory of Squirrel and the driver is registered succesfully. Also when I run the SQLLine.py utility in the localhost it loads the SQL commandline to Phoenix succesfully and I can run the commands. Added the phoenix core jars to the $HBASE_HOME/lib folders as well.
2015-06-15 12:48:53,766 [pool-7-thread-1] INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Process identifier=hconnection-0x776a1002 connecting to ZooKeeper ensemble=10.58.126.245:2181
2015-06-15 12:48:53,766 [pool-7-thread-1] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=10.58.126.245:2181 sessionTimeout=90000 watcher=hconnection-0x776a10020x0, quorum=10.58.126.245:2181, baseZNode=/hbase
2015-06-15 12:48:58,287 [pool-7-thread-1-SendThread(10.58.126.245:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.58.126.245/10.58.126.245:2181. Will not attempt to authenticate using SASL (unknown error)
2015-06-15 12:48:58,301 [pool-7-thread-1-SendThread(10.58.126.245:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to 10.58.126.245/10.58.126.245:2181, initiating session
2015-06-15 12:48:58,314 [pool-7-thread-1-SendThread(10.58.126.245:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server 10.58.126.245/10.58.126.245:2181, sessionid = 0x14df5b87b120040, negotiated timeout = 90000
2015-06-15 12:49:58,100 [pool-7-thread-1] INFO org.apache.hadoop.hbase.client.RpcRetryingCaller - Call exception, tries=10, retries=35, started=59774 ms ago, cancelled=false, msg=
2015-06-15 12:50:20,456 [pool-7-thread-1] INFO org.apache.hadoop.hbase.client.RpcRetryingCaller - Call exception, tries=11, retries=35, started=82130 ms ago, cancelled=false, msg=
2015-06-15 12:50:36,114 [AWT-EventQueue-1] ERROR net.sourceforge.squirrel_sql.client.gui.db.ConnectToAliasCallBack - Unexpected Error occurred attempting to open an SQL connection.
java.util.concurrent.TimeoutException
at java.util.concurrent.FutureTask.get(FutureTask.java:201)
at net.sourceforge.squirrel_sql.client.mainframe.action.OpenConnectionCommand.awaitConnection(OpenConnectionCommand.java:132)
at net.sourceforge.squirrel_sql.client.mainframe.action.OpenConnectionCommand.access$100(OpenConnectionCommand.java:45)
at net.sourceforge.squirrel_sql.client.mainframe.action.OpenConnectionCommand$2.run(OpenConnectionCommand.java:115)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

I have the same problem, don't find the solution yet, but I managed to use the "thin" client instead.
Start queryserver https://phoenix.apache.org/server.html should listen on port 8765
Copy JAR phoenix-4.6.0-HBase-1.1-thin-client to Squirel lib folder
Create new driver, the class name is "org.apache.phoenix.queryserver.client.Driver"
Connect with this driver (my URI: jdbc:phoenix:thin:url=http://docker:8765)

Spring JMS: each DefaultMessageListenerContainer deployment takes longer than one before

We're using Active MQ 5.6.0, and have configured a number of JMS message listeners (17 currently), each using a Spring 3.1.1 DefaultMessageListenerContainer, deploying to a Jetty 7.
First of all, message routing and handling works and performs fine. The problem is the time it takes to deploy the Spring configuration: each deployment takes longer than the one before it, so we're talking something like polynomial rather than linear time. That's going to be crippling if we ever get to 30 or so.
The example (trimmed) log file shows the 17 deployments appearing to fight one another for connection, until - after over 2.5 minutes - all are connected:
22:33:21,147 - (Def...r) - JMS message listener invoker needs to establish shared Connection
22:33:22,148 - (Def...r) - JMS message listener invoker needs to establish shared Connection
22:33:23,160 - (Def...r) - JMS message listener invoker needs to establish shared Connection
22:33:23,166 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerA' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:33:24,160 - (Def...r) - JMS message listener invoker needs to establish shared Connection
22:33:24,172 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerB' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:33:25,173 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerC' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:33:26,184 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerD' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:33:27,166 - (Def...r) - JMS message listener invoker needs to establish shared Connection
[*snip*]
22:35:25,135 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerI' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:26,146 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerH' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:27,167 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerD' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:28,178 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerE' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:29,179 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerF' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:31,191 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerG' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:32,192 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerH' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:33,203 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerI' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:34,214 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerJ' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:35,225 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerK' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:36,229 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerL' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:37,230 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerM' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:38,242 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerN' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:40,266 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerC' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:41,277 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerB' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:42,280 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerA' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:44,292 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerN' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:45,293 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerM' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:46,294 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerL' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:47,300 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerK' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:48,282 - (Def...r) - JMS message listener invoker needs to establish shared Connection
22:35:48,311 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerJ' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:49,322 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerI' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:50,322 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerH' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:51,323 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerG' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:53,345 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerF' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:54,366 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerE' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:55,367 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerD' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:56,378 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerH' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:57,389 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerI' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:58,394 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerJ' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:35:59,400 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerK' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:00,396 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerL' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:01,409 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerM' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:02,412 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerN' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:03,421 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerO' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:04,422 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerA' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:05,426 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerB' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:06,437 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerC' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:09,471 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerO' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:10,472 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerN' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:11,483 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerM' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:12,483 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerL' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:13,459 - (Def...r) - JMS message listener invoker needs to establish shared Connection
22:36:13,494 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerK' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:14,515 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerJ' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:15,516 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerI' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:16,518 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerH' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:17,530 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerD' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:18,541 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerE' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:19,542 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerF' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:21,546 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerG' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:22,557 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerH' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:23,569 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerI' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:24,580 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerJ' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:25,581 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerK' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:26,582 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerL' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:27,593 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerM' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:28,594 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerN' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:29,605 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerO' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:30,616 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerP' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:32,638 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerC' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:33,639 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerB' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:34,650 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerA' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:36,674 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerP' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:37,685 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerO' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:38,805 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerN' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:39,806 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerM' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:40,653 - (Def...r) - JMS message listener invoker needs to establish shared Connection
22:36:40,807 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerL' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:41,824 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerK' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:42,825 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerJ' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:43,837 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerI' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:44,838 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerH' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:45,851 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerG' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:47,863 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerF' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:48,874 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerE' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:49,875 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerD' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:50,876 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerH' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:51,891 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerI' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:52,902 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerJ' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:53,913 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerK' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:54,924 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerL' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:55,925 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerM' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:56,926 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerN' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:57,937 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerO' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:58,935 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerP' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:36:59,936 - (Def...r) - Could not refresh JMS Connection for destination 'queue://HandlerQ' - retrying in 5000 ms. Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
22:37:00,520 - (Def...r) - Successfully refreshed JMS Connection
22:37:00,520 - (Def...r) - Successfully refreshed JMS Connection
22:37:00,520 - (Def...r) - Successfully refreshed JMS Connection
22:37:00,520 - (Def...r) - Successfully refreshed JMS Connection
22:37:00,520 - (Def...r) - Successfully refreshed JMS Connection
22:37:00,520 - (Def...r) - Successfully refreshed JMS Connection
22:37:00,520 - (Def...r) - Successfully refreshed JMS Connection
22:37:00,520 - (Def...r) - Successfully refreshed JMS Connection
22:37:00,520 - (Def...r) - Successfully refreshed JMS Connection
22:37:00,520 - (Def...r) - Successfully refreshed JMS Connection
22:37:00,520 - (Def...r) - Successfully refreshed JMS Connection
22:37:00,520 - (Def...r) - Successfully refreshed JMS Connection
For the sake of brevity, I won't post each Spring bean, but here are the important bits:
<jee:jndi-lookup id="srcConnectionFactory" jndi-name="${jndi.jms.connfactory}">
<jee:environment>
java.naming.factory.initial = ${jndi.jms.naming.factory.initial}
java.naming.provider.url = ${jndi.jms.naming.url}
</jee:environment>
</jee:jndi-lookup>
<bean id="connectionFactory" class="org.springframework.jms.connection.CachingConnectionFactory">
<property name="targetConnectionFactory" ref="srcConnectionFactory"/>
<property name="cacheConsumers" value="false"/>
<property name="reconnectOnException" value="true"/>
<property name="sessionCacheSize" value="20"></property>
</bean>
<jee:jndi-lookup id="handlerA_Dest" jndi-name="handlerA">
<jee:environment>
java.naming.factory.initial = ${jndi.jms.naming.factory.initial}
java.naming.provider.url = ${jndi.jms.naming.url}
queue.handlerA = handlerA
</jee:environment>
</jee:jndi-lookup>
[x 17]
<bean id="handlerA_Container" class="org.springframework.jms.listener.DefaultMessageListenerContainer"
p:connectionFactory-ref="connectionFactory"
p:messageListener-ref="handlerA"
p:destination-ref="handlerA_Dest">
<property name="concurrentConsumers"><value>10</value></property>
<property name="maxConcurrentConsumers"><value>10</value></property>
<property name="idleConsumerLimit"><value>5</value></property>
</bean>
[x 17]
Any idea why these containers should be contending at all?
Unfortunately, the MessageSelector approach is not (any longer) an option for us. It's one Listener per type-of-Message.
Update It's true that, strictly speaking, the MQ broker is indeed down at the start of this deployment, but it's deployed into the same Jetty (this all happens in the Maven integration test phase), so even if there was a brief period where the broker started up second, I wouldn't expect it to be 2-3 minutes late.
What I'm seeing looks really looks like the broker deployment is being 'blocked' by this Spring deployment...

This is nothing to do with Spring or contention between clients.
The broker wasn't available from 22:33:23,166 until 22:37:00,520.
Cause: Could not connect to broker URL: tcp://0.0.0.0:61616. Reason: java.net.ConnectException: Connection refused: connect
Looks to me like they all connected within 1 millisecond as soon as the broker was available.

Resources