Http Read timeouts on query calls to Elastic Search cluster - elasticsearch

I'm troubleshooting an issue with our application hosted in K8s pod, which tries to query ES cluster through ES Jest Client. Most of the queries were successful, but randomly a call fails with read timeout. Any help will be appreciated.
Jest Client version:6.3.1
ES cluster is hosted in AWS
Below i'm providing apache http debug logs.
o.a.h.c.protocol.RequestAuthCache - Auth cache not set in the context
o.a.h.i.c.BasicHttpClientConnectionManager - Get connection for route
{}->http://dev-app-elasticsearch-lb.xyz.com:9200
o.a.h.impl.execchain.MainClientExec - Executing request POST
/_search?search_type=dfs_query_then_fetch&preference=_primary_first
HTTP/1.1 o.a.h.impl.execchain.MainClientExec - Target auth state:
UNCHALLENGED o.a.h.impl.execchain.MainClientExec - Proxy auth state:
UNCHALLENGED org.apache.http.headers - http-outgoing-16 >> POST
/_search?search_type=dfs_query_then_fetch&preference=_primary_first
HTTP/1.1 org.apache.http.headers - http-outgoing-16 >>
Content-Length: 2531 org.apache.http.headers - http-outgoing-16 >>
Content-Type: application/json; charset=UTF-8
org.apache.http.headers - http-outgoing-16 >> Host:
dev-app-elasticsearch-lb.xyz.com:9200 org.apache.http.headers -
http-outgoing-16 >> Connection: Keep-Alive org.apache.http.headers -
http-outgoing-16 >> User-Agent: Apache-HttpClient/4.5 (Java/1.8.0_181)
org.apache.http.headers - http-outgoing-16 >> Accept-Encoding:
gzip,deflate org.apache.http.wire - http-outgoing-16 >> "POST
/_search?search_type=dfs_query_then_fetch&preference=_primary_first
HTTP/1.1[\r][\n]" org.apache.http.wire - http-outgoing-16 >>
"Content-Length: 2531[\r][\n]" org.apache.http.wire -
http-outgoing-16 >> "Content-Type: application/json;
charset=UTF-8[\r][\n]" org.apache.http.wire - http-outgoing-16 >>
"Host: dev-app-elasticsearch-lb.xyz.com:9200[\r][\n]"
org.apache.http.wire - http-outgoing-16 >> "Connection:
Keep-Alive[\r][\n]" org.apache.http.wire - http-outgoing-16 >>
"User-Agent: Apache-HttpClient/4.5 (Java/1.8.0_181)[\r][\n]"
org.apache.http.wire - http-outgoing-16 >> "Accept-Encoding:
gzip,deflate[\r][\n]" org.apache.http.wire - http-outgoing-16 >>
"[\r][\n]" org.apache.http.wire - http-outgoing-16 >> "{[\n]"
....<<outgoing query details>>
o.a.h.i.c.PoolingHttpClientConnectionManager - Closing connections
idle longer than 60000 MILLISECONDS
o.a.h.i.c.PoolingHttpClientConnectionManager - Closing connections
idle longer than 60000 MILLISECONDS
o.a.h.i.c.PoolingHttpClientConnectionManager - Closing connections
idle longer than 60000 MILLISECONDS
o.a.h.i.c.PoolingHttpClientConnectionManager - Closing connections
idle longer than 60000 MILLISECONDS
c.t.c.util.concurrent.DatabaseLock - Cleaning up expired locks from DS
0 o.a.h.i.c.BasicHttpClientConnectionManager - Closing connection
o.a.h.i.c.DefaultManagedHttpClientConnection - http-outgoing-6: Close
connection o.a.h.i.n.c.PoolingNHttpClientConnectionManager - Closing
connections idle longer than 15 MINUTES
o.a.h.i.n.c.PoolingNHttpClientConnectionManager - Closing connections
idle longer than 15 MINUTES
o.a.h.i.n.c.PoolingNHttpClientConnectionManager - Closing connections
idle longer than 15 MINUTES```
Any help would be greatly appreciated.
Thanks
Naresh

Related

Spring cloud AWS SQS Listener constant sending request

I have Implement annotation-driven-queue-listener using spring cloud AWS API.
<aws-messaging:annotation-driven-queue-listener max-number-of-messages="10" task-executor="messagingTaskExecutor" wait-time-out="20"
visibility-timeout="100" amazon-sqs="sqsClient" region="us-west-1" />
In a Below log file I can see request constantly send to us-east-1 region Though I have provided region us-west-1. Request send Interval is 3 request per second which is too much
amz-sdk-invocation-id:4865f324-322f-53f2-9f4d-f989e106e96c
amz-sdk-retry:0/0/500
**host:sqs.us-east-1.amazonaws.com**
user-agent:aws-sdk-java/1.11.18 Mac_OS_X/10.11.6 Java_HotSpot(TM)_64-Bit_Server_VM/25.111-b14/1.8.0_111 AmazonSQSBufferedAsyncClient/1.11.18
x-amz-date:20170210T030639Z
amz-sdk-invocation-id;amz-sdk-retry;host;user-agent;x-amz-date
937d63370e14d7439a711514a14d0d371c3f6fc73208e1a9d54bd99f194e42a7"
2017-02-10 08:36:39 DEBUG AWS4Signer:33 - AWS4 String to Sign: '"AWS4-HMAC-SHA256
20170210T030639Z
20170210/us-east-1/sqs/aws4_request
07187f106ed69b4b3ad36be48ba53811adc903a39388c5b0a1677ad2af93f38b"
2017-02-10 08:36:39 DEBUG RequestAddCookies:122 - CookieSpec selected: default
2017-02-10 08:36:39 DEBUG RequestAuthCache:76 - Auth cache not set in the context
2017-02-10 08:36:39 DEBUG PoolingHttpClientConnectionManager:249 - Connection request: [route: {s}->https://sqs.us-east-1.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]
2017-02-10 08:36:39 DEBUG PoolingHttpClientConnectionManager:282 - Connection leased: [id: 23][route: {s}->https://sqs.us-east-1.amazonaws.com:443][total kept alive: 0; route allocated: 1 of 50; total allocated: 1 of 50]
2017-02-10 08:36:39 DEBUG MainClientExec:207 - Stale connection check
2017-02-10 08:36:39 DEBUG wire:86 - http-outgoing-23 << "[read] I/O error: Read timed out"
2017-02-10 08:36:39 DEBUG DefaultManagedHttpClientConnection:90 - http-outgoing-23: set socket timeout to 50000
2017-02-10 08:36:39 DEBUG MainClientExec:255 - Executing request POST /189178718836/CMR HTTP/1.1
2017-02-10 08:36:39 DEBUG MainClientExec:266 - Proxy auth state: UNCHALLENGED
2017-02-10 08:36:39 DEBUG headers:135 - http-outgoing-23 >> POST /189178718836/CMR HTTP/1.1
2017-02-10 08:36:39 DEBUG headers:138 - http-outgoing-23 >> Host: sqs.us-east-1.amazonaws.com
2017-02-10 08:36:39 DEBUG headers:138 - http-outgoing-23 >> Authorization: AWS4-HMAC-SHA256 Credential=something/20170210/us-east-1/sqs/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;host;user-agent;x-amz-date, Signature=0d110d904500fd10a5262c5beea63df72eff28627801d0c00f74d499d4750092
2017-02-10 08:36:39 DEBUG headers:138 - http-outgoing-23 >> X-Amz-Date: 20170210T030639Z
2017-02-10 08:36:39 DEBUG headers:138 - http-outgoing-23 >> User-Agent: aws-sdk-java/1.11.18 Mac_OS_X/10.11.6 Java_HotSpot(TM)_64-Bit_Server_VM/25.111-b14/1.8.0_111 AmazonSQSBufferedAsyncClient/1.11.18
2017-02-10 08:36:39 DEBUG headers:138 - http-outgoing-23 >> amz-sdk-invocation-id: 4865f324-322f-53f2-9f4d-f989e106e96c
2017-02-10 08:36:39 DEBUG headers:138 - http-outgoing-23 >> amz-sdk-retry: 0/0/500
2017-02-10 08:36:39 DEBUG headers:138 - http-outgoing-23 >> Content-Type: application/x-www-form-urlencoded; charset=utf-8
2017-02-10 08:36:39 DEBUG headers:138 - http-outgoing-23 >> Content-Length: 110
2017-02-10 08:36:39 DEBUG headers:138 - http-outgoing-23 >> Connection: Keep-Alive
2017-02-10 08:36:39 DEBUG wire:72 - http-outgoing-23 >> "POST /189178718836/CMR HTTP/1.1[\r][\n]"
2017-02-10 08:36:39 DEBUG wire:72 - http-outgoing-23 >> "Host: sqs.us-east-1.amazonaws.com[\r][\n]"
2017-02-10 08:36:39 DEBUG wire:72 - http-outgoing-23 >> "Authorization: AWS4-HMAC-SHA256 Credential=AKIAJRLXVVM5WJ5X7CFA/20170210/us-east-1/sqs/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;host;user-agent;x-amz-date, Signature=0d110d904500fd10a5262c5beea63df72eff28627801d0c00f74d499d4750092[\r][\n]"
2017-02-10 08:36:39 DEBUG wire:72 - http-outgoing-23 >> "X-Amz-Date: 20170210T030639Z[\r][\n]"
2017-02-10 08:36:39 DEBUG wire:72 - http-outgoing-23 >> "User-Agent: aws-sdk-java/1.11.18 Mac_OS_X/10.11.6 Java_HotSpot(TM)_64-Bit_Server_VM/25.111-b14/1.8.0_111 AmazonSQSBufferedAsyncClient/1.11.18[\r][\n]"
2017-02-10 08:36:39 DEBUG wire:72 - http-outgoing-23 >> "amz-sdk-invocation-id: 4865f324-322f-53f2-9f4d-f989e106e96c[\r][\n]"
2017-02-10 08:36:39 DEBUG wire:72 - http-outgoing-23 >> "amz-sdk-retry: 0/0/500[\r][\n]"
2017-02-10 08:36:39 DEBUG wire:72 - http-outgoing-23 >> "Content-Type: application/x-www-form-urlencoded; charset=utf-8[\r][\n]"
2017-02-10 08:36:39 DEBUG wire:72 - http-outgoing-23 >> "Content-Length: 110[\r][\n]"
2017-02-10 08:36:39 DEBUG wire:72 - http-outgoing-23 >> "Connection: Keep-Alive[\r][\n]"
2017-02-10 08:36:39 DEBUG wire:72 - http-outgoing-23 >> "[\r][\n]"
2017-02-10 08:36:39 DEBUG wire:86 - http-outgoing-23 >> "Action=ReceiveMessage&Version=2012-11-05&AttributeName.1=All&MessageAttributeName.1=All&MaxNumberOfMessages=10"
can anybody tell me why spring cloud aws api itself constantly sending this request to us-east-1 region?

Consolidate multiple similar lines in log with bash

Sometimes I would get errors such as the lines below in web server error log:
2016/09/19 04:32:49 [error] 18703#18703: *1148440 limiting requests, excess: 10.897 by zone "flood", client: 10.20.30.40, server: domain.com, request: "HEAD / HTTP/1.1", host: "domain.com"
2016/09/19 04:32:49 [error] 18703#18703: *1148443 limiting requests, excess: 10.891 by zone "flood", client: 10.20.30.40, server: domain.com, request: "HEAD / HTTP/1.1", host: "domain.com"
2016/09/19 04:32:49 [error] 18703#18703: *1148441 limiting requests, excess: 10.888 by zone "flood", client: 10.20.30.40, server: domain.com, request: "HEAD / HTTP/1.1", host: "domain.com"
2016/09/19 04:32:49 [error] 18703#18703: *1148444 limiting requests, excess: 10.885 by zone "flood", client: 10.20.30.40, server: domain.com, request: "HEAD / HTTP/1.1", host: "domain.com"
2016/09/19 04:32:49 [error] 18703#18703: *1148445 limiting requests, excess: 10.873 by zone "flood", client: 10.20.30.40, server: domain.com, request: "HEAD / HTTP/1.1", host: "domain.com"
2016/09/19 04:32:49 [error] 18704#18704: *1148446 limiting requests, excess: 10.845 by zone "flood", client: 10.20.30.40, server: domain.com, request: "HEAD / HTTP/1.1", host: "domain.com"
2016/09/19 04:32:49 [error] 18703#18703: *1148447 limiting requests, excess: 10.844 by zone "flood", client: 10.20.30.40, server: domain.com, request: "HEAD / HTTP/1.1", host: "domain.com"
2016/09/19 04:32:49 [error] 18703#18703: *1148448 limiting requests, excess: 10.839 by zone "flood", client: 10.20.30.40, server: domain.com, request: "HEAD / HTTP/1.1", host: "domain.com"
2016/09/19 04:32:49 [error] 18703#18703: *1148449 limiting requests, excess: 10.836 by zone "flood", client: 10.20.30.40, server: domain.com, request: "HEAD / HTTP/1.1", host: "domain.com"
I've a bash script that parses log files each day, but it just dumps all the lines into the report.
function error_log_errors {
ERRORLOG_ERRORS=$(cat "${ERRORLOG}" | grep "${ERRORLOG_DATE}" | awk '{print substr($0, index($0,$6))}')
}
How can I consolidate multiple of such similar lines into:
(20) 04:32 Limited requests (zone: flood) from IP address: 10.20.30.40, request: "HEAD / HTTP/1.1", host: domain.com
The columns are:
Number of times of occurrence
Time it first started
Event (corresponding entry in log: limiting requests, excess: 10.897 by zone "flood")
IP address
Request (corresponding entry in log: request: "HEAD / HTTP/1.1")
Host name
Thanks for any advice.
Note: Please note that the log file also contains other unrelated lines as well. Thanks.

Using PhpStorm 2016.1. Error while connecting via FTPS - Could not list the contents of folder

I'm on a Mac and using the latest version of PhpStorm 2016.1. I had version 6.0.4 and was having problems connecting via FTPS (I'm using credentials which are working on FileZilla). I looked here and other places online and it said that enabling Passive mode and 'Always use LIST command' should work. But this version did not have the latter. So I decided to upgrade and now I've got the latest version.
I tried connecting again, and this time the option did indeed come up but unfortunately I'm still getting the same error!
Just to be sure, the configuration I have in FileZilla is:
Host: ****
Port: 21
Protocol: FTP - File Transfer Protocol
Encryption: Require explicit FTP over TLS
Logon Type: Normal
User: ****
Password: ****
Here are the log entries -
2016-08-31 21:38:11,396 [ 431750] DEBUG - t.connections.RemoteConnection - Opening connection to ftps://www.myfinder.com.au:21/ in #30e0690c
2016-08-31 21:38:11,475 [ 431829] DEBUG - ains.plugins.webDeployment.ftp - 220 ATP Login
2016-08-31 21:38:11,475 [ 431829] DEBUG - ains.plugins.webDeployment.ftp - AUTH TLS
2016-08-31 21:38:11,643 [ 431997] DEBUG - ains.plugins.webDeployment.ftp - 234 Using authentication type TLS
2016-08-31 21:38:11,843 [ 432197] DEBUG - ains.plugins.webDeployment.ftp - USER datar
2016-08-31 21:38:11,870 [ 432224] DEBUG - ains.plugins.webDeployment.ftp - 331 Password required for datar
2016-08-31 21:38:11,871 [ 432225] DEBUG - ains.plugins.webDeployment.ftp - PASS ***
2016-08-31 21:38:11,960 [ 432314] DEBUG - ains.plugins.webDeployment.ftp - 230 Logged on
2016-08-31 21:38:11,960 [ 432314] DEBUG - ains.plugins.webDeployment.ftp - TYPE I
2016-08-31 21:38:11,976 [ 432330] DEBUG - ains.plugins.webDeployment.ftp - 200 Type set to I
2016-08-31 21:38:11,977 [ 432331] DEBUG - ains.plugins.webDeployment.ftp - PBSZ 0
2016-08-31 21:38:12,070 [ 432424] DEBUG - ains.plugins.webDeployment.ftp - 200 PBSZ=0
2016-08-31 21:38:12,070 [ 432424] DEBUG - ains.plugins.webDeployment.ftp - PROT P
2016-08-31 21:38:12,285 [ 432639] DEBUG - ains.plugins.webDeployment.ftp - 200 Protection level set to P
2016-08-31 21:38:12,286 [ 432640] DEBUG - ains.plugins.webDeployment.ftp - SYST
2016-08-31 21:38:12,409 [ 432763] DEBUG - ains.plugins.webDeployment.ftp - 215 UNIX emulated by FileZilla
2016-08-31 21:38:12,410 [ 432764] DEBUG - ains.plugins.webDeployment.ftp - PASV
2016-08-31 21:38:12,480 [ 432834] DEBUG - ains.plugins.webDeployment.ftp - 227 Entering Passive Mode (168,1,11,106,199,178)
2016-08-31 21:38:12,509 [ 432863] DEBUG - ains.plugins.webDeployment.ftp - LIST
2016-08-31 21:38:12,583 [ 432937] DEBUG - ains.plugins.webDeployment.ftp - 150 Opening data channel for directory listing of "/"
2016-08-31 21:38:12,933 [ 433287] DEBUG - ains.plugins.webDeployment.ftp - QUIT
2016-08-31 21:38:13,021 [ 433375] DEBUG - ains.plugins.webDeployment.ftp - 450 TLS session of data connection has not resumed or the session does not match the control connection

OSGI Disable org.apache.http.wire log

I am using osgi httpcore;version=4.4.1 . I need to stop the debug logging of "org.apache.http.wire"
Is there any runtime config to disable this logging?
I have tried log4j.logger.org.apache.http.wire=ERROR in run time with no effect.
[Thread-522] DEBUG org.apache.http.wire - << " <volumeSize>6</volumeSize>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <deleteOnTermination>true</deleteOnTermination>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <volumeType>standard</volumeType>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <encrypted>false</encrypted>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " </ebs>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " </item>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " </blockDeviceMapping>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <virtualizationType>paravirtual</virtualizationType>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <hypervisor>xen</hypervisor>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " </item>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <item>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <imageId>ami-30697651</imageId>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <imageLocation>057448758665/CentOS 7.2 x86_64 with cloud-init (PV)</imageLocation>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <imageState>available</imageState>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <imageOwnerId>057448758665</imageOwnerId>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <creationDate>2015-12-15T16:22:38.000Z</creationDate>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <isPublic>true</isPublic>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <architecture>x86_64</architecture>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <imageType>machine</imageType>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <kernelId>aki-fc8f11cc</kernelId>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <name>CentOS 7.2 x86_64 with cloud-init (PV)</name>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <description>Base CentOS 7.2.1511 with cloud-init ( use user ec2-user ), paravirtual and EBS backed. This image has been created with ComodIT. See how to build yours at http://www.comodit.com/2012/07/12/how-to-build-your-custom-centos-image-on-ec2</description>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <rootDeviceType>ebs</rootDeviceType>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <rootDeviceName>/dev/sda</rootDeviceName>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <blockDeviceMapping>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <item>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <deviceName>/dev/sda</deviceName>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <ebs>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <snapshotId>snap-955a440c</snapshotId>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <volumeSize>6</volumeSize>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <deleteOnTermination>true</deleteOnTermination>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <volumeType>gp2</volumeType>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <encrypted>false</encrypted>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " </ebs>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " </item>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " </blockDeviceMapping>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <virtualizationType>paravirtual</virtualizationType>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <hypervisor>xen</hypervisor>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " </item>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <item>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <imageId>ami-3181cf01</imageId>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <imageLocation>057448758665/comodit-bootstrap-hvm</imageLocation>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <imageState>available</imageState>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <imageOwnerId>057448758665</imageOwnerId>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <creationDate>2014-10-24T11:01:30.000Z</creationDate>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <isPublic>true</isPublic>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <architecture>x86_64</architecture>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <imageType>machine</imageType>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <name>comodit-bootstrap-hvm</name>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <description>comodit-bootstrap-hvm</description>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <rootDeviceType>ebs</rootDeviceType>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <rootDeviceName>/dev/sda1</rootDeviceName>[\n]"
[Thread-522] DEBUG org.apache.http.wire - << " <blockDeviceMapping>[\n]"

Parsing of XML fails under Grizzly using JAX-RS on a chunked POST

I have encountered a problem when using Apache HttpClient 4.3.6 to POST XML to Grizzly under Jersey 2.15. I switched to use the Apache HttpClient last night so I could introduce support for chunked POST operations and support for 100-Continue (for reasons I will not get into). The message flow seems to be correct, but when it comes to JAX-RS annotated servlet to be invoked I get the "Content is not allowed in prolog." error on entry. Here is the method signature:
#POST
#Path("/callback")
#Produces({ MediaType.APPLICATION_XML, NsiConstants.NSI_DDS_V1_XML })
#Consumes({ MediaType.APPLICATION_XML, NsiConstants.NSI_DDS_V1_XML })
public Response notification(NotificationListType notify) {
…
}
Before introducing the Apache HttpClient with the chunked POST and 100-continue this worked fine. I changed the input parameter to be of type String and discovered the start of the data stream contains "800\r\n" followed by the complete XML. The "800\r\n" happens to be the first chunk length. There are multiple chunks but only the first chunk length is in the stream. If I manually remove this I can parse the XML without issue, but would rather not have to do this. Can anyone provide guidance?
Here is a debug trace of the initial interaction:
2015-01-27 13:00:51,879 DEBUG (org.apache.http.impl.conn.LoggingManagedHttpClientConnection:124) - http-outgoing-3 >> POST /dds/callback HTTP/1.1
2015-01-27 13:00:51,879 DEBUG (org.apache.http.impl.conn.LoggingManagedHttpClientConnection:127) - http-outgoing-3 >> Accept: application/xml
2015-01-27 13:00:51,879 DEBUG (org.apache.http.impl.conn.LoggingManagedHttpClientConnection:127) - http-outgoing-3 >> Content-Type: application/xml
2015-01-27 13:00:51,879 DEBUG (org.apache.http.impl.conn.LoggingManagedHttpClientConnection:127) - http-outgoing-3 >> User-Agent: Jersey/2.15 (Apache HttpClient 4.3.6)
2015-01-27 13:00:51,880 DEBUG (org.apache.http.impl.conn.LoggingManagedHttpClientConnection:127) - http-outgoing-3 >> Transfer-Encoding: chunked
2015-01-27 13:00:51,880 DEBUG (org.apache.http.impl.conn.LoggingManagedHttpClientConnection:127) - http-outgoing-3 >> Host: localhost:8402
2015-01-27 13:00:51,880 DEBUG (org.apache.http.impl.conn.LoggingManagedHttpClientConnection:127) - http-outgoing-3 >> Connection: Keep-Alive
2015-01-27 13:00:51,880 DEBUG (org.apache.http.impl.conn.LoggingManagedHttpClientConnection:127) - http-outgoing-3 >> Expect: 100-continue
2015-01-27 13:00:51,881 DEBUG (org.apache.http.impl.conn.LoggingManagedHttpClientConnection:127) - http-outgoing-3 >> Accept-Encoding: gzip,deflate
2015-01-27 13:00:51,881 DEBUG (org.apache.http.impl.conn.Wire:72) - http-outgoing-3 >> "POST /dds/callback HTTP/1.1[\r][\n]"
2015-01-27 13:00:51,881 DEBUG (org.apache.http.impl.conn.Wire:72) - http-outgoing-3 >> "Accept: application/xml[\r][\n]"
2015-01-27 13:00:51,881 DEBUG (org.apache.http.impl.conn.Wire:72) - http-outgoing-3 >> "Content-Type: application/xml[\r][\n]"
2015-01-27 13:00:51,882 DEBUG (org.apache.http.impl.conn.Wire:72) - http-outgoing-3 >> "User-Agent: Jersey/2.15 (Apache HttpClient 4.3.6)[\r][\n]"
2015-01-27 13:00:51,882 DEBUG (org.apache.http.impl.conn.Wire:72) - http-outgoing-3 >> "Transfer-Encoding: chunked[\r][\n]"
2015-01-27 13:00:51,882 DEBUG (org.apache.http.impl.conn.Wire:72) - http-outgoing-3 >> "Host: localhost:8402[\r][\n]"
2015-01-27 13:00:51,882 DEBUG (org.apache.http.impl.conn.Wire:72) - http-outgoing-3 >> "Connection: Keep-Alive[\r][\n]"
2015-01-27 13:00:51,883 DEBUG (org.apache.http.impl.conn.Wire:72) - http-outgoing-3 >> "Expect: 100-continue[\r][\n]"
2015-01-27 13:00:51,883 DEBUG (org.apache.http.impl.conn.Wire:72) - http-outgoing-3 >> "Accept-Encoding: gzip,deflate[\r][\n]"
2015-01-27 13:00:51,883 DEBUG (org.apache.http.impl.conn.Wire:72) - http-outgoing-3 >> "[\r][\n]"
2015-01-27 13:00:51,884 DEBUG (org.apache.http.impl.conn.Wire:72) - http-outgoing-3 << "HTTP/1.1 100 Continue[\r][\n]"
2015-01-27 13:00:51,885 DEBUG (org.apache.http.impl.conn.Wire:72) - http-outgoing-3 << "[\r][\n]"
2015-01-27 13:00:51,886 DEBUG (org.apache.http.impl.conn.LoggingManagedHttpClientConnection:113) - http-outgoing-3 << HTTP/1.1 100 Continue
2015-01-27 13:00:51,945 DEBUG (org.apache.http.impl.conn.Wire:72) - http-outgoing-3 >> "800[\r][\n]"
2015-01-27 13:00:51,945 DEBUG (org.apache.http.impl.conn.Wire:72) - http-outgoing-3 >> "<?xml version="1.0" encoding="UTF-8"?>[\n]"
…
2015-01-27 13:00:51,968 DEBUG (org.apache.http.impl.conn.Wire:72) - http-outgoing-3 >> "0[\r][\n]"
2015-01-27 13:00:51,968 DEBUG (org.apache.http.impl.conn.Wire:72) - http-outgoing-3 >> "[\r][\n]"

Resources