haproxy + spring boot writeAddress(..) failed: Connection reset by peer - spring

I'm running HAproxy with backend Spring Boot Rest controllers. My spring log shows constant errors that look like the following:
[reactor-http-epoll-26] ERROR o.s.w.s.a.HttpWebHandlerAdapter - [9df8bfcf] Error [io.netty.channel.unix.Errors$NativeIoException: writeAddress(..) failed: Connection reset by peer] for HTTP GET "/api/v1/status", but ServerHttpResponse already committed (200 OK)
HAproxy performs an HTTP check on the url /api/v1/status. What would be the reason that I'm getting these errors?
HAProxy Config
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
# daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
ssl-default-bind-ciphers EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EX$
ssl-default-bind-options no-sslv3 no-tlsv10
tune.ssl.default-dh-param 4096
defaults
mode http
log global
option httplog
option http-server-close
option forwardfor except 127.0.0.0
option redispatch
retries 3
timeout http-request 30s
timeout queue 1m
timeout connect 30s
timeout client 30s
timeout server 30s
frontend https-in
bind *:443 ssl crt /etc/cert.pem
default_backend api
backend api
mode http
option httpchk GET /api/v1/status HTTP/1.0
http-check expect status 200
balance roundrobin
server api1 127.0.0.1:8001 check fall 3 rise 2
server api2 127.0.0.1:8002 check fall 3 rise 2

HAproxy is doing GET request, reads http response code and closing connection.
Boot is trying to send remaining parts (http headers and some json payload) but connection is already closed.
Just replace GET with OPTIONS in the line:
option httpchk GET /api/v1/status HTTP/1.0

Related

Haproxy 503 service unavailable

We are running Haproxy and 3 nginx processes inside a single docker on the host network to access it from localhost. We have installed HaProxy version 2.6.6. 3 nginx servers are running behind it as backend servers. We are able to access them individually What is wrong with the haproxy configuration here? we are always getting the below error on curl (curl http://localhost:80) requests. Please suggest
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
pidfile /usr/local/etc/haproxy/haproxy.pid
defaults
timeout http-request 2000
timeout queue 1000
timeout check 2000
timeout connect 2000
timeout client 5000
timeout server 5000
log global
option dontlognull
option clitcpka
option srvtcpka
option tcpka
unique-id-format %[uuid()]
unique-id-header X-Request-ID
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %ID %{+Q}r"
frontend http-in
mode http
option forwardfor
bind *:80
use_backend webservers
backend webservers
mode http
server localhost-01 localhost:8000
server localhost-02 localhost:8001
server localhost-03 localhost:8002

When I started HAProxy under Windows Server, I got the following error

haproxy.exe -f haproxy.cfg -d
When I run HAProxy, I get an error:
'''
Available polling systems :
poll : pref=200, test result OK
select : pref=150, test result FAILED
Total: 2 (1 usable), will use poll.
Available filters :
[SPOE] spoe
[CACHE] cache
[FCGI] fcgi-app
[COMP] compression
[TRACE] trace
Using poll() as the polling mechanism.
[NOTICE] (1036) : haproxy version is 2.4.0-6cbbecf
[ALERT] (1036) : Starting proxy warelucent: cannot bind socket (Address already in use) [0.0.0.0:5672]
[ALERT] (1036) : [haproxy.main()] Some protocols failed to start their listeners! Exiting.
'''
In the meantime, no other services are running, and I have the RabbitMQ service open.
My haproxy.cfg file is as follows:
'''
#logging options
global
log 127.0.0.1 local0 info
maxconn 1500
daemon
quiet
nbproc 20
defaults
log global
mode tcp
#if you set mode to tcp,then you nust change tcplog into httplog
option tcplog
option dontlognull
retries 3
option redispatch
maxconn 2000
timeout connect 10s
timeout client 10s
timeout server 10s
#front-end IP for consumers and producters
listen warelucent
bind 0.0.0.0:5672
#配置TCP模式
mode tcp
#balance url_param userid
#balance url_param session_id check_post 64
#balance hdr(User-Agent)
#balance hdr(host)
#balance hdr(Host) use_domain_only
#balance rdp-cookie
#balance leastconn
#balance source //ip
#简单的轮询
balance roundrobin
server one 1.1.1.1:5672 check inter 5000 rise 2 fall 2
server two 2.2.2.2:5672 check inter 5000 rise 2 fall 2
server three 3.3.3.3:5672 check inter 5000 rise 2 fall 2
listen stats
bind 127.0.0.1:8100
mode http
option httplog
stats enable
stats uri /rabbitmq-stats
stats refresh 5s
'''
Most of the Internet is due to the version, but I checked the official website, the version is the latest, and I also started the RabbitMQ service, so I don't know where the error is at present
(Address already in use) [0.0.0.0:5672]
it means that the port 5672 (RabbitMQ is already in use. Most likely you have a rabbitmq node running in that machine.
So just change the HA-PROXY port.

HA Proxy fails to recognize Tomcat:8080 running on a mac

Can someone help be understand why haproxy fails to detect my tomcat instance running on port 8080? I can reach the server just fine when browsing to 127.0.0.1:8080
On start up haproxy spits out the following error:
WARNING] 141/163609 (23336) : Server cim/cim is DOWN, reason: Layer7 wrong status, code: 404, info: "Not Found", check duration: 1ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 141/163609 (23336) : backend 'cim' has no server available!
Here's the haproxy config.
global
maxconn 128
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
tune.ssl.default-dh-param 2048
defaults
mode http
log global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
option httplog
option forwardfor
option httpclose
option redispatch
option httpchk HEAD / HTTP/1.0
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 20s
timeout check 10s
frontend in-http
bind *:80
acl restricted_urls path_beg /trm/ws/v2
use_backend cim if restricted_urls
backend cim
balance roundrobin
server cim 127.0.0.1:8080 check
Tomcat is definitely up and running, and I can telnet to 127.0.0.1 8080 and receive an http response.

No such server found while disabling HAproxy disable server from cluster

My haproxy.cfg
global
log 127.0.0.1 local0
maxconn 20000
user haproxy
group haproxy
stats socket /var/run/haproxy/haproxy.sock level admin
stats timeout 2m
listen admin
bind *:8080
stats enable
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
# timeout http-request 5s
timeout connect 5000
timeout client 60000
timeout server 60000
frontend http-in
bind *:80
default_backend monkey
backend monkey
stats enable
stats uri /haproxy?stats
stats realm Strictly\ Private
stats auth admin:hello
stats auth Another_User:hi
mode http
compression algo gzip
compression type text/html text/plain text/css application/json
balance roundrobin
option httpclose
option forwardfor
default-server inter 1s fall 1
server cd-test-1 1.2.3.4:80 check
server cd-test-2 5.6.7.8:80 check
I have been using socat to disable a node from HAproxy cluster.
below is the command
echo "disable server monkey/cd-test-1"| socat stdio /var/run/haproxy/haproxy.sock
The above disables my node from haproxy. But if I use the ip address(1.2.3.4) instead of "cd-test-1" it returns No such server.
I am using ansible to automated this. I use {{inventory_hostname}} and delegate the command to my HAproxy server. Hence the issue.
- name: Disable {{ inventory_hostname }} in haproxy and letting the services drain
shell: echo "disable server monkey/{{inventory_hostname}}"| socat stdio /var/run/haproxy/haproxy.sock
become_user: root
delegate_to: "{{ item }}"
with_items: groups.haproxy_backend
This returns "No such server." and moves along.
Can someone please help me find the issue with using the IP instead of the name of server. Might be doing something very silly. Any help is appreciated.
Disabling and enabling HAproxy using socat, the server alias name has to be mentioned.
Otherwise, we will be getting a No such server error.

https is not working behind haproxy

I have to put haproxy in front of my already running Apache web-server. Both haproxy and apache web-server are on separate Cent-OS6.4 machines.
I had installed haproxy-1.5-dev19.el6.x86_64 and it is working fine with http, but getting
below error with https:-
"502 Bad Gateway: The server returned an invalid or incomplete response".
haproxy logs are shown below:
Nov 7 05:49:56 localhost haproxy[9925]: XX.XX.XXX.XX:51949
[07/Nov/2013:05:49:55.204] https-in~ abc-https/server1
1595/0/1/-1/1597 502 714 - - PHNN 2/2/0/0/0 0/0 "GET / HTTP/1.1"
Nov 7 05:49:57 localhost haproxy[9925]: XX.XX.XXX.XX:51947
[07/Nov/2013:05:49:55.972] https-in~ abc-https/server1
1523/0/1/-1/1525 502 714 - - PHNN 1/1/0/0/0 0/0 "GET /favicon.ico HTTP/1.1"
SSL logs on webserver (request behind proxy):
10.0.0.218 - - [06/Nov/2013:22:42:34 -0800] **"GET /"** 400 510
10.0.0.218 - - [06/Nov/2013:22:42:34 -0800] "GET /" 400 510
SSL logs on webserver (direct request):
XX.XX.XX.XX - - [06/Nov/2013:22:48:42 -0800] **"GET / HTTP/1.1"** 200 19553
As you can see the difference between proxy and without proxy at webserver.
Below is my haproxy.cfg file:
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 40000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor
option redispatch
retries 10
timeout http-request 60s
timeout queue 60s
timeout connect 60s
timeout client 60s
timeout server 60s
timeout http-keep-alive 60s
timeout check 60s
maxconn 30000
Listen stats 0.0.0.:8880
stats enable
stats hide-version
stats uri /
Stats realm HAProxy\ Statistics
stats auth XXXXX:XXXXX
frontend http-in
bind *:80
acl url_static path_beg -i /static /images /javascript /stylesheets
acl url_static path_end -i .jpg .gif .png .css .js
tcp-request connection accept if { src -f /etc/haproxy/whitelist.lst }
tcp-request connection reject if { src_conn_cur ge 200 } tcp-request
connection track-sc1 src
use_backend http-in-static if url_static
default_backend http-in-bk
frontend https-in
bind *:443 ssl crt /home/ec2-user/ev/haproxy.pem
http-request add-header X-Proto https if { ssl_fc }
use_backend abc-https if {ssl_fc}
backend abc-https
server server1 10.0.0.16:443 check
backend http-in-static
server static 10.0.0.16:80 check inter 100 weight 1
backend http-in-bk
acl abuse src_http_err_rate(http-in) ge 100
acl flag_abuser src_inc_gpc0(http-in)
tcp-request content reject if abuse flag_abuser
server server1 10.0.0.16:80 check inter 100 weight 1
There is only one webserver which is already running and I have to implement haproxy in front of that.
Where I am doing wrong? Kindly help me to resolve this issue.
Regards,
Komal Pal
You are decrypting the SSL traffic and then sending the plaintext HTTP to an HTTPS socket on your webserver.
In this setup you would normally send it to port 80 on the webserver, because you have already decrypted it.
If you want to re-encrypt you must change your "server xxx" line to have the flag "ssl" on it as well.

Resources