https is not working behind haproxy - https

I have to put haproxy in front of my already running Apache web-server. Both haproxy and apache web-server are on separate Cent-OS6.4 machines.
I had installed haproxy-1.5-dev19.el6.x86_64 and it is working fine with http, but getting
below error with https:-
"502 Bad Gateway: The server returned an invalid or incomplete response".
haproxy logs are shown below:
Nov 7 05:49:56 localhost haproxy[9925]: XX.XX.XXX.XX:51949
[07/Nov/2013:05:49:55.204] https-in~ abc-https/server1
1595/0/1/-1/1597 502 714 - - PHNN 2/2/0/0/0 0/0 "GET / HTTP/1.1"
Nov 7 05:49:57 localhost haproxy[9925]: XX.XX.XXX.XX:51947
[07/Nov/2013:05:49:55.972] https-in~ abc-https/server1
1523/0/1/-1/1525 502 714 - - PHNN 1/1/0/0/0 0/0 "GET /favicon.ico HTTP/1.1"
SSL logs on webserver (request behind proxy):
10.0.0.218 - - [06/Nov/2013:22:42:34 -0800] **"GET /"** 400 510
10.0.0.218 - - [06/Nov/2013:22:42:34 -0800] "GET /" 400 510
SSL logs on webserver (direct request):
XX.XX.XX.XX - - [06/Nov/2013:22:48:42 -0800] **"GET / HTTP/1.1"** 200 19553
As you can see the difference between proxy and without proxy at webserver.
Below is my haproxy.cfg file:
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 40000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor
option redispatch
retries 10
timeout http-request 60s
timeout queue 60s
timeout connect 60s
timeout client 60s
timeout server 60s
timeout http-keep-alive 60s
timeout check 60s
maxconn 30000
Listen stats 0.0.0.:8880
stats enable
stats hide-version
stats uri /
Stats realm HAProxy\ Statistics
stats auth XXXXX:XXXXX
frontend http-in
bind *:80
acl url_static path_beg -i /static /images /javascript /stylesheets
acl url_static path_end -i .jpg .gif .png .css .js
tcp-request connection accept if { src -f /etc/haproxy/whitelist.lst }
tcp-request connection reject if { src_conn_cur ge 200 } tcp-request
connection track-sc1 src
use_backend http-in-static if url_static
default_backend http-in-bk
frontend https-in
bind *:443 ssl crt /home/ec2-user/ev/haproxy.pem
http-request add-header X-Proto https if { ssl_fc }
use_backend abc-https if {ssl_fc}
backend abc-https
server server1 10.0.0.16:443 check
backend http-in-static
server static 10.0.0.16:80 check inter 100 weight 1
backend http-in-bk
acl abuse src_http_err_rate(http-in) ge 100
acl flag_abuser src_inc_gpc0(http-in)
tcp-request content reject if abuse flag_abuser
server server1 10.0.0.16:80 check inter 100 weight 1
There is only one webserver which is already running and I have to implement haproxy in front of that.
Where I am doing wrong? Kindly help me to resolve this issue.
Regards,
Komal Pal

You are decrypting the SSL traffic and then sending the plaintext HTTP to an HTTPS socket on your webserver.
In this setup you would normally send it to port 80 on the webserver, because you have already decrypted it.
If you want to re-encrypt you must change your "server xxx" line to have the flag "ssl" on it as well.

Related

Haproxy 503 service unavailable

We are running Haproxy and 3 nginx processes inside a single docker on the host network to access it from localhost. We have installed HaProxy version 2.6.6. 3 nginx servers are running behind it as backend servers. We are able to access them individually What is wrong with the haproxy configuration here? we are always getting the below error on curl (curl http://localhost:80) requests. Please suggest
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
pidfile /usr/local/etc/haproxy/haproxy.pid
defaults
timeout http-request 2000
timeout queue 1000
timeout check 2000
timeout connect 2000
timeout client 5000
timeout server 5000
log global
option dontlognull
option clitcpka
option srvtcpka
option tcpka
unique-id-format %[uuid()]
unique-id-header X-Request-ID
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %ID %{+Q}r"
frontend http-in
mode http
option forwardfor
bind *:80
use_backend webservers
backend webservers
mode http
server localhost-01 localhost:8000
server localhost-02 localhost:8001
server localhost-03 localhost:8002

HAProxy and websockets never connect

I thought with HAProxy 2.4 supporting WebSockets over h2, I would finally be able to get websockets over HAProxy to work... But no...
When my browser tries to connect, it instantly gives up on the "connect?transport=webSockets" and tries SSE. This "connect" also is instantly abandoned and yet it "start?transport=SSE" just fine.
Logging in HAProxy says it never even got the requests for connect.
Nov 17 17:10:19 127.0.0.1 haproxy[26334]: 192.168.126.78:1228 [17/Nov/2021:17:10:18.520] www-httpsStrict~ test_backend/werdc2020 557/0/0/2/558 200 704 - - --VN 2/1/0/0/0 0/0 "GET https://myserver.com/signalr/negotiate?clientProtocol=2.1&connectionData=%5B%7B%22name%22%3A%22testservice%22%7D%5D&_=1637143818090 HTTP/2.0"
Nov 17 17:10:19 127.0.0.1 haproxy[26334]: 192.168.126.78:1228 [17/Nov/2021:17:10:19.075] www-httpsStrict~ test_backend/werdc2020 52/0/0/1/52 404 1392 - - --VN 2/1/0/0/0 0/0 "GET https://myserver.com/favicon.ico HTTP/2.0"
Nov 17 17:10:19 127.0.0.1 haproxy[26334]: 192.168.126.78:1228 [17/Nov/2021:17:10:19.279] www-httpsStrict~ test_backend/werdc2020 96/0/0/7/103 200 339 - - --VN 3/2/1/1/0 0/0 "GET https://myserver.com/signalr/start?transport=serverSentEvents&clientProtocol=2.1&connectionToken=%2FZ99uK3N88wIyXYxQlQc4w42M3jcf0Tz6tbLcm7CXhOcTgTu7qgbfsMxn9l6GNP%2FML39o%2BabOZ4GSQDgAxo7oYXIacee8Ku2Nd1QXvWRalisPY%2BTIgddfJsWdE828LaH&connectionData=%5B%7B%22name%22%3A%22testservice%22%7D%5D&_=1637143818091 HTTP/2.0"
Nov 17 17:10:19 127.0.0.1 haproxy[26334]: 192.168.126.78:1228 [17/Nov/2021:17:10:19.375] www-httpsStrict~ test_backend/werdc2020 65/0/0/20/84 200 348 - - --VN 3/2/1/1/0 0/0 "POST https://myserver.com/signalr/send?transport=serverSentEvents&clientProtocol=2.1&connectionToken=%2FZ99uK3N88wIyXYxQlQc4w42M3jcf0Tz6tbLcm7CXhOcTgTu7qgbfsMxn9l6GNP%2FML39o%2BabOZ4GSQDgAxo7oYXIacee8Ku2Nd1QXvWRalisPY%2BTIgddfJsWdE828LaH&connectionData=%5B%7B%22name%22%3A%22testservice%22%7D%5D HTTP/2.0"
If I connect directly on the server it works just fine...
HAProxy config:
global
log 127.0.0.1 local0 debug
log-tag haproxy
maxconn 10000
user haproxy
group haproxy
daemon
nbproc 1
nbthread 8
#cpu-map auto:all 0-1
stats socket /var/run/haproxy.sock mode 600 level admin
stats timeout 30s
pidfile /var/run/haproxy.pid
tune.ssl.default-dh-param 2048
tune.ssl.cachesize 100000
tune.ssl.lifetime 600
tune.ssl.maxrecord 1460
# intermediate configuration
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:!MD5:!aNULL:!DH:!RC4
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5s
timeout client 1m
timeout client-fin 1m
timeout server 1h
timeout tunnel 1h
frontend stats
bind 0.0.0.0:8080
mode http
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /
stats auth myuser:1111
stats refresh 60s
# Force front server MUST use redirect on HTTPS
frontend www-httpStrict
bind *:80
option splice-auto
# Test URI to see if its a letsencrypt request
acl letsencrypt-acl path_beg /.well-known
acl test hdr_sub(cookie) istest=true
redirect scheme https code 307 if !{ ssl_fc } !letsencrypt-acl
use_backend letsencrypt if letsencrypt-acl
use_backend test_backend if test
default_backend www-backendStrict
frontend www-httpsStrict
bind *:443 tfo ssl crt /etc/ssl/private alpn h2,http/1.1
option forwardfor
option splice-auto
acl test hdr_sub(cookie) istest=true
use_backend test_backend if test
default_backend www-backendStrict
backend letsencrypt
server nginx 127.0.0.1:8888
# this backend require haproxy open SSL tunel to port 443 on webservers
backend www-backendStrict
balance roundrobin
cookie MyWebFarm insert
option forwardfor
option splice-auto
option tcp-smart-connect
option httpchk
http-check connect ssl alpn h2 sni myserver.com
http-check send meth HEAD uri /login.aspx ver HTTP/2 hdr Host myserver.com
http-check expect status 200-399
http-request add-header X-Forwarded-Proto https if { ssl_fc }
retry-on all-retryable-errors
http-request disable-l7-retry if METH_POST
default-server ssl tfo verify none alpn h2,http/1.1 check allow-0rtt
server werdc01 192.168.1.5:443 cookie werc01 check
server werdc2020 192.168.1.6:443 cookie werdc2020 check
backend test_backend
balance roundrobin
cookie MyWebFarm insert
option forwardfor
option splice-auto
option tcp-smart-connect
option httpchk
http-check connect ssl alpn h2 sni myserver.com
http-check send meth HEAD uri /login.aspx ver HTTP/2 hdr Host myserver.com
http-check expect status 200-399
http-request add-header X-Forwarded-Proto https if { ssl_fc }
retry-on all-retryable-errors
http-request disable-l7-retry if METH_POST
default-server ssl verify none alpn h2,http/1.1 check allow-0rtt
server werdc2020 192.168.1.6:443 cookie werdc2020 check
I have the istest cookie set to always hit the test_backend...
Headers sent from browser:
GET /signalr/connect?transport=webSockets&clientProtocol=2.1&connectionToken=%2FZ99uK3N88wIyXYxQlQc4w42M3jcf0Tz6tbLcm7CXhOcTgTu7qgbfsMxn9l6GNP%2FML39o%2BabOZ4GSQDgAxo7oYXIacee8Ku2Nd1QXvWRalisPY%2BTIgddfJsWdE828LaH&connectionData=%5B%7B%22name%22%3A%22catiservice%22%7D%5D&tid=7 undefined
Host: myserver.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:94.0) Gecko/20100101 Firefox/94.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Sec-WebSocket-Version: 13
Origin: https://myserver.com
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Key: P1GaqaLxIxBfuXDCtxuQwg==
DNT: 1
Connection: keep-alive, Upgrade
Cookie: _ga=GA1.2.95033637.1627965955; .MyAuth=D758016C45CF1B9BDC38D17DCAF7A3CEE7E528B97352267A1B0D019BFE20964BF2900C655950E126CD7FC08CB421C1A997D1A5FDB0266316B920634FB2C23B63080B271B6331096902B47BF73661240D; MyWebFarm=werdc2020; ASP.NET_SessionId=w1zibnnuht4ohicpntyhn55q; istest=true
Sec-Fetch-Dest: websocket
Sec-Fetch-Mode: websocket
Sec-Fetch-Site: same-origin
Pragma: no-cache
Cache-Control: no-cache
Upgrade: websocket
TE: trailers
Why doesn't HAProxy see the "connect" requests?
Why doesn't it connect with WebSockets?
There is a github issue on this: https://github.com/haproxy/haproxy/issues/162
Although your question isn't very insightful, I'd recommend to strip down your config to the bare minimum to test websocket.
Here's a snippet to consider
option http-server-close
option http-use-htx
acl hdr_connection_upgrade hdr(Connection) -i upgrade
acl hdr_upgrade_websocket hdr(Upgrade) -i websocket
Why doesn't HAProxy see the "connect" requests?
I suppose, just because the CONNECT has not been successful. If you'd use tcpdump you pretty sure would see a request incoming.
Using the explanation from #sgohl this config works (at least until 2.5 fixes the issue):
global
log 127.0.0.1 local0 debug
log-tag haproxy
maxconn 10000
user haproxy
group haproxy
daemon
nbproc 1
nbthread 8
#cpu-map auto:all 0-1
stats socket /var/run/haproxy.sock mode 600 level admin
stats timeout 30s
pidfile /var/run/haproxy.pid
tune.ssl.default-dh-param 2048
tune.ssl.cachesize 100000
tune.ssl.lifetime 600
tune.ssl.maxrecord 1460
# intermediate configuration
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:!MD5:!aNULL:!DH:!RC4
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
defaults all
log global
mode http
option httplog
option dontlognull
timeout connect 5s
timeout client 1m
timeout client-fin 1m
timeout server 1h
timeout tunnel 4h
option splice-auto
option tcp-smart-connect
balance roundrobin
option forwardfor
retry-on all-retryable-errors
frontend stats from all
bind *:8080
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /
stats auth Myuser:11111
stats refresh 60s
listen www from all
bind *:80
bind *:443 tfo ssl crt /etc/ssl/private alpn h2,http/1.1 allow-0rtt
# Test URI to see if its a letsencrypt request
acs letsencrypt path_beg /.well-known
http-request redirect scheme https unless { ssl_fc } or letsencrypt
use_backend letsencrypt if letsencrypt
http-response set-header Strict-Transport-Security "max-age=10886400; includeSubDomains; preload;"
use_backend ws if { hdr(Connection) -i upgrade } or { hdr(Upgrade) -i websocket }
default_backend h2
backend letsencrypt
server letsencrypt 127.0.0.1:8888
defaults cg from all
cookie MyWebFarm insert
option httpchk
http-check connect ssl alpn h2 sni myserver.com
http-check send meth HEAD uri /login.aspx ver HTTP/2 hdr Host myserver.com
http-check expect status 200-399
backend h2 from cg
default-server ssl tfo verify none alpn h2 check allow-0rtt
http-request disable-l7-retry if METH_POST
use-server werdc2020 if { hdr_sub(cookie) istest=true }
server werdc01 192.168.1.5:443 cookie werc01
server werdc2020 192.168.1.6:443 cookie werdc2020
backend ws from cg
default-server ssl tfo verify none alpn http/1.1 check allow-0rtt
http-request disable-l7-retry if METH_POST
use-server werdc2020 if { hdr_sub(cookie) istest=true }
server werdc01 192.168.1.5:443 cookie werc01
server werdc2020 192.168.1.6:443 cookie werdc2020

haproxy + spring boot writeAddress(..) failed: Connection reset by peer

I'm running HAproxy with backend Spring Boot Rest controllers. My spring log shows constant errors that look like the following:
[reactor-http-epoll-26] ERROR o.s.w.s.a.HttpWebHandlerAdapter - [9df8bfcf] Error [io.netty.channel.unix.Errors$NativeIoException: writeAddress(..) failed: Connection reset by peer] for HTTP GET "/api/v1/status", but ServerHttpResponse already committed (200 OK)
HAproxy performs an HTTP check on the url /api/v1/status. What would be the reason that I'm getting these errors?
HAProxy Config
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
# daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
ssl-default-bind-ciphers EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EX$
ssl-default-bind-options no-sslv3 no-tlsv10
tune.ssl.default-dh-param 4096
defaults
mode http
log global
option httplog
option http-server-close
option forwardfor except 127.0.0.0
option redispatch
retries 3
timeout http-request 30s
timeout queue 1m
timeout connect 30s
timeout client 30s
timeout server 30s
frontend https-in
bind *:443 ssl crt /etc/cert.pem
default_backend api
backend api
mode http
option httpchk GET /api/v1/status HTTP/1.0
http-check expect status 200
balance roundrobin
server api1 127.0.0.1:8001 check fall 3 rise 2
server api2 127.0.0.1:8002 check fall 3 rise 2
HAproxy is doing GET request, reads http response code and closing connection.
Boot is trying to send remaining parts (http headers and some json payload) but connection is already closed.
Just replace GET with OPTIONS in the line:
option httpchk GET /api/v1/status HTTP/1.0

HA Proxy fails to recognize Tomcat:8080 running on a mac

Can someone help be understand why haproxy fails to detect my tomcat instance running on port 8080? I can reach the server just fine when browsing to 127.0.0.1:8080
On start up haproxy spits out the following error:
WARNING] 141/163609 (23336) : Server cim/cim is DOWN, reason: Layer7 wrong status, code: 404, info: "Not Found", check duration: 1ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 141/163609 (23336) : backend 'cim' has no server available!
Here's the haproxy config.
global
maxconn 128
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
tune.ssl.default-dh-param 2048
defaults
mode http
log global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
option httplog
option forwardfor
option httpclose
option redispatch
option httpchk HEAD / HTTP/1.0
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 20s
timeout check 10s
frontend in-http
bind *:80
acl restricted_urls path_beg /trm/ws/v2
use_backend cim if restricted_urls
backend cim
balance roundrobin
server cim 127.0.0.1:8080 check
Tomcat is definitely up and running, and I can telnet to 127.0.0.1 8080 and receive an http response.

HAProxy 1.5 Redirect with Exchange 2013 OWA

I've set up HAproxy version 1.5.4 as a NLB for 2 Exchange Multi-role servers.
I'm now trying to get redirection working when a user types in the flat hostname "webmail" into a browser. I would like it redirected to https://webmail.domain.com/owa
Here is my existing haproxy config.
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
defaults
log global
mode tcp
balance roundrobin
retries 3
option redispatch
maxconn 10000
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
listen OWA 10.20.100.120:443
option httpchk GET /owa/healthcheck.h
http-check expect status 200
server EX2013A.domain.com 10.20.100.126 check port 80
server EX2013B.domain.com 10.20.100.127 check port 80
listen EAC 10.20.100.131:443
option httpchk /eac/healthcheck.h
server EX2013A.domain.com 10.20.100.126 check port 80
server EX2013B.domain.com 10.20.100.127 check port 80
listen EWS 10.20.100.122:443
option httpchk get /ews/healthcheck.h
server EX2013A.domain.com 10.20.100.126 check port 80
server EX2013B.domain.com 10.20.100.127 check port 80
listen OAB 10.20.100.123:443
option httpchk get /oab/healthcheck.h
server EX2013A.domain.com 10.20.100.126 check port 80
server EX2013B.domain.com 10.20.100.127 check port 80
listen Autodiscover 10.20.100.132:443
option httpchk get /autodiscover/healthcheck.h
server EX2013A.domain.com 10.20.100.126 check port 80
server EX2013B.domain.com 10.20.100.127 check port 80
listen OA 10.20.100.133:443
option httpchk get /rpc/healthcheck.h
server EX2013A.domain.com 10.20.100.126 check port 80
server EX2013B.domain.com 10.20.100.127 check port 80
listen SMTP 10.20.100.120:25
option smtpchk
server EX2013A.domain.com 10.20.100.126 check port 25
server EX2013B.domain.com 10.20.100.127 check port 25
listen stats 0.0.0.0:4000
mode http
balance
timeout client 5000
timeout connect 4000
timeout server 30000
stats enable
stats hide-version
stats uri /stats
stats auth admin:password
Thanks
You did not precise which version of HAProxy you're using, so I'm assuming this is 1.5.
First, creating a listen section with the bound IP:port on the section description line is now deprecated.
Second, to make it clearer, I would also add the port 443 on the server line.
Please write your OWA section like this:
listen OWA
bind 10.20.100.120:443
option httpchk GET /owa/healthcheck.h
http-check expect status 200
server EX2013A.domain.com 10.20.100.126:443 check port 80
server EX2013B.domain.com 10.20.100.127:443 check port 80
Now, to perform the redirection, you need a frontend dedicated to answer to HTTP clear traffic only:
frontend OWA_http
bind 10.20.100.120:80
http-request redirect location https://%[req.hdr(Host)]/owa/
Baptiste

Resources