I am trying to loadbalance socket.io requests using HAProxy. The following is my configuration:
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
frontend all 0.0.0.0:8888
mode http
timeout client 120s
option forwardfor
# Fake connection:close, required in this setup.
option http-server-close
option http-pretend-keepalive
acl is_socketio path_beg /socket.io
use_backend socket-servers if is_socketio
default_backend http-servers
backend http-servers
balance roundrobin
option httpclose
option forwardfor
# Roundrobin switching
server node-1 172.14.2.25:8887 check
server node-2 172.14.2.28:8887 check
backend socket-servers
mode http
timeout server 120s
balance roundrobin
# based on cookie set in header
# haproxy will add the cookies for us
option forwardfor
cookie SERVERID insert indirect nocache
server node1 172.14.2.25:8887 cookie node1 weight 1 maxconn 1024 check
server node2 172.14.2.28:8887 cookie node2 weight 1 maxconn 1024 check
This configuration gives timeout when I try to create socket connection.
However, if I delete either on of the last two lines [server node1 172.14.2.25:8887 cookie node1 weight 1 maxconn 1024 check or server node2 172.14.2.28:8887 cookie node2 weight 1 maxconn 1024 check], the socket connects properly.
I do not understand what is wrong with the configuration. Any help will be highly appreciated.
TIA :)
In the configuration of backend socket-servers, I had to change balance roundrobin to balance source. That worked! :)
Related
I have simple WebSocket connection from my browser to a service inside Rancher.
I tried to connect to the service in 2 ways:
1) directly to the service:
browser ---> service
2) via Rancher's Load Balancer:
browser ---> Load Balancer ---> service
In the first case everything is fine: the connection is established and the messages are sent through it.
In the 2nd case the connection is dropped in after ~50s. Messages are send through the connection correctly in both directions.
What is the reason?
EDIT: I tested in on ws and wss protocol. In both cases there is the same issue.
Rancher Load Balancer internally uses HAProxy, which can be customized to your needs.
Here is an example HAProxy config for websockets:
global
maxconn 4096
ssl-server-verify none
defaults
mode http
balance roundrobin
option redispatch
option forwardfor
timeout connect 5s
timeout queue 5s
timeout client 36000s
timeout server 36000s
frontend http-in
mode http
bind *:443 ssl crt /etc/haproxy/certificate.pem
default_backend rancher_servers
# Add headers for SSL offloading
http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Forwarded-Ssl on if { ssl_fc }
acl is_websocket hdr(Upgrade) -i WebSocket
acl is_websocket hdr_beg(Host) -i ws
use_backend rancher_servers if is_websocket
backend rancher_servers
server websrv1 <rancher_server_1_IP>:8080 weight 1 maxconn 1024
server websrv2 <rancher_server_2_IP>:8080 weight 1 maxconn 1024
server websrv3 <rancher_server_3_IP>:8080 weight 1 maxconn 1024
Reference: https://rancher.com/docs/rancher/v1.6/en/installing-rancher/installing-server/basic-ssl-config/#example-haproxy-configuration
Only the relevant config can be used in "Custom haproxy.cfg" section of the LB.
See screenshot:
Here is the link for more documentation for custom haproxy in Rancher: https://rancher.com/docs/rancher/v1.6/en/cattle/adding-load-balancers/#custom-haproxy-configuration
We are trying to configure our Socket.io socket servers behind HAProxy and over HAProxy we are using Google Cloud Load Balancer, so that HAProxy is not the single point of failure. As mentioned in this post by https://medium.com/google-cloud/highly-available-websockets-on-google-cloud-c74b35ee20bc#.o6xxj5br8. Also depicted in the picture below.
At the Google cloud load balancer we are using TCP Load balancing with SSL Proxy with Proxy Protocol ON.
The HAProxy is configured to use Cookies so that a client always connects to the same server. However since cookies might not be available on all our clients system, we decided to use load balancing algorithm as source in HAProxy. Here is the HAProxy configuration
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
maxconn 16384
tune.ssl.default-dh-param 2048
user haproxy
group haproxy
daemon
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3
defaults
mode http
log global
option httplog
option http-server-close
option dontlognull
option redispatch
option contstats
retries 3
backlog 10000
timeout client 25s
timeout connect 5s
timeout server 25s
timeout tunnel 3600s
timeout http-keep-alive 1s
timeout http-request 15s
timeout queue 30s
timeout tarpit 60s
default-server inter 3s rise 2 fall 3
option forwardfor
frontend public
bind *:443 ssl crt /etc/ssl/private/key.pem ca-file /etc/ssl/private/cert.crt accept-proxy
maxconn 50000
default_backend ws
backend ws
timeout check 5000
option tcp-check
option log-health-checks
balance source
cookie QUIZIZZ_WS_COOKIE insert indirect nocache
server ws1 socket-server-1:4000 maxconn 4096 weight 10 check rise 1 fall 3 check cookie ws1 port 4000
server ws2 socket-server-1:4001 maxconn 4096 weight 10 check rise 1 fall 3 check cookie ws2 port 4001
server ws3 socket-server-2:4000 maxconn 4096 weight 10 check rise 1 fall 3 check cookie ws3 port 4000
server ws4 socket-server-2:4001 maxconn 4096 weight 10 check rise 1 fall 3 check cookie ws4 port 4001
This is however giving connection errors on around 5% of our clients as compared to our old single server system. Any suggestions?
Edit: Connection errors means that the client was not able to connect to the socket server and the socket.io client was throwing connection errors.
Thanks in advance.
I use haproxy as banlancer for mariadb galera cluster,
it can connect and do some operation normally,but after serval seconds I want to do some search again but a got lost connection during query error.see the error below image.
lost connection img
here is my haproxy.cfg
defaults
log global
mode tcp
option tcplog
option dontlognull
option http-server-close
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 50000ms
timeout server 50000ms
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
and the balance for mariadb cluster in haproxy.cfg
listen mariadb_cluster_writes 0.0.0.0:50613
## A failover pool for writes to ensure writes only hit one node at a time.
mode tcp
#option httpchk
option tcpka
option mysql-check user haproxy
server node65 172.27.12.65:3306 check weight 2
server node64 172.27.12.64:3306 check weight 1
server node67 172.27.12.67:3306 check weight 1
listen mariadb_cluster_reads 0.0.0.0:50614
## A load-balanced pool for reads to utilize all nodes for reads.
mode tcp
balance leastconn
#option httpchk
option tcpka
option mysql-check user haproxy
server node65 172.27.12.65:3306 check weight 1
server node64 172.27.12.64:3306 check weight 1
server node67 172.27.12.67:3306 check weight 1
Does anyone have idea?
I think i find the reason.Haproxy itself has a timeout for server and client,I set the sever timeout and client timeout as same as mysql timeout which is 8 hour.Now it seems like this:
defaults
log global
mode tcp
option tcplog
option dontlognull
option http-server-close
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 480m
timeout server 480m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
Hope can help others.
however i use mariadb cluster,it is master and mater
architecture,all of node can insert,so I changed the haproxy.cfg,now it seems like this:
# Global settings
global
log 127.0.0.1 local2
maxconn 4000
daemon
defaults
log global
mode tcp
#option tcplog
option dontlognull
option tcp-smart-accept
option tcp-smart-connect
option redispatch
retries 3
timeout connect 5s
timeout client 480m
timeout server 480m
listen admin_stats
bind 0.0.0.0:50613
mode http
maxconn 10
stats refresh 30s
stats uri /stats
stats realm XingCloud\ Haproxy
stats auth admin:admin
listen galera_back
bind :50614
balance leastconn
server node65 172.27.12.65:3306 check weight 1
server node64 172.27.12.64:3306 check weight 1
server node67 172.27.12.67:3306 check weight 1
I have 2 HAProxys that are load balancing user requests with roundrobin algorithm successfully to my 2 WebServers.
When a webserver fails, HAProxy sends the request to the next available server, but for some reason I'm not able to keep the user session, and therefore information isn't displayed properly.
How can I make it so the session is saved during a failover?
Here's my haproxy.cfg:
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 2000
user haproxy
group haproxy
daemon
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
timeout connect 5000
timeout client 10000
timeout server 1000
listen app 192.168.1.100:80
mode http
stats enable
stats uri /haproxy?stats
stats realm Strictly\ Private
stats auth admin:admin
balance roundrobin
cookie LSW_WEB insert
option httpclose
option forwardfor
option httpchk HEAD / HTTP/1.0
server server1 192.168.1.93:80 cookie LSW_WEB01 check
server server2 192.168.1.94:80 cookie LSW_WEB02 check
This has to do with how you share sessions between your servers not with haproxy itself.
The problem lies in your application or how you are storing sessions.
I am setting up a keepalived HAProxy balancer of http traffic in the frontend of a three appserver-nodes with nginx/php-fpm. So my chain of services would be:
-----> HAProxy -----> nginx -----> php-FPM ----> webapp
Well, the problem is that haproxy can't see my backend servers when is set it to use cookies to ensure session affinity.
here is my haproxy.conf file:
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
defaults
log global
mode http
option httplog
option dontlognull
contimeout 5000
clitimeout 50000
srvtimeout 50000
frontend http-in
bind *:80
default_backend servers
backend servers
option httpchk OPTIONS /
option forwardfor
option http-server-close
balance roundrobin
cookie PHPSESSID prefix indirect nocache
stats enable
stats refresh 10s
stats hide-version
stats scope .
stats uri /lb?stats
stats auth admin:admin2013
server nodo1 10.10.200.19:80 check cookie nodo1
server nodo2 10.10.200.20:80 check cookie nodo2
server nodo3 10.10.200.21:80 check cookie nodo3
Anyone has any idea why this is happening? I already checked my php.ini file and the session.name variable has the correct value (PHPSESSID in this case).
In the other hand, i'm planning use a redis db as a session storage, i'm thinking that if I choose this, would not be necessary to use session affinity in haproxy, but i'm worry that in this case, the user could been jumping on each server in every request.
Finally I found the error, i had to remove the line:
option httpchk OPTIONS /
this was all the problem, now is working like a charm!
Now I'll continue with the connection of redis to ensure that the session of the user will not been lost in a eventually node crash.