When I started HAProxy under Windows Server, I got the following error - windows

haproxy.exe -f haproxy.cfg -d
When I run HAProxy, I get an error:
'''
Available polling systems :
poll : pref=200, test result OK
select : pref=150, test result FAILED
Total: 2 (1 usable), will use poll.
Available filters :
[SPOE] spoe
[CACHE] cache
[FCGI] fcgi-app
[COMP] compression
[TRACE] trace
Using poll() as the polling mechanism.
[NOTICE] (1036) : haproxy version is 2.4.0-6cbbecf
[ALERT] (1036) : Starting proxy warelucent: cannot bind socket (Address already in use) [0.0.0.0:5672]
[ALERT] (1036) : [haproxy.main()] Some protocols failed to start their listeners! Exiting.
'''
In the meantime, no other services are running, and I have the RabbitMQ service open.
My haproxy.cfg file is as follows:
'''
#logging options
global
log 127.0.0.1 local0 info
maxconn 1500
daemon
quiet
nbproc 20
defaults
log global
mode tcp
#if you set mode to tcp,then you nust change tcplog into httplog
option tcplog
option dontlognull
retries 3
option redispatch
maxconn 2000
timeout connect 10s
timeout client 10s
timeout server 10s
#front-end IP for consumers and producters
listen warelucent
bind 0.0.0.0:5672
#配置TCP模式
mode tcp
#balance url_param userid
#balance url_param session_id check_post 64
#balance hdr(User-Agent)
#balance hdr(host)
#balance hdr(Host) use_domain_only
#balance rdp-cookie
#balance leastconn
#balance source //ip
#简单的轮询
balance roundrobin
server one 1.1.1.1:5672 check inter 5000 rise 2 fall 2
server two 2.2.2.2:5672 check inter 5000 rise 2 fall 2
server three 3.3.3.3:5672 check inter 5000 rise 2 fall 2
listen stats
bind 127.0.0.1:8100
mode http
option httplog
stats enable
stats uri /rabbitmq-stats
stats refresh 5s
'''
Most of the Internet is due to the version, but I checked the official website, the version is the latest, and I also started the RabbitMQ service, so I don't know where the error is at present

(Address already in use) [0.0.0.0:5672]
it means that the port 5672 (RabbitMQ is already in use. Most likely you have a rabbitmq node running in that machine.
So just change the HA-PROXY port.

Related

Haproxy 503 service unavailable

We are running Haproxy and 3 nginx processes inside a single docker on the host network to access it from localhost. We have installed HaProxy version 2.6.6. 3 nginx servers are running behind it as backend servers. We are able to access them individually What is wrong with the haproxy configuration here? we are always getting the below error on curl (curl http://localhost:80) requests. Please suggest
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
pidfile /usr/local/etc/haproxy/haproxy.pid
defaults
timeout http-request 2000
timeout queue 1000
timeout check 2000
timeout connect 2000
timeout client 5000
timeout server 5000
log global
option dontlognull
option clitcpka
option srvtcpka
option tcpka
unique-id-format %[uuid()]
unique-id-header X-Request-ID
log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %ID %{+Q}r"
frontend http-in
mode http
option forwardfor
bind *:80
use_backend webservers
backend webservers
mode http
server localhost-01 localhost:8000
server localhost-02 localhost:8001
server localhost-03 localhost:8002

haproxy + spring boot writeAddress(..) failed: Connection reset by peer

I'm running HAproxy with backend Spring Boot Rest controllers. My spring log shows constant errors that look like the following:
[reactor-http-epoll-26] ERROR o.s.w.s.a.HttpWebHandlerAdapter - [9df8bfcf] Error [io.netty.channel.unix.Errors$NativeIoException: writeAddress(..) failed: Connection reset by peer] for HTTP GET "/api/v1/status", but ServerHttpResponse already committed (200 OK)
HAproxy performs an HTTP check on the url /api/v1/status. What would be the reason that I'm getting these errors?
HAProxy Config
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
# daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
ssl-default-bind-ciphers EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EX$
ssl-default-bind-options no-sslv3 no-tlsv10
tune.ssl.default-dh-param 4096
defaults
mode http
log global
option httplog
option http-server-close
option forwardfor except 127.0.0.0
option redispatch
retries 3
timeout http-request 30s
timeout queue 1m
timeout connect 30s
timeout client 30s
timeout server 30s
frontend https-in
bind *:443 ssl crt /etc/cert.pem
default_backend api
backend api
mode http
option httpchk GET /api/v1/status HTTP/1.0
http-check expect status 200
balance roundrobin
server api1 127.0.0.1:8001 check fall 3 rise 2
server api2 127.0.0.1:8002 check fall 3 rise 2
HAproxy is doing GET request, reads http response code and closing connection.
Boot is trying to send remaining parts (http headers and some json payload) but connection is already closed.
Just replace GET with OPTIONS in the line:
option httpchk GET /api/v1/status HTTP/1.0

Outbound connection intermittent failed to response

I'm experiencing intermittent failed to response when make an outbound connection such as RPC call, it is logged by my application (Java) like this :
org.apache.http.NoHttpResponseException: RPC_SERVER.com:443 failed to respond !
Outbound connection flow
Kubernetes Node -> ELB for internal NGINX -> internal NGINX ->[Upstream To]-> ELB RPC server -> RPC server instance
This problem is not occurred on usual EC2 (AWS).
I'm able to reproduce on my localhost by doing this
Run main application which act as client in port 9200
Run RPC server in port 9205
Client will make a connection to server using port 9202
Run $ socat TCP4-LISTEN:9202,reuseaddr TCP4:localhost:9205 that will listen on port 9202 and then forward it to 9205 (RPC Server)
Add rules on iptables using $ sudo iptables -A INPUT -p tcp --dport 9202 -j DROP
Trigger a RPC calling, and it will return the same error message as I desrcibe before
Hypothesis
Caused by NAT on kubernetes, as far as I know, NAT is using conntrack, conntrack and break the TCP connection if it was idle for some period of time, client will assume the connection is still established although it isn't. (CMIIW)
I also have tried scaling kube-dns into 10 replica, and the problem still occurred.
Node Specification
Use calico as network plugin
$ sysctl -a | grep conntrack
net.netfilter.nf_conntrack_acct = 0
net.netfilter.nf_conntrack_buckets = 65536
net.netfilter.nf_conntrack_checksum = 1
net.netfilter.nf_conntrack_count = 1585
net.netfilter.nf_conntrack_events = 1
net.netfilter.nf_conntrack_expect_max = 1024
net.netfilter.nf_conntrack_generic_timeout = 600
net.netfilter.nf_conntrack_helper = 1
net.netfilter.nf_conntrack_icmp_timeout = 30
net.netfilter.nf_conntrack_log_invalid = 0
net.netfilter.nf_conntrack_max = 262144
net.netfilter.nf_conntrack_tcp_be_liberal = 0
net.netfilter.nf_conntrack_tcp_loose = 1
net.netfilter.nf_conntrack_tcp_max_retrans = 3
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 3600
net.netfilter.nf_conntrack_tcp_timeout_established = 86400
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
net.netfilter.nf_conntrack_timestamp = 0
net.netfilter.nf_conntrack_udp_timeout = 30
net.netfilter.nf_conntrack_udp_timeout_stream = 180
net.nf_conntrack_max = 262144
Kubelet config
[Service]
Restart=always
Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
Environment="KUBELET_CLOUD_ARGS=--cloud-provider=aws"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_EXTRA_ARGS $KUBELET_CLOUD_ARGS
Kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"clean", BuildDate:"2017-08-31T09:14:02Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.7", GitCommit:"8e1552342355496b62754e61ad5f802a0f3f1fa7", GitTreeState:"clean", BuildDate:"2017-09-28T23:56:03Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Kube-proxy Log
W1004 05:34:17.400700 8 server.go:190] WARNING: all flags other than --config, --write-config-to, and --cleanup-iptables are deprecated. Please begin using a config file ASAP.
I1004 05:34:17.405871 8 server.go:478] Using iptables Proxier.
W1004 05:34:17.414111 8 server.go:787] Failed to retrieve node info: nodes "ip-172-30-1-20" not found
W1004 05:34:17.414174 8 proxier.go:483] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
I1004 05:34:17.414288 8 server.go:513] Tearing down userspace rules.
I1004 05:34:17.443472 8 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_max' to 262144
I1004 05:34:17.443518 8 conntrack.go:52] Setting nf_conntrack_max to 262144
I1004 05:34:17.443555 8 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I1004 05:34:17.443584 8 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I1004 05:34:17.443851 8 config.go:102] Starting endpoints config controller
I1004 05:34:17.443888 8 config.go:202] Starting service config controller
I1004 05:34:17.443890 8 controller_utils.go:994] Waiting for caches to sync for endpoints config controller
I1004 05:34:17.443916 8 controller_utils.go:994] Waiting for caches to sync for service config controller
I1004 05:34:17.544155 8 controller_utils.go:1001] Caches are synced for service config controller
I1004 05:34:17.544155 8 controller_utils.go:1001] Caches are synced for endpoints config controller
$ lsb_release -s -d
Ubuntu 16.04.3 LTS
Check the value of sysctl net.netfilter.nf_conntrack_tcp_timeout_close_wait inside the pod that contains your program. It is possible that the value on the node that you listed (3600) isn't the same as the value inside the pod.
If the value in the pod is too small (e.g. 60), and your Java client half-closes the TCP connection with a FIN when it finishes transmitting, but the response takes longer than the close_wait timeout to arrive, nf_conntrack will lose the connection state and your client program will not receive the response.
You may need to change the behavior of the client program to not use a TCP half-close, OR modify the value of net.netfilter.nf_conntrack_tcp_timeout_close_wait to be larger. See https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/.

HA Proxy fails to recognize Tomcat:8080 running on a mac

Can someone help be understand why haproxy fails to detect my tomcat instance running on port 8080? I can reach the server just fine when browsing to 127.0.0.1:8080
On start up haproxy spits out the following error:
WARNING] 141/163609 (23336) : Server cim/cim is DOWN, reason: Layer7 wrong status, code: 404, info: "Not Found", check duration: 1ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 141/163609 (23336) : backend 'cim' has no server available!
Here's the haproxy config.
global
maxconn 128
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
tune.ssl.default-dh-param 2048
defaults
mode http
log global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
option httplog
option forwardfor
option httpclose
option redispatch
option httpchk HEAD / HTTP/1.0
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 20s
timeout check 10s
frontend in-http
bind *:80
acl restricted_urls path_beg /trm/ws/v2
use_backend cim if restricted_urls
backend cim
balance roundrobin
server cim 127.0.0.1:8080 check
Tomcat is definitely up and running, and I can telnet to 127.0.0.1 8080 and receive an http response.

No such server found while disabling HAproxy disable server from cluster

My haproxy.cfg
global
log 127.0.0.1 local0
maxconn 20000
user haproxy
group haproxy
stats socket /var/run/haproxy/haproxy.sock level admin
stats timeout 2m
listen admin
bind *:8080
stats enable
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
# timeout http-request 5s
timeout connect 5000
timeout client 60000
timeout server 60000
frontend http-in
bind *:80
default_backend monkey
backend monkey
stats enable
stats uri /haproxy?stats
stats realm Strictly\ Private
stats auth admin:hello
stats auth Another_User:hi
mode http
compression algo gzip
compression type text/html text/plain text/css application/json
balance roundrobin
option httpclose
option forwardfor
default-server inter 1s fall 1
server cd-test-1 1.2.3.4:80 check
server cd-test-2 5.6.7.8:80 check
I have been using socat to disable a node from HAproxy cluster.
below is the command
echo "disable server monkey/cd-test-1"| socat stdio /var/run/haproxy/haproxy.sock
The above disables my node from haproxy. But if I use the ip address(1.2.3.4) instead of "cd-test-1" it returns No such server.
I am using ansible to automated this. I use {{inventory_hostname}} and delegate the command to my HAproxy server. Hence the issue.
- name: Disable {{ inventory_hostname }} in haproxy and letting the services drain
shell: echo "disable server monkey/{{inventory_hostname}}"| socat stdio /var/run/haproxy/haproxy.sock
become_user: root
delegate_to: "{{ item }}"
with_items: groups.haproxy_backend
This returns "No such server." and moves along.
Can someone please help me find the issue with using the IP instead of the name of server. Might be doing something very silly. Any help is appreciated.
Disabling and enabling HAproxy using socat, the server alias name has to be mentioned.
Otherwise, we will be getting a No such server error.

Resources