When I open the haproxy statistics report page of my http proxy server, I saw something like this:
Cum. connections: 280073
Cum. sessions : 3802
Cum. HTTP requests: 24245
I'm not using 'appsession' and any other cookie related command in the configuration. So what's 'session' means here?
I guess haproxy identify http session by this order:
Use cookie or query string if it's exists in the configuration.
Use SSL/TLS session.
Use ip address and TCP connection status.
Am I Right?
I was asking myself the very same question this morning.
Searching through http://www.haproxy.org/download/1.5/doc/configuration.txt I came accross this very short definition (hidden in a parameter description) :
A session is a connection that was accepted by the layer 4 rules.
In your case, you're obviously using Haproxy as a layer7/HTTP loadbalancer. If a session is a TCP connection, due to client-side / frontend Keep-Alive, it's normal to have more HTTP reqs than sessions.
Then I guess the high connection number shows that a lot of incoming connections were rejected even before being considered by the HTTP layer. For instance via IP-based ACLs.
As a far as I understand, the 'session' word was introduced to make sure two different concepts were not mixed :
a (TCP) connection : it's a discrete event
a (TCP) session : it's a state which tracks various metadata and has some duration; most importantly Haproxy workload (CPU and memory) should be mostly related to the number of sessions (both arrival rate and concurrent number)
In fact sessions were not introduced after but before connections. An end-to-end connection was called a "session". With the introduction of SSL, proxy protocol and layer4 ACLs, it was needed to cut the end-to-end sessions in smaller parts, hence the introduction of "connections". Zerodeux has perfectly explained what you're observing.
Related
So I read this article https://www.baeldung.com/httpclient-timeout and it says that connection timeout adds to its own penalty if the underlying service's DNS that httpclient tries to connect to has multiple IPs configured to it.
So if I have a connection timeout set to 100ms and the called service DNS has 5 IPs mapped to it then, I am looking at a max connection timeout of 500ms assuming what works is the last IP.
Is there a way to have a cap on this connection timeout regardless what the underlying service topology is as being a client, I will always be agnostic to it.
As far as I understood, you don't have a code-wise case to run in 5 or more IPs situation rather curiosity. So here my experience :
Since you're using RestTemplate which by default uses SimpleClientHttpRequestFactory.
And as the definition of connection time out goes :
The connection timeout is the timeout in making the initial
connection; i.e. completing the TCP connection handshake and getting
connected to the requested Server.
So, as far as theory goes :
Regardless of the underlying service topology, RestTemplate will try to make connection as per the connection timeout value.
And in order to figure out the almost exact timeout in your case, you must run some latency test, print the time differences which restTemplate is taking to get 200 OK.
Also, SimpleClientHttpRequestFactory internally uses HttpURLConnection which has default timeout of infinite (0/-1).
Yes, it has also been observed in rare cases, the connection keeps trying unless Thread.interrupt() explicitly being called to end.
Thus it becomes vital to describe your read-time-out and connection-time-out values and in this way you cap your connection to the limits you defined.
Hope this helps.
It is a SpringBoot website and deployed in one Linux server. We use Jmeter to do the load test.
We mock 500 users to visit the webiste index page simultaneously. The index page is very simple html, no database connection,so it is a quite short connection.
After about 2 minutes, Jmeter starts to throw timeout exception as bleow
I guess this is because of website reaching its capacity and running out of connection.
I get one quesiton here, why does website reach its capacity 2 minutes later after Jemter starts. If its TCP connection capacity for this website is 1000, I guess it will reach 1000 very soon after the Jmeter starts, not 2 minutes.
Besides, I see many TCP connections are in TIME_WAIT status in Linux server. I guess this may be related with the connection timeout?
Edit: Someone thinks it is running of port. Someone thinks it is running out of connection. And someone thinks it is running out of processing thread(eg. What does this messge java.net.ConnectException/Connection timed out mean in log.jtl file of Jmeter?). I don't know which one is the exact reason...
Most probably this is due to underlying Linux TCP/IP kernel stack configuration, as per Linux TCP/IP tuning for scalability article:
By default, a connection is supposed to stay in the TIME_WAIT state for twice the msl. Its purpose is to make sure any lost packets that arrive after a connection is closed do not confuse the TCP subsystem (the full details of this are beyond the scope of this article, but ask me if you’d like details). The default msl is 60 seconds, which puts the default TIME_WAIT timeout value at 2 minutes. Which means you’ll run out of available ports if you receive more than about 400 requests a second, or if we look back to how nginx does proxies, this actually translates to 200 requests per second. Not good for scaling.
SO double check timeouts along with maximum number of ports/sockets/files on the Linux server - my expectation is that the aforementioned parameters need to be tuned for high loads.
It's also a good practice to have monitoring of baseline OS health metrics in place (CPU, RAM, Network, Disk, swap usage, etc.). You can use i.e. JMeter PerfMon Plugin or JMeter SSHMon Listener for this.
I have written a oracle client program using OCI library.
client send a request to server and hung because server crashed and not intimated to client.
How can i find server status from client side(using OCI API).?
Thanks
I think Oracle db module for Asterisk had a nice DCD(dead connection detection) implemented. There are various approaches(server side, client side).
In your case the easiest way would be to use TCP keepalive. Use enable=broken directive in tnsnames.ora.
Purpose
The keepalive feature on the supported TCP transports can be enabled
for a net service client by embedding (ENABLE=BROKEN) under the
DESCRIPTION parameter in the connect string. Keepalive allows the
caller to detect a dead remote server, although typically it will take
2 hours or more to notice. Operating system TCP configurables, which
vary by platform, define the actual keepalive timing details.
net_service_name=
(DESCRIPTION=
(enable=broken)
(ADDRESS=(PROTOCOL=tcp)(HOST=sales1-svr)(PORT=1521))
(ADDRESS=(PROTOCOL=tcp)(HOST=sales2-svr)(PORT=1521)))
(CONNECT_DATA=(SERVICE_NAME=sales.us.example.com))
Just beware you will also need root privileges. With default setting Linux kernel starts sending keepalive packets after 2 hours of inactivity. So you also have to change tcp_keepalive_time and tcp_keepalive_intvl in /etc/sysctl.conf. This is global server side settings and Oracle can not yet set keepalive interval for a single TCP connection.
One more comment: I recall there is some function called OCIPing.
This one can be used for testing too. But I'm not sure how to distinguish long running queries from dead server situation.
Is it possible to route dns to different servers based on the protocol of the request without using a proxy server?
For example wss://example.com goes to 1 server and https://example.com goes to a different one.
In principle no, this is not possible. Although there are exceptions.
When you an application and request a network address, e.g. example.com the application will (typically) pass on a request to the OS to open a connection to that address.
Because the OS can only make a connection to an IP Address, the first thing it does is to make a DNS request to find out the address it can connect to. There is no mechanism at all, to tell that DNS request what protocol is being requested. At the point the OS makes the DNS request it is simply, 'What is the address for example.com' there is no space in the body of the message for more information.
In that sense it is not possible.
However there are a few services that use SRV records to find the server they need to connect to.
In these cases the application will say to the OS 'get me the SRV record for _somefancyservice.example.com' When it has that record, it will then send another request to the OS to open a connection the whatever it found in the SRV record.
So you could, in theory, write an application that performed the SRV record query first, and then opened a connection to whatever was returned.
I wrote a more detailed answer specifically about wss, which you may find interesting.
Most proxy servers perform the job of forwarding data to an appropriate "real" server. However, I am in the process of designing a distributed system in which when the "proxy" receives a TCP/IP socket connection, the remote system actually connects with a real server which the proxy nominates. All subsequent data flows from remote to the real server.
So is it possible to "forward" the socket connection request so that the remote system connects with the real server?
(I am assuming for the moment that nothing further can be done with the remote system. Ie the proxy can't respond to the connection by sending the IP address of the actual server and the remote connections with that. )
This will be under vanilla Windows (not Server), so can't use cunning stuff like TCPCP.
I assume your "remote system" is the one that initiates connection attempts, i.e. client of the proxy.
If I get this right: when the "remote system" wants to connect somewhere, you want the "proxy server" to decide where the connection will really go ("real server"). When the decision is made, you don't want to involve the proxy server any further - the data of the connection should not pass the proxy, but go directly between the "remote system" and the "real server".
Problem is, if you want the connection to be truly direct, the "remote system" must know the IP address of of the "real server", and vice versa.
(I am assuming for the moment that nothing further can be done with
the remote system. Ie the proxy can't respond to the connection by
sending the IP address of the actual server and the remote connections
with that. )
Like I said, not possible. Why is it a problem to have the "proxy" send back the actual IP address?
Is it security - you want to make sure the connection really goes where the proxy wanted? If that's the case, you don't have an option - you have to compromise. Either the proxy forwards all the data, and it knows where the data is going, or let the client connect itself, but you don't have control where it connects.
Most networking problems can be solved as long as you have complete control over the entire network. Here, for instance, you could involve routers on the path between the "remote system" and the "real client", to make sure the connection is direct and that it goes where the proxy wanted. But this is complex, and probably not an option in practice (since you may not have control over those routers).
A compromise may be to have several "relay servers" distributed around the network that will forward the connections instead of having the actual proxy server forward them. When a proxy makes a decision, it finds the best (closest) relay server, tells it about the connection, then orders the client to connect to the relay server, which makes sure the connection goes where the proxy intended it to go.
There might be a way of doing this but you need to use a Windows driver to achieve it. I've not tried this when the connection comes from an IP other than localhost, but it might work.
Take a look at NetFilter SDK. There's a trial version which is fully functional up to 100000 TCP and UDP connections. The other possibility is to write a Windows driver yourself, but this is non-trivial.
http://www.netfiltersdk.com
Basically it works as follows:
1) You create a class which inherits from NF_EventHandler. In there you can provide your own implementation of methods like tcpConnectRequest to allow you to redirect TCP connections somewhere else.
2) You initialize the library with a call to nf_init. This provides the link between the driver and your proxy, as you provide an instance of your NF_EventHandler implementation to it.
There are also some example programs for you to see the redirection happening. For example, to redirect a connection on port 80 from process id 214 to 127.0.0.0:8081, you can run:
TcpRedirector.exe -p 80 -pid 214 -r 127.0.0.1:8081
For your proxy, this would be used as follows:
1) Connect from your client application to the proxy.
2) The connection request is intercepted by NetFilterSDK (tcpConnectRequest) and the connection endpoint is modified to connect to the server the proxy chooses. This is the crucial bit because your connection is coming from outside and this is the part that may not work.
Sounds like routing problem, one layer lower than TCP/IP;
You're actually looking for ARP like proxy:
I'd say you need to manage ARP packets, chekcing the ARP requests:
CLIENT -> WHOIS PROXY.MAC
PROXY -> PROXY.IP is SERVER.IP
Then normal socket connection via TCP/IP from client to server.