After configuring application to work with TLS CPU consumption has got up to 10%.
I suppose it is because of TLS Handshake that happens every time.
On standalone environment I don't have such an effect. But when I am trying to use HAProxy LB it seems to me that session is cached for one node however when request came to another it need to perform handshake again.
How can I configure LB or tune it in order to avoid extra handshakes?
Tried to increase session cache it does't help.
tune.ssl.cachesize
tune.ssl.lifetime
Related
I have load runner app which is running some flow. Connection to my app is over https. Problem is that CPU is extremely high when TLS is enabled. I tried to configure keep alive setting and increase connection timeout - it didn't help.
Is there any way to disable handshake between load runner and my app, because it is not something I am testing. For example when i use "curl" I could use flag --insecure. I didn't sound such configuration in load runner.
Which CPU?
CPU on Server where TLS handshake is taking place?
CPU on load generator host, running exclusive of the controller, for a single virtual user and above?
CPU on combined controller, load generator for all virtual users?
As to disabling? It is recommended that your load use the same mechanism as users. If users connect insecurely in production, then by all means use the same HTTP (vs HTTPS) connection leveraged by end users. Otherwise the use of resources on your server infrastructure will be very different than production. This will make your test both less predictive and of lower diagnostic value for bottlenecks in resource utilization.
Our infrastructure is composed by
1 F5 load balancer
3 nodes
We have an application which uses websockets, so when a user goes to our site, it opens a websocket to the balancer which it connects to the first available node, and it works as expected.
Our truobles arrives with maintenance tasks, when we have to update our software, we need to turn offline 1 node at a time, deploy the new release and then turn it on again. Doing this task, the balancer drops the open websocket connections to the node and the clients retries to connect after few seconds to the first available nodes, creating an inconvenience for the client because he could miss a signal (or more).
How we can keep the connection between the client and the balancer, changing the backend websocket server? Is the load balancer enough to achieve our goal or we need to change our infrastructure?
To avoid this kind of problems I recommend to read about the Azure SignalR. With this you don't need to thing about stuff like load balancer, redis backplane and other infrastructures that you possibly need to a WebSockets connection.
Basically the clients will not connected to your node directly but redirected to Azure SignalR. You can read more about it here: https://learn.microsoft.com/en-us/azure/azure-signalr/signalr-overview
Since it is important to your application to maintain the connection, I don't see how any other way to archive no connection drop to your nodes, since you need to shut them down.
It's important to understand that the F5 is a full TCP proxy. This means that the F5 is the server to the client and the client to the server. If you are using the websockets protocol then you must apply a websockets profile to the F5 Virtual Server in order for the websockets application to be handled properly by the Load Balancer.
Details of the websockets profile can be found here: https://support.f5.com/csp/article/K14754
If a websockets and an HTTP profile are applied to the Virtual Server - meaning that you have websockets and web traffic using the same port and LB nodes - then the F5 will allow the websockets traffic as passthrough. Also keep in mind that if this is an HTTPS virtual sever that you will need to ensure a client and server side HTTPS profile (SSL offload) are applied to the Virtual Server.
While there are a variety of ways that you can fiddle with load balancers to minimize the downtime caused by a software upgrade, none of them solve the problem, which is that your application-layer protocol seems to not tolerate some small network outages.
Even if you have a perfect load balancer and your software deploys cause zero downtime, the customer's computer may be on flaky wifi which causes a network dropout for half a second - or going over ethernet and someone reconfigures some routing on their LAN, etc.
I'd suggest having your server maintain a queue of messages for clients (up to some size/time limit) so that when a client drops a connection - whether it be due to load balancers/upgrades - or any other reason, it can continue without disruption.
We run a latency-sensitive system. We found one significant cause of latency: some processes were making blocking DNS lookups to remote nameservers. To mitigate this, we have installed a local caching DNS resolver, specially dnsmasq.
But we still see occasional significant pauses where queries to the local DNS cache (dnsmasq) can take a long time. These are caused by TTL expiry; in these cases dnsmasq queries its upstream server before responding to the local process.
We would like to eliminate these pauses, too. I would like our local DNS cache to always respond immediately, even if the response is stale. The cache should query its upstream server asynchronously. For example, if the cache serves a stale response, it could refresh this asynchronously. Or a more sophisticated policy would be to refresh the cache asynchronously shortly before the TTL expires.
But I can't find any such setting for dnsmasq, or for any other caching DNS servers I've looked at. Are any DNS servers designed to run in this configuration?
Knot resolver with configuration modules = { 'predict' } will start asynchronous refresh of records that are put into answer at a moment when their TTL is close to expiration.
Note that version 2.0.0 has a bug that defeats this refresh for records without DNSSEC signatures (will be fixed in the next release).
Unbound DNS Server also does this with a prefetch option - yes/no.
We have a web server and a client, both written in go, that interact with each other. We want HAProxy to load balance requests between several instance of the server, but it's not working. The client will always connect to the same server while it's still up.
If I look at the output of "netstat -anp", I can see that there is a persistent connection that was established between the client and the sever through HAProxy. I tried setting the Connection Header in the response to "close", but that didn't work at all.
Needless to say, I'm completely confused by this. My first question is, is this a problem with the client, server, or HAProxy? How does one force the client to disconnect? Am I missing something regarding this? Curl works fine, so I know that HAProxy does load balance, but curl also completely shuts down when finished, hence why I'm suspecting it's the persistent connection that's causing me issues since the client and server are long running.
Just as an FYI, I'm using go-martini on the server.
Thanks.
HTTP/1.1 uses KeepAlive by default. Since the connections aren't closed, HAProxy can't balance the requests between different backends.
You have a couple options:
Force the connection to close after each request in your code. Setting Request.Close = true on either the client or the server will send a Connection: close header, telling both sides to close the tcp connection.
Alternatively you could have HAPoxy alter the requests by setting http-server-close so the backend is closed after each request, or http-closeto shutdown both sides after each request.
http-server-close is usually the best option, since that still maintains persistent connections for the client, while proxying each request individually.
I want to stop serving requests to my back end servers if the load on those servers goes above a certain level. Anyone who is already surfing the site will still get routed but new connection will be sent to a static server busy page until the load drops below a pre determined level.
I can use cookies to let the current customers in but I can't find information on how to to routing based on a custom load metric.
Can anyone point me in the right direction?
Nginx has an HTTP Upstream module for load balancing. Checking the responsiveness of the backend servers is done with the max_fails and fail_timeout options. Routing to an alternate page when no backends are available is done with the backup option. I recommend translating your load metrics into the options that Nginx supplies.
Let's say though that Nginx is still seeing the backend as being "up" when the load is higher than you want. You may be able to adjust that further by tuning the max connections of the backend servers. So, maybe the backend servers can only handle 5 connections before the load is too high, so you tune it only allow 5 connections. Then on the front-end, Nginx will time-out immediately when trying to send a sixth connection, and mark that server as inoperative.
Another option is to handle this outside of Nginx. Software like Nagios can not only monitor load, but can also proactively trigger actions based on the monitor it does.
You can generate your Nginx configs from a template that has options to mark each upstream node as up or down. When a monitor detects that the upstream load is too high, it could re-generate the Nginx config from the template as appropriate and then reload Nginx.
A lightweight version of the same idea could done with a script that runs on the same machine as your Nagios server, and performs simple monitoring as well as the config file updates.