We run a latency-sensitive system. We found one significant cause of latency: some processes were making blocking DNS lookups to remote nameservers. To mitigate this, we have installed a local caching DNS resolver, specially dnsmasq.
But we still see occasional significant pauses where queries to the local DNS cache (dnsmasq) can take a long time. These are caused by TTL expiry; in these cases dnsmasq queries its upstream server before responding to the local process.
We would like to eliminate these pauses, too. I would like our local DNS cache to always respond immediately, even if the response is stale. The cache should query its upstream server asynchronously. For example, if the cache serves a stale response, it could refresh this asynchronously. Or a more sophisticated policy would be to refresh the cache asynchronously shortly before the TTL expires.
But I can't find any such setting for dnsmasq, or for any other caching DNS servers I've looked at. Are any DNS servers designed to run in this configuration?
Knot resolver with configuration modules = { 'predict' } will start asynchronous refresh of records that are put into answer at a moment when their TTL is close to expiration.
Note that version 2.0.0 has a bug that defeats this refresh for records without DNSSEC signatures (will be fixed in the next release).
Unbound DNS Server also does this with a prefetch option - yes/no.
Related
I have a simple static website on Heroku free-tier. Heroku puts servers on the sleep after some time of no traffic, and then the first user coming again needs to wait some ˜30 seconds till the server boots again.
My question is whether having the website cached by Cloudflare would bridge this 30 second waiting time, please?
By default, Cloudflare respects the origin web server’s cache headers unless overridden via an Edge Cache TTL Page Rule.
You might be able to instruct Cloudflare to cache all of your static assets by making Your server respond with the right header. You will be able to clear Cloudflare’s cache from their dashboard every time you update your website but the website might still be cached forever on your past visitors' browsers.
You can bypass this problem by making your server respond with reasonable cache headers and instructing Cloudflare to cache it forever using their custom edge cache page rule (and clearing the cache manually every time you update your website)
Note that if your website uses non-static requests (I.e login, signup, filling forms) this won’t work at all
After configuring application to work with TLS CPU consumption has got up to 10%.
I suppose it is because of TLS Handshake that happens every time.
On standalone environment I don't have such an effect. But when I am trying to use HAProxy LB it seems to me that session is cached for one node however when request came to another it need to perform handshake again.
How can I configure LB or tune it in order to avoid extra handshakes?
Tried to increase session cache it does't help.
tune.ssl.cachesize
tune.ssl.lifetime
How reliable is DNS as a mechanism to switch between servers? With low TTL's, testing seems great, but I was wondering how reliable this would be for a public, production system?
My concern with this strategy is that I'm not sure if DNS record caching can be overridden by DNS proxies, and some providers may use this to save traffic. What if we use integration with other systems (e.g. a web service for mobile)?
While I understand how RFC compliant DNS is meant to work, I'm not actually sure how complaint networks are in general. (and DNS rr works great too for distribution, but this is specifically for switch-overs).
It's "reliable enough" for most of your audience.
Yes, recursive DNS resolvers do allow administrative override of TTLs (e.g.cache-min-ttl) that disobeys yours if you had set it too low. Also, there are software stacks that cache records forever in their default configuration (Java < 1.6).
You should always be prepared for some residual traffic to the old host even long after the switch over. In my experience, though, they're mostly poorly-written crawlers. If you want to be 100% certain of not missing any traffic, proxy all traffic from the old host to the new. Nginx/Apache can be easily made to do that.
You can query a recursive server for the remaining TTL that it is going to cache a record. Type this repeatedly, and you'll see that the TTL decreases with time:
dig #208.67.222.222 stackoverflow.com
Once the record expires, it should start at the TTL that you configured in your zone.
This way, you can at least test it against the public resolvers to see if it is obeying your TTL.
I want to stop serving requests to my back end servers if the load on those servers goes above a certain level. Anyone who is already surfing the site will still get routed but new connection will be sent to a static server busy page until the load drops below a pre determined level.
I can use cookies to let the current customers in but I can't find information on how to to routing based on a custom load metric.
Can anyone point me in the right direction?
Nginx has an HTTP Upstream module for load balancing. Checking the responsiveness of the backend servers is done with the max_fails and fail_timeout options. Routing to an alternate page when no backends are available is done with the backup option. I recommend translating your load metrics into the options that Nginx supplies.
Let's say though that Nginx is still seeing the backend as being "up" when the load is higher than you want. You may be able to adjust that further by tuning the max connections of the backend servers. So, maybe the backend servers can only handle 5 connections before the load is too high, so you tune it only allow 5 connections. Then on the front-end, Nginx will time-out immediately when trying to send a sixth connection, and mark that server as inoperative.
Another option is to handle this outside of Nginx. Software like Nagios can not only monitor load, but can also proactively trigger actions based on the monitor it does.
You can generate your Nginx configs from a template that has options to mark each upstream node as up or down. When a monitor detects that the upstream load is too high, it could re-generate the Nginx config from the template as appropriate and then reload Nginx.
A lightweight version of the same idea could done with a script that runs on the same machine as your Nagios server, and performs simple monitoring as well as the config file updates.
Is that called "clustering" of servers? When a web request is sent, does it go through the main server, and if the main server can't handle the extra load, then it forwards it to the secondary servers that can handle the load? Also, is one "server" that's up and running the application called an "instance"?
[...] Is that called "clustering" of servers?
Clustering is indeed using transparently multiple nodes that are seen as a unique entity: the cluster. Clustering allows you to scale: you can spread your load on all the nodes and, if you need more power, you can add more nodes (short version). Clustering allows you to be fault tolerant: if one node (physical or logical) goes down, other nodes can still process requests and your service remains available (short version).
When a web request is sent, does it go through the main server, and if the main server can't handle the extra load, then it forwards it to the secondary servers that can handle the load?
In general, this is the job of a dedicated component called a "load balancer" (hardware, software) that can use many algorithms to balance the request: round-robin, FIFO, LIFO, load based...
In the case of EC2, you previously had to load balance with round-robin DNS and/or HA Proxy. See Introduction to Software Load Balancing with Amazon EC2. But for some time now, Amazon has launched load balancing and auto-scaling (beta) as part of their EC2 offerings. See Elastic Load Balancing.
Also, is one "server" that's up and running the application called an "instance"?
Actually, an instance can be many things (depending of who's speaking): a machine, a virtual machine, a server (software) up and running, etc.
In the case of EC2, you might want to read Amazon EC2 Instance Types.
Here is a real example:
This specific configuration is hosted at RackSpace in their Managed Colo group.
Requests pass through a Cisco Firewall. They are then routed across a Gigabit LAN to a Cisco CSS 11501 Content Services Switch (eg Load Balancer). The Load Balancer matches the incoming content to a content rule, handles the SSL decryption if necessary, and then forwards the traffic to one of several back-end web servers.
Each 5 seconds, the load balancer requests a URL on each webserver. If the webserver fails (two times in a row, IIRC) to respond with the correct value, that server is not sent any traffic until the URL starts responding correctly.
Further behind the webservers is a MySQL master / slave configuration. Connections may be mad to the master (for transactions) or to the slaves for read only requests.
Memcached is installed on each of the webservers, with 1 GB of ram dedicated to caching. Each web application may utilize the cluster of memcache servers to cache all kinds of content.
Deployment is handled using rsync to sync specific directories on a management server out to each webserver. Apache restarts, etc.. are handled through similar scripting over ssh from the management server.
The amount of traffic that can be handled through this configuration is significant. The advantages of easy scaling and easy maintenance are great as well.
For clustering, any web request would be handled by a load balancer, which being updated as to the current loads of the server forming the cluster, sends the request to the least burdened server. As for if it's an instance.....I believe so but I'd wait for confirmation first on that.
You'd' need a very large application to be bothered with thinking about clustering and the "fun" that comes with it software and hardware wise, though. Unless you're looking to start or are already running something big, it wouldn't' be anything to worry about.
Yes, it can be required for clustering. Typically as the load goes up you might find yourself with a frontend server that does url rewriting, https if required and caching with squid say. The requests get passed on to multiple backend servers - probably using cookies to associate a session with a particular backend if necessary. You might have the database on a separate server also.
I should add that there are other reasons why you might need multiple servers, for instance there may be a requirement that the database is not on the frontend server for security reasons