There is a long delay between "forked new backend" and "connection received", from about 200 to 13000 ms. Postgres 12.2, Windows Server 2016.
During this delay the client is waiting for the network packet to start the authentication. Example:
14:26:33.312 CEST 3184 DEBUG: forked new backend, pid=4904 socket=5340
14:26:33.771 CEST 172.30.100.238 [unknown] 4904 LOG: connection received: host=* port=56983
This was discussed earlier here:
Postegresql slow connect time on Windows
But I have not found a solution.
After rebooting the server the delay is much shorter, about 50 ms. Then it gradually increases in the course of a few hours. There are about 100 clients connected.
I use ip addresses only in "pg_hba.conf". "log_hostname" is off.
There is BitDefender running on the server but switching it off did not help. Further, Postgres files are excluded from BitDefender checks.
I used Process Monitor which revealed the following: Forking the postgres.exe process needs 3 to 4 ms. Then, after loading DLLs, postgres.exe is looking for custom and extended locale info of 648 locales. It finds none of these. This locale search takes 560 ms (there is a gap of 420 ms, though). Perhaps this step can be skipped by setting a connection parameter. After reading some TCP/IP parameters, there are no events for 388 ms. This time period overlaps the 420 ms mentioned above. Then postgres.exe creates a thread. The total connection time measured by the client was 823 ms.
Locale example, performed 648 times:
"02.9760160","RegOpenKey","HKLM\System\CurrentControlSet\Control\Nls\CustomLocale","REPARSE","Desired Access: Read"
"02.9760500","RegOpenKey","HKLM\System\CurrentControlSet\Control\Nls\CustomLocale","SUCCESS","Desired Access: Read"
"02.9760673","RegQueryValue","HKLM\System\CurrentControlSet\Control\Nls\CustomLocale\bg-BG","NAME NOT FOUND","Length: 532"
"02.9760827","RegCloseKey","HKLM\System\CurrentControlSet\Control\Nls\CustomLocale","SUCCESS",""
"02.9761052","RegOpenKey","HKLM\System\CurrentControlSet\Control\Nls\ExtendedLocale","REPARSE","Desired Access: Read"
"02.9761309","RegOpenKey","HKLM\System\CurrentControlSet\Control\Nls\ExtendedLocale","SUCCESS","Desired Access: Read"
"02.9761502","RegQueryValue","HKLM\System\CurrentControlSet\Control\Nls\ExtendedLocale\bg-BG","NAME NOT FOUND","Length: 532"
"02.9761688","RegCloseKey","HKLM\System\CurrentControlSet\Control\Nls\ExtendedLocale","SUCCESS",""
No events for 388 ms:
"03.0988152","RegCloseKey","HKLM\System\CurrentControlSet\Services\Tcpip6\Parameters\Winsock","SUCCESS",""
"03.4869332","Thread Create","","SUCCESS","Thread ID: 2036"
Related
I try to get a system working with a modem and the following services :
permanent Data connection (2G/3G/4G) using old serial interface (not qmi/wmi)
permanent polling in AT command for getting SMS and get monitoring information like signal, provider, cell...
Does this modem can be connected 24/24 without any interruption ? Actually I have fast every day one or two small interruption of data (1 to 10 minutes).
The main question is, does a Modem can be connected to the provider 24/24 without interrupt, or it's a standard behavior to be disconnected sometimes ?
Additional note : I have multiples devices, and the PPPD binary fail in 10 minutes on all devices.
I have collect some logs, and I can see the system is disconnected each ~720 minutes = 12 hours :
Sep 14 20:23:02 daemon.info pppd[1905]: Connect time 718.6 minutes.
Sep 15 08:23:02 daemon.info pppd[19903]: Connect time 719.9 minutes.
Sep 15 20:23:03 daemon.info pppd[2493]: Connect time 719.9 minutes.
Sep 16 08:23:03 daemon.info pppd[16865]: Connect time 719.9 minutes.
Sep 16 20:23:03 daemon.info pppd[31234]: Connect time 719.8 minutes.
Sep 17 08:23:03 daemon.info pppd[13827]: Connect time 719.8 minutes.
Depending of the provider / cells. The system will de-register your modem each X hours (in my case 12 hours).
The best solution I have found is to set the option --persist on pppd. The system take ~1 minutes to re-register, but PPPD stay alive.
I am getting the Socket Exception while running load test on self-provisioned test rig.
I am trigger those load tests in agent machine(self-provisioned test rig) from my local machine.
Note : For first 2 to 3 minutes test iterations are passing , after that we are getting the Socket Exception.
Below is the error message :
A connection attempt failed because the connected party did not
properly respond after a period of time, or established connection
failed because connected host has failed to respond.
Below are the stack trace details :
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult) at
System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure,
Socket s4, Socket s6, Socket& socket, IPAddress& address,
ConnectSocketState state, IAsyncResult asyncResult, Exception&
exception)
Run Time - 20min
Sample rate - 10sec
warm up duration 10sec
number of agents used - 2
Load pattern :
initial load - 10user
max user count - 300
step duration - 10sec
step user count - 10
Although, by Changing above values I am still getting the exception in the same way.
I am using Visual studio 2015 enterprise.
The question states: start with 10 users, every 10 seconds add 10 users to a maximum of 300. Thus after 29 increments there will be 300 users and that will take 29*10 seconds which is 4m50s. The test will thus (attempt to) run with the maximum load of 300 users for the remaining 15m10s.
Given that all tests pass for the first 2 or 3 minutes plus the the error message, that suggests that you are overloading some part of the network. It might be the agents, it might be the servers or it might be on the connections between them. Some network components have a maximum number of active connections and the 300 users might be too many.
Increasing the load so rapidly means you do not clearly know what the limiting value. The sampling rate (at 10 seconds) seems high. At each sampling interval a lot of data is transferred (i.e. the sample data) and that can swamp parts of the network. You should look at the network counters for the agents and controller, also the servers if available.
I recommend changing the load test steps to add 10 users every 30 seconds, so it takes about 15 minutes to reach 300 users. It may also be worth reducing the sample rate to every 20 seconds.
I have used IdMappedPortTCP in a particular program to allow generic port-forwarding for many years. I'm testing an upgraded build/component environment, and I ran into a problem. First, here's the old & new version info:
OS: W2kSP4 --> same (Hey, why is everyone laughing?)
Delphi: 5 --> 7
Project Indy: 9.0.0.14 --> 9.[Latest SVN]
I'm testing it by inserting it in a telnet session using the standard Windows console telnet client, and a Linux server, and I'm seeing an odd change in behavior.
Direct connection: Client connects, immediately sees server greeting
Old Indy: Same as direct
New Indy: Client connects, sees nothing. Press key, sees server greeting + keystroke.
Here's a comparison of the event chain:
Old:
6/08/2017 6:47:16 PM - DEBUG: MappedPort-Connect
6/08/2017 6:47:16 PM - TCP Port Fwd: Connect: 127.0.0.1:4325 --> 127.0.0.1:23
6/08/2017 6:47:16 PM - DEBUG: MappedPort-OutboundConnect
6/08/2017 6:47:16 PM - TCP Port Fwd: Outbound Connect: 192.168.214.11:4326 --> 192.168.210.101:23
6/08/2017 6:47:16 PM - DEBUG: MappedPort-OutboundData
6/08/2017 6:47:16 PM - DEBUG: MappedPort-Execute
6/08/2017 6:47:16 PM - DEBUG: MappedPort-OutboundData
6/08/2017 6:47:16 PM - DEBUG: MappedPort-Execute
6/08/2017 6:47:16 PM - DEBUG: MappedPort-OutboundData
...
New:
6/08/2017 6:41:34 PM - DEBUG: MappedPort-Connect
6/08/2017 6:41:34 PM - TCP Port Fwd: Connect: 127.0.0.1:1085 --> 127.0.0.1:23
6/08/2017 6:41:34 PM - DEBUG: MappedPort-OutboundConnect
6/08/2017 6:41:34 PM - TCP Port Fwd: Outbound Connect: 192.168.214.59:1086 --> 192.168.210.101:23
6/08/2017 6:47:36 PM - DEBUG: MappedPort-Execute
6/08/2017 6:47:36 PM - DEBUG: MappedPort-OutboundData
6/08/2017 6:47:36 PM - DEBUG: MappedPort-Execute
6/08/2017 6:47:36 PM - DEBUG: MappedPort-OutboundData
6/08/2017 6:47:36 PM - DEBUG: MappedPort-Execute
In the first one, you see OutboundData right after connecting. In the second, nothing happens after connect until I sent a keystroke (6 minutes later), at which time you see the Execute and then the first OutboundData event.
This caused me to wonder: Is it really connecting to the server and only delaying the output, or is the connection itself being delayed?
My first conclusion was that the connection itself was being delayed, and here's why. The server has a 1-minute timeout at the login prompt. If you connect and get the greeting but just sit there, the server disconnects after a minute. With the new Indy version, I sat there after the connect event for 6 full minutes, then got the server greeting with no problem.
However... NETSTAT shows the connection to the remote server established soon after the connection event is logged! So, I'm left to conclude that the connection is indeed established, but perhaps some initial character is being "eaten" or something that's causing getty to not engage until it gets a keystroke?
Any suggestions? Are you aware of anything that changed that I might look for -- something that I ought to be doing but am not? Any insights are appreciated.
(Barring any good leads, I guess the next step in my sleuthing might be to sniff both machines w/ WireShark to see what's going on after the connection.)
Update: Wireshark (single leg)
A packet capture from outside the machines showing traffic between MappedPort & the server (but not traffic between the client & MappedPort) shows that the telnet server sends a "Do Authenticate", to which the client (via MappedPort) replies w/ a "Will Authenticate". This is followed by the server sending authenticate suboption (and the client agreeing) then all the other telnet options. Finally, after seeing the login text, the client sends "do echo" and they both sit there until after 1min, at which time the server sends a TCP FIN to close the connection. That's the "good old" version.
On the new version, the client doesn't respond to the "Will Authenticate", and they both sit there indefinitely. (Hmmm, I wonder what that ties up in terms of server resources -- could be good DOS attack. It is an old telnet daemon, though, so it's probably been fixed by now...) When I finally sent the first keystroke, that's all it sent in that packet. THEN the client sends the "will authenticate" (without additional prodding from the server) and the negotiation continues exactly as normal; the last packet from the server (containing echo parameters) also then includes the echoed character which was typed. So it's like the client doesn't see the initial "do authenticate" packet from the server, but once you start typing, goes ahead and responds as if it had just heard it (once it sends the keystroke).
6/13 Update: Wireshark (both legs)
I captured both legs of the "broken" conversation and analyzed it. Interesting behavior. Bottom line:
As soon as the the server gets the TCP connection, it sends back a Telnet-DoAuth invitation. IdMappedPortTCP holds onto that packet and doesn't pass it on to the client -- yet. Once the client finally sends the first keystroke (seconds or minutes later), Id passes it on to the server. THEN Id passes the DoAuth packet that it got from the server on to the client.
Here's a more detailed accounting of the packets:
65 11-59 TCP Syn
67 59-11 TCP SynAck
69 11-59 TCP Ack
71 59-101 TCP Syn
73 101-59 TCP SynAck
74 59-101 TCP Ack
76 101-59 DoAuth
77 59-101 TCP Ack
nothing for 23 seconds
79 11-59 Data:\r\n (I pressed Enter)
81 59-101 Data:\r\n
83 59-11 DoAuth
85 11-59 WillAuth
87 101-59 TCP Ack
88 59-101 WillAuth
90 101-59 TCP Ack
91 101-59 Authentication option
92 59-11 Authentication option
94 11-59 Authentication option reply
96 59-101 Authentication option reply
98 101-59 Will/do Encryption/terminal/env options
99 59-101 Will/do Encryption/terminal/env options
101 11-59 Don't encrypt
103 59-101 Don't encrypt
105 101-59 TCP Ack
106 59-11 TCP Ack
108 11-59 Won't/will list
110 59-101 Won't/will list
112 101-59 TCP Ack
113 101-59 Do window size
114 59-11 Do window size
Packet dump line format: Pkt# From-To Payload
(Don't mind the packet# skips; the client & proxy are both running on VMs hosted by the machine where I was running the capture from, so Wireshark saw two copies of packets. I only included the pkt# so I can reference the original dump later if I want.)
From/To Machines:
10 = Linux client (see below)
11 = Windows client
59 = proxy
101 = server
An Interesting diversion: Linux Client
Though all my testing has been using various Windows clients (because that's what's used in production), I "accidentally" used Linux (because that's what I run on my workstation, where I ran Wireshark) because it was convenient. That client behaves differently -- more aggressively -- and thus avoids the problem. Here's what a dump of that looks like:
1 10-59 TCP Syn
2 59-10 TCP SynAck
3 10 59 TCP Ack
4 10-59 Do/Will list
5 59-101 TCP Syn
7 101-59 TCP SynAck
8 59-101 TCP Ack
10 59-101 Do/Will list
12 101-59 TCP Ack
13 101-59 DoAuth
14 59-10 DoAuth
15 10-59 TCP Auth
16 10-59 WontAuth
17 59-101 WontAuth
19 101-59 Will/Do list
20 59-10 Will/Do list
21 10-50 Do window size
22 59-101 Do window size
As you can see, the client doesn't wait for the telnet server to speak first -- as soon as the TCP connection is established, it sends a full Do/Will list. This is in turn passed onto the server once Id opens that connection. The server sends back the same "DoAuth" that it did initially before; the difference being that this time, having already passed traffic from the client, Id passes it on immediately. The client then sends auth flags, and things move right along.
So, if the client speaks first, IdMappedPortTCP does okay; it's only when the server speaks first that it holds onto its message and doesn't pass it on to the client until the client says something.
9/27 Update: Found the Code Change
Downgrading to 9.0.0.14 fixed the problem. Comparing the two versions' source code for IdMappedPortTCP.pas I found that the only difference is that the newer version added a block of code to procedure TIdMappedPortThread.OutboundConnect:
DoOutboundClientConnect(Self);
FNetData := Connection.CurrentReadBuffer;
if Length(FNetData) > 0 then begin
DoLocalClientData(Self);
FOutboundClient.Write(FNetData);
end;//if
except
(The first and last lines existed already, and are only shown for context.)
I confirmed that adding that code to 9.0.0.14 produced the problem.
I checked the SVN repo, and you added the offending code on 9/7/2008. The commit comment is:
Updated TIdMappedPortThread.OutboundConnect() to check for pending
data in the inbound client's InputBuffer after the OnOutboundConnect
event handler exits.
I don't fully understand the reason for or implications of the change -- obviously you had a good reason for doing it -- but it does appear to produce the effect I described ("holding onto" the server's initial output until the client sends something).
In Indy 9, TIdTCPConnection.CurrentReadBuffer() calls TIdTCPConnection.ReadFromStack() before then returning whatever data is stored in the TIdTCPConnection.InputBuffer property:
function TIdTCPConnection.CurrentReadBuffer: string;
begin
Result := '';
if Connected then begin
ReadFromStack(False); // <-- here
end;
Result := InputBuffer.Extract(InputBuffer.Size);
end;
Regardless of what may already be in the InputBuffer, ReadFromStack() waits for the socket to receive new data to append to the InputBuffer. It does not exit until new data actually arrives, or the specified ReadTimeout interval elapses. The TIdTCPConnection.ReadTimeout property is set to 0 by default, so when CurrentReadBuffer() calls ReadFromStack(), it ends up using an infinite timeout:
function TIdTCPConnection.ReadFromStack(const ARaiseExceptionIfDisconnected: Boolean = True;
ATimeout: Integer = IdTimeoutDefault; const ARaiseExceptionOnTimeout: Boolean = True): Integer;
// Reads any data in tcp/ip buffer and puts it into Indy buffer
// This must be the ONLY raw read from Winsock routine
// This must be the ONLY call to RECV - all data goes thru this method
var
i: Integer;
LByteCount: Integer;
begin
if ATimeout = IdTimeoutDefault then begin
if ReadTimeOut = 0 then begin
ATimeout := IdTimeoutInfinite; // <-- here
end else begin
ATimeout := FReadTimeout;
end;
end;
...
end;
So, when TIdMappedPortTCP.OutboundConnect() calls CurrentReadBuffer() after connecting its OutboundClient to the server, it does indeed wait for data to arrive from the client before then reading data from the server. To avoid that, you can set a non-infinite ReadTimeout value in the TIdMappedPortTCP.OnConnect or TIdMappedPortTCP.OnOutboundConnect event, eg:
AThread.Connection.ReadTimeout := 1;
In Indy 10, this problem was fixed in TIdMappedPortTCP by avoiding this initial wait on the client data after connecting to the server. I have now updated TIdMappedPortTCP in Indy 9 to do the same.
I am building an autocomplete functionality and realized the amount of time taken between the client and server is too high (in the range of 450-700ms)
My first stop was to check if this is result of server delay.
But as you can see these Nginx logs are almost always 0.001 milliseconds (request time is the last column). It’s hardly a cause of concern.
So it became very evident that I am losing time between the server and the client. My benchmarks are Google Instant's response times. Which almost often is in the range of 30-40 milliseconds. Magnitudes lower.
Although it’s easy to say that Google's has massive infrastructural capabilities to deliver at this speed, I wanted to push myself to learn if this is possible for someone who is not that level. If not 60 milliseconds, I want to shave off 100-150 milliseconds.
Here are some of the strategies I’ve managed to learn.
Enable httpd slowstart and initcwnd
Ensure SPDY if you are on https
Ensure results are http compressed
Etc.
What are the other things I can do here?
e.g
Does have a persistent connection help?
Should I reduce the response size dramatically?
Edit:
Here are the ping and traceroute numbers. The site is served via cloudflare from a Fremont Linode machine.
mymachine-Mac:c name$ ping site.com
PING site.com (160.158.244.92): 56 data bytes
64 bytes from 160.158.244.92: icmp_seq=0 ttl=58 time=95.557 ms
64 bytes from 160.158.244.92: icmp_seq=1 ttl=58 time=103.569 ms
64 bytes from 160.158.244.92: icmp_seq=2 ttl=58 time=95.679 ms
^C
--- site.com ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 95.557/98.268/103.569/3.748 ms
mymachine-Mac:c name$ traceroute site.com
traceroute: Warning: site.com has multiple addresses; using 160.158.244.92
traceroute to site.com (160.158.244.92), 64 hops max, 52 byte packets
1 192.168.1.1 (192.168.1.1) 2.393 ms 1.159 ms 1.042 ms
2 172.16.70.1 (172.16.70.1) 22.796 ms 64.531 ms 26.093 ms
3 abts-kk-static-ilp-241.11.181.122.airtel.in (122.181.11.241) 28.483 ms 21.450 ms 25.255 ms
4 aes-static-005.99.22.125.airtel.in (125.22.99.5) 30.558 ms 30.448 ms 40.344 ms
5 182.79.245.62 (182.79.245.62) 75.568 ms 101.446 ms 68.659 ms
6 13335.sgw.equinix.com (202.79.197.132) 84.201 ms 65.092 ms 56.111 ms
7 160.158.244.92 (160.158.244.92) 66.352 ms 69.912 ms 81.458 ms
mymachine-Mac:c name$ site.com (160.158.244.92): 56 data bytes
I may well be wrong, but personally I smell a rat. Your times aren't justified by your setup; I believe that your requests ought to run much faster.
If at all possible, generate a short query using curl and intercept it with tcpdump on both the client and the server.
It could be a bandwidth/concurrency problem on the hosting. Check out its diagnostic panel, or try estimating the traffic.
You can try and save a response query into a static file, then requesting that file (taking care as not to trigger the local browser cache...), to see whether the problem might be in processing the data (either server or client side).
Does this slowness affect every request, or only the autocomplete ones? If the latter, and no matter what nginx says, it might be some inefficiency/delay in recovering or formatting the autocompletion data for output.
Also, you can try and serve a static response bypassing nginx altogether, in case this is an issue with nginx (and for that matter: have you checked out nginx' error log?).
One approach I didn't see you mention is to use SSL sessions: you can add the following into your nginx conf to make sure that an SSL handshake (very expensive process) does not happen with every connection request:
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
See "HTTPS server optimizations" here:
http://nginx.org/en/docs/http/configuring_https_servers.html
I would recommend using New Relic if you aren't already. It is possible that the server-side code you have could be the issue. If you think that might be the issue, there are quite a few free code profiling tools.
You may want to consider an option to preload autocomplete options in the background while the page is rendered and then save a trie or whatever structure you use on the client in the local storage. When the user starts typing in the autocomplete field you would not need to send any requests to the server but instead query local storage.
Web SQL Database and IndexedDB introduce databases to the clientside.
Instead of the common pattern of posting data to the server via
XMLHttpRequest or form submission, you can leverage these clientside
databases. Decreasing HTTP requests is a primary target of all
performance engineers, so using these as a datastore can save many
trips via XHR or form posts back to the server. localStorage and
sessionStorage could be used in some cases, like capturing form
submission progress, and have seen to be noticeably faster than the
client-side database APIs.
For example, if you have a data grid component or an inbox with
hundreds of messages, storing the data locally in a database will save
you HTTP roundtrips when the user wishes to search, filter, or sort. A
list of friends or a text input autocomplete could be filtered on each
keystroke, making for a much more responsive user experience.
http://www.html5rocks.com/en/tutorials/speed/quick/#toc-databases
I've been developing a web-site that uses Django and MySQL; what I want to know is how many HTTP requests my server can handle serving certain pages.
I have been using siege but I'm not sure if that's a good benchmarking tool.
ab, the Apache HTTP server benchmarking tool. Many options. An example of use with ten concurrent requests:
% ab -n 20 -c 10 http://www.bortzmeyer.org/
...
Benchmarking www.bortzmeyer.org (be patient).....done
Server Software: Apache/2.2.9
Server Hostname: www.bortzmeyer.org
Server Port: 80
Document Path: /
Document Length: 208025 bytes
Concurrency Level: 10
Time taken for tests: 9.535 seconds
Complete requests: 20
Failed requests: 0
Write errors: 0
Total transferred: 4557691 bytes
HTML transferred: 4551113 bytes
Requests per second: 2.10 [#/sec] (mean)
Time per request: 4767.540 [ms] (mean)
Time per request: 476.754 [ms] (mean, across all concurrent requests)
Transfer rate: 466.79 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 22 107 254.6 24 854
Processing: 996 3301 1837.9 3236 8139
Waiting: 23 25 1.3 25 27
Total: 1018 3408 1795.9 3269 8164
Percentage of the requests served within a certain time (ms)
50% 3269
66% 4219
...
(In that case, network latency was the main slowness factor.)
ab reports itself in the User-Agent field so, in the log of the HTTP server, you'll see something like:
2001:660:3003:8::4:69 - - [28/Jul/2009:12:22:45 +0200] "GET / HTTP/1.0" 200 208025 "-" "ApacheBench/2.3" www.bortzmeyer.org
ab is a widely used benchmarking tool that comes with apache httpd
Grinder is pretty good. It lets you simulate coordinated load from several client machines, which is more meaningful than from a single machine.
There's also JMeter.
I've used httperf and it's quite easy to use. There's a peepcode screencast on how to use it as well.