I had some networking issues on my Windows server, and find out (by using NETSTAT) that I have more than 90,000 (!) connections in TIME_WAIT which didn't closed.
I've changed the TcpTimedWaitDelay param in registry, but apparently a server restart is needed.
Because it's a single production DB server, I can't afford it at the moment.
Is there any way killing a TIME_WAIT connection?
Any other suggestions?
Thanks!
Roei
I have more than 90,000 (!) connections in TIME_WAIT which didn't closed.
No you don't. These represent connections which have already closed, and whose local port is hanging around for TCP security reasons. They will only be that way for a couple of minutes each. Just wait.
Related
Just cross-posting this from github.
I am using xorm 0.4.3 with go-mysql. We are on Golang 1.4.
We have specified maxIdleConnetions and maxOpenConnections in xorm as below:-
var orm *xorm.Engine
...
orm.SetMaxOpenConns(50)
orm.SetMaxIdleConns(5)
And we are using the same single xorm instance to query Mysql.
But still we are seeing lot of connections in TCP Connection Establised state which is way over the numbers I have configured in maxIdleConnetions and maxOpenConnections state when we lsof:-
app 8747 10568 sandeshsharma 16u IPv4 691032 0t0 TCP 127.0.0.1:57337->127.0.0.1:mysql (ESTABLISHED)
We have also observed that even if we stop the MySQL, the connection numbers still remain fixed but in the CLOSED_WAIT state. If we shutdown the app then all connections go away.
app 8747 10844 sandeshsharma 38u IPv4 505058 0t0 TCP 127.0.0.1:54160->127.0.0.1:mysql (CLOSE_WAIT)
However in mysql process list it is showing the correct number of connections as I have specified in maxIdleConnetions and maxOpenConnections.
Can some one please explain me this behaviour? Why are we observing so much TCP connections even though we have specified maxIdleConnetions and maxOpenConnections to 5 & 50 respectively?
First of all, Go 1.4 is too old. Use latest Go 1.6.
This answer is written with knowledge of Go 1.6. So some details may be different in your case.
There are four state in connection: connecting, idle, inuse, and closing.
MaxOpenConnections limits number of connection in connecting, idle, inuse state.
So, if your application closes and reopen connection quickly, it can happen.
Since TCP is CLOSED_WAIT state in MySQL server side, your app is waiting EOF from connection. I suppose your app is under very high load
and slow at reading EOF and closing connection. Until read EOF and close connection, TCP state is ESTABLISHED on client side, regardless TCP state in server side.
I recommend you to update Go and "go-sql-driver/mysql", and set MaxIdleConns equal to MaxOpenConns to avoid high reconnect rate.
Instead, you can use SetConnMaxLifetime (new API in Go 1.6) to close
connections when your app is idle.
Is it possible that windows leaks sockets connection and these sockets are not shown in tcpview and netstat?
After running a few applications that perform many network connections, my windows machine enters a state in whitch it in not able to open any new socket connection. Even to itself (localhosts).
For example, telnet to a local application failed because windows can't create new sockets.
Closing and restarting the network applications does not helps. Only full windows restart solves the problem.
netstat (& tcpview) indicates that there are only some dozens of connections.
Thanks for your help.
No, it is not possible for those apps to miss leaked connections. Something else is going on. Maybe you are not looking at their detailed views, like seeing closed sockets that are in TIME_WAIT state. If you cannot open new socket connections, you mostly likely are encountering port exhaustion. Wait some time for ports to time out and become available again. Or stop wasting ports in the first place.
Ive searched internet but didnt got the answer can any1 explain me the difference between them
A TCP "connection" is a 4-tuple. Local IP, Local Port, Remote IP, and Remote Port. Each end maintains this identification within its TCP stack, with the senses reversed (Local vs. Remote).
The combination of these 4 values must be unique. This explains the problems people often have writing a TCP client that reuses a socket to reconnect to the same server.
A "closed" connection leaves this ID in the tables at each end for some time, in TIME_WAIT state. This is an artifact of a TCP mechansim that deals with maintaining connection integrity even if the physical layer connection breaks, keeps pending packets from being recevied by a second connection, etc. TIME_WAIT can last up to 4 minutes.
Unless the client resets its socket's LocalPort to 0 (which is a request for an automatic ephemeral port assignment) it can fail if it tries to reconnect before TIME_WAIT expires. Since this is 0 for a newly created socket, programmers often overlook this requirement prior to calling Connect.
LocalPort isn't just an issue for listening sockets.
A server listens on a localport, while a client sends data from the localport.
The client remoteport should be the same as the server localport.
i.e.:
Server listens on port n (local port relative to server)
Client connects to server on port n (remote port relative to client)
To answer your question, the difference is in name, based on perspective.
This seems to be a good place to start with VB6 socket communication
I see a number of connections with CLOSE_WAIT status on my production server, and there are few questions. Please advice.
I know that the windows register has the following parameter:
TcpTimedWaitDelay in the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
According to the Microsoft stuff (http://social.technet.microsoft.com/Forums/en-US/windowsserver2008r2networking/thread/4288d218-fbf9-4489-b869-384a05dea83d/) , the value of the TIME_WAIT by default is 4 minutes. I set up the value in 30 seconds, but it looks like nothing is changed. Moreover, even after 4 mins CLOSE_WAIT connections are still there.
My questions is how I can change the value of the TIME_WAIT, and see it in action? Should I restart my server?
Is there any other settings to control the CLOSE_WAIT status in the Windows environment?
Regards,
Cyril
CLOSE_WAIT means that the peer has closed the connection and you haven't. The operating system is waiting for you (the local application) to close it.
So close it. Somewhere or other you have missed out.
You're confusing CLOSE_WAIT and TIME_WAIT. They're not the same.
See here: http://www.serverframework.com/asynchronousevents/2011/01/time-wait-and-its-design-implications-for-protocols-and-scalable-servers.html for some details on TIME_WAIT and why you might not really want to play with shortening the timeout.
And see here: http://www.sunmanagers.org/pipermail/summaries/2006-January/007068.html and here: http://blogs.msdn.com/b/spike/archive/2008/10/09/tcp-connections-hanging-in-the-close-wait-and-fin-wait-2-state.aspx for details on why you might be collecting sockets stuck in CLOSE_WAIT - in summary, you're possibly not closing your sockets correctly.
I Will suggest you to use Spring WS because I was facing same issue in my project and I switched to Spring Web Service than my problem resolved.
See the following code,
<bean id="viewCustomerInfo"
class="org.springframework.remoting.jaxws.JaxWsPortProxyFactoryBean"
p:serviceInterface="com.javaplex.CustomerInfoInterface"
p:wsdlDocumentUrl="http://127.0.0.1:8080/portal/CustomerInfoPort?wsdl"
p:namespaceUri="http://ws.customergen.com/" p:serviceName="CustomerInfo"
p:portName="CustomerInfoPort">
here is the complete article how to setup spring based beans which give your optimum performance.
http://www.javaplex.com/spring-jax-ws-client-for-best-performance/
Why is it that TCP connections to a loopback interface end up in TIME_WAIT (socket closed with SO_DONTLINGER set), but identical connections to a different host do not end up in TIME_WAIT (they are reset/destroyed immediately)?
Here are scenarios to illustrate:
(A) Two applications, a client and a server, are both running on the same Windows machine. The client connects to the server via the server's loopback interface (127.0.0.1, port xxxx), sends data, receives data, and closes the socket (SO_DONTLINGER is set).
Let's say that the connections are very short-lived, so the client app is establishing and destroying a large number of connections each second. The end result is that the sockets end up in TIME_WAIT, and the client eventually exhausts its max number of sockets (on Windows, this is ~3900 by default, and we are assuming that this value will not be changed in the registry).
(B) Same two applications as scenario (A), but the server is on a different host (the client is still running on Windows). The connections are identical in every way, except that they are not destined for 127.0.0.1, but some other IP instead. Here the connections on the client machine do NOT go into TIME_WAIT, and the client app can continue to make connections indefinitely.
Why the discrepancy?
The TIME_WAIT state only occurs at one end of the connection -- the end that closes first. For the loopback interface both ends are on the same machine so you will always see TIME_WAIT.
In your other case, try looking at the other machine. I think you'll see the TIME_WAIT sockets there.