I would like to see how a program responds when it's connection is severed. Aside from disabling the network card, is there a way to sever a tcp connection in Windows without killing the process, or the thread that owns the connections?
The closest thing that I've found to generating an OS error is to use something like TcpView to look at what sockets are open and sever them. I'm not sure exactly what it does to sever the connection, but it does close it in a way that an application can see.
TCPView by SysInternals lets you close a connection (and see all open connections).
One thing I've seen done is to have the network code written in such a way that a connection can be severed remotely. A product I once worked on was written that way. We even had a set of torture tests that would randomly break the connections. The product was meant to be transactional, and it was instructive to see how it behaved.
Of course, we then found a customer whose network was actually breaking connections all the time, and were very glad we'd tested so hard.
Why not just unplug the network cable?
Related
Is it possible that windows leaks sockets connection and these sockets are not shown in tcpview and netstat?
After running a few applications that perform many network connections, my windows machine enters a state in whitch it in not able to open any new socket connection. Even to itself (localhosts).
For example, telnet to a local application failed because windows can't create new sockets.
Closing and restarting the network applications does not helps. Only full windows restart solves the problem.
netstat (& tcpview) indicates that there are only some dozens of connections.
Thanks for your help.
No, it is not possible for those apps to miss leaked connections. Something else is going on. Maybe you are not looking at their detailed views, like seeing closed sockets that are in TIME_WAIT state. If you cannot open new socket connections, you mostly likely are encountering port exhaustion. Wait some time for ports to time out and become available again. Or stop wasting ports in the first place.
I'm using beanstalkd to offload some work to other machines. The setup is a bit unusual, the server is on the internet (public ip) but the consumers are behind adsl lines on some peoples homes. So there is a linux server as client going out through a dynamic ip and connecting to the server to get a job. It's all PHP and I'm using pheanstalk library.
Everything runs smoothly for some time, but then the adsl changes the IP (every 24h hours the provider forces a disconnect-reconnect) the client just hangs, never to go out of "reserve".
I thought that putting a timeout on the reserve would help it, but it didn't. As it seems, the client issues a command and blocks, it never checks the timeout. It just issues a reserve-with-timeout (instead of a simple reserve) and it is the servers responsibility to return a TIME_OUT as the timeout occurs. The problem is, the connection is broken (but the TCP/IP doesn't know about that yet until any of the sides try to talk to the other side) and if the client blocked reading, it will never return.
The library seems to have support for some kind of timeouts locally (for example when trying to connect to server), but it does not seem to contemplate this scenario.
How could I detect the stale connection and force a reconnect? Is there some kind of keepalive on the protocol (and on the pheanstalk itself)?
Thanks!
You could try to close each connection right after the request is answered and reopen a new connection each time.
There is no close() function but you deleting the Pheanstaly Object with unset($pheanstalk) will close it.
This explanation is quite helpful:
Pheanstalk (PHP client for beanstalk) - how do connections work?
I haven't tried it yet, but I came up with the idea of connecting to the beanstalk server through an SSH tunnel. We can enable the ServerAliveCountMax and ServerAliveInterval options on the tunnel, so that a network or server failure will cause the tunnel to close. This should then cause the pheanstalk client to report an error.
I am trying to reverse engineer a third-party TCP client / server Windows XP, SP 3 app for which I have no source available. My main line of attack is to use WireShark to capture TCP traffic.
When I issue a certain GUI command on the client side, the client creates a TCP connection to the server, sends some data, and tears down the connection. The server port is 1234, and the client port is assigned by the OS and therefore varies.
WireShark is showing that the message corresponding to the GUI command I issued gets sent twice. The two messages bear a different source port, but they have the same destination port (1234, as mentioned previosuly).
The client side actually consists of several processes, and I would like to determine which processes are sending these messages. These processes are long-lived, so their PIDs are stable and known. However, the TCP connections involved are transient, lasting only a few milliseconds or so. Though I've captured the client-side port numbers in WireShark and though I know all of the PIDs involved, the fact the connections are transient makes it difficult to determine which PID opened the port. (If the connections were long-lived, I could use netstat to map port numbers to PIDs.) Does anybody have any suggestions on how I can determine which processes are creating these transient connections?
I can think of two things:
Try sysinternals' tcpview program. It gives a detailed listing of all tcp connections opened by all the processes in the system. If a process creates connections, you will be able to see them flash (both connect and disconnect are flashed) in tcpview and you will know which processes to start looking into.
Try running the binary under a debugger. Windbg supports multi-process debugging (so does visual studio I think). You may have only export symbols to work with but that should still work for calls made to system dlls. Try breaking on any suspected windows APIs you know will be called by the process to create the connections. MSDN should have the relevant dlls for most system APIs documented.
Start here... post a follow-up if you get stuck again.
I ended up creating a batch file that runs netstat in a tight loop and appends its output to a text file. I ran this batch file while running the system, and by combing through all of the netstat dumps, I was able to find a dump that contained the PIDs associated with the ports.
stupid problem. I get those from a client connecting to a server. Sadly, the setup is complicated making debugging complex - and we run out of options.
The environment:
*Client/Server system, both running on the same machine. The client is actually a service doing some database manipulation at specific times.
* The cnonection comes from C# going through OleDb to an EasySoft JDBC driver to a custom written JDBC server that then hosts logic in C++. Yeah, compelx - but the third party supplier decided to expose the extension mechanisms for their server through a JDBC interface. Not a lot can be done here ;)
The Symptom:
At (ir)regular intervals we get a "Address already in use: connect" told from the JDBC driver. They seem to come from one particular service we run.
Now, I did read all the stuff about port exhaustion. This is why we have a little tool running now that counts ports and their states every minute. Last time this happened, we had an astonishing 370 ports in use, with the count rising to about 900 AFTER the error. We aleady patched the registry (it is a windows machine) to allow more than the 5000 client ports standard, but even then, we are far far from that limit to start with.
Which is why I am asking here. Ayneone an ide what ELSE could cause this?
It is a Windows 2003 Server machine, 64 bit. The only other thing I can see that may cause it (but this functionality is supposedly disabled) is Symantec Endpoint Protection that is installed on the server - and being capable of actinc as a firewall, it could possibly intercept network traffic. I dont want to open a can of worms by pointing to Symantec prematurely (if pointing to Symantec can ever be seen as such). So, anyone an idea what else may be the cause?
Thanks
"Address already in use", aka WSAEADDRINUSE (10048), means that when the client socket prepared to connect to the server socket, it first tried to bind itself to a specific local IP/Port pair that was already in use by another socket, either an active one or one that has been closed but is still in the FD_WAIT state. This has nothing to do with the number of ports that are available.
I'm having the same issue on a Windows 2000 Server with a .Net application connecting to a SQL Server 7.0. There's like 10 servers with the same configuration and only one is showing this error several times a day. With a small test program I'm able to reproduce the error by just establishing a TCP connection on the SQL Server listening port. Running CurrPorts (http://www.nirsoft.net/utils/cports.html) shows there's still plenty of available ports in range 1024-5000.
I'm out of ideas and would like to know if you've found a solution since you've posted your question.
Edit : I finally found the solution : a worm was present on the server (WORM_DOWNAD.A) and exhausted local ports without being noticed.
I am trying to simulate a scenario where connection to the server of one process is down while the connection to another server is up. Just pulling the network cable won't work in my case since I need another process connection to stay up.
Is there any tool for this kind of job? I am on Windows. Thanks!
There's a few layers which you can simulate this at. The easiest would be if your two servers listen on two distinct TCP ports. In that case, you could run two tcp proxies, and stop/pause one when you want to simulate a failure. For Windows I would suggest using tcpTrace to do this.
Another option would be to have the two servers bound to two virtual NICs, which are bridged to the physical NIC. Of course if you have two physical NICs, you could bind each server process to a different physical NIC.
At a lower level, you can ran a WAN simulator. Most simulators allow you to impair specific types of traffic or specific ports. One such simulator is Packetstorm.
One other method which I would suggest is attaching a debugger to one process, and halting all threads on the process with the debugger. Often, a process doesn't die, but gets stuck in garbage collection, or in a loop. As the sockets don't close, many 'high availability' solutions won't automatically failover.
One approach would be to mock the relevant network connection code for the purposes of testing. In this case you would probably want to mock it returning whatever it usually would if the connection was down.
A poor man's approach if you can use sleep/hibernate mode on your machine :
Set an Outbound rule in the Windows Firewall to disallow connection for a particular Program.
Already connected sockets stay connected: put the machine in sleep/hibernate mode for a brief moment to force those sockets to disconnect.
When the system is restored, the program cannot establish new connections.
New connections are made possible as soon as you disable the firewall rule.
Note that it does not simulate network outage because each connection fails immediately with an permission error. But it prevents a process to establish connections.