I was running a very big application on Windows 2003 server. It creates almost 900 threads and a single thread who is operating on a socket. It's a C++ application which I had compiled with Visual Studio environment.
After almost 17-20 hours of testing, I get 10055 socket error while sending the data.
Apart from this error my application runs excellently without any error or issue. It's a quad core system with 4 GiB of RAM and this application occupies around 30-40% CPU (on all 4 CPUs) in all of its running.
Can anyone here help me to pass through this. I had searched almost everything on google regarding this error but could not get anything relevant to my case.
I think, it's impossible to say mo than:
Error 10055 means that Windows has run
out of TCP/IP socket buffers because
too many connections are open at once.
http://kbase.pscs.co.uk/index.php?article=93
https://wiki.pscs.co.uk/how_to:10055
I have seen this symptom before in an IOCP socket system. I had to throttle outgoing async socket sends so that not too much data gets queued in the kernel waiting to be sent on the socket.
Although the error text says this happens due to number of connections, that's not my experience. If you write a tight loop doing async sends on a single socket, with no throttling, you can hit this very quickly.
Possibly #Len Holgate has something to add here, he's my "goto guy" for Windows sockets problems.
It creates almost 900 threads
That's partially your problem. Each thread is likely using the default 1MB of stack. You start to approach a GB of thread overhead. Chances of running out of memory are high. The whole point of using IOCP is so that you don't have to create a "thread per connection". You can just create several threads (from 1x - 4x the number of CPUs) to listen on the completion port handler and have each thread service a different request to maximize scalability.
I recall reading an article linked off of Stack Overflow that buffers you post for pending IOCP operations are setup such that the operating system WILL NOT let the memory swap out from physical memory to disk. And then you can run out of system resources when the connection count gets high.
The workaround, if I recall correctly, is to post a 0 byte buffer (or was it a 1 byte buffer) for each socket connection. When data arrives, your completion port handler will return, and that's a hint to your code to post a larger buffer. If I can find the link, I'll share it.
Related
Our application has a feature to actively connect to the customers' internal factory network and send a message when inspection events occur. The customer enters the IP address and port number of their machine and application into our software.
I'm using a TClientSocket in blocking mode and have provided callback functions for the OnConnect and OnError events. Assuming the abovementioned feature has been activated, when the application starts I call the following code in a separate thread:
// Attempt active connection
try
m_socketClient.Active := True;
except
end;
// Later...
// If `OnConnect` and socket is connected...send some data!
// If `OnError`...call `m_socketClient.Active := True;` again
When IP + port are valid, the feature works well. But if not, after several thousand errors (and many hours or even days) eventually Windows socket error 10055 (WSAENOBUFS) occurs and the application crashes.
Various articles such as this one from ServerFramework and this one from Microsoft talk about exhausting the Windows non-paged pool and mention (1) actively managing the number outstanding asynchronous send operations and (2) releasing the data buffers that were used for the I/O operations.
My question is how to achieve this and is three-fold:
A) Am I doing something wrong that memory is being leaked? For example, is there some missing cleanup code in the OnError handler?
B) How do you monitor I/O buffers to see if they are being exhausted? I've used Process Explorer to confirm my application is the cause of the leak, but ideally I'd need some programmatic way of measuring this.
C) Apart from restarting the application, is there a way to ask Windows to clear out or release I/O operation data buffers?
Code samples in Delphi, C/C++, C# fine.
A) The cause of the resource leak was a programming error. When the OnError event occurs, Socket.Close() should be called to release low-level resources associated with the socket.
B) The memory leak does not show up in the standard Working Set memory use of the process. Open handles belonging to your process need to be monitored which is possible with GetProcessHandleCount. See this answer in Delphi which was tested and works well. This answer in C++ was not tested but the answer is accepted so should work. Of course, you should be able to use GetProcessHandleCount directly in C++.
C) After much research, I must conclude that just like a normal memory leak, you cannot just ask Windows to "clean up" after you! The handle resource has been leaked by your application and you must find and fix the cause (see A and B above).
I have a strange problem which occurs only with some specific Windows versions (unfortunately I do not have such a Windows variant but only bug reports from a user which can see this problem).
What I'm doing: I send data via TCP/IP from a client application (on a PC) to an embedded device. This is a continuous stream of data out of one threads context and here always a bunch of data are collected so that the packages have a size of a bit less of multiples of 1460 bytes to have as most as effective TCP-packets.
Now a stop-condition can occur, where one has to react immediately. In this case an other TCP-package is sent to the device using the same network connection but out of an other thread. This package contains a payload of only a few bytes but is sent 3..4 times to ensure reaction of the device.
Both threads use the same sending-function, but it is locked by a mutex so that no concurrent accesses can happen.
Now following problem occurs on this one users PC: when stop-packages have been sent, all following send-attempts to the same socket fail with a system error code of 10054 (remote end hung up). Amazingly the log files of the embedded device clearly say the connection was not closed by it.
Since I can't reproduce this problem, I stab in the dark here.
The problem appears only on this users Windows 7, it does not happen with Linux or with my Windows versions (including a Windows 7). Thus any idea/suggestion/comment is welcome: are there any bugs/conditions known where a Windows TCP/IP connection can fail in this way?
Thanks for every comment and idea!
I profiled an application. Basically every thread reads an XML file from a network share, deserializes an object, logs to local files, asynchronously logs to db and calls a web service.
Amount of Threads is about 14 on a 24 core machine.
Redgate profiler shows me the multithreaded application is waiting for synchronisation 70% of the time. Is this an alarming signal or to be expected? Further if you can give advice how to approach analysing such a profiler log please share your knowledge.
Waiting for synchronization just means that a thread is suspended while waiting for another thread to complete an operation. Whether or not you should be concerned about this depends on how long you expect the operation on that thread to take to reach completion.
If the stack indicates a read/write, then it may just mean the disk is slow, for example. Maybe you can minimize that by changing your code; maybe it's just a flaky network or disk drive.
I need several client programs (stereo DSP audio generators) to be able to continuously, bidirectionally communicate with an external peripheral on an I2C bus, at a data rate of around 16kB/s with updates occurring every 1ms, all running on a 700MHz CPU. The programs will need simultaneous access to read and write but I don't care about locking on write.
I'm envisaging a daemon to manage the raw I2C communication, with the client audio programs communicating with the daemon via one of the following IPC options:
DBUS
Berkeley/POSIX sockets
Memory mapped file
With DBUS I have performance concerns, and with Berkeley/POSIX sockets I'm not sure about handling multiple clients. It's also important that no locking occurs as the daemon communication has to happen in the same thread as the audio rendering.
Memory mapping appears to suit the task. 10 bytes should do it, I'd need 4 bytes for input, 4 bytes for output, some way of telling that daemon that it should write the output bytes now, and some way of telling the daemon that it should currently be continuously updating the input bytes. But as I understand it memory mapping relies on buffering by the operating system and so I'm not sure what would happen if my daemon updates the input bytes while my client app is in the middle of a read() operation.
What's the best option for inter-process communication in my scenario?
we are having latency issues in one of our network application. Most of the time requests are being handled within 100ms. But sometime it can take up to a few seconds for no apparent reason.
So I hooked up some monitoring tools and looked up what was happening (Wireshark to monitor the network externally through port replication and Process Monitor to see what was happening on the local machine).
I was able to match tcp packets and they usually where within a millisecond of eachother in both logs file. But in one occurence, the last packet of a series was delayed by more then 250ms in Process Monitor compared to wireshark (and the application erratic behavior - due to latency - was being observed).
Since Wireshark was hooked up on another computer I'm quite sure that what was being monitored was accurate : all the packed did reach the network card on time.
As for Process Monitor I'm not totally sure about how it work : when is the network data being registered? Is it when it reach the network card? When it is made available to the application? When the application reads the data?
During these 250ms there were a few other events being registered which let me believe that Process Monitor was recording correctly and that this 250ms delay wasn't "created" by it.
Any help regarding the behavior of Process Monitor, the current method I use to dig down the problem or what you think could be the problem would be much appreciated.
Option 2
Perhaps you're experiencing the infamous 250ms delays that the GC cause from time to time (link). You can accurately measure GC suspensions using a specialized CLR host (link)
Option 1 - was ruled out
Since you are using TCP, I'd suggest that you'll turn on the NoDelay option on your socket just to eliminate the possibility that you're suffering from a clash between Nagle's Algorithm and the Delayed ACKs Algorithm. If you're experiencing "batching" of packets while sometimes a packet is "delayed" for about 200ms, then it just might be the issue.
A more in-depth explanation of this behavior can be found here.