I am writing a massive UDP network application.
Running traffic at 10gigabits per second.
I have a very high "System Interrupts" CPU usage in task manager.
Reading about what this means, I see:
What Is the “System Interrupts” Process?
System Interrupts is an official part of Windows and, while it does
appear as a process in Task Manager, it’s not really a process in the
traditional sense. Rather, it’s an aggregate placeholder used to
display the system resources used by all the hardware interrupts
happening on your PC.
However most articles say that a high value corresponds with failing hardware.
However, since the "system interrupts" entry correlates to high IRQ usage, maybe this should be high considering my large UDP network usage.
Also, is all of this really happenning on one CPU core? Or is this an aggregate of all things happening across all CPU cores.
If you have many individual datagrams being sent over UDP, it's certainly going to cause a lot of hardware interrupts, and a lot of CPU usage. 10 Gb is certainly in the range of "lots of CPU" if your datagrams are relatively small.
Each CPU has its own hardware interrupts. You can see how spread out the load is over cores on the performance tab - the red line is the kernel CPU time, which includes hardware interrupts and other low-level socket handling by the OS.
Related
I currently have a problem, that some packages are getting dropped in my local network.
But just on one device.
Ping to local router
Here you can see a ping to my router. I only have this problem on my pc. Mobilephone and Laptops are completly fine.
I tried a network card and two Wlan usb sticks all with the same problem.
Does somebody a clue on what could cause these problems?
*OS: Windows 10 21H2
*CPU usage ideling around 4-10%
*RAM usage 40%
*Network usage 0-1%
Your question is a bit broad - there are so many things that can disturb a network connection, from physical issues (i.e. cable defects, WIFI interference), to driver problems, CPU bottlenecks, etc. That being said, I would tip at a CPU bottleneck (app using most or all of your CPU), but even that is by no means certain.
Take a look at you CPU usage with TaskManager or ProcessExplorer (from the Sysinternals package). They both also show network usage. If your machine shows excessive CPU (constantly over 30% with frequent peeks), then you might want to explorer the reasons for that, and there can be many.
Using those same tools you can also try to identify apps that are possibly using alot of network bandwidth.
Windows has much happening in the background and those processes require resources (CPU, RAM, Network, Harddisk, etc.). Should any of those resources be limited then, you can easily see issues as you describe as there is a certain interdependence between those resources, i.e. you have many apps running with limited RAM, that leads to paging and as the hard disk is slow, the CPU is then busy with data shoveling and can't keep up with the NIC requests.
However, I am theorizing here. Supply some hard data (machine config, OS Info, Network info/config, task list, CPU usage, etc.) and we can continue.
I have a PLC that sends UDP packets every 24ms. "Simultaneously" (i.e. within a matter of what should be a few tens or at most hundreds of microseconds), the same PLC triggers a camera to snap an image. There is a Windows 8.1 system that receives both the images and the UDP packets, and an application running on it that should be able to match each image with the UDP packet from the PLC.
Most of the time, there is a reasonably fixed latency between the two events as far as the Windows application is concerned - 20ms +/- 5ms. But sometimes the latency rises, and never really falls. Eventually it goes beyond the range of the matching buffer I have, and the two systems reset themselves, which always starts back off with "normal" levels of latency.
What puzzles me is the variability in this variable latency - that sometimes it will sit all day on 20ms +/- 5ms, but on other days it will regularly and rapidly increase, and our system resets itself disturbingly often.
What could be going on here? What can be done to fix it? Is Windows the likely source of the latency, or the PLC system?
I 99% suspect Windows, since the PLC is designed for real time response, and Windows isn't. Does this sound "normal" for Windows? If so, even if there are other processes contending for the network and/or other resources, why doesn't Windows ever seem to catch up - to rise in latency when contention occurs, but return to normal latency levels after the contention stops?
FYI: the Windows application calls SetPriorityClass( GetCurrentProcess(), REALTIME_PRIORITY_CLASS ) and each critical thread is started with AfxBeginThread( SomeThread, pSomeParam, THREAD_PRIORITY_TIME_CRITICAL ). There is as little as possible else running on the system, and the application only uses about 5% of the available Quad-core processor (with hyperthreading, so 8 effective processors). There is no use of SetThreadAffinityMask() although I am considering it.
So, you have two devices, PLC and camera, which send data to the same PC using UDP.
I 90% suspect networking.
Either just buffering / shaping mechanism in your switch/router (by the way, I hope your setup is isolated, i.e. you have not just plugged your hardware into a busy corporate network), or network stack in either of devices, or maybe some custom retransmission mechanism in the PLC. Both IP and Ethernet protocols were never meant to guarantee low latencies.
To verify, use Wireshark to view the network traffic.
For best experiment, you can use another PC with three network cards.
Plug your three devices (windows client, PLC, camera) into that PC, and configure network bridge between the 3 cards. This way that second PC will act as an Ethernet switch, and you’ll be able to use Wireshark to capture all network traffic that goes through that.
The answer turned out to be a complex interaction between multiple factors, most of which don't convey any information useful to others... except as examples of "just because it seems to have been running fine for 12 months doesn't give you licence to assume everything was actually OK."
Critical to the issue was that the PLC was a device from Beckhoff to which several I/O modules were attached. It turns out that the more of these modules are attached, the less ability the PLC has to transmit UDP packets, despite having plenty of processor time and network bandwidth available. It looks like a resource contention issue of some kind which we have not resolved - we have simply chosen to upgrade to a more powerful PLC device. That device is still subject to the same issue, but the issue occurs if you try to transmit at roughly every 10ms, not 24ms.
The issue arose because our PLC application was operating right on the threshold of its UDP transmit capabilities. The PLC has to step its way through states in a state machine to do the transmit. With a PLC cycle of 2ms, the fastest it could ever go through the states in the state machine with the I/O modules we had attached turned out to be every 22ms.
Finally, what was assumed at first to be an insignificant and unrelated change on PLC startup tipped it over the edge and left it occasionally unable to keep up with the normal 24ms transmit cycle. So it would fall progressively further behind, giving the appearance of an increasingly latency.
I'm accepting #Soonts answer because careful analysis of some Wireshark captures was the the key to unlocking what was going on.
Is there some tool which allows one to control the MS-Windows (XP-SP3 32-bit in my case) scheduler, s.t. a target application (which I'd like to test), operates as if it is running on a slower CPU. Say my physical host is a 2.4GHzv Dual-Core, but I'd like the application to run as if, it is running on a 800MHz/1.0GHz CPU.
I am aware of some such programs which allowed old DOS games to run slower, but AFAIK, they take the approach of consuming CPU cycles to starve the application. I do not want such a thing, and also would like to have higher precision control on the clock.
I don't believe you'll find software that directly emulates the different CPUs. But something like ProcessLasso would let you control a programs CPU usage. Thus simulating, in a way, a slower clock speed.
I also found this blog entry with many other ways to throttle your CPU: Windows CPU throttling techniques
Additionally, if you have access to VMWare you could setup a resource pool with a limited CPU reservation.
I would like a software environment in which I can test the speed of my software on hardware with specific resources. For example, how fast does this program run on an 800MHz x86 with 24 Mb of RAM, when my host hardware is a 3GHz quad core amd64 with 12GB of RAM? Emulators such as qemu make a great point of running "almost as fast" as the underlying hardware; I would like to make it run slower. Is there a way to do that?
I have never tried it, but perhaps you could achieve what you want to some extent by combining an emulator like QEMU or VirtualBox on Linux with something like this:
http://cpulimit.sourceforge.net/
If you can limit the CPU time available to the emulator you might be able to simulate the results of execution on a slower computer. Keep in mind, though, that this would only affect the execution speed (or so I hope, anyway).
The CPU instruction set and other system features would remain unchanged. This means that emulating a specific processor accurately would be difficult if not impossible.
In addition, using something like cpulimit, which works using SIGSTOP and SIGCONT to repeatedly stop/restart the emulator process might cause side-effects, such as timing inconsistencies, video display artifacts etc.
In your emulator, keep a virtual "clock" and increment it appropriately as you execute each instruction. From there you can simply report how long it took in virtual time to execute, or you can have your emulator sleep now and again to keep execution speed roughly where it would be in the target.
I have a web application, and my users are complaining about performance. I have been able to narrow it down to JavaScript in IE6 issues, which I need to resolve. I have found the excellent dynaTrace AJAX tool, but my problem is that I don't have any issues on my dev machine.
The problem is that my users' computers are ancient, so timings which are barely noticable on my machine are perhaps 3-5 times longer on theirs, and suddenly the problem is a lot larger. Is it possible somehow to degrade the performance of my dev machine, or preferrably of a VM running on my dev machine, to the specs of my customers' computers?
I don't know of any virtualization solutions that can do this, but I do know that the computer/CPU emulator Bochs allows you to specify a limit on the number of emulated instructions per second, which you can use to simulate slower CPUs.
I am not sure if you can cpu bound it, but in VirutalBox or Parallel, you can bound the memory usage. I assume if you only give it about 128MB then it will be very slow. You can also limit the throughput on the network with a lot of tools. I guess the only thing I am not sure about is the CPU. That's tricky. Curious to know what you find. :)
You could get a copy of VMWare Workstation and choke the CPU of your VM.
With most virtual PC software you can limit the amount of RAM, but you are not able to set the CPU to a slower speed as it does not emulate a CPU, but uses the host CPU.
You could go with some emulation software like bochs that will let you setup an x89 processor environment.
You may try Fossil Toys
* PC Speed
PC CPU speed monitor / benchmark. With logging facility.
* Memory Load Test
Test application/operating system behaviour under low memory conditions.
* CPU Load Test
Test application/operating system behaviour under high CPU load conditions.
Although it doesn't simulate a specific CPU clock speed.