I'm trying to send some self-packed Ethernet packets via the Winpcap API pcap_sendpacket(), but I got two identical packets after invoking the API once. The two packets can be captured on Wireshark for debugging purpose, with identical data and continuous frame numbers.
The environment is Win7 64bit. And it is wierd that the same code base running on another Win7 64bit will show only one packet on Wireshark.
Edit:
[2016.1.24 19:30]
I'm sorry I can only post the pcap related code parts due to the confidential thing
// first, enum the device list
pcap_if_t *m_alldevs;
char errbuf[PCAP_ERRBUF_SIZE];
if (pcap_findalldevs(&m_alldevs, errbuf) == -1)
{
// log error ...
for(pcap_if_t *d = m_alldevs; d != NULL; d = d->next)
{
// second, open the interface
// use flag PCAP_OPENFLAG_MAX_RESPONSIVENESS to get response quickly
// set timeout to 1000ms
errbuf[PCAP_ERRBUF_SIZE];
pcap_t* fp = pcap_open(d->name, 65536, PCAP_OPENFLAG_PROMISCUOUS|PCAP_OPENFLAG_MAX_RESPONSIVENESS, 1000, NULL, errbuf);
// third, get the interface device then release all the device
pcap_freealldevs(m_alldevs);
// 4th, send data
// unsigned char* buf;
// int size;
pcap_sendpacket(fp, buf, size);
And for the packet, the packet is handcrafted, with size between 64 and 1500, has an IEEE 802.3 type frame header, the two mac fields are customized.
On the machine that has the error, the version of the Winpcap is "4.1.0.2980", Wireshark is "64bit 1.12.3"; I will check the other machine that does not have the error tomorrow.
Edit:
[2016.1.26 10:30]
The version of the Winpcap is "4.1.0.2980", the same as on the machine with error. The version of Wireshark is "64bit 1.12.8". Both OS are Win7 Enterprise 64bit.
I had the same problem.
My steps to resolve it:
uninstall winpcap & npcap. I have both on my local machine
install only npcap
use delayed dll loading according to https://nmap.org/npcap/guide/npcap-devguide.html section "For software that want to use Npcap first when Npcap and WinPcap coexist".
Related
EDIT: Issue reported here: https://github.com/signal11/hidapi/issues/276
Inkling is a pen-device from Wacom. (InklingReader) is an open source project that gets real-time data from it.
I'm trying to tidy up InklingReader to use HIDAPI rather than libusb (as it works at higher level: HID rather than raw USB, so is much more compact & suitable. Also libusb fails on recent OSX).
HID API a small lib: one .h, one (per-platform) .c.
My code looks like this:
unsigned short inklingVendorId = 0x056a, inklingProductId = 0x0221;
if (hid_init() == FAIL) return;
handle = hid_open(inklingVendorId, inklingProductId, nullptr);
On Windows hid_open fails. Single stepping reveals the fail-point here:
// path = "\\\\?\\hid#vid_056a&pid_0221&mi_00&col01#8&1ea90857&0&0000#"
// "{4d1e55b2-f16f-11cf-88cb-001111000030}"
//
static HANDLE open_device(const char *path, BOOL enumerate)
{
HANDLE handle;
DWORD desired_access = (enumerate)? 0: (GENERIC_WRITE | GENERIC_READ);
DWORD share_mode = FILE_SHARE_READ|FILE_SHARE_WRITE;
// enumerate = 0
handle = CreateFileA(path,
desired_access,
share_mode,
NULL,
OPEN_EXISTING,
FILE_FLAG_OVERLAPPED,/*FILE_ATTRIBUTE_NORMAL,*/
0);
int err = GetLastError(); // 5 i.e. ERROR_ACCESS_DENIED
return handle; // returns 0xffffffff i.e. INVALID_HANDLE
}
Now the HIDAPI author says "HIDAPI won't work with keyboards and mice on Windows. Windows as a security measure doesn't allow the opening of Mouse and Keyboard HIDs." (here)
And if I enumerate HID devices:
struct hid_device_info *devs, *cur_dev;
devs = hid_enumerate(inklingVendorId, inklingProductId);
cur_dev = devs;
while (cur_dev) {
DBG2("Device Found\n type: %04hx %04hx\n path: %s\n serial_number: %ls", cur_dev->vendor_id, cur_dev->product_id, cur_dev->path, cur_dev->serial_number);
DBG2("");
DBG2(" Manufacturer: %ls", cur_dev->manufacturer_string);
DBG2(" Product: %ls", cur_dev->product_string);
DBG2(" Release: %hx", cur_dev->release_number);
DBG2(" Interface: %d", cur_dev->interface_number);
DBG2(" Usage Page: %d", cur_dev->usage_page);
DBG2(" Usage: %d", cur_dev->usage);
DBG2("");
cur_dev = cur_dev->next;
}
hid_free_enumeration(devs);
... I get not one but TWO entries:
Device Found
type: 056a 0221
path: \\?\hid#vid_056a&pid_0221&mi_00&col01#8&1ea90857&0&0000#{4d1e55b2-f16f-11cf-88cb-001111000030}
serial_number: 2B0400001C90C22A0002DD07FE8B022A
Manufacturer: Wacom, Inc.
Product: MSC Device
Release: 1256
Interface: 0
Usage Page: 1
Usage: 2
Device Found
type: 056a 0221
path: \\?\hid#vid_056a&pid_0221&mi_00&col02#8&1ea90857&0&0001#{4d1e55b2-f16f-11cf-88cb-001111000030}
serial_number: 2B0400001C90C22A0002DD07FE8B022A
Manufacturer: Wacom, Inc.
Product: MSC Device
Release: 1256
Interface: 0
Usage Page: 13
Usage: 2
(Note: OSX only reports the SECOND entry! On OSX there is no problem!)
Comparing path:
path: \?\hid#vid_056a&pid_0221&mi_00&col01#8&1ea90857&0&0000#...
path: \?\hid#vid_056a&pid_0221&mi_00&col02#8&1ea90857&0&0001#...
As per http://www.usb.org/developers/hidpage/Hut1_12v2.pdf,
UsagePage/Usage = 1/2 = {Generic Desktop Controls}/{Mouse}.
UsagePage/Usage = 13/2 = {Digitizers}/{Pen}.
(EDIT: Sometimes the first path is the 1/2 and the second is the 13/2, other times it's swapped).
And HIDAPI is only taking the first one it finds.
So it looks like this should be the solution. The Inkling was exposing 2 'devices' and hidapi was taking the wrong (mouse) one, and Windows doesn't allow access to Mouse or Keyboard Devices.
So I tweak the code...
while (cur_dev) {
if (cur_dev->vendor_id == vendor_id &&
cur_dev->product_id == product_id &&
cur_dev->usage_page == 13)
{
... to get the correct entry, it should work right?
Nope, CreateFileA just raises a different error:
usage_page== 1 => Error code 5 (ERROR_ACCESS_DENIED)
usage_page==13 => Error code 32 (ERROR_SHARING_VIOLATION)
Meh. This is rather upsetting. I seem to be at a dead-end!
I've tried fiddling with CreateFileA's params, e.g. replacing GENERIC_READ | GENERIC_WRITE with STANDARD_RIGHTS_READ | STANDARD_RIGHTS_WRITE -- now it happily creates a handle. But subsequent hid_read-s fail to collect any data.
Googling, https://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/af869f90-7bda-483d-ba2d-51680073fe9f/createfile-returns-invalid-handle-while-trying-to-access-hid-device-on-windows-8-desktop-app?forum=wdk seems to contain a couple of suggested workarounds:
both toaster and firefly can work in the HID stack. toaster shows how
to address the filter through a raw PDO, firefly shows how to access
it with WMI. From a C perspective, I think the raw PDO is much simpler
to code to, WMI is a bit nasty and complicated.
firefly
toaster
The author is recommending something in toaster, but it is a big CodeBase and I don't have experience with Windows Driver programming.
It looks as though I'm going to have to dig through a lot of very unfamiliar territory to get anything working, so before a start out I am asking here. If nobody answers and I figure it out, I will answer my own question.
The only other thing I can think of it is that maybe another process is already engaging this path. Maybe if I can terminate this process, the CreateFileA might succeed? Roel's libusb approach involves detaching kernel driver: https://github.com/roelj/inklingreader/blob/master/src/usb/online-mode.c#L98
PS Somewhere I read that if another process has already opened this device, our open has to match the permissions of this previous open. And I also read that Windows automatically opens all HID Devices upon detection.
Find out which process has an exclusive lock on a USB device handle
PPS maybe one idea is to try an alternative HID lib What is the best usb library to communicate with usb HID devices on Windows?
PPPS maybe I need to run my code as admin. But that's not a good solution.
I have seen similar behavior. The ERROR_SHARING_VIOLATION problem started to occur after upgrading to Windows 10 Anniversary Edition. The problem is only seen for USB HID devices connected when Windows is started. If you unplug and plug the USB device after Windows has started then CreateFile is successful. I haven't yet found a root cause or a solution.
You're right: ERROR_SHARING_VIOLATION will occur if some other app already opened this device. You need to call CreateFileW API like this:
DWORD desired_access = GENERIC_WRITE | GENERIC_READ;
DWORD share_mode = FILE_SHARE_READ | FILE_SHARE_WRITE;
::CreateFileW(deviceInterfacePath, desired_access, share_mode, 0, OPEN_EXISTING, 0, 0);
If you don't provide dwShareMode then it means that you're trying to open device exclusively. Which can fail if other app (new Windows version that maybe supports these kind of devices natively) already opened this device for its use.
Note about Keyboard and Mouse devices: you can also call ::CreateFileW without even setting desired_access (use zero value): in this case you can use HidD_GetManufacturerString/HidD_GetProductString/HidD_GetSerialNumberString/HidD_GetAttributes (and maybe some others) HID methods with returned handle. But you cannot read/write data to such device. This should be useful if you need to acquire name or VID/PID for HID keyboard/mouse.
Here is list of HID device types and their access modes on Windows.
i am opening a USB device:
for communication using CreateFile:
HANDLE hUsb = CreateFile("\\.\LCLD9",
GENERIC_READ | GENERIC_WRITE,
0,
null,
OPEN_EXISTING,
FILE_FLAG_OVERLAPPED,
0);
The call succeeds (i.e. hUsb is not equal to INVALID_HANDLE_VALUE). But then it comes time to do what we do with every serial port:
SetupComm (set receive and transit buffer sizes)
SetCommState (set flow-control, baud rate, etc)
SetCommTimeouts (set timeouts)
Each of these calls returns a GetLastError() code of 1. E.g.:
SetupComm(hUsb, 1024, 1024);
Why are operations to configure the serial device failing when using a "USB" serial device, but work when using a "virtual COM port"? Do USB devices not support such baud rates, buffers, flow control, and timeouts?
If this is a limitation/feature of Universal Serial devices, how can i detect that a handle refers to a "Universal Serial Device", rather than a "COMM Port"? For example, the user is the one who specifies which port to use:
\.\COM5
\.\LCLD9
Other serial functions that fail when talking to Universal Serial Bus serial device:
GetCommModemStatus (with error code 1)
ReadFile (with error code 6)
PurgeComm (with error code 6)
WriteFile (with error code 6)
Which begs the larger question, how do you communicate with a USB device once it's been opened with CreateFile?
No, USB devices do not use these things. If your device is an actual USB-to-RS232 (or other slow serial), then you should be opening the COM port it associated with. Let the drivers handle the work of sending that data.
USB communication is not like COM ports. You can think of it more as an external PCI bus than a simple send-whatever-data-you-want line.
Turns out that i don't have to do anything with Comm, because it's not a COM port. The reason my WriteFile was failing was because i was attempting to write to \\.\LCLD9 rather than \\.\LCLD9\.
The trailing backslash is critical; even though CreateFile returns success both ways.
void WriteToDisplay(String s)
{
//Open the display
var hLineDisplay = CreateFile("\\.\LCLD9\", GENERIC_WRITE, 0, nil, OPEN_EXISTING, 0, 0);
//Write the command
DWORD bytesWritten;
WriteFile(hLineDisplay, s, s.Length, ref bytesWritten, nil);
FileClose(hLineDisplay);
}
Anyone using Logic Controls LD9000 USB Line Display, the above is how you write to the display.
After reverse engineering their .NET Line Display driver i will also mention that the name of the port you use, e.g.:
\\.\LCLD9\
\\.\LCPD6\
\\.\LCPD3\
can be inferred from the full devicePath returned using the Windows Setup APIs. For example, my pole display's full device path is:
\\?\USB#VID_0FA8&PID_A090#6&DF2EE03&0&1#{A5DCBF10-6530-11D2-901F-00C04FB951ED}
\______/
|
ProductID
The rule is to check the device path for Product IDs. In my case PID_A090 means it will be available as file \\.\LCLD9\. Other product IDs and their associated file paths:
Contains DeviceName (trailing backslash is not optional)
======== ===============================================
PID_A030 \\.\LCPD3\
PID_A060 \\.\LCPD6\
PID_A090 \\.\LCLD9\
Note: Any code is released into the public domain. No attribution required.
I have a computer which is connected with external devices via serial communication (i.e. RS-232/RS-422 of physical or emulated serial ports). They communicate with each other by frequent data exchange (30Hz) but with only small data packet (less than 16 bytes for each packet).
The most critical requirement of the communication is low latency or delay between transmitting and receiving.
The data exchange pattern is handshake-like. One host device initiates communication and keeps sending notification on a client device. A client device needs to reply every notification from the host device as quick as possible (this is exactly where the low latency needs to be achieved). The data packets of notifications and replies are well defined; namely the data length is known.
And basically data loss is not allowed.
I have used following common Win API functions to do the I/O read/write in a synchronous manner:
CreateFile, ReadFile, WriteFile
A client device uses ReadFile to read data from a host device. Once the client reads the complete data packet whose length is known, it uses WriteFile to reply the host device with according data packet. The reads and writes are always sequential without concurrency.
Somehow the communication is not fast enough. Namely the time duration between data sending and receiving takes too long. I guess that it could be a problem with serial port buffering or interrupts.
Here I summarize some possible actions to improve the delay.
Please give me some suggestions and corrections :)
call CreateFile with FILE_FLAG_NO_BUFFERING flag? I am not sure if this flag is relevant in this context.
call FlushFileBuffers after each WriteFile? or any action which can notify/interrupt serial port to immediately transmit data?
set higher priority for thread and process which handling serial communication
set latency timer or transfer size for emulated devices (with their driver). But how about the physical serial port?
any equivalent stuff on Windows like setserial/low_latency under Linux?
disable FIFO?
thanks in advance!
I solved this in my case by setting the comm timeouts to {MAXDWORD,0,0,0,0}.
After years of struggling this, on this very day I finally was able to make my serial comms terminal thingy fast enough with Microsoft's CDC class USB UART driver (USBSER.SYS, which is now built in in Windows 10 making it actually usable).
Apparently the aforementioned set of values is a special value that sets minimal timeouts as well as minimal latency (at least with the Microsoft driver, or so it seems to me anyway) and also causes ReadFile to return immediately if no new characters are in the receive buffer.
Here's my code (Visual C++ 2008, project character set changed from "Unicode" to "Not set" to avoid LPCWSTR type cast problem of portname) to open the port:
static HANDLE port=0;
static COMMTIMEOUTS originalTimeouts;
static bool OpenComPort(char* p,int targetSpeed) { // e.g. OpenComPort ("COM7",115200);
char portname[16];
sprintf(portname,"\\\\.\\%s",p);
port=CreateFile(portname,GENERIC_READ|GENERIC_WRITE,0,0,OPEN_EXISTING,0,0);
if(!port) {
printf("COM port is not valid: %s\n",portname);
return false;
}
if(!GetCommTimeouts(port,&originalTimeouts)) {
printf("Cannot get comm timeouts\n");
return false;
}
COMMTIMEOUTS newTimeouts={MAXDWORD,0,0,0,0};
SetCommTimeouts(port,&newTimeouts);
if(!ComSetParams(port,targetSpeed)) {
SetCommTimeouts(port,&originalTimeouts);
CloseHandle(port);
printf("Failed to set COM parameters\n");
return false;
}
printf("Successfully set COM parameters\n");
return true;
}
static bool ComSetParams(HANDLE port,int baud) {
DCB dcb;
memset(&dcb,0,sizeof(dcb));
dcb.DCBlength=sizeof(dcb);
dcb.BaudRate=baud;
dcb.fBinary=1;
dcb.Parity=NOPARITY;
dcb.StopBits=ONESTOPBIT;
dcb.ByteSize=8;
return SetCommState(port,&dcb)!=0;
}
And here's a USB trace of it working. Please note the OUT transactions (output bytes) followed by IN transactions (input bytes) and then more OUT transactions (output bytes) all within 3 milliseconds:
And finally, since if you are reading this, you might be interested to see my function that sends and receives characters over the UART:
unsigned char outbuf[16384];
unsigned char inbuf[16384];
unsigned char *inLast = inbuf;
unsigned char *inP = inbuf;
unsigned long bytesWritten;
unsigned long bytesReceived;
// Read character from UART and while doing that, send keypresses to UART.
unsigned char vgetc() {
while (inP >= inLast) { //My input buffer is empty, try to read from UART
while (_kbhit()) { //If keyboard input available, send it to UART
outbuf[0] = _getch(); //Get keyboard character
WriteFile(port,outbuf,1,&bytesWritten,NULL); //send keychar to UART
}
ReadFile(port,inbuf,1024,&bytesReceived,NULL);
inP = inbuf;
inLast = &inbuf[bytesReceived];
}
return *inP++;
}
Large transfers are handled elsewhere in code.
On a final note, apparently this is the first fast UART code I've managed to write since abandoning DOS in 1998. O, doest the time fly when thou art having fun.
This is where I found the relevant information: http://www.egmont.com.pl/addi-data/instrukcje/standard_driver.pdf
I have experienced similar problem with serial port.
In my case I resolved the problem decreasing the latency of the serial port.
You can change the latency of every port (which by default is set to 16ms) using control panel.
You can find the method here:
http://www.chipkin.com/reducing-latency-on-com-ports/
Good Luck!!!
I am currently in the process of writing a Linux kernel module in C. The module provides an extremely basic driver for a USB light (the device consists of three colored LEDs). I have managed to get the driver to load and unload without problems and also create the device (/dev/wn0, /dev/wn1, etc.). However, I keep getting errors when attempting to write to the device:
$ echo "1" >/dev/wn0
bash: echo: write error: Broken pipe
The entire code for the module is here. However, the interesting part is the wn_set_color() function:
/* Create the data buffer to be sent to the device. */
u8 buf[8] = {
red, green, blue, 0, 0, 0, 0x1F, 0x05
};
/* Send the data to the device. */
return usb_control_msg(udev,
usb_sndctrlpipe(udev, 0),
0, 0, 0, 0,
buf, 8, 0);
For some reason, it returns -32 instead of sending the data to the device.
I am completely new to Linux kernel programming so I'm likely doing something silly. If you can shed some light on this at all, it would be greatly appreciated.
Edit: here is some further information:
lsusb -v output is here
the bDescriptorType member of the usb_endpoint_descriptor class contains '5' for the single endpoint exposed by the device (bEndpointAddress is 129 - or 0x81 in hex)
here is a screengrab of one of the control URBs sent to the device
usb_control_msg() eventually calls down to usb_submit_urb(). The Documentation/usb/error-codes.txt file describes the errors that this function can return:
-EPIPE The pipe type specified in the URB doesn't match the
endpoint's actual type.
If usb_submit_urb() succeeded, then usb_control_msg() returns an urb->status value. This lists under EPIPE:
-EPIPE (**) Endpoint stalled. For non-control endpoints,
reset this status with usb_clear_halt().
(**) This is also one of several codes that different kinds of host
controller use to indicate a transfer has failed because of device
disconnect. In the interval before the hub driver starts disconnect
processing, devices may receive such fault reports for every request.
Have you checked for any messages in the kernel log?
I have a feeling it has to do with your usb_sndctrlpipe call. The definition of this function is as follows: unsigned int usb_sndctrlpipe(struct usb_device *dev, unsigned int
endpoint).
You seem to be passing the device pointer appropriately, however your pass in the value 0 for your control endpoint, which as you mention, is not the address of your endpoint. I would recommend defining a constant at the beginning with the hex value of your endpoint and passing that to your calls.
However, I believe you have a bigger problem.
Looking at your lsusb, it seems that your endpoint is not actually a control endpoint, but an interrupt endpoint. This changes the functions that you need to call to communicate. For example, instead of usb_sndctrlpipe you will need usb_rcvintpipe(struct usb_device *dev, unsigned int endpoint) to generate the pipe (since it is an IN endpoint as listed in your lsusb) and use a different function instead of usb_control_msg. Unfortunately, from what I can gather, it seems like there are no functions available to automatically construct interrupt urbs so you will need to create a urb struct as described in section 13.3.2.1 of http://www.makelinux.net/ldd3/chp-13-sect-3. Even worse news is that (unless I am missing something) because your only endpoint seems to be the interrupt IN endpoint, it would seem that you can only receive interrupts from the device and are not able to send anything to the device. Do you know for sure that changing the colors of the lamp via usb is a functionality supported by the device?
More information can be found at the following:
http://www.beyondlogic.org/usbnutshell/usb4.shtml (thorough information on endpoints and how to read the descriptors)
http://www.makelinux.net/ldd3/chp-13-sect-3 and
http://www.makelinux.net/ldd3/chp-13-sect-5 (function definitions for usb communication)
I have a very simple Winsock2 TCP client - full listing below - which simply blasts a bunch of bytes. However, it's running very slowly over the network; the data just trickles by.
Here's what I've tried and found (both Windows PCs are on the same LAN):
Running this app from one machine to the other is slow - it takes ~50s to send 8MB.
Two different servers - netcat and a custom-written one (just as simple as the below client) - yielded the same results.
taskmgr shows both the CPU and network being barely-utilized.
Running this app with the server on the same machine is fast - it takes ~1-2s to send 8MB.
A different client, netcat, works just fine - it takes ~7s to send 20MB of data. (I used the nc that comes with Cygwin.)
Varying the buffer size (1*4096, 16*4096, and 128*4096) made little difference.
Running almost the same code on Linux boxes on a different LAN worked just fine.
Adding a bunch of print statements around the send call shows that we spend most of our time blocking on it.
On the server side, we see a bunch of receives of <= 4K chunks (regardless of what size buffers the sender is pushing). However, this happens with other clients as well, like netcat, which runs at full speed.
Any ideas? Thanks in advance for any tips.
#include <winsock2.h>
#include <iostream>
using namespace std;
enum { bytecount = 8388608 };
enum { bufsz = 16*4096 };
int main(int argc, TCHAR* argv[])
{
WSADATA wsaData;
WSAStartup(MAKEWORD(2,2), &wsaData);
struct sockaddr_in sa;
memset(&sa, 0, sizeof sa);
sa.sin_family = AF_INET;
sa.sin_port = htons(9898);
sa.sin_addr.s_addr = inet_addr("157.54.144.70");
if (sa.sin_addr.s_addr == -1) {
cerr << "inet_addr: " << WSAGetLastError() << endl;
return 1;
}
char *blob = new char[bufsz];
for (int i = 0; i < bufsz; ++i) blob[i] = (char) i;
SOCKET s = socket(AF_INET, SOCK_STREAM, IPPROTO_IP);
if (s == INVALID_SOCKET) {
cerr << "socket: " << WSAGetLastError() << endl;
return 1;
}
int res = connect(s, reinterpret_cast<sockaddr*>(&sa), sizeof sa);
if (res != 0) {
cerr << "connect: " << WSAGetLastError() << endl;
return 1;
}
int sent;
for (int j = 0; j < bytecount; j += sent) {
sent = send(s, blob, bufsz, 0);
if (sent < 0) {
cerr << "send: " << WSAGetLastError() << endl;
return 1;
}
}
closesocket(s);
return 0;
}
Here are the things you can do to get a better picture.
You can check how much time it spends inside the "connect", "send" API calls. You can see if connect call is a problem. You can do it with profiler, but if your application is very slow, you will be able to see it while debugging.
Try running Wireshark (or Ethereal) to dump you network traffic so that you see that TCP packets are transferred with some lattency. If responses come fast then it has to do with your system only. If you find delays, than it is routing/network problem.
You can run "route print" to check how your PC is sending traffic to destination machine (157.54.144.70). You would be able to see if gateway is used and check routing priority for the different routes.
Try sending smaller chunks. (I mean changing "bufsz" to 1024). Is there any correlation between performance and buffer size?
Check if there is antivirus, firewall applications installed? Make sure to turn it off. You can try to run the same app in safe mode with network support.
The application looks fine, and you said it works fine with linux.
I dont know whether this will help you, but I would have compared -
1) The mtu values of the windows with the linux system.
2) checked the tcp receive mem size in windows and Linux.
3) checked whether the network card speed of both the systems are same.
I watched packets going by using Microsoft Network Monitor (netmon) with the nice TCP Analyzer visualizer, and it turned out that tons of packets were getting lost and needing to be retransmitted - hence the slow speeds, because of retransmission timeouts (RTOs).
A colleague helped me debug this:
Well, from this trace on the receiver side, it definitely looks like some packets are not making it through to the receiver. I also see what appear to be some mangled packets (things like partial TCP headers, etc) in these traces.
Even in the “good” trace (the receiver's view of the netcat client), I see some mangled packets (wrong TCP data length, etc). The errors aren’t as frequent as in the other trace, however.
Given that these machines are on the same subnet, there is no router in the way which could be dropping packets. That leaves the two NICs, the Ethernet cables, and the Ethernet switches. You could try to isolate the bad machine by adding a third machine into the mix and try the same test with the new machine replacing first the sender and then the receiver. Use a different physical port for the third machine. If either of the original machines has a switch between it and the floor jack, try removing that switch from the equation. You could also try an Ethernet reversing cable between the original two machines (or a different Ethernet switch that you plug the two machines into directly) and see if the problem persists.
Since the problem appears to be packet content dependent, I doubt the problem is in the cabling. Given that the sender has an NVidia nForce chipset Ethernet and the receiver has a Broadcom Ethernet, my money is on the sender’s NIC being the culprit. If it does seem to be the fault of a particular NIC, try turning off special features of the NIC like checksum offloading or large-send offload.
I tried using a third box as the sender (identical to original sender, a Shuttle XPC with nForce chipset), and this worked smoothly - TCP Analyzer showed very smooth-running TCP sessions. This suggests to me that the problem was actually due to a buggy NIC/driver on the original sender box, or bad Ethernet cable.