IdMappedPortTCP now requires "prodding" after telnet connection - delphi-7

Question

I have used IdMappedPortTCP in a particular program to allow generic port-forwarding for many years. I'm testing an upgraded build/component environment, and I ran into a problem. First, here's the old & new version info:
OS: W2kSP4 --> same (Hey, why is everyone laughing?)
Delphi: 5 --> 7
Project Indy: 9.0.0.14 --> 9.[Latest SVN]
I'm testing it by inserting it in a telnet session using the standard Windows console telnet client, and a Linux server, and I'm seeing an odd change in behavior.
Direct connection: Client connects, immediately sees server greeting
Old Indy: Same as direct
New Indy: Client connects, sees nothing. Press key, sees server greeting + keystroke.
Here's a comparison of the event chain:
Old:
6/08/2017 6:47:16 PM - DEBUG: MappedPort-Connect
6/08/2017 6:47:16 PM - TCP Port Fwd: Connect: 127.0.0.1:4325 --> 127.0.0.1:23
6/08/2017 6:47:16 PM - DEBUG: MappedPort-OutboundConnect
6/08/2017 6:47:16 PM - TCP Port Fwd: Outbound Connect: 192.168.214.11:4326 --> 192.168.210.101:23
6/08/2017 6:47:16 PM - DEBUG: MappedPort-OutboundData
6/08/2017 6:47:16 PM - DEBUG: MappedPort-Execute
6/08/2017 6:47:16 PM - DEBUG: MappedPort-OutboundData
6/08/2017 6:47:16 PM - DEBUG: MappedPort-Execute
6/08/2017 6:47:16 PM - DEBUG: MappedPort-OutboundData
...
New:
6/08/2017 6:41:34 PM - DEBUG: MappedPort-Connect
6/08/2017 6:41:34 PM - TCP Port Fwd: Connect: 127.0.0.1:1085 --> 127.0.0.1:23
6/08/2017 6:41:34 PM - DEBUG: MappedPort-OutboundConnect
6/08/2017 6:41:34 PM - TCP Port Fwd: Outbound Connect: 192.168.214.59:1086 --> 192.168.210.101:23
6/08/2017 6:47:36 PM - DEBUG: MappedPort-Execute
6/08/2017 6:47:36 PM - DEBUG: MappedPort-OutboundData
6/08/2017 6:47:36 PM - DEBUG: MappedPort-Execute
6/08/2017 6:47:36 PM - DEBUG: MappedPort-OutboundData
6/08/2017 6:47:36 PM - DEBUG: MappedPort-Execute
In the first one, you see OutboundData right after connecting. In the second, nothing happens after connect until I sent a keystroke (6 minutes later), at which time you see the Execute and then the first OutboundData event.
This caused me to wonder: Is it really connecting to the server and only delaying the output, or is the connection itself being delayed?
My first conclusion was that the connection itself was being delayed, and here's why. The server has a 1-minute timeout at the login prompt. If you connect and get the greeting but just sit there, the server disconnects after a minute. With the new Indy version, I sat there after the connect event for 6 full minutes, then got the server greeting with no problem.
However... NETSTAT shows the connection to the remote server established soon after the connection event is logged! So, I'm left to conclude that the connection is indeed established, but perhaps some initial character is being "eaten" or something that's causing getty to not engage until it gets a keystroke?
Any suggestions? Are you aware of anything that changed that I might look for -- something that I ought to be doing but am not? Any insights are appreciated.
(Barring any good leads, I guess the next step in my sleuthing might be to sniff both machines w/ WireShark to see what's going on after the connection.)
Update: Wireshark (single leg)
A packet capture from outside the machines showing traffic between MappedPort & the server (but not traffic between the client & MappedPort) shows that the telnet server sends a "Do Authenticate", to which the client (via MappedPort) replies w/ a "Will Authenticate". This is followed by the server sending authenticate suboption (and the client agreeing) then all the other telnet options. Finally, after seeing the login text, the client sends "do echo" and they both sit there until after 1min, at which time the server sends a TCP FIN to close the connection. That's the "good old" version.
On the new version, the client doesn't respond to the "Will Authenticate", and they both sit there indefinitely. (Hmmm, I wonder what that ties up in terms of server resources -- could be good DOS attack. It is an old telnet daemon, though, so it's probably been fixed by now...) When I finally sent the first keystroke, that's all it sent in that packet. THEN the client sends the "will authenticate" (without additional prodding from the server) and the negotiation continues exactly as normal; the last packet from the server (containing echo parameters) also then includes the echoed character which was typed. So it's like the client doesn't see the initial "do authenticate" packet from the server, but once you start typing, goes ahead and responds as if it had just heard it (once it sends the keystroke).
6/13 Update: Wireshark (both legs)
I captured both legs of the "broken" conversation and analyzed it. Interesting behavior. Bottom line:
As soon as the the server gets the TCP connection, it sends back a Telnet-DoAuth invitation. IdMappedPortTCP holds onto that packet and doesn't pass it on to the client -- yet. Once the client finally sends the first keystroke (seconds or minutes later), Id passes it on to the server. THEN Id passes the DoAuth packet that it got from the server on to the client.
Here's a more detailed accounting of the packets:
65 11-59 TCP Syn
67 59-11 TCP SynAck
69 11-59 TCP Ack
71 59-101 TCP Syn
73 101-59 TCP SynAck
74 59-101 TCP Ack
76 101-59 DoAuth
77 59-101 TCP Ack
nothing for 23 seconds
79 11-59 Data:\r\n (I pressed Enter)
81 59-101 Data:\r\n
83 59-11 DoAuth
85 11-59 WillAuth
87 101-59 TCP Ack
88 59-101 WillAuth
90 101-59 TCP Ack
91 101-59 Authentication option
92 59-11 Authentication option
94 11-59 Authentication option reply
96 59-101 Authentication option reply
98 101-59 Will/do Encryption/terminal/env options
99 59-101 Will/do Encryption/terminal/env options
101 11-59 Don't encrypt
103 59-101 Don't encrypt
105 101-59 TCP Ack
106 59-11 TCP Ack
108 11-59 Won't/will list
110 59-101 Won't/will list
112 101-59 TCP Ack
113 101-59 Do window size
114 59-11 Do window size
Packet dump line format: Pkt# From-To Payload
(Don't mind the packet# skips; the client & proxy are both running on VMs hosted by the machine where I was running the capture from, so Wireshark saw two copies of packets. I only included the pkt# so I can reference the original dump later if I want.)
From/To Machines:
10 = Linux client (see below)
11 = Windows client
59 = proxy
101 = server
An Interesting diversion: Linux Client
Though all my testing has been using various Windows clients (because that's what's used in production), I "accidentally" used Linux (because that's what I run on my workstation, where I ran Wireshark) because it was convenient. That client behaves differently -- more aggressively -- and thus avoids the problem. Here's what a dump of that looks like:
1 10-59 TCP Syn
2 59-10 TCP SynAck
3 10 59 TCP Ack
4 10-59 Do/Will list
5 59-101 TCP Syn
7 101-59 TCP SynAck
8 59-101 TCP Ack
10 59-101 Do/Will list
12 101-59 TCP Ack
13 101-59 DoAuth
14 59-10 DoAuth
15 10-59 TCP Auth
16 10-59 WontAuth
17 59-101 WontAuth
19 101-59 Will/Do list
20 59-10 Will/Do list
21 10-50 Do window size
22 59-101 Do window size
As you can see, the client doesn't wait for the telnet server to speak first -- as soon as the TCP connection is established, it sends a full Do/Will list. This is in turn passed onto the server once Id opens that connection. The server sends back the same "DoAuth" that it did initially before; the difference being that this time, having already passed traffic from the client, Id passes it on immediately. The client then sends auth flags, and things move right along.
So, if the client speaks first, IdMappedPortTCP does okay; it's only when the server speaks first that it holds onto its message and doesn't pass it on to the client until the client says something.
9/27 Update: Found the Code Change
Downgrading to 9.0.0.14 fixed the problem. Comparing the two versions' source code for IdMappedPortTCP.pas I found that the only difference is that the newer version added a block of code to procedure TIdMappedPortThread.OutboundConnect:
DoOutboundClientConnect(Self);
FNetData := Connection.CurrentReadBuffer;
if Length(FNetData) > 0 then begin
DoLocalClientData(Self);
FOutboundClient.Write(FNetData);
end;//if
except
(The first and last lines existed already, and are only shown for context.)
I confirmed that adding that code to 9.0.0.14 produced the problem.
I checked the SVN repo, and you added the offending code on 9/7/2008. The commit comment is:
Updated TIdMappedPortThread.OutboundConnect() to check for pending
data in the inbound client's InputBuffer after the OnOutboundConnect
event handler exits.
I don't fully understand the reason for or implications of the change -- obviously you had a good reason for doing it -- but it does appear to produce the effect I described ("holding onto" the server's initial output until the client sends something).

Answer 1

In Indy 9, TIdTCPConnection.CurrentReadBuffer() calls TIdTCPConnection.ReadFromStack() before then returning whatever data is stored in the TIdTCPConnection.InputBuffer property:
function TIdTCPConnection.CurrentReadBuffer: string;
begin
Result := '';
if Connected then begin
ReadFromStack(False); // <-- here
end;
Result := InputBuffer.Extract(InputBuffer.Size);
end;
Regardless of what may already be in the InputBuffer, ReadFromStack() waits for the socket to receive new data to append to the InputBuffer. It does not exit until new data actually arrives, or the specified ReadTimeout interval elapses. The TIdTCPConnection.ReadTimeout property is set to 0 by default, so when CurrentReadBuffer() calls ReadFromStack(), it ends up using an infinite timeout:
function TIdTCPConnection.ReadFromStack(const ARaiseExceptionIfDisconnected: Boolean = True;
ATimeout: Integer = IdTimeoutDefault; const ARaiseExceptionOnTimeout: Boolean = True): Integer;
// Reads any data in tcp/ip buffer and puts it into Indy buffer
// This must be the ONLY raw read from Winsock routine
// This must be the ONLY call to RECV - all data goes thru this method
var
i: Integer;
LByteCount: Integer;
begin
if ATimeout = IdTimeoutDefault then begin
if ReadTimeOut = 0 then begin
ATimeout := IdTimeoutInfinite; // <-- here
end else begin
ATimeout := FReadTimeout;
end;
end;
...
end;
So, when TIdMappedPortTCP.OutboundConnect() calls CurrentReadBuffer() after connecting its OutboundClient to the server, it does indeed wait for data to arrive from the client before then reading data from the server. To avoid that, you can set a non-infinite ReadTimeout value in the TIdMappedPortTCP.OnConnect or TIdMappedPortTCP.OnOutboundConnect event, eg:
AThread.Connection.ReadTimeout := 1;
In Indy 10, this problem was fixed in TIdMappedPortTCP by avoiding this initial wait on the client data after connecting to the server. I have now updated TIdMappedPortTCP in Indy 9 to do the same.