I have jmeter , where a single thread contains two mqtt gateway connection sampler & each sampler have three publishers connected to iothub.
Jmeter reference:
When I run the thread in loop 6frames / second for 10 seconds, I could see all 60 frames published successful in JMeter.
But when I check data count at iothub, first gate way point have received only 6 frames ( some data get missed it seems, problem with jmeter I assume ) & second gateway have received 42 frames. Second part led to major confusion, when it have to receive maximum of 30 frames, but received 42.
Diagram reference:
Each gateway (A &B) include the Connection panel with :
Iothub URL
Mqtt v 3.1.1
Username: iothuburl/device ID
Pwd: SAS token ( generated SAS from connection string available at iothubowner page from azure portal).
Each Gateway (A&B) include
three publishers & Each includes 200 JSON objects and size doesn't exceed 55kb.
Publisher QoS: 0
Operation:
For every one second, each gateway publish 3 frames ( total 600 JSON objects).
As I have mentioned 2 gateway, so total 6 frames with 1200 JSON object get published successful in JMeter.
But the data is missing at iothub.
note: while running two gateway in single thread, i could both gateway connection sharing the common connection string ID.
Any clue, where did I miss the major configuration, any help would be greatly appreciated. Thanks.
Change the QoS=1 in publisher panel. Though we have few latency time to wait for acknowledgement, but the simulation works fine without any loss of connection/data.
In a Windows Server 2016 Standard,
I had to install Multipont to add licenses
But Multipoint takes the port 80
With the following command
netsh http show servicestate
you can see
Server session ID: FF00000520000001
Version: 2.0
State: Active
Properties:
Max bandwidth: 4294967295
Timeouts:
Entity body timeout (secs): 120
Drain entity body timeout (secs): 120
Request queue timeout (secs): 120
Idle connection timeout (secs): 120
Header wait timeout (secs): 120
Minimum send rate (bytes/sec): 150
URL groups:
URL group ID: FE00000540000001
State: Active
Request queue name: Request queue is unnamed.
Properties:
Max bandwidth: inherited
Max connections: inherited
Timeouts:
Timeout values inherited
Number of registered URLs: 1
Registered URLs:
HTTP://+:80/MULTIPOINT/IMULTIPOINTCERTIFICATEREQUEST/
How can I change the port number?
Does anyone know when, specifically, the clock for $upstream_response_time begins and ends?
The documentation seems a bit vague:
keeps time spent on receiving the response from the upstream server; the time is kept in seconds with millisecond resolution. Times of several responses are separated by commas and colons like addresses in the $upstream_addr variable.
There is also an $upstream_header_time value, which adds more confusion.
I assume $upstream_connect_time stops once the connection is established, but before it is accepted upstream?
After this what does $upstream_response_time include?
Time spent waiting for upstream to accept?
Time spent sending the request?
Time spent sending the response header?
A more specific definition is in their blog.
$request_time – Full request time, starting when NGINX reads the first
byte from the client and ending when NGINX sends the last byte of the
response body
$upstream_connect_time – Time spent establishing a
connection with an upstream server
$upstream_header_time – Time
between establishing a connection to an upstream server and receiving
the first byte of the response header
$upstream_response_time – Time
between establishing a connection to an upstream server and receiving
the last byte of the response body
So
$upstream_header_time is included in $upstream_response_time.
Time spent connecting to upstream is not included in both of them.
Time spent sending response to client is not included in both of them.
I've investigated and debug the behavior around this, and it turned out as follows:
start time
end time
$upstream_connect_time
before Nginx establishes TCP connection with upstream server
before Nginx sends HTTP request to upstream server
$upstream_header_time
before Nginx establishes TCP connection with upstream server
after Nginx receives and processes headers in HTTP response from upstream server
$upstream_response_time
before Nginx establishes TCP connection with upstream server
after Nginx receives and processes HTTP response from upstream server
source code
I'll explain how values are different between $upstream_connect_time and $upstream_response_time, as it's what I was primarily interested in.
The value of u->state->connect_time, which represents $upstream_connect_time in millisecond, is ingested in the following section: https://github.com/nginx/nginx/blob/3334585539168947650a37d74dd32973ab451d70/src/http/ngx_http_upstream.c#L2073
if (u->state->connect_time == (ngx_msec_t) -1) {
u->state->connect_time = ngx_current_msec - u->start_time;
}
Whereas the value of u->state->repponse_time, which represents $upstream_response_time in millisecond, is set in the following section: https://github.com/nginx/nginx/blob/3334585539168947650a37d74dd32973ab451d70/src/http/ngx_http_upstream.c#L4432
if (u->state && u->state->response_time == (ngx_msec_t) -1) {
u->state->response_time = ngx_current_msec - u->start_time;
You can notice that both of values are calculated based on u->start_time, which is the time just before the connection is established, defined in https://github.com/nginx/nginx/blob/3334585539168947650a37d74dd32973ab451d70/src/http/ngx_http_upstream.c#L1533
(note that ngx_event_connect_peer is a function to establish TCP connection between nginx workerprocesses and upstream servers).
Therefore, both values include the time taken to establish the TCP connection. You can check this by doing a live debug with, for example, gdbserver.
I have used IdMappedPortTCP in a particular program to allow generic port-forwarding for many years. I'm testing an upgraded build/component environment, and I ran into a problem. First, here's the old & new version info:
OS: W2kSP4 --> same (Hey, why is everyone laughing?)
Delphi: 5 --> 7
Project Indy: 9.0.0.14 --> 9.[Latest SVN]
I'm testing it by inserting it in a telnet session using the standard Windows console telnet client, and a Linux server, and I'm seeing an odd change in behavior.
Direct connection: Client connects, immediately sees server greeting
Old Indy: Same as direct
New Indy: Client connects, sees nothing. Press key, sees server greeting + keystroke.
Here's a comparison of the event chain:
Old:
6/08/2017 6:47:16 PM - DEBUG: MappedPort-Connect
6/08/2017 6:47:16 PM - TCP Port Fwd: Connect: 127.0.0.1:4325 --> 127.0.0.1:23
6/08/2017 6:47:16 PM - DEBUG: MappedPort-OutboundConnect
6/08/2017 6:47:16 PM - TCP Port Fwd: Outbound Connect: 192.168.214.11:4326 --> 192.168.210.101:23
6/08/2017 6:47:16 PM - DEBUG: MappedPort-OutboundData
6/08/2017 6:47:16 PM - DEBUG: MappedPort-Execute
6/08/2017 6:47:16 PM - DEBUG: MappedPort-OutboundData
6/08/2017 6:47:16 PM - DEBUG: MappedPort-Execute
6/08/2017 6:47:16 PM - DEBUG: MappedPort-OutboundData
...
New:
6/08/2017 6:41:34 PM - DEBUG: MappedPort-Connect
6/08/2017 6:41:34 PM - TCP Port Fwd: Connect: 127.0.0.1:1085 --> 127.0.0.1:23
6/08/2017 6:41:34 PM - DEBUG: MappedPort-OutboundConnect
6/08/2017 6:41:34 PM - TCP Port Fwd: Outbound Connect: 192.168.214.59:1086 --> 192.168.210.101:23
6/08/2017 6:47:36 PM - DEBUG: MappedPort-Execute
6/08/2017 6:47:36 PM - DEBUG: MappedPort-OutboundData
6/08/2017 6:47:36 PM - DEBUG: MappedPort-Execute
6/08/2017 6:47:36 PM - DEBUG: MappedPort-OutboundData
6/08/2017 6:47:36 PM - DEBUG: MappedPort-Execute
In the first one, you see OutboundData right after connecting. In the second, nothing happens after connect until I sent a keystroke (6 minutes later), at which time you see the Execute and then the first OutboundData event.
This caused me to wonder: Is it really connecting to the server and only delaying the output, or is the connection itself being delayed?
My first conclusion was that the connection itself was being delayed, and here's why. The server has a 1-minute timeout at the login prompt. If you connect and get the greeting but just sit there, the server disconnects after a minute. With the new Indy version, I sat there after the connect event for 6 full minutes, then got the server greeting with no problem.
However... NETSTAT shows the connection to the remote server established soon after the connection event is logged! So, I'm left to conclude that the connection is indeed established, but perhaps some initial character is being "eaten" or something that's causing getty to not engage until it gets a keystroke?
Any suggestions? Are you aware of anything that changed that I might look for -- something that I ought to be doing but am not? Any insights are appreciated.
(Barring any good leads, I guess the next step in my sleuthing might be to sniff both machines w/ WireShark to see what's going on after the connection.)
Update: Wireshark (single leg)
A packet capture from outside the machines showing traffic between MappedPort & the server (but not traffic between the client & MappedPort) shows that the telnet server sends a "Do Authenticate", to which the client (via MappedPort) replies w/ a "Will Authenticate". This is followed by the server sending authenticate suboption (and the client agreeing) then all the other telnet options. Finally, after seeing the login text, the client sends "do echo" and they both sit there until after 1min, at which time the server sends a TCP FIN to close the connection. That's the "good old" version.
On the new version, the client doesn't respond to the "Will Authenticate", and they both sit there indefinitely. (Hmmm, I wonder what that ties up in terms of server resources -- could be good DOS attack. It is an old telnet daemon, though, so it's probably been fixed by now...) When I finally sent the first keystroke, that's all it sent in that packet. THEN the client sends the "will authenticate" (without additional prodding from the server) and the negotiation continues exactly as normal; the last packet from the server (containing echo parameters) also then includes the echoed character which was typed. So it's like the client doesn't see the initial "do authenticate" packet from the server, but once you start typing, goes ahead and responds as if it had just heard it (once it sends the keystroke).
6/13 Update: Wireshark (both legs)
I captured both legs of the "broken" conversation and analyzed it. Interesting behavior. Bottom line:
As soon as the the server gets the TCP connection, it sends back a Telnet-DoAuth invitation. IdMappedPortTCP holds onto that packet and doesn't pass it on to the client -- yet. Once the client finally sends the first keystroke (seconds or minutes later), Id passes it on to the server. THEN Id passes the DoAuth packet that it got from the server on to the client.
Here's a more detailed accounting of the packets:
65 11-59 TCP Syn
67 59-11 TCP SynAck
69 11-59 TCP Ack
71 59-101 TCP Syn
73 101-59 TCP SynAck
74 59-101 TCP Ack
76 101-59 DoAuth
77 59-101 TCP Ack
nothing for 23 seconds
79 11-59 Data:\r\n (I pressed Enter)
81 59-101 Data:\r\n
83 59-11 DoAuth
85 11-59 WillAuth
87 101-59 TCP Ack
88 59-101 WillAuth
90 101-59 TCP Ack
91 101-59 Authentication option
92 59-11 Authentication option
94 11-59 Authentication option reply
96 59-101 Authentication option reply
98 101-59 Will/do Encryption/terminal/env options
99 59-101 Will/do Encryption/terminal/env options
101 11-59 Don't encrypt
103 59-101 Don't encrypt
105 101-59 TCP Ack
106 59-11 TCP Ack
108 11-59 Won't/will list
110 59-101 Won't/will list
112 101-59 TCP Ack
113 101-59 Do window size
114 59-11 Do window size
Packet dump line format: Pkt# From-To Payload
(Don't mind the packet# skips; the client & proxy are both running on VMs hosted by the machine where I was running the capture from, so Wireshark saw two copies of packets. I only included the pkt# so I can reference the original dump later if I want.)
From/To Machines:
10 = Linux client (see below)
11 = Windows client
59 = proxy
101 = server
An Interesting diversion: Linux Client
Though all my testing has been using various Windows clients (because that's what's used in production), I "accidentally" used Linux (because that's what I run on my workstation, where I ran Wireshark) because it was convenient. That client behaves differently -- more aggressively -- and thus avoids the problem. Here's what a dump of that looks like:
1 10-59 TCP Syn
2 59-10 TCP SynAck
3 10 59 TCP Ack
4 10-59 Do/Will list
5 59-101 TCP Syn
7 101-59 TCP SynAck
8 59-101 TCP Ack
10 59-101 Do/Will list
12 101-59 TCP Ack
13 101-59 DoAuth
14 59-10 DoAuth
15 10-59 TCP Auth
16 10-59 WontAuth
17 59-101 WontAuth
19 101-59 Will/Do list
20 59-10 Will/Do list
21 10-50 Do window size
22 59-101 Do window size
As you can see, the client doesn't wait for the telnet server to speak first -- as soon as the TCP connection is established, it sends a full Do/Will list. This is in turn passed onto the server once Id opens that connection. The server sends back the same "DoAuth" that it did initially before; the difference being that this time, having already passed traffic from the client, Id passes it on immediately. The client then sends auth flags, and things move right along.
So, if the client speaks first, IdMappedPortTCP does okay; it's only when the server speaks first that it holds onto its message and doesn't pass it on to the client until the client says something.
9/27 Update: Found the Code Change
Downgrading to 9.0.0.14 fixed the problem. Comparing the two versions' source code for IdMappedPortTCP.pas I found that the only difference is that the newer version added a block of code to procedure TIdMappedPortThread.OutboundConnect:
DoOutboundClientConnect(Self);
FNetData := Connection.CurrentReadBuffer;
if Length(FNetData) > 0 then begin
DoLocalClientData(Self);
FOutboundClient.Write(FNetData);
end;//if
except
(The first and last lines existed already, and are only shown for context.)
I confirmed that adding that code to 9.0.0.14 produced the problem.
I checked the SVN repo, and you added the offending code on 9/7/2008. The commit comment is:
Updated TIdMappedPortThread.OutboundConnect() to check for pending
data in the inbound client's InputBuffer after the OnOutboundConnect
event handler exits.
I don't fully understand the reason for or implications of the change -- obviously you had a good reason for doing it -- but it does appear to produce the effect I described ("holding onto" the server's initial output until the client sends something).
In Indy 9, TIdTCPConnection.CurrentReadBuffer() calls TIdTCPConnection.ReadFromStack() before then returning whatever data is stored in the TIdTCPConnection.InputBuffer property:
function TIdTCPConnection.CurrentReadBuffer: string;
begin
Result := '';
if Connected then begin
ReadFromStack(False); // <-- here
end;
Result := InputBuffer.Extract(InputBuffer.Size);
end;
Regardless of what may already be in the InputBuffer, ReadFromStack() waits for the socket to receive new data to append to the InputBuffer. It does not exit until new data actually arrives, or the specified ReadTimeout interval elapses. The TIdTCPConnection.ReadTimeout property is set to 0 by default, so when CurrentReadBuffer() calls ReadFromStack(), it ends up using an infinite timeout:
function TIdTCPConnection.ReadFromStack(const ARaiseExceptionIfDisconnected: Boolean = True;
ATimeout: Integer = IdTimeoutDefault; const ARaiseExceptionOnTimeout: Boolean = True): Integer;
// Reads any data in tcp/ip buffer and puts it into Indy buffer
// This must be the ONLY raw read from Winsock routine
// This must be the ONLY call to RECV - all data goes thru this method
var
i: Integer;
LByteCount: Integer;
begin
if ATimeout = IdTimeoutDefault then begin
if ReadTimeOut = 0 then begin
ATimeout := IdTimeoutInfinite; // <-- here
end else begin
ATimeout := FReadTimeout;
end;
end;
...
end;
So, when TIdMappedPortTCP.OutboundConnect() calls CurrentReadBuffer() after connecting its OutboundClient to the server, it does indeed wait for data to arrive from the client before then reading data from the server. To avoid that, you can set a non-infinite ReadTimeout value in the TIdMappedPortTCP.OnConnect or TIdMappedPortTCP.OnOutboundConnect event, eg:
AThread.Connection.ReadTimeout := 1;
In Indy 10, this problem was fixed in TIdMappedPortTCP by avoiding this initial wait on the client data after connecting to the server. I have now updated TIdMappedPortTCP in Indy 9 to do the same.
Standard squid config only logs one CONNECT line for any https transaction. What is being counted/timed by the reported bytes and duration fields in that line?
Got an answer via the squid-users mailing list [1]:
Unless you are using SSL-Bump or such to process the contents specially.
The duration is from the CONNECT message arriving to the time TCP close
is used to end the tunnel. The size should be the bytes sent to the
client (excluding the 200 reply message itself) during that time.
[1] http://lists.squid-cache.org/pipermail/squid-users/2016-July/011714.html