Gatling 3.4 bug with timeout after silent check - websocket

Question

I experience a very weird behavior of Gatling with websocket silent check:
The .await(600 seconds)(check) fails with timeout after some milliseconds.
At first I explain my situation. The limitation of Gatling websocket does not allow to handle ping requests from server. So I have to cheat and invented a fancy protocol. The code with comments is below:
client may send an INITIAL event or NON-INITIAL event (not first in a sequence).
the minimum interval between INITIAL events is 3 seconds
each request initialized by client results with 2 responses: "calculation started" and "calculation result"
ping request from server may come at any time. When we pause between client events, we still may receive ping request
when we receive a ping request, we must respond.
the usual sequence of events does not depend on ping, so if ping arrived, then we must wait for one more message
exec(session => dump(session, s"The spin action: event=$eventType oneRound=$oneRound" )).
exec(_.remove(ATTR_PING_SEQ_ID))
.doIfOrElse("CLIENT_INITIAL_EVENT".equals(eventType)) {
exec(session => dump(session, s"Sending CLIENT_INITIAL_EVENT and expect 2 or 3 responses. 3 responses mean that one of them is a ping. wait for each response for ${Config.waitForResponseSec} seconds" )).
exec(
clientActionBuilder
// first response: "started calculation"
.await(Config.waitForResponseSec seconds)(check1)
// second response: "calculated result"
.await(Config.waitForResponseSec seconds)(check2)
// wait for the delay between client initial events.
// We cannot just wait because PING may come within this time and we must handle it!
// Most probably the ping will not come, so we ignore the timeout and make the check silent
.await(Config.minTimeBetweenClientInitialEventsMillis milliseconds)(check3.silent)
)
.exec(session => dump(session, s"SPIN 2 or 3 responses got" ))
// we waited for 3 messages, so if the ping came, we just send pong and do not wait for anything else
.doIf(session => Utils.getStringSessionAttribute(session, ATTR_PING_SEQ_ID, "0") != "0"){
exec(session => dump(session, s"Sending PONG for CLIENT_INITIAL_EVENT" )).
exec(pongBuilder)
}
} {
exec(session => dump(session, s"Sending NON-INITIAL_CLIENT_EVENT and expect 2 responses" )).
exec(
clientActionBuilder
// first response: "started calculation"
.await(Config.waitForResponseSec seconds)(check4)
// second response: "calculation result"
.await(Config.waitForResponseSec seconds)(check5)
)
.exec(session => dump(session, s"NON-INITIAL_CLIENT_EVENT 2 responses got" ))
// we waited for 2 messages. If ping came, then it came instead of a
// "started calculation" or "calculation result", so we have to wait for one more message
.doIf(session => Utils.getStringSessionAttribute(session, ATTR_PING_SEQ_ID, "0") != "0"){
exec(session => dump(session, s"Sending PONG for NON-INITIAL_CLIENT_EVENT" )).
exec(pongBuilder.await(Config.waitForResponseSec seconds)(check6))
.exec(session => dump(session, s"NON-INITIAL_CLIENT_EVENT PONG response got" ))
}
}
.exitHereIfFailed
.exec(_.remove(ATTR_PING_SEQ_ID))
The check for each message is the same. It is cloned because I want to see in logs which concrete check has timed out:
val checkX = ws.checkTextMessage("myCheckX")
.matching(jsonPath(matchingCondition).exists)
jsonPath("$.body.data.nextActions[0]").optional.saveAs(ATTR_NEXT_ACTION),
).check(regex("\"cId\":(.*?),(\"name\":\"Ping\")").optional.saveAs(ATTR_PING_SEQ_ID))
the actual messages are very simple:
val clientActionBuilder = ws("requestClientAction").sendText(
"""{
| "header":
| {
| "name": "Action",
| "cId": ${cId},
| "dType": 2
| },
| "body":
| {
| "type": "#TYPE#",
| "seqId":${seqId},
| "data":{
| }
| }
|}
""".stripMargin.replaceAll("[\\s\n\r]", "").replace("#TYPE#", eventType)
)
val pongBuilder = ws("requestPong").sendText(
"""{
| "header":
| {
| "name": "Ping",
| "cId": ${pingSeqId},
| "dType": 1
| },
| "body": {}
|}
""".stripMargin.replaceAll("[\\s\n\r]", "")
)
the client actions are sent in a loop untill timeout:
asLongAs(
session =>
!timeoutIsOver(startTime, testDurationMillis)
) {
exec(doClientAction())
}
The logic works as expected until the ping request comes from server. After that the ws await timeout breaks. Here is what I see in logs:
DUMP---> The client action: event=INITIAL_EVENT oneRound=false
DUMP---> Sending CLIENT_INITIAL_EVENT and expect 2 or 3 responses. wait for each response for 600 seconds
20:33:18.772 [INFO ] i.g.h.a.w.WsSendTextFrame - Sending text frame {"header":{"name":"Action","cId":103,"dType":2},"body":{}} with websocket 'gatling.http.webSocket': Scenario 'doUntilTimeout', UserId #1
20:33:18.773 [DEBUG] i.g.h.a.w.f.WsIdleState - Send text frame requestClientAction {"header":{"name":"Action","cId":103,"dType":2},"body":{}}
20:33:18.773 [DEBUG] i.g.h.c.i.WebSocketHandler - ctx.write msg=TextWebSocketFrame(data: UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeHeapByteBuf(ridx: 0, widx: 164, cap: 512))
20:33:18.773 [TRACE] i.n.h.c.h.w.WebSocket08FrameEncoder - Encoding WebSocket Frame opCode=1 length=164
20:33:18.773 [DEBUG] i.g.h.a.w.f.WsIdleState - Trigger check after sending text frame
20:33:18.787 [TRACE] i.n.h.c.h.w.WebSocket08FrameDecoder - Decoding WebSocket Frame opCode=1
20:33:18.787 [TRACE] i.n.h.c.h.w.WebSocket08FrameDecoder - Decoding WebSocket Frame length=80
20:33:18.787 [DEBUG] i.g.h.c.i.WebSocketHandler - Read msg=TextWebSocketFrame(data: PooledUnsafeDirectByteBuf(ridx: 0, widx: 80, cap: 80))
20:33:18.788 [DEBUG] i.g.h.a.w.f.WsPerformingCheckState - Received matching message {"header":{"cId":103,"name":"ClientAction","code":1,"dType":2}}
20:33:18.789 [DEBUG] i.g.h.a.w.f.WsPerformingCheckState - Current check success
20:33:18.789 [DEBUG] i.g.h.a.w.f.WsPerformingCheckState - Perform next check sequence
20:33:19.233 [TRACE] i.n.h.c.h.w.WebSocket08FrameDecoder - Decoding WebSocket Frame opCode=1
20:33:19.233 [TRACE] i.n.h.c.h.w.WebSocket08FrameDecoder - Decoding WebSocket Frame length=1480
20:33:19.233 [DEBUG] i.g.h.c.i.WebSocketHandler - Read msg=TextWebSocketFrame(data: PooledUnsafeDirectByteBuf(ridx: 0, widx: 1480, cap: 1480))
20:33:19.235 [DEBUG] i.g.h.a.w.f.WsPerformingCheckState - Received matching message {"header":{"cId":37,"name":"ClientEvent","dType":2,"dId":1270},"body":{...}}
20:33:19.237 [DEBUG] i.g.h.a.w.f.WsPerformingCheckState - Current check success
20:33:19.238 [DEBUG] i.g.h.a.w.f.WsPerformingCheckState - Perform next check sequence
20:33:20.871 [TRACE] i.n.h.c.h.w.WebSocket08FrameDecoder - Decoding WebSocket Frame opCode=1
20:33:20.871 [TRACE] i.n.h.c.h.w.WebSocket08FrameDecoder - Decoding WebSocket Frame length=65
20:33:20.871 [DEBUG] i.g.h.c.i.WebSocketHandler - Read msg=TextWebSocketFrame(data: PooledUnsafeDirectByteBuf(ridx: 0, widx: 65, cap: 65))
20:33:20.872 [DEBUG] i.g.h.a.w.f.WsPerformingCheckState - Received matching message {"header":{"cId":38,"name":"Ping","dType":2}}
20:33:20.872 [DEBUG] i.g.h.a.w.f.WsPerformingCheckState - Current check success
20:33:20.872 [DEBUG] i.g.h.a.w.f.WsPerformingCheckState - Check sequences completed successfully
DUMP---> 2 or 3 responses got
DUMP---> Sending PONG for CLIENT_INITIAL_EVENT
20:33:20.873 [INFO ] i.g.h.a.w.WsSendTextFrame - Sending text frame {"header":{"name":"Ping","cId":38,"dType":1},"body":{}} with websocket 'gatling.http.webSocket': Scenario 'doUntilTimeout', UserId #1
20:33:20.873 [DEBUG] i.g.h.a.w.f.WsIdleState - Send text frame requestPong {"header":{"name":"Ping","cId":38,"dType":1},"body":{}}
20:33:20.873 [DEBUG] i.g.h.c.i.WebSocketHandler - ctx.write msg=TextWebSocketFrame(data: UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeHeapByteBuf(ridx: 0, widx: 65, cap: 256))
20:33:20.873 [TRACE] i.n.h.c.h.w.WebSocket08FrameEncoder - Encoding WebSocket Frame opCode=1 length=65
....
20:33:20.876 [INFO ] i.g.h.a.w.WsSendTextFrame - Sending text frame {"header":{"name":"ClientAction","cId":104,"dType":2},"body":{"type":"NON-INITIAL_CLIENT_EVENT","seqId":304,"data":{...}}} with websocket 'gatling.http.webSocket': Scenario 'doUntilTimeout', UserId #1
20:33:20.876 [DEBUG] i.g.h.a.w.f.WsIdleState - Send text frame requestClientAction {"header":{"name":"CLientAction","cId":104,"dType":2},"body":{"type":"NON-INITIAL_CLIENT_EVENT","seqId":304,"data":{...}}}
20:33:20.876 [DEBUG] i.g.h.c.i.WebSocketHandler - ctx.write msg=TextWebSocketFrame(data: UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeHeapByteBuf(ridx: 0, widx: 167, cap: 512))
20:33:20.876 [TRACE] i.n.h.c.h.w.WebSocket08FrameEncoder - Encoding WebSocket Frame opCode=1 length=167
20:33:20.877 [DEBUG] i.g.h.a.w.f.WsIdleState - Trigger check after sending text frame
20:33:20.897 [TRACE] i.n.h.c.h.w.WebSocket08FrameDecoder - Decoding WebSocket Frame opCode=1
20:33:20.897 [TRACE] i.n.h.c.h.w.WebSocket08FrameDecoder - Decoding WebSocket Frame length=81
20:33:20.897 [DEBUG] i.g.h.c.i.WebSocketHandler - Read msg=TextWebSocketFrame(data: PooledUnsafeDirectByteBuf(ridx: 0, widx: 81, cap: 81))
20:33:20.898 [DEBUG] i.g.h.a.w.f.WsPerformingCheckState - Received matching message {"header":{"cId":104,"name":"ClientAction","code":1,"dType":2}}
20:33:20.899 [DEBUG] i.g.h.a.w.f.WsPerformingCheckState - Current check success
20:33:20.899 [DEBUG] i.g.h.a.w.f.WsPerformingCheckState - Perform next check sequence
....
20:33:21.535 [TRACE] i.n.h.c.h.w.WebSocket08FrameDecoder - Decoding WebSocket Frame opCode=1
20:33:21.535 [TRACE] i.n.h.c.h.w.WebSocket08FrameDecoder - Decoding WebSocket Frame length=2275
20:33:21.535 [DEBUG] i.g.h.c.i.WebSocketHandler - Read msg=TextWebSocketFrame(data: PooledUnsafeDirectByteBuf(ridx: 0, widx: 2275, cap: 2275))
20:33:21.537 [DEBUG] i.g.h.a.w.f.WsPerformingCheckState - Received matching message {"header":{"cId":39,"name":"ClientEvent","dType":2},"body":{...}}
20:33:21.540 [DEBUG] i.g.h.a.w.f.WsPerformingCheckState - Current check success
20:33:21.540 [DEBUG] i.g.h.a.w.f.WsPerformingCheckState - Check sequences completed successfully
DUMP---> NON-INITIAL_CLIENT_EVENT 2 responses got
....
DUMP---> Sending CLIENT_INITIAL_EVENT and expect 2 or 3 responses. wait for each response for 600 seconds
20:33:21.542 [INFO ] i.g.h.a.w.WsSendTextFrame - Sending text frame {"header":{"name":"ClientAction","cId":105,"dType":2},"body":{...}}} with websocket 'gatling.http.webSocket': Scenario 'doUntilTimeout', UserId #1
20:33:21.542 [DEBUG] i.g.h.a.w.f.WsIdleState - Send text frame requestClientAction {"header":{"name":"ClientAction","cId":105,"dType":2},"body":{...}}}
20:33:21.542 [DEBUG] i.g.h.c.i.WebSocketHandler - ctx.write msg=TextWebSocketFrame(data: UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeHeapByteBuf(ridx: 0, widx: 164, cap: 512))
20:33:21.542 [TRACE] i.n.h.c.h.w.WebSocket08FrameEncoder - Encoding WebSocket Frame opCode=1 length=164
20:33:21.543 [DEBUG] i.g.h.a.w.f.WsIdleState - Trigger check after sending text frame
20:33:21.558 [TRACE] i.n.h.c.h.w.WebSocket08FrameDecoder - Decoding WebSocket Frame opCode=1
20:33:21.559 [TRACE] i.n.h.c.h.w.WebSocket08FrameDecoder - Decoding WebSocket Frame length=81
20:33:21.559 [DEBUG] i.g.h.c.i.WebSocketHandler - Read msg=TextWebSocketFrame(data: PooledUnsafeDirectByteBuf(ridx: 0, widx: 81, cap: 81))
20:33:21.560 [DEBUG] i.g.h.a.w.f.WsPerformingCheckState - Received matching message {"header":,"cId":105,"name":"ClientAction","code":1,"dType":2}}
20:33:21.560 [DEBUG] i.g.h.a.w.f.WsPerformingCheckState - Current check success
20:33:21.561 [DEBUG] i.g.h.a.w.f.WsPerformingCheckState - Perform next check sequence
20:33:21.742 [DEBUG] i.g.h.a.w.f.WsPerformingCheckState - Check timeout
20:33:21.743 [DEBUG] i.g.h.a.w.f.WsPerformingCheckState - Check timeout, failing it and performing next action
DUMP---> 2 or 3 responses got
20:33:21.744 [DEBUG] i.g.c.a.Exit - End user #1
20:33:21.748 [DEBUG] i.g.c.c.i.Injector - End user #doUntilTimeout
20:33:21.748 [INFO ] i.g.c.c.i.Injector - All users of scenario doUntilTimeout are stopped
20:33:21.749 [INFO ] i.g.c.c.i.Injector - Stopping
20:33:21.749 [INFO ] i.g.c.c.Controller - Injector has stopped, initiating graceful stop
I received the first message from web socket at 20:33:21.560
Then the second "await" started. It should timeout after 600 seconds,
but in fact I see the timeout right away at 20:33:21.743
I looks like a bug in Gatling. Something like timeout property reset to zero
Thanks in advance!
Andrei