I'm creating a secured websocket connection between my server using okhttp library. But my websocket connections keep getting closed after 3 minutes. So I added below code to ensure ping/pong:
okHttpClient = new OkHttpClient.Builder()
.pingInterval(20, TimeUnit.SECONDS)
.build();
Request request = new Request.Builder().url(freeSwitchHost).build();
vertoWebSocket = okHttpClient.newWebSocket(request, pstnWebSocketListener);
okHttpClient.dispatcher().executorService().shutdown();
This is the code I am using to send ping for my server. But unfortunately I am not able to capture any data in pcap that ping is being sent by app. Is there a way to capture the ping being sent by okhttp websocket?
When ping seems to be not working, I tried below code to send keep-alive. But I'm not able to capture that too in pcap as I'm using secured websocket connections.
okHttpClient = new OkHttpClient.Builder()
.connectionPool(new ConnectionPool(100, 30, TimeUnit.SECONDS))
.build();
This is the code I'm using to send keep-alive. But still same response. I'm not sure whether my websocket connection is sending keep-alive or ping pong. Is there way to identify ping and keep-alive is being sent by my websocket.
Sadly, I haven't used this library, so I will assume that there are no errors in the code.
I would suggest you use two really useful tools in Wireshark:
Packet lengths (Statistics > Packet Lengths)
IO Graphs (Statistics > I/O Graphs)
Record a nice Pcap, where the channel is idle so that you can observe the keepalive packets clearly. If this is not an option, you can always use a filter to show you only small packets frame.len<250.
With the packet lengths, I would look for small packets (around 80-159), since the keepalive packets are not that big, and also look at the "Rate" field, as it may help you see if there are patterns.
With the IO Graphs, I would set the interval at something smaller than 100ms and look for peaks 3 seconds apart (which were caused by the keepalive packets).
Related
This question already has answers here:
Why HTTP/2 does multiplexing altough tcp does same thing?
(3 answers)
Closed last year.
This passage claims that the binary frame layer becomes the base for multiplexing in http for TCP connection, which is confusing to me.
https://developers.google.com/web/fundamentals/performance/http2#design_and_technical_goals
The confusing part is the HTTP client can just send more requests in one TCP connection without waiting for the response and receive the response for the corresponding request. That is the "frame" is the request and response. So why should it add the binary frame?
Let's have a look at what you're suggesting:
the HTTP client can just send more requests in one TCP connection without waiting for the response
So far, so good: I can send "GET /foo" and then immediately "GET /bar" on the same connection.
and receive the response for the corresponding request
So, the server replies "200 OK" with some HTML content, and ... wait, is that for "/foo" or "/bar"? The key word in your own description is "corresponding" - we need some way of saying "this response corresponds to request #1".
And then, halfway through sending the first response, the server finishes handling the other request, and is ready to send part of a different response; but if it jumps in with "200 OK", that's going to appear to be part of the response it's already sending. So we also need to be able to say "this is the start of a new response", and "this content is the continuation of response #2".
To do that, we need a new abstraction: a frame, with a header which can encode details like "the next 100 bytes are the start of response #2, which corresponds to request #1". (I'm not sure if that's exactly how an HTTP/2 frame works, but I think it's roughly the principle.)
We could do that and still keep the protocol human readable (which is what we really mean by "text-based" vs "binary") but there's going to be a lot of these frame headers, so the shorter we can make them, the better. So if we're interested in performance, we can give up on "human readable" as a requirement, and we end up with a binary framing protocol like HTTP/2.
I have a question about Spring Reactive WebClient...
Few days ago I decided to play with the new reactive stuff in Spring Framework and I made one small project for scraping data only for personal purposes. (making multiple requests to one webpage and combining the results).
I started using the new reactive WebClient for making requests but the problem I found is that the client not emitting response for every request. Sounds strange. Here is what I did for fetching data:
private Mono<String> fetchData(String uri) {
return this.client
.get()
.uri(uri)
.header("X-Fsign","SW9D1eZo")
.retrieve()
.bodyToMono(String.class)
.timeout(Duration.ofSeconds(35))
.log("category", Level.ALL, SignalType.ON_ERROR, SignalType.ON_COMPLETE, SignalType.CANCEL, SignalType.REQUEST);
}
And the function that calls fetchData:
public Mono<List<Stat>> fetch() {
return fetchData(URL)
.map(this::extractUrls)
.doOnNext(System.out::println)
.doOnNext(s-> System.out.println("all ids are "+s.size()))
.flatMapIterable(q->q)
.map(s -> s.substring(7, 15))
.map(s -> "http://d.flashscore.com/x/feed/d_hh_" + s + "_en_1") // list of N-length urls
.flatMap(this::fetchData)
.map(this::extractHeadToHead)
.collectList();
}
and the subscriber:
FlashScoreService bean = ctx.getBean(FlashScoreService.class);
bean.fetch().subscribe(s->{
System.out.println("finished !!! " + s.size()); //expecting same N-length list size
},Throwable::printStackTrace);
The problem is if I made a little bit more requests > 100.
I didn't get responses for all of them, no error is thrown or error response code is returned and subscribe method is invoked with size different from the number of requests.
The requests I made are based on List of Strings (urls) and after all responses are emitted I should receive all of them as list because I'm using collectList(). When I execute 100 requests, I expect to receive list of 100 responses but actually I'm receiving sometimes 100, sometimes 96 etc ... May be something fails silently.
This is easy reproducible here is my github project link.
Sample output:
all ids are 176
finished !!! 171
Please give me suggestions how I can debug or what I'm doing wrong. Help is appreciated.
Update:
The log shows if I pass 126 urls for example:
onNext(ReactorClientHttpResponse{request=[GET/some_url],status=200}) is called 121 times. May be here is the problem.
onComplete() is called 126 times which is the exact same length of the passed list of urls
but how it's possible some of the requests to be completed without calling onNext() or onError( ) ? (success and error in Mono)
I think the problem is not in the WebClient but somewhere else. Environment or server blocking the request, but may be I should receive some error log.
ps. Thanks for the help !
This is a tricky one. Debugging the actual HTTP frames received, it seems we're really not getting responses for some requests. Debugging a little more with Wireshark, it looks like the remote server is requesting the end of the connection with a FIN, ACK TCP packet and that the client acknowledges it. The problem is this connection is still taken from the pool to send another GET request after the first FIN, ACK TCP packet.
Maybe the remote server is closing connections after they've served a number of requests; in any case it's perfectly legal behavior. Note that I'm not reproducing this consistently.
Workaround
You can disable connection pooling on the client; this will be slower and apparently doesn't trigger this issue. For that, use the following:
this.client = WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(new Consumer<HttpClientOptions.Builder>() {
#Override
public void accept(HttpClientOptions.Builder builder) {
builder.disablePool();
}
}))
.build();
Underlying issue
The root problem is that the HTTP client should not onComplete when the TCP connection is closed without sending a response. Or better, the HTTP client should not reuse a connection while it's being closed. I'll report back here when I'll know more.
I have a really weird problem that is driving me crazy.
I have a Ruby server and a Flash client (Action Script 3). It's a multiplayer game.
The problem is that everything is working perfect and then, suddenly, a random player stops receiving data. When the server closes the connection because of inactivity, about 20-60 seconds later, the client receives all the buffered data.
The client uses XMLsocket for retrieving data, so the way the client receives data is not the problem.
socket.addEventListener(Event.CONNECT, connectHandler);
function connectHandler(event)
{
sendData(sess);
}
function sendData(dat)
{
trace("SEND: " + dat);
addDebugData("SEND: " + dat)
if (socket.connected) {
socket.send(dat);
} else {
addDebugData("SOCKET NOT CONNECTED")
}
}
socket.addEventListener(DataEvent.DATA, dataHandler);
function dataHandler(e:DataEvent) {
var data:String = e.data;
workData(data);
}
The server flushes data after every write, so is not a flushing problem:
sock.write(data + DATAEOF)
sock.flush()
DATAEOF is null char, so the client parses the string.
When the server accepts a new socket, it sets sync to true, to autoflush, and TCP_NODELAY to true too:
newsock = serverSocket.accept
newsock.sync = true
newsock.setsockopt(Socket::IPPROTO_TCP, Socket::TCP_NODELAY, true)
This is my research:
Info: I was dumping netstat data to a file each second.
When the client stops receiving data, netstat shows that socket status is still ESTABLISHED.
Some seconds after that, send-queue grows accordingly to data sent.
tcpflow shows that packets are sent 2 times.
When the server closes the socket, socket status changes to FIN_WAIT1, as expected. Then, tcpflow shows that all buffered data is sent to the client, but the client don't receives data. some seconds after that, connection dissapears from netstat and tcpflow shows that the same data is sent again, but this time the client receives the data so starts sending data to the server and the server receives it. But it's too late... server has closed connection.
I don't think it's an OS/network problem, because I've changed from a VPS located in Spain to Amazon EC2 located in Ireland and the problem still remains.
I don't think it's a client network problem too, because this occurs dozens of times per day, and the average quantity of online users is about 45-55, with about 400 unique users a day, so the ratio is extremely high.
EDIT:
I've done more research. I've changed the server to C++.
When a client stops sending data, after a while the server receives a "Connection reset by peer" error. In that moment, tcpdump shows me that the client sent a RST packet, this could be because the client closed the connection and the server tried to read, but... why the client closed the connection? I think the answer is that the client is not the one closing the connection, is the kernel. Here is some info: http://scie.nti.st/2008/3/14/amazon-s3-and-connection-reset-by-peer
Basically, as I understand it, Linux kernels 2.6.17+ increased the maximum size of the TCP window/buffer, and this started to cause other gear to wig out, if it couldn’t handle sufficiently large TCP windows. The gear would reset the connection, and we see this as a “Connection reset by peer” message.
I've followed the steps and now it seems that the server is closing connections only when the client losses its connection to internet.
I'm going to add this as an answer so people know a bit mroe about this.
I think the answer is that the kernel is the one closing the connection. Here is some info: http://scie.nti.st/2008/3/14/amazon-s3-and-connection-reset-by-peer
Basically, as I understand it, Linux kernels 2.6.17+ increased the maximum size of the TCP window/buffer, and this started to cause other gear to wig out, if it couldn’t handle sufficiently large TCP windows. The gear would reset the connection, and we see this as a “Connection reset by peer” message.
I've followed the steps and now it seems that the server is closing connections only when the client losses its connection to internet.
I am programming a client application sending TCP/IP packets to a server. Because of timeout issues I want to start a timer as soon as the ACK-Package is returned (so there can be no timeout while the package has not reached the server). I want to use the winapi.
Setting the Socket to blocking mode doesn't help, because the send command returns as soon as the data is written into the buffer (if I am not mistaken). Is there a way to block send till the ACK was returned, or is there any other way to do this without writing my own TCP-implementation?
Regards
It sounds like you want to do the minimum implementation to achieve your goal. In this case you should set your socket to blocking, and following the send which blocks until all data is sent, you call recv which in turn will block until the ACK packet is received or the server end closes or aborts the connection.
If you wanted to go further with your implementation you'd have to structure your client application in such a way that supports asynchronous communication. There are a few techniques with varying degrees of complexity; polling using select() simple, event model using WSASelectEvent/WSAWaitForMultipleEvents challenging, and the IOCompletionPort model which is very complicated.
peudocode... Will wait until ack is recevied, after which time you can call whatever functionallity you want -i chose some made up function send_data.. which would then send information over the socket after receiving the ack.
data = ''
while True
readable, writable, errors = select([socket])
if socket in readble
data += recv(socket)
if is_ack(data)
timer.start() #not sure why you want this
break
send_data(socket)
so.. I'm doing a small multiplayer game with blocking UDP and IO.select. To my problem.. (In the server) reading from a UDP socket (packet, sender = #socket.recvfrom(1000)) which have just sent a packet to a dead client results in a ICMP unreachable (and exception Errno::ECONNRESET in ruby). The problem is that I can't find any way whatsoever to extract the IP of that ICMP.. so I can clean out that dead client.
Anyone know how to achieve this?
thanks
You'll need to call recvmsg for the socket, and pass MSG_ERRQUEUE as the flag.
The original destination address of the datagram that caused the error is supplied via msg_name.
It's worth noting that the source IP address of the ICMP packet will not always be the same address as your client. Any router that handles packets for this connection could be the source, and the payload of the ICMP packet would contain the IP header + the first 8 bytes of the packet it relates to.