I'm using Closure client-side and Tornado server side. I create a socket:
this.socket = goog.net.WebSocket(true)
and then open it:
this.socket.open(theSocketUrl)
Every works fine including messages being passed correctly. However, once per minute (once every 60 to 61 seconds), the socket closes and then reopens. There are no errors server-side and the Closure socket error event doesn't get called.
I've added logging to tornado and something seems to be calling on_connection_close() which then calls the socket's method on_close(). The close() method itself does not get called.
Any idea why this might be happening?
Are you using nginx or some other reverse-proxy in front of your tornado server? I've seen this happen when the proxy timeout elapses and then nginx closes the connection, causing the behavior you're seeing.
You can change the proxy_send_timeout and proxy_read_timeout in nginx to prevent this. Just make sure that when you edit proxy.conf, you include it from your main nginx.conf.
Related
I have Delphi application that uses Indy HTTP components (that uses Windows socket). And from time to time I am receiving #10060 socket error (WSAETIMEDOUT - An attempt to connect timed out without establishing a connection) upon execution of Indy procedure:
CheckForSocketError(IdWinsock2.Connect(ASocket, #LAddr, SizeOf(LAddr)));
...
connect : TconnectProc;
...
TconnectProc = function ( const s: TSocket; const name: PSockAddr; const namelen: Integer): Integer; stdcall;
Actually all this is just wrap around Windows connect function https://learn.microsoft.com/en-us/windows/win32/api/winsock2/nf-winsock2-connect that gives error and message WSAETIMEDOUT. So - my question is - can this be programming error at all? Even if I have server running on the other computer and even that server has trouble serving request, even in such cases the low level connect should execute normally, if the server can not serve the GET/POST request, then, of course, the errors should be but those errors should spring in only during execution of other Socket functions not in the connect function, isn't so?
I am trying to solve my problem https://serverfault.com/questions/973648/is-it-possible-that-unencrypted-traffic-can-cause-windows-socket-10057-10060?noredirect=1#comment1266907_973648 and now I am seeking whats happening in my code.
My server side code is very simple - it is just TIdHttpServer component with implemented (I provide event name only here):
MyForm.IdHTTPServerCommandGet(AContext: TIdContext;
ARequestInfo: TIdHTTPRequestInfo; AResponseInfo: TIdHTTPResponseInfo);
So - what can be worng with my implemention, what can lead to the appearance of WSAETIMEDOUT for connect? Yes, my procedure can be long sometimes, but it for years it returned the answer sucessfully and there were no communication errors. And I guess, that connect function can even not depend (does not use/raise) OnCommandGet event, so, I have no control how the server side handles socket connect function from the client?
It may be possible that this is connected with TCP (not HTTP) keepalive, maybe some Windows updates have reduced client-side settings of Windows TCP keepalive for the clients and now this manifests as such error.
Indy TCP clients, like TIdHTTP, have a public ConnectTimeout property, which is set to 0 (infinite) by default. If no timeout is specified, a hard-coded 2 minute timeout is used if the client's TIdTCPClient.Connect() method is called in the main UI thread and TIdAntiFreeze is active, otherwise no timeout is used.
If a timeout is used, Indy calls Winsock's connect() function in a worker thread and waits for that thread to terminate. If Indy's timeout elapses, the socket is closed to abort the connect(), and then EIdConnectTimeout is raised to the caller. If connect() exits before Indy's timeout elapses, an exception is raised to the caller only if connect() failed.
If no timeout is used, Indy calls Winsock's connect() directly, waits for it to exit on its own accord, and then raises an exception only if failed.
So, the ONLY way you can get a WSATIMEDOUT error from Indy when it is calling Winsock's connect() function is if Winsock itself timed out internally before Indy's own timeout elapses. That does not necessarily indicate a problem in your code. It just means that the Host you are trying to connect to is simply not reachable at that moment in time. If the server were reachable, but could not accept your connection, you would get a different error, such as WSAECONNREFUSED.
If your server is behind a firewall or router, make sure it is not blocking connections from reaching your server. Try running a packet sniffer on the server machine, such as WireShark, and make sure the 3-way TCP handshake from TIdHTTP is reaching the server machine correctly.
All required changes have been done to respective files like:
stalecheck=true,
keepalive is checked from HTTP request defaults,
retrycount=1,
hc.parameters file changes,
Socket timeout is 240000
Still we see "java.net.SocketException: Connection reset" in response data however I see the valid requests been passed to Server.
The issue wasnt till we reach 3000 users, worked smoothly till 3000 users.
Connection Reset has a lot of meaning, possible reasons are:
One of the server components is not able to handle load so it closes connections on its side
On JMeter side, check that you running in NON GUI mode and that neither JMeter JVM nor injector machine are overloaded which could explain this. See:
https://jmeter.apache.org/usermanual/get-started.html#non_gui
We have a client application which keeps sending some plain text to the server every 5 seconds.
Every 5 seconds, we create a HTTPSClientSession with server IP and Port, and then send the data. Session is closed afer send as HTTPSClientSession is created as a local variable in the sending function.
What we have observed is that in some cases, sending data works for some time and after that it stops working.
Strange thing is that if we disable wifi and enable it again, client resumes sending data.
Any help about how to solve this is appreciated.
Most of the times POCO exception returns "Host not found" , "Timeout" or "Software caused connection abort".
Thanks
A few weeks ago I wrote a small program which created a socket to an apache webserver and made a request.
Back then I did not know that this web server had a KeepAliveTimeout of 5 seconds.
After my first request I waited 1 minute. After this I wanted to reuse my first socket for another webserver request, but got an error.
From Beej's Guide to Network Programming I learned that if recv returns 0, then the other side has closed its connection:
Wait! recv() can return 0. This can mean only one thing: the remote side has closed
the connection on you! A return value of 0 is recv()'s way of letting you know this
has occurred.
My questions are now:
What does Apache send when the KeepAliveTimeout is over - a FIN or a RST packet?
I know that using a TCP connection for 2 unrelated HTTP requests like in this scenario might
not be the best thing. But in order to understand TCP more the next question is:
After my first successful http request, and before sending the next HTTP request over the same socket, would there be somehow a possibility to get informed about this keepalivetimeout TCPsocket termination of the server other than receiving 0 from the next recv() call?
It will send a FIN. If you write a request to the server after that, send() will return -1 with errno/WSAGetLastError() = ECONNRESET.
would there be somehow a possibility to get informed about this keepalivetimeout tcp socket termination of the server
Yes, by reading the proper response header parameter, namely Keep-Alive: timeout=delta-seconds:
'timeout' Parameter
A host sets the value of the timeout parameter to the time that the host will allows an idle connection to remain open before it is closed. A connection is idle if no data is sent or received by a host.
The value of the timeout parameter is a single integer in seconds.
A host MAY keep an idle connection open for longer than the time that it indicates, but it SHOULD attempt to retain a connection for at least as long as indicated.
As you can see, it's up to the host to decide. Given it only SHOULD try to keep the connection open as long as promised, but it isn't required that it does in order to conform to the spec, so the server might decide to close and reuse the connection to serve another pending client.
I'm using beanstalkd to offload some work to other machines. The setup is a bit unusual, the server is on the internet (public ip) but the consumers are behind adsl lines on some peoples homes. So there is a linux server as client going out through a dynamic ip and connecting to the server to get a job. It's all PHP and I'm using pheanstalk library.
Everything runs smoothly for some time, but then the adsl changes the IP (every 24h hours the provider forces a disconnect-reconnect) the client just hangs, never to go out of "reserve".
I thought that putting a timeout on the reserve would help it, but it didn't. As it seems, the client issues a command and blocks, it never checks the timeout. It just issues a reserve-with-timeout (instead of a simple reserve) and it is the servers responsibility to return a TIME_OUT as the timeout occurs. The problem is, the connection is broken (but the TCP/IP doesn't know about that yet until any of the sides try to talk to the other side) and if the client blocked reading, it will never return.
The library seems to have support for some kind of timeouts locally (for example when trying to connect to server), but it does not seem to contemplate this scenario.
How could I detect the stale connection and force a reconnect? Is there some kind of keepalive on the protocol (and on the pheanstalk itself)?
Thanks!
You could try to close each connection right after the request is answered and reopen a new connection each time.
There is no close() function but you deleting the Pheanstaly Object with unset($pheanstalk) will close it.
This explanation is quite helpful:
Pheanstalk (PHP client for beanstalk) - how do connections work?
I haven't tried it yet, but I came up with the idea of connecting to the beanstalk server through an SSH tunnel. We can enable the ServerAliveCountMax and ServerAliveInterval options on the tunnel, so that a network or server failure will cause the tunnel to close. This should then cause the pheanstalk client to report an error.