While writing an app that uses socket.io, I'm finding that the heartbeat debug messages drown out the debug messages I want to see. What's the best way to shut off the debug messages for only the heartbeat?
Removing heartbeat debug messages from socket.io
Set up your socket.io with dynamic environmental configurations, and for the development environment turn down the 'heartbeat interval'
Here's some tips, most of them come from the Configuring-Socket.IO page of the socketIO v09 wiki.
var io = require('socket.io').listen(80);
// This is sugar for if process.env.NODE_ENV==='production'
io.configure('production', function(){
// send minified client
io.enable('browser client minification');
// apply etag caching logic based on version number
io.enable('browser client etag');
io.enable('browser client gzip'); // gzip the file
io.set('log level', 1); // reduce logging
// enable all transports
// (optional if you want flashsocket support,
// please note that some hosting
// providers do not allow you to create servers that listen on a
// port different than 80 or their
// default port)
io.set('transports', [
'websocket'
, 'flashsocket'
, 'htmlfile'
, 'xhr-polling'
, 'jsonp-polling'
]);
});
io.configure('development', function(){
io.set('heartbeats', false); //removes heartbeats
io.set('log level', 1); // reduces all socket.io logging, including heartbeats.
});
So do that, and then you're good on the heartbearts being on (the default) in production, and while you're at it -- you got the basic suggested production settings.
Also, depending on framework, or where exactly in the express/socket.io/top-level-app way of making things, you might have a setup like.
app.io.enable('browser client minification');
or
app.configure(function() {
app.enable('junny')
app.set('foo', 'bar')
});
So, to then control the heartbeats, first make sure you're getting the most out of your express/api logging, so load your app like this:
$ DEBUG=* node ./app.js
Then detect if you're in the development env (most top level way):
if (process.env.NODE_ENV === 'development') {
io.set('heartbeat interval', 60*750);
}
// 60*750 = 45000ms = 45 seconds.
That would be turn the heartbeat interval to 45 seconds.
One can find all options at the socket.io configuration wiki page, a blockquote most relevant to this question:
heartbeat timeout defaults to 60 seconds
The timeout for the client, we should receive a heartbeat from the
server within this interval. This should be greater than the heartbeat
interval. This value is sent to the client after a successful
handshake.
heartbeat interval defaults to 25 seconds
The timeout for the server when it should send a new heartbeat to the
client.
If you want the heartbeats/message still then adjust those times upward, you'll have to test on your own system, but follow the general guideline/ratio and then adjust/test from there.
Related
Under load in production we receive "RabbitMQ.Client.Exceptions.ConnectFailureException" connection failed and "MassTransit.RequestTimeoutException" timeout waiting for response. The consumer does receive the message and send it back. It's like the web app isn't listening, or unable to accept the connection.
We're running an ASP.NET web application ( not MVC ) on .NET Framework 4.6.2 on Windows Server 2019 on IIS. We're using MassTransit 7.0.4. In production, under load, we can get some exceptions dealing with sockets on RabbitMQ or timeouts from masstransit. It's difficult to reproduce them in Dev. RabbitMQ is in a mirror, it seems to happen once we turn on a high-load service that bumps from 140 message/sec to 250 message/sec.
I have a few questions about the code architecture, and then if anyone else is running into these kinds of timeout issues.
Questions:
Should I have static scope for the IBusControl? IE, should it be static inside Global asax? And does it matter at all if it's a singleton underneath?
Should I create a new IBusControl and start it per request ( maybe stick it in Application BeginRequest ). Would that make a difference?
Would adding another worker process affect the total number of open connections I'm able to make -- If this is a resource issue ( exhausting threads, connections or some resource ).
Exceptions:
MassTransit.RequestTimeoutException
Timeout Waiting for response
Stacktrace:
System.Runtime.ExceptionServices.ExceptionDispathInfo.Throw
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
MassTransit.Clients.ResponseHandlerConnectionHandle`1+<GetTask>d_11.MoveNext
System.Threading.ExecutionContext.RunInternal
RabbitMQ.Client.Exceptions.ConnectFailureException
Connection failed
Statcktrace:
RabbitMQ.Client.Impl.SocketFrameHandler.ConnectOrFail
RabbitMQ.Client.Impl.SocketFrameHandler.ConnectUsingAddressFamily
RabbitMQ.Client.Impl.SocketFrameHandler..ctor
RabbitMQ.Client.ConnectionFactory.CreateFrameHandler
RabbitMQ.Client.EndPointResolverExtensions.SelectOne
RabbitMQ.Client.ConnectionFactory.CreateConnection
How Our Code Works ( overview )
Static IBusControl that is instantiated the first time someone tries to produce a message. The whole connection and send code is a little large to put in here ( connection factory and other metric classes, but below are the interesting parts ).
Static IBusControl B;
B = Bus.Factory.CreateUsingRabbitMq(x =>
{
hostAddress = host.HostAddress;
x.Host(new Uri(host.HostAddress), h =>
{
h.Username(host.UserName);
h.Password(host.Password);
});
x.Durable = false;
x.SetQueueArgument("x-message-ttl", 600000);
});
B.Start(new TimeSpan(0, 0, 10));
// Then send the Actual Messages
// Generic with TRequest and TResponse : class BaseMessage
// Pulling the code out of a few different classes
string serviceAddressString = string.Format("{0}/{1}?durable={2}", HostAddress, ChkMassTransit.QueueName(typeof(TRequest), typeof(TResponse)), false ? "true" : "false");
Uri serviceAddress = new Uri(serviceAddressString);
RequestTimeout rt = RequestTimeout.After(0, 0, 0, 0, timeout.Value);
IRequestClient<TRequest> reqClient = B.CreateRequestClient<TRequest>(serviceAddress, rt);
var v = reqClient.GetResponse<TResponse>(request, sendInfo.CT, sendInfo.RT);
if ( v.Wait(timeoutMS) ) { /*do some stuff*/ }
First, I find your lack of async disturbing. Using Wait or anything like it on TPL-based code is a recipe for death and destruction, pain and suffering, dogs and cats living together, etc.
Yes, you should have a single bus instance that is started when the application starts. Since you're doing request/response, set AutoStart = true on the bus configurator to make sure it's all warmed up and ready.
Never, no, one bus only!
Each bus instance only has a single connection, so you shouldn't see any resource issues related to capacity on RabbitMQ.
MassTransit 7.0.4 is really old, you might consider the easy upgrade 7.3.1 and see if that improves things for you. It's the last version of the v7 codebase available.
We set the reconnectionDelayMax (docs-ref) to max. 10 seconds.
So when a user is offline for a while socket.io will retry only every 10 seconds - which is good, but it also means, that a user must wait for up to 10 seconds when they get online.
Which is basically also okay, but can be improved in some cases:
We want to listen to the navigator.ononline event and then "reset" the reconnection logic: i.e. try to reconnect immediately (with the initial reconnectionDelay)
Is this possible in socket.io (client v2.x)
You could try adding something like this that would fire when the browser goes online and (re)connect the socket
window.addEventListener('online', function () {
const socket = io('http://localhost:3000');
});
I'm writing a new SPA application that will subscribes to several rooms for several types of information updates.
In my production setup I'll use two servers behind a load balancer for reliability.
In the event of disconnect - Does the client have to resend the request for rooms subscriptions on the reconnect event callback, or is there a way to have the server reconnect the client automatically (even when the client reconnects to a different server due to server failure) ?
Socket.io will unsubscribe your users from all rooms on a disconnect. It will unsubscribe you from the server side. I played around with this a little. The server can store your user's rooms in redis or a database under the user ID and, upon connecting, check to see if that user should be in any of these rooms. At which time your user can join them from the server side without ever having to do anything from the client.
The problem is that this list of rooms must be constantly stored and updated. It's just another thing that has to work seamlessly on the backend. It's a lot of tests to consider all the possibilities that could mess up your organization. Like, what if they log in on another device, you have to clear the rooms and put in new ones, but if the user opens his laptop again and it reconnects, now he has to get back in those rooms from his laptop. ...It's totally doable/solvable, but I only did this on the front end:
// rejoin if there's a disconnect
mySocket.on('reconnect', () => {
mySocket.emit('subscribe', 'theRoom')
})
...and no further hassle. If you added some more details about why it's necessary to do it from the server..?
From my experience, I found this to be the easiest and useful solution:
Client side:
// the next 3 functions will be fired automatically on a disconnect.
// the disconnect (the first function) is not required, but you know,
// you can use it make some other good stuff.
socket.on("disconnect", function() {
console.log("Disconnected");
});
socket.on("reconnect", function() {
// do not rejoin from here, since the socket.id token and/or rooms are still
// not available.
console.log("Reconnecting");
});
socket.on("connect", function() {
// thats the key line, now register to the room you want.
// info about the required rooms (if its not as simple as my
// example) could easily be reached via a DB connection. It worth it.
socket.emit("registerToRoom", $scope.user.phone);
});
Server side:
io.on('connection', function(socket){
socket.on("registerToRoom", function(userPhone) {
socket.join(userPhone);
});
});
And thats it. Very simple and straight forward.
You also can add in the connected socket (the last function) some more updates to the user display, such as refreshing its index or something else.
Socket.io does have a reconnect event - Docs here
Something like the below should work
socket.on('reconnect', () => attemptReconnection())
The attempt reconnection callback would look something like:
const attemptReconnection = () => socket.emit('joinRoom', roomId)
I have started to learn web sockets. It is must learn technology in today's time.
But i am curious to learn more about it. My basic question is How many WebSocket connection can be created on Client Side.
My Typically Application is html UI based and on the server i have rest based services. I need to track whether
Session timeout has happed or not
Whether Connection to the server is lost or not ? A kind of pooling program to check with connections is alive or not.
So I am creating 2 websocket objects on client and different url for them.
I hope i have implemented it correctly ?
Basically Browser closes the old websocket connection once you opened to new connection to SAME URL(ws://127.0.0.1:8080/WebSocket-context-root/getResource). You can keep small hack like "ws://127.0.0.1:8080/WebSocket-context-root/getResource/"+k. where k is any number/any random string. On server side just ignore the path variable k.
In this way you can open many number of connection at same time. Browser restriction of max-number-connection per domain is not applying here (Tested on Firefox). I tried max 25 parallel connections.
You can use websocket.readyState to check the status of the web socket connection.
onclose Event of the Web socket have reason code for closed connection.
User below code to test number of active connections.
var x=0
var intervalID = setInterval(function () {
websocket = new WebSocket("ws://127.0.0.1:8080/WebSocketApi/web/chat/"+x);
websocket.onopen = function (evt) {
console.log('open')
}
websocket.onmessage = function (evt) {
console.log('msg');
}
websocket.onclose= function (evt) {
console.log('closed');
}
if (++x === 15) {
window.clearInterval(intervalID);
}
}, 1);
How long can the browser wait before an error is shown before server answers for request? Can this time be unlimited?
If you are using a jQuery $.ajax call you can set the timeout property to control the amount of time before a request returns with a timeout status. The timeout is set in milliseconds, so just set it to a very high value. You can also set it to 0 for "unlimited" but in my opinion you should just set a high value instead.
Note: unlimited is actually the default but most browsers have default timeouts that will be hit.
When an ajax call is returned due to timeout it will return with an error status of "timeout" that you can handle with a separate case if needed.
So if you want to set a timeout of 3 seconds, and handle the timeout here is an example:
$.ajax({
url: "/your_ajax_method/",
type: "GET",
dataType: "json",
timeout: 3000, //Set your timeout value in milliseconds or 0 for unlimited
success: function(response) { alert(response); },
error: function(jqXHR, textStatus, errorThrown) {
if(textStatus==="timeout") {
alert("Call has timed out"); //Handle the timeout
} else {
alert("Another error was returned"); //Handle other error type
}
}
});
Yes and no. Yes the server can do it or be configured to do so, no the browsers (i dont know about version/distributor specifics) may have timeouts enabled.
There are 2 solutions though for achieving/emulating this over HTTP:
If this is simple a long running script and you're waiting for results this isnt the way to go, you should instead do as previous poster mentioned and use async processing with server polling for the results, this would be a much more sure fire solution. For example: a thumbnail script from an image processor server side: the user uploads an image, the server immediately returns a 200 and a "Job ID". The client (javascript^^) can then use the JobID to request the job status/result.
If your goal is to have something like a realtime connection between browser and server (1 way connection, once the request is made by the browser no further info can be sent without using new requests (ajax^^)), this is called long polling/reverse ajax and can be used for real-time communication over http. There are several techniques using 2 long polled requests in parallel so that once one of them timeout the second one becomes the active and the first one attempts to reconnect.
Can you explain a bit more about what you're trying to achieve - do you have a long running process on a server, do you want to change the settings on just a local machine or are you after a way to manage it for large numbers of users?
How long the browser will wait depends on a number of factors e.g. where the timeout occurs - is it at the TCP level, the server or the local browser?
If you've got a long running process on a server and you want to update a webpage afterwards the typical way to handle it is to run the long process asynchronously and notify the client when it's complete e.g. have an ajax call that polls the server, or use HTTP 1.1 and serve out a notification stream to the client.
In either case it's still possible for the connection to be closed so the client will still need the ability to re-open it.
I found, that in case of a normal (HTML page) request, browsers run to timeout after cca. 30 secs. It's important, because other participiants probably follows it: proxies, routers (do routers play in this game? I'm not sure). I am using 4 sec long server-side delay (if there's nothing to send to the client), and my AJAX client performs another HTTP request immediatelly (I am on local network, there's no internet lag). 4 sec is long enough to not to overload the server and network with frequented polls, and is short enough for the case, when somehow one poll falls out of the row which the client can't detect and handle.
Also, there're other issues with comet (long HTTP request): browser's limit on number of simultaneous HTTP request, handling of client-side events (must sent to the server immediatelly), server/network down detection and recovery, multi user handling etc.