I have a endpoint in koa, that loads much data from the database and then calculates some results based on it. This can take some seconds.
What happens to a request, if the browser cancels it? Like the browser tab gets closed etc.
For example, would cache.c be filled even if the browser canceled or is koa smart enough to simply stop any actions it started for this request?
const cache = {}
router.get('/data', function *(next) {
if (cache.c) return this.body = cache.c
// somewhere here the browser cancels the request
//---
const a = yield db.getA()
const b = yield db.getB()
cache.c = yield calculateC(a,b)
//---
this.body = cache.c
})
Your generator function will get yielded/evaluated in its entirety regardless of when the request socket closes (e.g. user closes the tab).
Halting further generator execution just because the socket closes isn't an assumption Koa can make.
For instance, imagine if Koa did auto-halt generator execution and the user closes the socket before yield query('ROLLBACK') has a chance to clean up the db connection. Now you have a botched db connection stuck in the pool. Maybe there are even better reasons than that.
So, yes, Koa keeps executing your generator function until it's finished, at which point the response bubbles back up through your middleware and back to Koa's control where it goes nowhere since the socket is closed.
If you want to know when the user has closed the tab so you can implement your own short-circuiting behavior, you can listen for the "close" event on the request socket:
const cache = {}
router.get('/data', function *(next) {
if (cache.c) return this.body = cache.c;
var closed = false;
this.request.socket.on('close', function() {
closed = true;
});
if (closed) return;
const a = yield db.getA();
if (closed) return;
const b = yield db.getB();
if (closed) return;
cache.c = yield calculateC(a,b);
this.body = cache.c;
})
Related
I want to confirm if I am not slowing down the code below. My goal is to increase the performance of the application. I was considering a promise.all but not sure if it is necessary as I believe the code is written now, all the fetch requests are running simultaneously?
The test functions don't need to wait for each other. I don't want to create a funnel where each test function waits for the other one to finish or each fetch request is waiting for the previous one to finish. The goal is to have them all running together. Is my code doing that at the moment? Thank you for your assistance.
function test1() {
//some code here
fetch(URL)
.then(checkStatusandContentType)
.then(HtmlToObject)
.then(subTest1=> //work with the data here)
.catch(error => {console.log('Request failed', error);});
}
function test2() {
//some code here
fetch(URL)
.then(checkStatusandContentType)
.then(HtmlToObject)
.then(subTest2 => //work with the data here)
.catch(error => {console.log('Request failed', error);});
fetch(URL)
.then(checkStatusandContentType)
.then(HtmlToObject)
.then(subTest3 => //work with the data here)
.catch(error => {console.log('Request failed', error);});
fetch(URL)
.then(checkStatusandContentType)
.then(HtmlToObject)
.then(subTest4 => //work with the data here)
.catch(error => {console.log('Request failed', error);});
}
//....and hundreds like the above test functions down here
const checkStatusandContentType = async response => {
const isJson = response.headers.get('content-type')?.includes('application/json');
const isHtml = response.headers.get('content-type')?.includes('text/html');
const data = isJson ? await response.json()
: isHtml ? await response.text()
: null;
// check for error response
if (!response.ok) {
// get error message from body or default to response status
const error = (data && data.message) || response.status;
return Promise.reject(error);
}
return data;
}
const HtmlToObject = data => {
const stringified = data;
const processCode = stringified.substring(stringified.lastIndexOf("<X1>") + 4, stringified.indexOf("</X1>"));
//CONTENT EXTRACT
data = JSON.parse(processCode);
return data;
};
TL;DR fetch and XmlHTTPRequest perform the same.
You said you want to increase your application's performance. It's usually wise to dream up a way of measuring the performance when you do that.
Your performance measurement may be for just one desktop user with an excellent connection to the network. Or, it may be for hundreds of mobile devices using your app at the same time.
Browser HTML / Javascript apps using XmlHTTPRequest (XHR for short) or fetch requests are often designed to display something useful to your user, and then use the received data to make that display even more useful. If your measure of performance is how long it takes for a single user to see something useful, you may already have acceptable peformance. But if you have hundreds of mobile users, performance is harder to define.
You asked about whether XHR or fetch has better performance. From the point of view of your server, they are the same: both generate requests that your server must satisfy. They both generate the same requests; your server can't tell them apart. fetch requests are easier to code, as you have discovered.
Your code runs many requests. You showed us three but you said you have many more. Browsers restrict the number of concurrent outbound requests, and others wait for an available slot. Here's information about concurrent requests in an answer. Most browsers allow six concurrent requests to any given domain, and ten overall.
So, your concurrent fetch operations (or concurrent XHR operations, it doesn't matter which) will hit your server with six connections at once. That's fine for low-volume applications with good bandwidth. But if your app scales up to many users or must work over limited (mobile) bandwidth, you should consider whether you will overload your users' networks or your server.
Reducing the number of requests coming from your browser app, and perhaps returning more complete information in each request, is a good way to manage this server and network load.
Having two problems that I would appreciate some advise on. Have used puppeteer in the past in node, but for some reason, running into a problem on the sharp version.
Basically I'm crawling a webpage with a WaitUntil set to WaitUntilNavigation.Networkidle0, the longest wait period. In my node code, this runs and loads my website correctly, but in the C# version, I get the page without angular loaded. From the best I can tell it is not waiting and returning the initial Load state. Below is my code.
if (BROWSER == null)
{
await new BrowserFetcher().DownloadAsync(BrowserFetcher.DefaultRevision);
BROWSER = await Puppeteer.LaunchAsync(new LaunchOptions
{
Headless = true,
Args = new string[] { "--no-sandbox", "--disable-accelerated-2d-canvas", "--disable-gpu", "--proxy-server='direct://'", "--proxy-bypass-list=*" }
});
}
if (page == null)
{
page = await BROWSER.NewPageAsync();
await page.SetUserAgentAsync("PScraper-SiteCrawler");
await page.SetViewportAsync(new ViewPortOptions() { Width = 1024, Height = 842 });
var response = await page.GoToAsync(url, new NavigationOptions() { Referer = "PScraper-SiteCrawler", Timeout = timeoutMilliseconds, WaitUntil = new[] { WaitUntilNavigation.Networkidle0 } });
}
Timeout is set to 30 seconds, or 30,000 milliseconds. I then get the html of the page doing
await reponse.TextAsync()
My second question is unrelated, but likely simpler to solve. One route I was considering was using the page.WaitForSelectorAsync() method. This appears to wait until the content I'm looking for is loaded, but I haven't been able to figure out how to get the entire html of the page after this is done from the ElementHandle return.
Would appreciate some help here, tried a couple routes and haven't been able to figure out whats causing the difference between the node and C# code.
Solved my problem. The issue was how I was getting the html of the page.
I was using...
await reponse.TextAsync()
Apparently, this gets me only the initial response. When I changed my html get to the following line of code everything worked as expected.
await page.GetContentAsync()
i've currently started using sailsJS with angularJs at frontend alognwith socket for realtime communiction.
Sailsjs gives built-in support to websocket through "sails.io.js".On client side after adding this library this code is added to angular's chat controller.
Client side code
io.socket.get('/chat',{token:token},function(users){
console.log(users);
});
chatController's action on sails side is like this.
Server side code
chat: function (req, res) {
console.log(req.isSocket);
//this gives true when called through client.
})
infact very new to sails so i want suggestion that how to maintain connected user's list because m not using redis as storage purpose.adapter is memory.array is not a good idea because it'll vanish when restart a server.m using sails version of 0.11.0.
thanx in advance.
I'm somewhat new but learning fast, these suggestions should get you there unless someone else responds with greatness...
They changed it in 11 but in 10.5 I use sockets.js in config folder and on connect I store the session data in an array with their socket.
I created a service in APIs/service that contains the array and socket associate function.
For v11 you can't do that exactly the same, but you can make your first 'hello' from the client call a function in a controller that calls the associate function.
A couple tips would be don't let the client just tell you who they are, as in don't just take the username from the params but get it from req.session
(This assumes you have user auth setup)
In my case I have
in api/services/Z.js (putting the file here makes it's functions globally accessible)
var socketList = [];
module.exports = {
associateSocket: function(session, socket) { // send in your username(string) socket(object) id(mongoId) and this will push to the socketlist for lookups
sails.log.debug("associate socket called!",socketList.length)
var iHateYou = socketList
//DEBUG
var sList = socketList
var util = require('util')
if (session.authenticated){
var username = session.user.auth.username
var userId = session.user.id
// sails.log.debug("Z: associating new user!",username,userId,socket)
if (username && socket && userId) {
sList[sList.length]= {
username: session.user.auth.username,
socket: socket,
userId: session.user.id,
};
sails.log.debug('push run!!! currentsocketList length',socketList.length)
} else sails.log("Z.associateSocket called with invalid data", username, userId, authId, socket)
}else{sails.log.warn("Z.associateSocket: a socket attempted to associate itself without being logged in")}
},
}
in my config/sockets.js
onConnect: function(session, socket) {
Z.associateSocket(session,socket)
if (session.user && session.user.auth){
sails.log("config/sockets.js: "+session.user.auth.username+" CONNECT! session:",session)
}else sails.log.warn('connect called on socket without an auth, the client thinks it already has a session, so we need to fix this')
// By default, do nothing.
},
Then you can make add some functions to your services file to do lookups based on username and passwords, remove sockets that are disconnecting and the like (I'm using waterlock for my auth at the moment, although debating the switch back to sails-generate-auth)
Remove your onConnect and dicconnect function from config/sockets.js.
I'm trying to group all my socket.io connection into groups.
I want 1 group for each sails.js session.
My first goal is authentificate all tabs in a same time.
So I tried to do this with onConnect in config/sockets.js like that :
onConnect: function(session, socket) {
// By default: do nothing
// This is a good place to subscribe a new socket to a room, inform other users that
// someone new has come online, or any other custom socket.io logic
if (typeof session.socket == 'undefined'){
session.socket = [];
}
session.socket.push(socket.id);
session.save();
console.log(session, socket);
},
// This custom onDisconnect function will be run each time a socket disconnects
onDisconnect: function(session, socket) {
// By default: do nothing
// This is a good place to broadcast a disconnect message, or any other custom socket.io logic
if(Array.isArray(session.socket)){
var i = session.socket.indexOf(socket.id);
if(i != -1) {
session.socket.splice(i, 1);
session.save();
}
}
console.log(session, socket);
},
But I realize that session doesn't save my modifications.
I tried a session.save but sailsjs doesn't know req !
Session.set(sessionKey, req.session, function (err) {
I want to access to sails.js sesion but I don't know how to do it.
I tried to search a solution but now, after 6 hours of search I think it's time to requiered some help !
Thanks and sorry for my poor english (I'm french).
There appears to be a bug in the implementation of onConnect and onDisconnect in Sails v0.9.x. You can work around it for now by adding the following line before a call to session.save in those methods:
global.req = {}; global.req.session = session;
then changing session.save() to:
session.save(function(){delete global.req;});
That will provide the missing req var as a global, and then delete the global (for safety) after the session is saved.
Note that this issue only affects sessions in the onConnect and onDisconnect methods; inside of controller code session.save should work fine.
Thanks for pointing this out!
So I'm building a multipart form uploader over ajax on node.js, and sending progress events back to the client over socket.io to show the status of their upload. Everything works just fine until I have multiple clients trying to upload at the same time. Originally what would happen is while one upload is going, when a second one starts up it begins receiving progress events from both of the forms being parsed. The original form does not get affected and it only receives progress updates for itself. I tried creating a new formidable form object and storing it in an array along with the socket's session id to try to fix this, but now the first form stops receiving events while the second form gets processed. Here is my server code:
var http = require('http'),
formidable = require('formidable'),
fs = require('fs'),
io = require('socket.io'),
mime = require('mime'),
forms = {};
var server = http.createServer(function (req, res) {
if (req.url.split("?")[0] == "/upload") {
console.log("hit upload");
if (req.method.toLowerCase() === 'post') {
socket_id = req.url.split("sid=")[1];
forms[socket_id] = new formidable.IncomingForm();
form = forms[socket_id];
form.addListener('progress', function (bytesReceived, bytesExpected) {
progress = (bytesReceived / bytesExpected * 100).toFixed(0);
socket.sockets.socket(socket_id).send(progress);
});
form.parse(req, function (err, fields, files) {
file_name = escape(files.upload.name);
fs.writeFile(file_name, files.upload, 'utf8', function (err) {
if (err) throw err;
console.log(file_name);
})
});
}
}
});
var socket = io.listen(server);
server.listen(8000);
If anyone could be any help on this I would greatly appreciate it. I've been banging my head against my desk for a few days trying to figure this one out, and would really just like to get this solved so that I can move on. Thank you so much in advance!
Can you try putting console.log(socket_id);
after form = forms[socket_id]; and
after progress = (bytesReceived / bytesExpected * 100).toFixed(0);, please?
I get the feeling that you might have to wrap that socket_id in a closure, like this:
form.addListener(
'progress',
(function(socket_id) {
return function (bytesReceived, bytesExpected) {
progress = (bytesReceived / bytesExpected * 100).toFixed(0);
socket.sockets.socket(socket_id).send(progress);
};
})(socket_id)
);
The problem is that you aren't declaring socket_id and form with var, so they're actually global.socket_id and global.form rather than local variables of your request handler. Consequently, separate requests step over each other since the callbacks are referring to the globals rather than being proper closures.
rdrey's solution works because it bypasses that problem (though only for socket_id; if you were to change the code in such a way that one of the callbacks referenced form you'd get in trouble). Normally you only need to use his technique if the variable in question is something that changes in the course of executing the outer function (e.g. if you're creating closures within a loop).