I have some very simple Node.js code that I want to emit as chunked responses. The code is:
const WAIT = 500;
const records = [...];
function getList(req, res, url) {
var list = records.slice().map(rec => JSON.stringify(rec) + "\n");
var len = list.reduce((size, cur) => { size += Buffer.byteLength(cur); return size; }, 0);
function next() {
var cur = list.shift();
if(!cur) {
res.end();
return;
}
res.write(cur);
setTimeout(next, WAIT);
}
res.writeHead(200, {
"Access-Control-Allow-Origin": "*",
"Content-Type": "application/x-ndjson",
"Content-Length": len.toString(),
"X-Accel-Buffering": "no"
});
setTimeout(next, WAIT);
}
const PORT = process.env.PORT || 8080;
require("http").createServer(getList).listen(PORT);
This should write out this data as ndjson chunks until the array is emptied.
Locally and in a generic VPS this works as expected. On Heroku it does not emit the chunks in a streaming fashion, but rather dumps out them all at the end.
Does anyone have an idea of what I might be doing wrong?
After the comments section - setting the Transfer-Encoding: Chunked header explicitly resolved this problem. Digital Ocean might be doing something smart with chunking while you need to specify this explicitly in Heroku.
Documentation regarding the header: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding
Another potential problem why this might occur in your system: https://github.com/expressjs/compression/issues/56 (If you're using an older version of expressjs - response flushing contained a bug)
Related
I am using FastImage for caching image and it loads image very fast after caching data as expected. But my server is generating new uri (s3 presigned url) each time for same image.
So, FastImage is considering it as new image and tries to download everytime which affects my app performance.
My question is, Is there any optimistic way to render images fast as possible without caching it?
If you have chance to modify the server side application, you can create Authorization headers instead of creating presigned urls.
This function should help.
import aws4 from 'aws4';
export function getURIWithSignedHeaders(imagePath) {
if(!imagePath){
return null;
}
const expires = 86400; // 24 hours
const host = `${process.env.YOUR_S3_BUCKET_NAME}.s3.${process.env.YOUR_S3_REGION}.amazonaws.com`;
// imagePath should be something like images/3_profileImage.jpg
const path = `/${imagePath}?X-Amz-Expires=${expires}`;
const opts = {
host,
path,
headers: {
'Content-Type': 'image/jpeg'
}
};
const { headers } = aws4.sign(opts, {accessKeyId: process.env.YORU_ACCESS_KEY_ID, secretAccessKey: process.env.YOUR_SECRET_ACCESS_KEY});
return {
uri: `https://${host}${path}`,
headers: {
Authorization: headers['Authorization'],
'X-Amz-Content-Sha256': headers['X-Amz-Content-Sha256'],
'X-Amz-Date': headers['X-Amz-Date'],
'Content-Type': 'image/jpeg',
}
}
}
See: 609974221
I'm using an express api (my back-end) and an angular app (my front-end).
One express js end point (let's call it '/foo') is processing a lot of files,
i send data using res.write() after each treatment so the http response body is update.
I would like to get this update on my angular app.
I was using ajax in a previous version and it worked fine with ajax call :
xhrFields: {
// Getting on progress streaming response
onprogress: function(e)
{
var progressResponse;
var response = e.currentTarget.response;
if(lastResponseLength === false)
{
progressResponse = response;
lastResponseLength = response.length;
}
else
{
progressResponse = response.substring(lastResponseLength);
lastResponseLength = response.length;
}
actualResponse += progressResponse
}
Unfortunatly i found nothing to get partial http body. I tried to use 'reportProgress' Parameter but it's not working.
For some more context my front-end angular code:
service.ts :
setHolidaysDirectory(holidaysKey: string, path: string): Observable<Object>{
const setHolidayDirectoryStreamHttpRequest =
new HttpRequest('POST', 'http://localhost:8089/holidays/pictures/edit', { 'key': holidaysKey,
'path': path
}, {headers: this._httpHeaders, reportProgress: true, responseType: 'text'});
// pipe stream answer
return this._http.request(setHolidayDirectoryStreamHttpRequest);
}
and my component just call the service and subscribe :
this._holidaysService
.setHolidaysDirectory(key, finalHolidaysForm.path)
.subscribe((stream) => {
console.log('new answer');
console.log(stream);
}, error => console.log(error));
But unfortunatly i got empty answer and all the http body is recovered after res.end() (server side)
Can anyone help pls !
Thank a lot !
I was in the middle of teaching myself some Ajax, and this lesson required building a simple file upload form locally. I'm running XAMPP on windows 7, with a virtual host set up for http://test. The solution in the book was to use node and an almost unknown package called "multipart" which was supposed to parse the form data but was crapping out on me.
I looked for the best package for the job, and that seems to be formidable. It does the trick and my file will upload locally and I get all the details back through Ajax. BUT, it won't play nice with the simple JS code from the book which was to display the upload progress in a progress element. SO, I looked around and people suggested using socket.io to emit the progress info back to the client page.
I've managed to get formidable working locally, and I've managed to get socket.io working with some basic tutorials. Now, I can't for the life of me get them to work together. I can't even get a simple console log message to be sent back to my page from socket.io while formidable does its thing.
First, here is the file upload form by itself. The script inside the upload.html page:
document.getElementById("submit").onclick = handleButtonPress;
var httpRequest;
function handleResponse() {
if (httpRequest.readyState == 4 && httpRequest.status == 200) {
document.getElementById("results").innerHTML = httpRequest.responseText;
}
}
function handleButtonPress(e) {
e.preventDefault();
var form = document.getElementById("myform");
var formData = new FormData(form);
httpRequest = new XMLHttpRequest();
httpRequest.onreadystatechange = handleResponse;
httpRequest.open("POST", form.action);
httpRequest.send(formData);
}
And here's the corresponding node script (the important part being form.on('progress')
var http = require('http'),
util = require('util'),
formidable = require('formidable');
http.createServer(function(req, res) {
if (req.url == '/upload' && req.method.toLowerCase() == 'post') {
var form = new formidable.IncomingForm(),
files = [],
fields = [];
form.uploadDir = './files/';
form.keepExtensions = true;
form
.on('progress', function(bytesReceived, bytesExpected) {
console.log('Progress so far: '+(bytesReceived / bytesExpected * 100).toFixed(0)+"%");
})
.on('file', function(name, file) {
files.push([name, file]);
})
.on('error', function(err) {
console.log('ERROR!');
res.end();
})
.on('end', function() {
console.log('-> upload done');
res.writeHead(200, "OK", {
"Content-Type": "text/html", "Access-Control-Allow-Origin": "http://test"
});
res.end('received files: '+util.inspect(files));
});
form.parse(req);
} else {
res.writeHead(404, {'content-type': 'text/plain'});
res.end('404');
}
return;
}).listen(8080);
console.log('listening');
Ok, so that all works as expected. Now here's the simplest socket.io script which I'm hoping to infuse into the previous two to emit the progress info back to my page. Here's the client-side code:
var socket = io.connect('http://test:8080');
socket.on('news', function(data){
console.log('server sent news:', data);
});
And here's the server-side node script:
var http = require('http'),
fs = require('fs');
var server = http.createServer(function(req, res) {
fs.createReadStream('./socket.html').pipe(res);
});
var io = require('socket.io').listen(server);
io.sockets.on('connection', function(socket) {
socket.emit('news', {hello: "world"});
});
server.listen(8080);
So this works fine by itself, but my problem comes when I try to place the socket.io code inside my form.... I've tried placing it anywhere it might remotely make sense, i've tried the asynchronous mode of fs.readFile too, but it just wont send anything back to the client - meanwhile the file upload portion still works fine. Do I need to establish some sort of handshake between the two packages? Help me out here. I'm a front-end guy so I'm not too familiar with this back-end stuff. I'll put this aside for now and move onto other lessons.
Maybe you can create a room for one single client and then broadcast the percentage to this room.
I explained it here: How to connect formidable file upload to socket.io in Node.js
I'm having the classic IE-caches-everything-in-Ajax issue. I have a bit of data that refreshes every minute.
Having researched the forums the solutions boil down to these options (http://stackoverflow.com/questions/5997857/grails-best-way-to-send-cache-headers-with-every-ajax-call):
add a cache-busting token to the query string (like ?time=[timestamp])
send a HTTP response header that specifically forbids IE to cache the request
use an ajax POST instead of a GET
Unfortunately the obvious querysting or "cache: false" setting will not work for me as the updated data file is hosted on Akamai Netstorage and cannot accept querystrings. I don't want to use POST either.
What I want to do is try send an HTTP response header that specifically forbids IE to cache the request or if anyone else knows another cache busting solution??
Does anyone know how this might be done? Any help would be much appreciated.
Here is my code:
(function ($) {
var timer = 0;
var Browser = {
Version: function () {
var version = 999;
if (navigator.appVersion.indexOf("MSIE") != -1) version = parseFloat(navigator.appVersion.split("MSIE")[1]);
return version;
}
}
$.fn.serviceboard = function (options) {
var settings = { "refresh": 60};
return this.each(function () {
if (options) $.extend(settings, options);
var obj = $(this);
GetLatesData(obj, settings.refresh);
if (settings.refresh > 9 && Browser.Version() > 6) {
timer = setInterval(function () { GetLatestData(obj, settings.refresh) }, settings.refresh * 1000);
}
});
};
function GetLatestData(obj, refresh) {
var _url = "/path/updated-data.htm";
$.ajax({
url: _url,
dataType: "html",
complete: function () {},
success: function (data) {
obj.empty().append(data);
}
}
});
}
})(jQuery);
Add a random number to the GET request so that IE will not identify it as "the same" in its cache. This number could be a timestamp:
new Date().getTime()
EDIT perhaps make the requested url:
var _url = "/path/updated-data.htm?" + new Date().getTime()
This shouldn't cause any errors I believe.
EDIT2 Sorry I just read your post a bit better and saw that this is not an option for you.
You say "is hosted on Akamai and cannot accept querystrings" but why not?
I've never heard of a page that won't accept an additional: "?blabla", even when it's html.
This was driving me crazy. I tried many cache busting techniques and setting cache headers. So many of these either did not work or were wild goose chases. The only solution I found which tested to work correctly was setting:
Header Pragma: no-cache
I hope it saves others with IE headaches.
I'm building a super simple server in node and in my onRequest listener I'm trying to determine if I should serve a static file (off the disk) or some json (probably pulled from mongo) based on the path in request.url.
Currently I'm trying to stat the file first (because I use mtime elsewhere) and if that doesn't fail then I read the contents from disk. Something like this:
fs.stat(request.url.pathname, function(err, stat) {
if (!err) {
fs.readFile(request.url.pathname, function( err, contents) {
//serve file
});
}else {
//either pull data from mongo or serve 404 error
}
});
Other than cacheing the result of fs.stat for the request.url.pathname, is there something that could speed this check up? For example, would it be just as fast to see if fs.readFile errors out instead of the stat? Or using fs.createReadStream instead of fs.readFile? Or could I potentially check for the file using something in child_process.spawn? Basically I just want to make sure I'm not spending any extra time messing w/ fileio when the request should be sent to mongo for data...
Thanks!
var fs = require('fs');
fs.exists(file, function(exists) {
if (exists) {
// serve file
} else {
// mongodb
}
});
I don't think you should be worrying about that, but rather how can you improve the caching mechanism. fs.stat is really ok for file checking, doing that in another child process would probably slow you down rather then help you here.
Connect implemented the staticCache() middleware a few months ago, as described in this blog post: http://tjholowaychuk.com/post/9682643240/connect-1-7-0-fast-static-file-memory-cache-and-more
A Least-Recently-Used (LRU) cache algo is implemented through the
Cache object, simply rotating cache objects as they are hit. This
means that increasingly popular objects maintain their positions while
others get shoved out of the stack and garbage collected.
Other resources:
http://senchalabs.github.com/connect/middleware-staticCache.html
The source code for staticCache
this snippet can help you
fs = require('fs') ;
var path = 'sth' ;
fs.stat(path, function(err, stat) {
if (err) {
if ('ENOENT' == err.code) {
//file did'nt exist so for example send 404 to client
} else {
//it is a server error so for example send 500 to client
}
} else {
//every thing was ok so for example you can read it and send it to client
}
} );
In case you want to serve a file using express, I would recommend to just use the sendFile error Handler of express
const app = require("express")();
const options = {};
options.root = process.cwd();
var sendFiles = function(res, files) {
res.sendFile(files.shift(), options, function(err) {
if (err) {
console.log(err);
console.log(files);
if(files.length === 0) {
res.status(err.status).end();
} else {
sendFiles(res, files)
}
} else {
console.log("Image Sent");
}
});
};
app.get("/getPictures", function(req, res, next) {
const files = [
"file-does-not-exist.jpg",
"file-does-not-exist-also.jpg",
"file-exists.jpg",
"file-does-not-exist.jpg"
];
sendFiles(res, files);
});
app.listen(8080);
If the file is not existent then it will go to the error that sends it self.
I made a github repo here https://github.com/dmastag/ex_fs/blob/master/index.js