can't seem to get progress events from node-formidable to send to the correct client over socket.io - ajax

So I'm building a multipart form uploader over ajax on node.js, and sending progress events back to the client over socket.io to show the status of their upload. Everything works just fine until I have multiple clients trying to upload at the same time. Originally what would happen is while one upload is going, when a second one starts up it begins receiving progress events from both of the forms being parsed. The original form does not get affected and it only receives progress updates for itself. I tried creating a new formidable form object and storing it in an array along with the socket's session id to try to fix this, but now the first form stops receiving events while the second form gets processed. Here is my server code:
var http = require('http'),
formidable = require('formidable'),
fs = require('fs'),
io = require('socket.io'),
mime = require('mime'),
forms = {};
var server = http.createServer(function (req, res) {
if (req.url.split("?")[0] == "/upload") {
console.log("hit upload");
if (req.method.toLowerCase() === 'post') {
socket_id = req.url.split("sid=")[1];
forms[socket_id] = new formidable.IncomingForm();
form = forms[socket_id];
form.addListener('progress', function (bytesReceived, bytesExpected) {
progress = (bytesReceived / bytesExpected * 100).toFixed(0);
socket.sockets.socket(socket_id).send(progress);
});
form.parse(req, function (err, fields, files) {
file_name = escape(files.upload.name);
fs.writeFile(file_name, files.upload, 'utf8', function (err) {
if (err) throw err;
console.log(file_name);
})
});
}
}
});
var socket = io.listen(server);
server.listen(8000);
If anyone could be any help on this I would greatly appreciate it. I've been banging my head against my desk for a few days trying to figure this one out, and would really just like to get this solved so that I can move on. Thank you so much in advance!

Can you try putting console.log(socket_id);
after form = forms[socket_id]; and
after progress = (bytesReceived / bytesExpected * 100).toFixed(0);, please?
I get the feeling that you might have to wrap that socket_id in a closure, like this:
form.addListener(
'progress',
(function(socket_id) {
return function (bytesReceived, bytesExpected) {
progress = (bytesReceived / bytesExpected * 100).toFixed(0);
socket.sockets.socket(socket_id).send(progress);
};
})(socket_id)
);

The problem is that you aren't declaring socket_id and form with var, so they're actually global.socket_id and global.form rather than local variables of your request handler. Consequently, separate requests step over each other since the callbacks are referring to the globals rather than being proper closures.
rdrey's solution works because it bypasses that problem (though only for socket_id; if you were to change the code in such a way that one of the callbacks referenced form you'd get in trouble). Normally you only need to use his technique if the variable in question is something that changes in the course of executing the outer function (e.g. if you're creating closures within a loop).

Related

is there a way to load external data in Lambda#Edge

i was trying to use Lambda#Edge to handle A/B testing in my site.
i wonder is there a way to let Lambada#Edge functions load an external config data from a url, eg. i created an api to return the traffic rates of A/B channels,i want control that config data outside, so that i can dynamically adjust the traffics flow to A or B channel and no need to modify the Lambda function.
what i did now is
var versions = [];
var isLoadingVersionData = false;
const https = require('https');
function loadVersions() {
if (isLoadingVersionData)
return null;
isLoadingVersionData=true;
https.get('https://example.com/getAbTestConfig', (res) => {
res.on('data', (d) => {
var parsedBody = JSON.parse(d);
if (parsedBody.status)
versions = parsedBody.data;
});
}).on('error', (e) => {
console.log(e);
});
}
//and load the function in handler
exports.handler = (event, context, callback) => {
context.callbackWaitsForEmptyEventLoop = false;
loadVersions();
}
i wonder this variable "versions" can be loaded correctly and shared in each later requests
do you have some more effectually solutions?
Why not maintain this data in S3 and use your Lambda#Edge to get the configuration from there? Further to reduce latency, you can front the S3 bucket containing traffic ratio with CloudFront and have your L#E make a call to CloudFront and get the desired value.
I was facing the same issue but not for A/B testing. I just created a json file in my lambda function to avoid delay of making http calls inside lambda functions. It works but the maintenance is not good, as every time when I need to change the Json file I need deploy the lambda function again.
While I was searching for it, I found same solution described for Mr.Ocean above, sounds like a good alternative to maintain the data in S3.

Telegram Bot Random Images (How to send random images with Telegram-Bot)

const TeleBot = require('telebot');
const bot = new TeleBot({
token: 'i9NhrhCQGq7rxaA' // Telegram Bot API token.
});
bot.on(/^([Hh]ey|[Hh]oi|[Hh]a*i)$/, function (msg) {
return bot.sendMessage(msg.from.id, "Hello Commander");
});
var Historiepics = ['Schoolfotos/grr.jpg', 'Schoolfotos/boe.jpg',
'Schoolfotos/tobinsexy.jpg'];
console.log('Historiepics')
console.log(Math.floor(Math.random() * Historiepics.length));
var foto = Historiepics[(Math.floor(Math.random() * Historiepics.length))];
bot.on(/aap/, (msg) => {
return bot.sendPhoto(msg.from.id, foto);
});
bot.start();
The result I'm getting from this is just one picture everytime, but if I ask for another random picture it keeps showing me the same one without change.
I recently figured this out, so I'll drop an answer for anyone that runs into this issue.
The problem is with Telegram's cache. They cache images server side so that they don't have to do multiple requests to the same url. This protects them from potentially getting blacklisted for too many requests, and makes things snappier.
Unfortunately if you're using an API like The Cat API this means you will be sending the same image over and over again. The simplest solution is just to somehow make the link a little different every time. This is most easily accomplished by including the current epoch time as a part of the url.
For your example with javascript this can be accomplished with the following modifications
bot.on(/aap/, (msg) => {
let epoch = (new Date).getTime();
return bot.sendPhoto(msg.from.id, foto + "?time=" + epoch);
});
Or something similar. The main point is, as long as the URL is different you won't receive a cached result. The other option is to download the file and then send it locally. This is what Telebot does if you pass the serverDownload option into sendPhoto.

Getting file contents when using DropzoneJS

I really love the DropZoneJS component and am currently wrapping it in an EmberJS component (you can see demo here). In any event, the wrapper works just fine but I wanted to listen in on one of Dropzone's events and introspect the file contents (not the meta info like size, lastModified, etc.). The file type I'm dealing with is an XML file and I'd like to look "into" it to validate before sending it.
How can one do that? I would have thought the contents would hang off of the file object that you can pick up on many of the events but unless I'm just missing something obvious, it isn't there. :(
This worked for me:
Dropzone.options.PDFDrop = {
maxFilesize: 10, // Mb
accept: function(file, done) {
var reader = new FileReader();
reader.addEventListener("loadend", function(event) { console.log(event.target.result);});
reader.readAsText(file);
}
};
could also use reader.reaAsBinaryString() if binary data!
Ok, I've answer my own question and since others appear interested I'll post my answer here. For a working demo of this you can find it here:
https://ui-dropzone.firebaseapp.com/demo-local-data
In the demo I've wrapped the Dropzone component in the EmberJS framework but if you look at the code you'll find it's just Javascript code, nothing much to be afraid of. :)
The things we'll do are:
Get the file before the network request
The key thing we need become familiar with is the HTML5 API. Good news is it is quite simple. Take a look at this code and maybe that's all you need:
/**
* Replaces the XHR's send operation so that the stream can be
* retrieved on the client side instead being sent to the server.
* The function name is a little confusing (other than it replaces the "send"
* from Dropzonejs) because really what it's doing is reading the file and
* NOT sending to the server.
*/
_sendIntercept(file, options={}) {
return new RSVP.Promise((resolve,reject) => {
if(!options.readType) {
const mime = file.type;
const textType = a(_textTypes).any(type => {
const re = new RegExp(type);
return re.test(mime);
});
options.readType = textType ? 'readAsText' : 'readAsDataURL';
}
let reader = new window.FileReader();
reader.onload = () => {
resolve(reader.result);
};
reader.onerror = () => {
reject(reader.result);
};
// run the reader
reader[options.readType](file);
});
},
https://github.com/lifegadget/ui-dropzone/blob/0.7.2/addon/mixins/xhr-intercept.js#L10-L38
The code above returns a Promise which resolves once the file that's been dropped into the browser has been "read" into Javascript. This should be very quick as it's all local (do be aware that if you're downloading really large files you might want to "chunk" it ... that's a more advanced topic).
Hook into Dropzone
Now we need to find somewhere to hook into in Dropzone to read the file contents and stop the network request that we no longer need. Since the HTML5 File API just needs a File object you'll notice that Dropzone provides all sorts of hooks for that.
I decided on the "accept" hook because it would give me the opportunity to download the file and validate all in one go (for me it's mainly about drag and dropping XML's and so the content of the file is a part of the validation process) and crucially it happens before the network request.
Now it's important you realise that we're "replacing" the accept function not listening to the event it fires. If we just listened we would still incur a network request. So to **overload* accept we do something like this:
this.accept = this.localAcceptHandler; // replace "accept" on Dropzone
This will only work if this is the Dropzone object. You can achieve that by:
including it in your init hook function
including it as part of your instantiation (e.g., new Dropzone({accept: {...})
Now we've referred to the "localAcceptHandler", let me introduce it to you:
localAcceptHandler(file, done) {
this._sendIntercept(file).then(result => {
file.contents = result;
if(typeOf(this.localSuccess) === 'function') {
this.localSuccess(file, done);
} else {
done(); // empty done signals success
}
}).catch(result => {
if(typeOf(this.localFailure) === 'function') {
file.contents = result;
this.localFailure(file, done);
} else {
done(`Failed to download file ${file.name}`);
console.warn(file);
}
});
}
https://github.com/lifegadget/ui-dropzone/blob/0.7.2/addon/mixins/xhr-intercept.js#L40-L64
In quick summary it does the following:
read the contents of the file (aka, _sendIntercept)
based on mime type read the file either via readAsText or readAsDataURL
save the file contents to the .contents property of the file
Stop the send
To intercept the sending of the request on the network but still maintain the rest of the workflow we will replace a function called submitRequest. In the Dropzone code this function is a one liner and what I did was replace it with my own one-liner:
this._finished(files,'locally resolved, refer to "contents" property');
https://github.com/lifegadget/ui-dropzone/blob/0.7.2/addon/mixins/xhr-intercept.js#L66-L70
Provide access to retrieved document
The last step is just to ensure that our localAcceptHandler is put in place of the accept routine that dropzone supplies:
https://github.com/lifegadget/ui-dropzone/blob/0.7.2/addon/components/drop-zone.js#L88-L95
using the FileReader() solution is working amazingly good for me:
Dropzone.autoDiscover = false;
var dz = new Dropzone("#demo-upload",{
autoProcessQueue:false,
url:'upload.php'
});
dz.on("drop",function drop(e) {
var files = [];
for (var i = 0; i < e.dataTransfer.files.length; i++) {
files[i] = e.dataTransfer.files[i];
}
var reader = new FileReader();
reader.onload = function(event) {
var line = event.target.result.split('\n');
for ( var i = 0; i < line.length; i++){
console.log(line);
}
};
reader.readAsText(files[files.length-1]);

Get Picture from Client - save on MongoDB, expressJS, nodeJS

I'm trying to Implement a simple Picture upload from the client to my mongoDB.
I've read many explanations but I can't find a way from start to finish.
My clientside -
function profilePic(input) {
if (input.files && input.files[0]) {
var file = input.files[0];
localStorage.setItem('picture', JSON.stringify(file));
}
}
Later on I take the this JSON from the LocalStorage and send it to my server side like this:
var request = false;
var result = null;
request = new XMLHttpRequest();
if (request) {
request.open("POST", "usersEditProf/");
request.onreadystatechange = function() {
if (request.readyState == 4 && request.status == 200) {
.....//More code to send to Server
request.setRequestHeader('content-type', 'application/json');
request.send(JSON.stringify(localStorage.getItem('picture)));
}
}
On my serverside:
app.post('/usersEditProf/',users.editProfile);
/** Edits the Profile - sends the new one **/
exports.editProfile = function(req, res) {
var toEdit = req.body;
var newPic = toEdit.picture;
And thats where I get lost. is newPic actually holding the picture? I doubt it...
Do I need to change the path? What is the new path I need to give the picture?
How do I put it in my DB? Do I need GridFS?
When trying to simply put that in my collection, it looks like this (example with a image called bar.jpg:
picture: "{\"webkitRelativePath\":\"\",\"lastModifiedDate\":\"2012-10-08T23:34:50.000Z\",\"name\":\"bar.jpg\",\"type\":\"image/jpeg\",\"size\":88929}",
If you want to upload a blob through XMLHTTPRequest(), you need to use an HTML 5 FormData object:
https://developer.mozilla.org/en-US/docs/Web/API/FormData
It alows you to specify a filename to push, then you handle the incoming file as you would with a mime form post. Note the limitations on browser support when you use the FormData object. Your alternative is a form POST to a hidden frame, which works OK but is not nearly as clean looking in code as FormData.

Sailsjs Session access during custom onConnect in socket.io

I'm trying to group all my socket.io connection into groups.
I want 1 group for each sails.js session.
My first goal is authentificate all tabs in a same time.
So I tried to do this with onConnect in config/sockets.js like that :
onConnect: function(session, socket) {
// By default: do nothing
// This is a good place to subscribe a new socket to a room, inform other users that
// someone new has come online, or any other custom socket.io logic
if (typeof session.socket == 'undefined'){
session.socket = [];
}
session.socket.push(socket.id);
session.save();
console.log(session, socket);
},
// This custom onDisconnect function will be run each time a socket disconnects
onDisconnect: function(session, socket) {
// By default: do nothing
// This is a good place to broadcast a disconnect message, or any other custom socket.io logic
if(Array.isArray(session.socket)){
var i = session.socket.indexOf(socket.id);
if(i != -1) {
session.socket.splice(i, 1);
session.save();
}
}
console.log(session, socket);
},
But I realize that session doesn't save my modifications.
I tried a session.save but sailsjs doesn't know req !
Session.set(sessionKey, req.session, function (err) {
I want to access to sails.js sesion but I don't know how to do it.
I tried to search a solution but now, after 6 hours of search I think it's time to requiered some help !
Thanks and sorry for my poor english (I'm french).
There appears to be a bug in the implementation of onConnect and onDisconnect in Sails v0.9.x. You can work around it for now by adding the following line before a call to session.save in those methods:
global.req = {}; global.req.session = session;
then changing session.save() to:
session.save(function(){delete global.req;});
That will provide the missing req var as a global, and then delete the global (for safety) after the session is saved.
Note that this issue only affects sessions in the onConnect and onDisconnect methods; inside of controller code session.save should work fine.
Thanks for pointing this out!

Resources