Currently I am developing an app that fire off hundreds of concurrent requests to external API service (like instagram for example) using ajax on client side. Response time is very fast.
However, I am migrating the request handling part to node backend using request + jsonstream but always get socket hang up error due to concurrency > 5 requests (even after changing maxsockets to higher values). Overall it is much much slower than doing API requests directly on client side using ajax.
My question is how can I make the proxy server faster/more responsive? Or maybe using ajax similar to when doing on client side but on node?
Server side: when client hits endpoint /fetchmedia/, node directs to this function.
var fetchInstagram = function(user_id, max_id, min_timestamp, max_timestamp, array, callback) {
http.request({
host: 'endpoint',
path: 'endpoint',
method: 'get'
}, function(res) {
var body = '';
res.on('data', function(chunk) {
body += chunk;
});
res.on('end', function() {
var data = JSON.parse(body);
array = array.concat(data.data);
if (data.pagination.next_max_id != undefined) {
fetchInstagram(user_id, data.pagination.next_max_id, min_timestamp, max_timestamp, array, callback);
} else {
callback(array);
}
});
}).on('error', function(e) {
console.log("Got error: ", e);
}).end();
Client-side: Backbone sends hundreds of requests (/fetchmedia) at the same time, which calls many fetchinstagram functions. The way I was doing before was sending ajax, which also hundreds concurrently but it handles very well. Node hangs up even with 20 users while ajax handles 1000+ users
Thanks
Related
server.js
var express = require('express');
var app = express();
var server = require('http').createServer(app);
var io = require('socket.io')(server);
app.use(express.static(__dirname + '/public'));
io.on('connection', function(client) {
console.log('Client connected...');
client.on('join', function(data) {
console.log(data);
io.emit('messages', 'Hello');
});
});
index.html
<script>
var socket = io.connect('http://localhost:7777');
socket.on('connect', function(data) {
socket.emit('join', 'Hello World from client');
});
socket.on('messages', function(data) {
alert(data);
});
</script>
I tried to implement very basic of Socket.io.
However, data sending from client to server is available but from server to client doesn't work.
In the command running server.js, 'Hello World from client' is printed. However, alert window doesn't work in the web browser.(I've also tried to console.log).
How to solve this?
Editted
I've put server.js codes in the app.get('/', function(req, res)){ ... }
Then, it doesn't work. Why it doesn't work in app.get?
Try this, I hope it works:
io.on('connection', function(client) {
console.log('Client connected...');
client.on('join', function(data) {
console.log(data);
io.emit('join', data); //this code sending data from server to client
});
});
If you're just trying to fetch some data with an Ajax call such as /test, then there is no need to use socket.io. That's just a classic request/response.
app.get('/test', function(req, res) {
// collect your data and then send it as a response
res.json(data);
});
If you're just trying to incorporate data into a web page that is requested, then you can use res.render() with the template engine of your choice (ejs, handlebars, pug, etc...). That would typically look like this:
app.get('/test', function(req, res) {
// collect your data and then pass it to res.render() to render your
// your template using that data
res.render('someTemplateName', data);
});
The main thing that socket.io is useful for is "pushing" data from server to client without a client request. So, if something happened on the server that the client was not aware of and the server wanted to tell the client about it, then socket.io would be used for that. The classic example is a chat app. Person A sends a chat message to the server that is addressed to Person B. The server receives that message and then needs to "push" it to Person B. That would be perfect for an already connected socket.io connection because the server can just push the data directly to the Person B client, something the server can't do with request/response (since there is no request from person B).
If you still think you need socket.io, then please describe exactly what you're trying to do with it (step by step what you're trying to send to the client).
socket.on("message",function (reply_data) {
console.log('inside on message functions ')
console.log(reply_data);
})
please change 'messages' to "message" that worked for me
I'm using a Web App (which is really big) so there are some parts of the application that I really don't know how they work.
I am a front end developer and I'm consuming a REST API implemented with .NET Web Api (as far as I know)
The request is simple - I use kendo Datasource to get the data from the server like this
var kendoDataSource = new kendo.data.DataSource({
// fake transport with local data
transport: {
read: function(options) {
// set results
options.success(lookupValues);
}
},
schema: {
parse: function (response) {
// sort case insensitive by name
response.sort(function (a, b) {
return (a.Name.toLowerCase() > b.Name.toLowerCase()) ? 1 : (a.Name.toLowerCase() < b.Name.toLowerCase()) ? -1 : 0;
});
return response;
}
},
// set the page size
pageSize: 25
});
and the request for the data
$http({ method: 'GET', url: 'REST/SystemDataSet/' + id + '/Values' }).success(function (response) {
// store data
lookupValues = response;
kendoDataSource.read();
// do some logic here
}).error(function(error) {
// logic
});
I do this in this way because there is some extra logic that manipulates the data.
This request in Chrome takes like 32 ms while it takes almost 9 seconds in IE.
The data retrieved is the same (you can see the Size of response), which is an array of JSon objects (Very simple)
I don't know exactly if there is a cache mechanism in the backend, but it shouldn't matter because I'm able to reproduce it like this every time (fast in Chrome, really really slow on IE)
Any ideas of what could be causing this behaviour ? As I understand, if there is a cache or something, it should be the same for every browser, so this should be happening on both and not only on IE - the backend is agnostic of the browser.
Here is some extra information I have from another request to check the distribution of time in the first IE request
As you can see, the biggest part is the "Request", which is the Time taken to send the request and receive the first response from the server.
Thanks in Advance
The problem is probably Windows Authentication turned on for the folder you are calling the ajax from...
Same principle applies here ...
http://docs.telerik.com/kendo-ui/web/upload/troubleshooting
Problem: Async uploads randomly fail when using IE10/11 with Windows Authentication
The upload either freezes indefinitely or times out if a 401 challenge is received on the HTTP POST.
Solution
For IE10 see KB2980019
No official fix for IE 11 as of November 6, 2014. See bug ID 819941
I was in the middle of teaching myself some Ajax, and this lesson required building a simple file upload form locally. I'm running XAMPP on windows 7, with a virtual host set up for http://test. The solution in the book was to use node and an almost unknown package called "multipart" which was supposed to parse the form data but was crapping out on me.
I looked for the best package for the job, and that seems to be formidable. It does the trick and my file will upload locally and I get all the details back through Ajax. BUT, it won't play nice with the simple JS code from the book which was to display the upload progress in a progress element. SO, I looked around and people suggested using socket.io to emit the progress info back to the client page.
I've managed to get formidable working locally, and I've managed to get socket.io working with some basic tutorials. Now, I can't for the life of me get them to work together. I can't even get a simple console log message to be sent back to my page from socket.io while formidable does its thing.
First, here is the file upload form by itself. The script inside the upload.html page:
document.getElementById("submit").onclick = handleButtonPress;
var httpRequest;
function handleResponse() {
if (httpRequest.readyState == 4 && httpRequest.status == 200) {
document.getElementById("results").innerHTML = httpRequest.responseText;
}
}
function handleButtonPress(e) {
e.preventDefault();
var form = document.getElementById("myform");
var formData = new FormData(form);
httpRequest = new XMLHttpRequest();
httpRequest.onreadystatechange = handleResponse;
httpRequest.open("POST", form.action);
httpRequest.send(formData);
}
And here's the corresponding node script (the important part being form.on('progress')
var http = require('http'),
util = require('util'),
formidable = require('formidable');
http.createServer(function(req, res) {
if (req.url == '/upload' && req.method.toLowerCase() == 'post') {
var form = new formidable.IncomingForm(),
files = [],
fields = [];
form.uploadDir = './files/';
form.keepExtensions = true;
form
.on('progress', function(bytesReceived, bytesExpected) {
console.log('Progress so far: '+(bytesReceived / bytesExpected * 100).toFixed(0)+"%");
})
.on('file', function(name, file) {
files.push([name, file]);
})
.on('error', function(err) {
console.log('ERROR!');
res.end();
})
.on('end', function() {
console.log('-> upload done');
res.writeHead(200, "OK", {
"Content-Type": "text/html", "Access-Control-Allow-Origin": "http://test"
});
res.end('received files: '+util.inspect(files));
});
form.parse(req);
} else {
res.writeHead(404, {'content-type': 'text/plain'});
res.end('404');
}
return;
}).listen(8080);
console.log('listening');
Ok, so that all works as expected. Now here's the simplest socket.io script which I'm hoping to infuse into the previous two to emit the progress info back to my page. Here's the client-side code:
var socket = io.connect('http://test:8080');
socket.on('news', function(data){
console.log('server sent news:', data);
});
And here's the server-side node script:
var http = require('http'),
fs = require('fs');
var server = http.createServer(function(req, res) {
fs.createReadStream('./socket.html').pipe(res);
});
var io = require('socket.io').listen(server);
io.sockets.on('connection', function(socket) {
socket.emit('news', {hello: "world"});
});
server.listen(8080);
So this works fine by itself, but my problem comes when I try to place the socket.io code inside my form.... I've tried placing it anywhere it might remotely make sense, i've tried the asynchronous mode of fs.readFile too, but it just wont send anything back to the client - meanwhile the file upload portion still works fine. Do I need to establish some sort of handshake between the two packages? Help me out here. I'm a front-end guy so I'm not too familiar with this back-end stuff. I'll put this aside for now and move onto other lessons.
Maybe you can create a room for one single client and then broadcast the percentage to this room.
I explained it here: How to connect formidable file upload to socket.io in Node.js
I've built a jQuery mobile app that gets its content from an external SQL server database via JSON and a server side script (ColdFusion CFC) that interfaces with the database. This app has been packaged as a native app using PhoneGap. I need to enable the jQuery mobile app to be able to write back to the external SQL server db.
Im new to mobile development but have several years of server side development using ColdFusion. I am guessing that the best way to do this is for the mobile app to send the results of the submitted form elements to a server side script for processing. I dont want the native app to send this "as a web page" but rather stay in the app to do it (via AJax I assume).
My server side script will be written in ColdFusion and handles input sanitation and database interaction...I just need to figure out what is the best way to submit from my jQuery app to the server side script, but do it while staying inside of my native application.
I'm pretty much doing the same thing. Server side Coldfuison8/MySQL, front end Jquery Mobile (, requireJS) with all forms submits routed through AJAX to avoid reloading a page.
I'm using a generic form submitter in my controller.js, which looks like this:
var ajaxFormSubmit =
function ( form, service, formdata, targetUrl, successHandler, dataHandler, errorHandler ){
$.ajax({
async: false,
type: "post",
url: service,
data: formdata,
dataType: "json",
success: function( objResponse ){
if (objResponse.SUCCESS == true ){
// alert("success!");
// this passes the response object to the success handler
// in case data needs to be ... handled.
dataHandler == "yes" ? successHandler( objResponse ) : successHandler();
} else {
// alert("AJAX failed!");
if ( errorHandler != "" ){
errorHandler();
}
}
},
error: function (XMLHttpRequest, textStatus, errorThrown) {
//alert("something else failed");
showErrors( [["server-request-error", "server_error"]], XMLHttpRequest, textStatus, errorThrown );
}
});
}
I'm returning results as a response object, which will contain Success = True/false, data (if there is any) and Error = error message.
A function call will look like this:
// the form
var form = $(this).closest('form'),
// trigger for cfcase inside my cfc
switcher = form.find('input[name="form_submitted"]').val(),
// which cfc to call
service = "../cfcs/form_handler_abc.cfc",
// the method in your cfc you are calling (validation/commit to database)
method = "process",
returnformat = "JSON",
// not using
targetUrl = "",
// serialized form plus any value you need to pass along
formdata = form.serialize()+"&method="+method+"&returnformat="+returnformat,
// specific error routine to run
errorHandler = function(){
// in my case, reset the whole form
cleanUp( $('form:jqmData(search="regular")'), "results" )
},
// inside my success handler I'm switching depending on submitted form
// `response` will be the AJAX object.response = whatever you send back from the server
successHandler = function( response ) {
switch (switcher) {
// form A - this is for a search form handling the results
case "getProducts":
// clean up
$('.ajaxContainer, .pagination').addClass('fade out').remove();
// AJAX data received
var makeUp = response.DATA;
// don't forget to trigger create to enhance all JQM elements you are sending
$('.results').append( makeUp ).trigger('create');
// redraw - will fire JQM updatelayout
$(window).trigger('dimensionchange');
// will set bindings on new elements
bindResults( $('.results').closest('div:jqmData(role="page")') );
break;
case "A":
// do sth else
break;
case "B":
// do sth else
break;
}
};
// now pass all of the above to the ajaxFormsubmit
ajaxFormSubmit( form, service, formdata, targetUrl, successHandler, handleData, errorHandler);
I have a number of CFCs, each with a main cfswitch and cfcase for each submitted form. I have built my backend using this sample. Took a while to get going, but now it's running more or less smooth.
Let me know if you have some questions regarding the above.
I am wondering what is the best way to stop duplciate submissions when using jquery and ajax?
I come up with 2 possible ways but not sure if these are the only 2.
On Ajax start disable all buttons till request is done. 2 problems I see with this though is I use jquery model dialog so I don't know how easy it would be to disable those button as I not sure if they have id's. Second I if the the request hangs the user has really no way to try again since all the buttons are disabled.
I am looking into something called AjaxQueue at this time I have no clue if it is what I need or how it works since the site where the plugin is apparently down for maintenance.
http://docs.jquery.com/AjaxQueue
Edit
I think this is a spin off of what I was looking at.
http://www.protofunc.com/scripts/jquery/ajaxManager/
The only problem I see with this ajaxManager is that I think I have to change all my $.post, $.get and $.ajax ones to their type.
But what happens if I need a special parameter from $.ajax? Or that fact I like using .post and .get.
Edit 2
I think it can take in all $.ajax options. I am still looking into it. However what I am unsure about now is can I use the same constructor for all requests that will use the same options.
First you have to construct/configure a new Ajaxmanager
//create an ajaxmanager named someAjaxProfileName
var someManagedAjax = $.manageAjax.create('someAjaxProfileName', {
queue: true,
cacheResponse: true
});
Or do I have to make the above every single time?
How about setting a flag when the user clicks the button? You will only clear the flag when the AJAX request completes successfully (in complete, which is called after the success and error callbacks), and you will only send an AJAX request if the flag is not set.
Related to AJAX queuing there is a plugin called jQuery Message Queuing that is very good. I've used it myself.
var requestSent = false;
jQuery("#buttonID").click(function() {
if(!requestSent) {
requestSent = true;
jQuery.ajax({
url: "http://example.com",
....,
timeout: timeoutValue,
complete: function() {
...
requestSent = false;
},
});
}
});
You can set a timeout value for long-running requests (value is in milliseconds) if you think your request has a possibility of hanging. If an timeout occurs, the error callback is called, after which the complete callback gets called.
You could store an active request in a variable, then clear it when there's a response.
var request; // Stores the XMLHTTPRequest object
$('#myButton').click(function() {
if(!request) { // Only send the AJAX request if there's no current request
request = $.ajax({ // Assign the XMLHTTPRequest object to the variable
url:...,
...,
complete: function() { request = null } // Clear variable after response
});
}
});
EDIT:
One nice thing about this, is that you could cancel long running requests using abort().
var request; // Stores the XMLHTTPRequest object
var timeout; // Stores timeout reference for long running requests
$('#myButton').click(function() {
if(!request) { // Only send the AJAX request if there's no current request
request = $.ajax({ // Assign the XMLHTTPRequest object to the variable
url:...,
...,
complete: function() { timeout = request = null } // Clear variables after response
});
timeout = setTimeout( function() {
if(request) request.abort(); // abort request
}, 10000 ); // after 10 seconds
}
});
$.xhrPool = {};
$.xhrPool['hash'] = []
$.ajaxSetup({
beforeSend: function(jqXHR,settings) {
var hash = settings.url+settings.data
if ( $.xhrPool['hash'].indexOf(hash) === -1 ){
jqXHR.url = settings.url;
jqXHR.data = settings.data;
$.xhrPool['hash'].push(hash);
}else{
console.log('Duplicate request cancelled!');
jqXHR.abort();
}
},
complete: function(jqXHR,settings) {
var hash = jqXHR.url+jqXHR.data
if (index > -1) {
$.xhrPool['hash'].splice(index, 1);
}
}
});