I have a page that displays some data using d3.js. Due to the heavy processing load, when the page load it freezes the browser for a few seconds.
I have determined that this "browser locking" behavior is due mostly to a line of the form:
selection.attr('d', linefn);
...where selection contains around 10K items.
I would like to replace this line with something like
function process_rest () {
if (selection.size() > 0) {
var next_item = first(selection); // function first() is hypothetical!
next_item.attr('d', linefn);
selection = rest(selection); // function rest() is hypothetical!
setTimeout(process_rest, 100);
return;
}
finish_up();
}
setTimeout(process_rest, 100);
I'm looking for an efficient way to implement either first and rest. My very naive guess would be something like:
function first(selection) {
return d3.select(selection[0][0]);
}
function rest(selection) {
selection[0] = selection[0].slice(1);
return selection;
}
...but, AFAIK, this is going "behind the API", or at least feels like it. Is there an "official" (i.e. documented) way to achieve the same result?
EDIT: deleted the shift variant (it's safer not to update selection until after the processing of the first element has been successfully completed).
You can simply use .each():
selection.each(function(d, i) {
setTimeout(function() { d3.select(this).attr("d", linefn); }, i * 100);
});
Related
I'm using Cypress for my automated tests. I'm trying to find a product on a page and click on it. If the product not displayed on the page, go to the next one until it is found.
I've been trying a lot of things already: while loop, each loop, simple cy.get but none of them work. Can anyone help me to solve this?
You'll need a recursive command, implementation of which will depend on your specific scenario. There's no one-size-fits-all solution, but generally it will look something like this:
function findElem ( targetSelector ) {
// first, we need to query a container (could be table, or a generic item
// container). The important thing is, this container must exist on every
// page you'll traverse.
cy.get('.someContainer').then( $container => {
// synchronously find the target element, however you want. In this
// example I use an attribute selector, but you can do whatever you
// want.
if ( $container.find(targetSelector).length ) {
return true;
} else {
return false;
}
}).then( found => {
if ( found ) {
return cy.log(`found elem "${targetSelector}"`);
} else {
// synchronously check if there's a next page/table
if ( Cypress.$('.nextPageButton').length ) {
// when know that there is a next page, click it using cypress
cy.get('.nextPageButton').click();
// here, assert that next page/table is loaded, possibly by
// asserting that current page/table is removed from DOM
// then, recurse
return findElem(targetSelector);
} else {
throw new Error(`Couldn't find elem "${targetSelector}" on any page`);
}
}
});
}
it('test', () => {
findElem(`[data-id="42"]`);
});
The crux of the solution is using a combination of commands to query a container, and synchronous inspection (using jQuery or whatever) to find the actual element. Learn more at Conditional Testing.
For a concrete implementation, you can refer to an older answer I gave to a similar question.
When writing custom functions to be used in spreadsheet cells, the default behavior for a sheet is to recalculate on edits, i.e. adding column or rows will cause a custom function to update.
This is a problem if the custom function calls a paid API and uses credits, the user will consuming API credits automatically.
I couldn't figure out a way to prevent this, so I decided to use the UserCache to cache the results for an arbitrary 25 minutes, and serve it back to the user should they happen to repeat the same function call. It's definitely not bulletproof but it's better than nothing I suppose. Apparently the cache can hold 10mb, but is this the right approach? Could I be doing something smarter?
var _ROOT = {
cache : CacheService.getUserCache(),
cacheDefaultTime: 1500,
// Step 1 -- Construct a unique name for function call storage using the
// function name and arguments passed to the function
// example: function getPaidApi(1,2,3) becomes "getPaidApi123"
stringifyFunctionArguments : function(functionName,argumentsPassed) {
var argstring = ''
for (var i = 0; i < argumentsPassed.length; i++) {
argstring += argumentsPassed[i]
}
return functionName+argstring
},
//Step 2 -- when a user calls a function that uses a paid api, we want to
//cache the results for 25 minutes
addToCache : function (encoded, returnedValues) {
var values = {
returnValues : returnedValues
}
Logger.log(encoded)
this.cache.put(encoded, JSON.stringify(values), this.cacheDefaultTime)
}
//Step 3 -- if the user repeats the exact same function call with the same
//arguments, we give them the cached result
//this way, we don't consume API credits as easily.
checkCache : function(encoded) {
var cached = this.cache.get(encoded);
try {
cached = JSON.parse(cached)
return cached.returnValues
} catch (e) {
return false;
}
}
}
Google Sheets already caches the values of custom functions, and will only run them again when either a) the inputs to the function have changed or b) the spreadsheet is being opened after being closed for a long time. I'm not able to replicate the recalculation you mentioned when adding and removing columns. Here's a simple example function I used to test that:
function rng() {
return Math.random();
}
Your approach of using an additional cache for expensive queries looks fine in general. I'd recommend using the DocumentCache instead of the UserCache, since all users of the document can and should see the same cell values.
I'd also recommend a more robust encoding of function signatures, since your current implementation is able to distinguish between the arguments [1, 2] and [12]. You could stringify the inputs and then base64 encode it for compactness:
function encode(functionName, argumentsPassed) {
var data = [functionName].concat(argumentsPassed);
var json = JSON.stringify(data);
return Utilities.base64Encode(json);
}
Is there any way to effectively delete rows in Parse that do something like this SQL statement?
DELETE FROM table WHERE delete_me = 1
I've tried this, but it's very slow:
var query = new Parse.Query('table');
query.equalTo('delete_me', 1);
query.each(function(obj) {
return obj.destroy();
}).then(function() {
// Done
}, function(error) {
// Error
});
Almost there: find() will get the objects satisfying the delete criteria, then destroyAll() will destroy them all.
var query = new Parse.Query('table');
query.equalTo('delete_me', 1);
query.find().then(function(results) {
return Parse.Object.destroyAll(results);
}).then(function() {
// Done
}, function(error) {
// Error
});
Edit - to delete a table with more than 1k, it takes a little extra work with promises. The idea is to cursor through the table, grouping finds in batches of 1k (or some smaller increment), execute those finds concurrently using Promise.when(), then destroy the results concurrently the same way...
var query = new Parse.Query('table');
query.equalTo('delete_me', 1);
query.count().then(function(count) {
var finds = [];
for (var i=0; i<count; i+=1000) {
finds.push(findSkip(i));
}
return Parse.Promise.when(finds);
}).then(function() {
var destroys = [];
_.each(arguments, function(results) {
destroys.push(Parse.Object.destroyAll(results));
});
return Parse.Promise.when(destroys);
}).then(function() {
// Done
}, function(error) {
// Error
});
// return a promise to find 1k rows starting after the ith row
function findSkip(i) {
var query = new Parse.Query('table');
query.limit(1000);
query.equalTo('delete_me', 1);
query.skip(i);
return query.find();
}
Edit 2 - This might be faster, but you'd need to discover empirically:
// return a promise to delete 1k rows from table, promise is fulfilled with the count deleted
function deleteABunch() {
var query = new Parse.Query('table');
query.limit(1000);
query.equalTo('delete_me', 1);
query.find().then(function(results) {
return Parse.Object.destroyAll(results).then(function() {
return results.length;
});
});
}
function deleteAll() {
return deleteABunch().then(function(count) {
return (count)? deleteAll() : Parse.Promise.as();
});
}
The 1802 request thing is the rate-limit (30/sec). The next idea is to batch the work into smaller transaction-count promises and run them serially, keeping the rate low but stretching them out over time. That's the gist of my suggestion above in a couple of forms (before I understood that you have ~500k rows).
Unfortunately, parse enforces a 10sec timeout limit, too. I think about ~1k rows deleted per sec is achievable, but I fear your 500k table will not yield to any method on the free tier. I think you have only these alternatives:
(a) throttle on the client - use some form of setTimeout(), to perform small enough, short enough batches. (This is how my app handles it, because the heavy work is done only by admins, and I can instruct them to not reload a page.).
(b) deploy your own node server which basically implements idea (a), calling parse.com in small enough steps to keep it happy but places no computational burden on the client.
(c) a parse.com background job that wakes periodically and nibbles away at it. You only get one of these on the free tier, and I imagine most of the time it will just wake frequently and waste electricity.
(d) pay.
I'll be able to do some actual code/test late today. If I learn anything new I'll post here. Best of luck.
I have a cron job that scrapes a list of items on a website and then inserts or updates records in a database. When I scrape the page, I want to create records for new ones that haven't been created yet, otherwise update any existing ones. Currently I'm doing something like this:
// pretend there is a "Widget" model defined
function createOrUpdateWidget(widgetConfig) {
return Widget.find(widgetConfig.id)
.then(function(widget) {
if (widget === null) {
return Widget.create(widgetConfig);
}
else {
widget.updateAttributes(widgetConfig);
}
});
}
function createOrUpdateWidgets(widgetConfigObjects) {
var promises = [];
widgetConfigObjects.forEach(function(widgetConfig) {
promises.push(createOrUpdateWidget(widgetConfig));
});
return Sequelize.Promise.all(promises);
}
createOrUpdateWidgets([...])
.done(function() {
console.log('Done!');
});
This seems to work fine, but I'm not sure if I'm doing this "correctly" or not. Do all promises that perform DB interactions need to run serially, or is how I have them defined ok? Is there a better way to do this kind of thing?
What you're doing is pretty idiomatic and perfectly fine, the only room for improvement is to utilize the fact Sequelize uses Bluebird for promises so you get .map for free, which lets you convert:
function createOrUpdateWidgets(widgetConfigObjects) {
var promises = [];
widgetConfigObjects.forEach(function(widgetConfig) {
promises.push(createOrUpdateWidget(widgetConfig));
});
return Sequelize.Promise.all(promises);
}
Into:
function createOrUpdateWidgets(widgetConfigObjects) {
return Sequelize.Promise.map(widgetConfig, createOrUpdateWidget)
}
Other than that minor improvement - you're chaining promises correctly and seem to have the correct hang of it.
I've build a livesearch with the jQuery.ajax() method. On every keyup events it receives new result data from the server.
The problem is, when I'm typing very fast, e.g. "foobar" and the GET request of "fooba" requires more time than the "foobar" request, the results of "fooba" are shown.
To handle this with the timeout parameter is impossible, I think.
Has anyone an idea how to solve this?
You can store and .abort() the last request when starting a new one, like this:
var curSearch;
$("#myInput").keyup(function() {
if(curSearch) curSearch.abort(); //cancel previous search
curSearch = $.ajax({ ...ajax options... }); //start a new one, save a reference
});
The $.ajax() method returns the XmlHttpRequest object, so just hang onto it, and when you start the next search, abort the previous one.
Assign a unique, incrementing ID to each request, and only show them in incrementing order. Something like this:
var counter = 0, lastCounter = 0;
function doAjax() {
++counter;
jQuery.ajax(url, function (result) {
if (counter < lastCounter)
return;
lastCounter = counter;
processResult(result);
});
}
You should only start the search when the user hasn't typed anything for a while (500ms or so). This would prevent the problem you're having.
An excellent jQuery plugin which does just that is delayedObserver:
http://code.google.com/p/jquery-utils/wiki/DelayedObserver
Make it so each cancels the last. That might be too much cancellation, but when typing slows, it will trigger.
That seems like an intense amount of traffic to send an ajax request for every KeyUp event. You should wait for the user to stop typing - presumably that they are done, for at least a few 100 milliseconds.
What I would do is this:
var ajaxTimeout;
function doAjax() {
//Your actual ajax request code
}
function keyUpHandler() {
if (ajaxTimeout !== undefined)
clearTimeout(ajaxTimeout);
ajaxTimeout = setTimeout(doAjax, 200);
}
You may have to play with the actual timeout time, but this way works very well and does not require any other plugins.
Edit:
If you need to pass in parameters, create an inline function (closure).
...
var fun = function() { doAjax(params...) };
ajaxTimeout = setTimeout(fun, 200);
You will want some kind of an ajax queue such as:
http://plugins.jquery.com/project/ajaxqueue
or http://www.protofunc.com/scripts/jquery/ajaxManager/
EDIT:Another option, study the Autocomplete plug-in code and emulate that.(there are several Autocomplete as well as the one in jquery UI
OR just implement the Autocomplete if that serves your needs