I have a cron job that scrapes a list of items on a website and then inserts or updates records in a database. When I scrape the page, I want to create records for new ones that haven't been created yet, otherwise update any existing ones. Currently I'm doing something like this:
// pretend there is a "Widget" model defined
function createOrUpdateWidget(widgetConfig) {
return Widget.find(widgetConfig.id)
.then(function(widget) {
if (widget === null) {
return Widget.create(widgetConfig);
}
else {
widget.updateAttributes(widgetConfig);
}
});
}
function createOrUpdateWidgets(widgetConfigObjects) {
var promises = [];
widgetConfigObjects.forEach(function(widgetConfig) {
promises.push(createOrUpdateWidget(widgetConfig));
});
return Sequelize.Promise.all(promises);
}
createOrUpdateWidgets([...])
.done(function() {
console.log('Done!');
});
This seems to work fine, but I'm not sure if I'm doing this "correctly" or not. Do all promises that perform DB interactions need to run serially, or is how I have them defined ok? Is there a better way to do this kind of thing?
What you're doing is pretty idiomatic and perfectly fine, the only room for improvement is to utilize the fact Sequelize uses Bluebird for promises so you get .map for free, which lets you convert:
function createOrUpdateWidgets(widgetConfigObjects) {
var promises = [];
widgetConfigObjects.forEach(function(widgetConfig) {
promises.push(createOrUpdateWidget(widgetConfig));
});
return Sequelize.Promise.all(promises);
}
Into:
function createOrUpdateWidgets(widgetConfigObjects) {
return Sequelize.Promise.map(widgetConfig, createOrUpdateWidget)
}
Other than that minor improvement - you're chaining promises correctly and seem to have the correct hang of it.
Related
So I think this is probably me mixing up sync/async code (Mainly because Cypress has told me so) but I have a function within a page object within Cypress that is searching for customer data. I need to use this data later on in my test case to confirm the values.
Here is my function:
searchCustomer(searchText: string) {
this.customerInput.type(searchText)
this.searchButton.click()
cy.wait('#{AliasedCustomerRequest}').then(intercept => {
const data = intercept.response.body.data
console.log('Response Data: \n')
console.log(data)
if (data.length > 0) {
{Click some drop downdowns }
return data < ----I think here is the problem
} else {
{Do other stuff }
}
})
}
and in my test case itself:
let customerData = searchAndSelectCustomerIfExist('Joe Schmoe')
//Do some stuff with customerData (Probably fill in some form fields and confirm values)
So You can see what I am trying to do, if we search and find a customer I need to store that data for my test case (so I can then run some cy.validate commands and check if the values exist/etc....)
Cypress basically told me I was wrong via the error message:
cy.then() failed because you are mixing up async and sync code.
In your callback function you invoked 1 or more cy commands but then
returned a synchronous value.
Cypress commands are asynchronous and it doesn't make sense to queue
cy commands and yet return a synchronous value.
You likely forgot to properly chain the cy commands using another
cy.then().
So obviously I am mixing up async/sync code. But since the return was within the .then() I was thinking this would work. But I assume in my test case that doesn't work since the commands run synchronously I assume?
Since you have Cypress commands inside the function, you need to return the chain and use .then() on the returned value.
Also you need to return something from the else branch that's not going to break the code that uses the method, e.g an empty array.
searchCustomer(searchText: string): Chainable<any[]> {
this.customerInput.type(searchText)
this.searchButton.click()
return cy.wait('#{AliasedCustomerRequest}').then(intercept => {
const data = intercept.response.body.data
console.log('Response Data: \n')
console.log(data)
if (data.length) {
{Click some drop downdowns }
return data
} else {
{Do other stuff }
return []
}
})
}
// using
searchCustomer('my-customer').then((data: any[]) => {
if (data.length) {
}
})
Finally "Click some drop downdowns" is asynchronous code, and you may get headaches calling that inside the search.
It would be better to do those actions after the result is passed back. That also makes your code cleaner (easier to understand) since searchCustomer() does only that, has no side effects.
you just need to add return before the cy.wait
here's a bare-bones example
it("test", () => {
function searchCustomer() {
return cy.wait(100).then(intercept => {
const data = {text: "my data"}
return data
})
}
const myCustomer = searchCustomer()
myCustomer.should("have.key", "text")
myCustomer.its("text").should("eq", "my data")
});
I'm learning composition API doing a simple todo app but then with multiple swim-lanes (todo, doing, done).
in useTasks.js composable I create "global" sourceTasks reactive, that later gets filled with data pulled from an API. Then it is reduced in tasks computed property, like this:
// useTasks.js
const sourceTasks = reactive({
list: []
});
export default function useTasks() {
const tasks = computed(() => {
return sourceTasks.list.reduce(divideIntoSwimLanes, [])
});
...
return {
tasks,
loadTasks,
createTask
}
}
Nothing too complicated.
Then I've got this SwimLane component, that well... uses the tasks :)
// SwimLane.vue - setup
async setup(props) {
const { status } = toRefs(props);
const { tasks, createTask } = useTasks();
return {
tasks,
label,
createTask
}
}
// SwimLane.vue - template
<single-task class="m-3" v-for="task in tasks[status]" :title="task.title" :id="task.id"/>
This works, but I don't find it elegant. I would prefer to create a new computed inside of SwimLane's setup, that holds the value of tasks for the given SwimLane. Putting it in the template obscures the logic.
I would expect this to work, but it does not, I think it loses the reactivity but I cant wrap my head around why:
// SwimLane.vue - alternative setup
const currentContextTasks = computed(() => {
return tasks.value[status]
});
return {
currentContextTasks
}
The problem feels a bit ridiculous, but my main concern is that I have misunderstood some core concept, hence this lengthy post.
This is like a biggest blunders ever. So right, the post was really helpful as a form of talking to the rubber duck.
What I did forgot to do is use the value of the status. Since it is a ref, I had to use it as follows:
const currentContextTasks = computed(() => {
return tasks.value[status.value] // 'status.value' and not just 'status'
});
Is there any way to effectively delete rows in Parse that do something like this SQL statement?
DELETE FROM table WHERE delete_me = 1
I've tried this, but it's very slow:
var query = new Parse.Query('table');
query.equalTo('delete_me', 1);
query.each(function(obj) {
return obj.destroy();
}).then(function() {
// Done
}, function(error) {
// Error
});
Almost there: find() will get the objects satisfying the delete criteria, then destroyAll() will destroy them all.
var query = new Parse.Query('table');
query.equalTo('delete_me', 1);
query.find().then(function(results) {
return Parse.Object.destroyAll(results);
}).then(function() {
// Done
}, function(error) {
// Error
});
Edit - to delete a table with more than 1k, it takes a little extra work with promises. The idea is to cursor through the table, grouping finds in batches of 1k (or some smaller increment), execute those finds concurrently using Promise.when(), then destroy the results concurrently the same way...
var query = new Parse.Query('table');
query.equalTo('delete_me', 1);
query.count().then(function(count) {
var finds = [];
for (var i=0; i<count; i+=1000) {
finds.push(findSkip(i));
}
return Parse.Promise.when(finds);
}).then(function() {
var destroys = [];
_.each(arguments, function(results) {
destroys.push(Parse.Object.destroyAll(results));
});
return Parse.Promise.when(destroys);
}).then(function() {
// Done
}, function(error) {
// Error
});
// return a promise to find 1k rows starting after the ith row
function findSkip(i) {
var query = new Parse.Query('table');
query.limit(1000);
query.equalTo('delete_me', 1);
query.skip(i);
return query.find();
}
Edit 2 - This might be faster, but you'd need to discover empirically:
// return a promise to delete 1k rows from table, promise is fulfilled with the count deleted
function deleteABunch() {
var query = new Parse.Query('table');
query.limit(1000);
query.equalTo('delete_me', 1);
query.find().then(function(results) {
return Parse.Object.destroyAll(results).then(function() {
return results.length;
});
});
}
function deleteAll() {
return deleteABunch().then(function(count) {
return (count)? deleteAll() : Parse.Promise.as();
});
}
The 1802 request thing is the rate-limit (30/sec). The next idea is to batch the work into smaller transaction-count promises and run them serially, keeping the rate low but stretching them out over time. That's the gist of my suggestion above in a couple of forms (before I understood that you have ~500k rows).
Unfortunately, parse enforces a 10sec timeout limit, too. I think about ~1k rows deleted per sec is achievable, but I fear your 500k table will not yield to any method on the free tier. I think you have only these alternatives:
(a) throttle on the client - use some form of setTimeout(), to perform small enough, short enough batches. (This is how my app handles it, because the heavy work is done only by admins, and I can instruct them to not reload a page.).
(b) deploy your own node server which basically implements idea (a), calling parse.com in small enough steps to keep it happy but places no computational burden on the client.
(c) a parse.com background job that wakes periodically and nibbles away at it. You only get one of these on the free tier, and I imagine most of the time it will just wake frequently and waste electricity.
(d) pay.
I'll be able to do some actual code/test late today. If I learn anything new I'll post here. Best of luck.
This is a sample functionality I need, is it possible??. I am facing problem in debugging it
I need those two functions to be ran before I do any other modification how to do it?
function getPromise() {
var deferred = $.Deferred();
$.when(getPromiseother()).done(function() {
deferred.resolve();
});
deferred.getPromiseother()
}
function getPromise() {
var deferrednext = $.Deferred();
$.when($.ajax(somefunction)).done(function() {
deferrednext.resolve();
});
deferrednext.promise()
}
$.when(getPromise).done(function() {
do something
});
This is a very clumsy way to do this - basically a major point of promises is that they compose and chain and you don't need to $.Deferred anywhere that's not converting a callback API to promises.
$.when($.ajax(somefunction), someOtherPromise).then(function(result, other){
// in here both promises have run to fulfillment and data is available
// so place the rest of your code here
});
I have a page that displays some data using d3.js. Due to the heavy processing load, when the page load it freezes the browser for a few seconds.
I have determined that this "browser locking" behavior is due mostly to a line of the form:
selection.attr('d', linefn);
...where selection contains around 10K items.
I would like to replace this line with something like
function process_rest () {
if (selection.size() > 0) {
var next_item = first(selection); // function first() is hypothetical!
next_item.attr('d', linefn);
selection = rest(selection); // function rest() is hypothetical!
setTimeout(process_rest, 100);
return;
}
finish_up();
}
setTimeout(process_rest, 100);
I'm looking for an efficient way to implement either first and rest. My very naive guess would be something like:
function first(selection) {
return d3.select(selection[0][0]);
}
function rest(selection) {
selection[0] = selection[0].slice(1);
return selection;
}
...but, AFAIK, this is going "behind the API", or at least feels like it. Is there an "official" (i.e. documented) way to achieve the same result?
EDIT: deleted the shift variant (it's safer not to update selection until after the processing of the first element has been successfully completed).
You can simply use .each():
selection.each(function(d, i) {
setTimeout(function() { d3.select(this).attr("d", linefn); }, i * 100);
});