Solution for Magento mass actions and large number of records problem? - magento

Question

Currently Magento has a problem with the way it handles mass actions. It returns a bit of JS that contains EVERY db id for the current collection and filter, regardless of pagination. This is to support the 'Select All' vs. 'Select All Visible' option in the grid header. This isn't such a problem when you have a smaller number of records, but if you have 850k records (orders in this case) it becomes a serious problem.
My question is, does anyone have an elegant solution to this problem?
I can think of several solutions, each with its own drawbacks, but I'm hoping someone has solved this in a simple manner that works as an add-on module. Paid or Open-Source solutions are both welcome suggestions.
Clarification:
I'm looking for an elegant/drop-in solution to the problem of 850k+ records using the grid widget in Magento. The stock Magento code makes the bone headed decision to return the id for every record that is matched by the current filter, even if they are not being displayed. This is not about offline processing of records, it's about using the grid widget for daily admin tasks.
One possible solution would be to store the results of the filtered search in a temp table and return a reference to the search result. Then you could change it from using the actual ids on a 'Select All' to using a specific callback for the action using the reference. This would preserve the current behavior.
So, to ask again, does anyone have a good solution to this already created?

Answer 1

I'm running heavy operations from within a shell script. I have a generic iterator (in my case products, but can be done with everything else), and I only implement a class that does the action on the product. My product_iterator shell script takes care of looping over the products, while processing only x products at once (to avoid memory leaks).

Answer 2

Well, the least invasive solution to the problem is to turn off the 'Select All' option for grids with large numbers of records. This is easily accomplished by extending the grid class and adding the following code:
protected function _prepareMassaction()
{
$this->getMassactionBlock()->setUseSelectAll(false);
return parent::_prepareMassaction();
}

Answer 3

I think that fbmc has the right general approach. Clearly, even if you did find a way to send back all 850k records, the framework would have a heart attack attempting to deal with them all. Run the logic in a shell script if this is what you need. For some jobs, you may even need to run them in batches or move down to actual SQL logic.
Unfortunately, some parts of the framework were not made to handle this kind of scale. You've found one of them.
Hope that helps!
Thanks,
Joe

Answer 4

Recently i have written an article about 'Adding new mass action to admin grid in Magento',
Hopefully you will like it:
http://www.blog.magepsycho.com/adding-new-mass-action-to-admin-grid-in-magento/
It describes the two way for adding Mass Action
1> Extending Grid Layouts _prepareMassaction() method
2> Using event: core_block_abstract_prepare_layout_before (more upgrade proof way)
Thanks
Regards