I need to manually save a very large number of order items to the database. I'm currently doing something like this:
for($x=0; $x<250000; $x++)
{
$orderItem = Mage::getModel('sales/order_item');
$data = array('a'=>1, 'b'=>2, 'c'=>3);
$orderItem->setData($data)->save();
}
Attempting to run this code from a shell script takes forever. What strategies can I use to speed up this code?
I'm not sure but take a look # Mage::getModel('core/resource_transaction')
See https://stackoverflow.com/a/4879133/1191288
You could try doing a batch save on ever X amount
$batch = 500;
$orderItem = Mage::getModel('sales/order_item');
$transactionSave = Mage::getModel('core/resource_transaction');
for($x=0; $x<250000; $x++){
$orderItem->setData( array('a'=>1, 'b'=>2, 'c'=>3));
$transactionSave->addObject($orderItem);
$orderItem->reset()
if ($x % $batch == 0){
$transactionSave->save();
$transactionSave = null;
$transactionSave = Mage::getModel('core/resource_transaction');
}
}
if($x % $batch > 0)
$transactionSave->save();
I think in this situation, your best bet to speed this up significantly would be to create custom database queries.
Edit: Related option: try to rewrite the model and resource model to implement a method bulkSave() which is like save() but only creates the corresponding query object and returns it, so it does not actually use the database. Then collect all the queries and run them in big transactions every thousand items or so.
Related
Is it possible to save a query bulider and use it multiple times?
for example, I have a model 'Tour'.
I create a long query buider and paginate it:
$tour = Tour::where(...)->orWhere(...)->orderBy(...)->paginate(10);
For example, 97 models qualify for the above query.
"Paginate" method outputs first 10 models qualifying for the query, but I also need to so some operations on all 97 models.
I don't want to 'repeat myself' writing this long query 2 times.
So I want something like:
$query = Tour::where(...)->orWhere(...)->orderBy(...);
$tour1 = $query->paginate(10);
$tour2 = $query->get();
Is that a correct way to do in Laravel? (my version is 5.4).
You need to use clone:
$query = Tour::where(...)->orWhere(...)->orderBy(...);
$query1 = clone $query;
$query2 = clone $query;
$tour1 = $query1->paginate(10);
$tour2 = $query2->get();
You can but it doesn't make any sense because every time a new query will be executed. So this code will work:
$query = Tour::where(...)->orWhere(...)->orderBy(...);
$tour1 = $query->paginate(10);
$tour2 = $query->get();
But if you want to execute just one query, you'll need to use collection methods for ordering, filtering and mapping the data. You'll also need to create Paginator instance manually:
$collection = Tour::where(...)->orWhere(...)->orderBy(...)->get();
$tour1 = // Make Paginator manually.
$tour2 = $collection;
$sortedByName = $collection->sortBy('name');
$activeTours = $collection->where('active', 1);
I have a complex SELECT with several time consuming calculated fields. I only want the first 100 rows, however, I also need the total rows found. This means executing the query once with LIMIT and once without.
I would like to be able to build two SELECT's simultaneously with CodeIgniter's QueryBuilder and eliminate the time consuming calculated fields on the second query.
I would like to know if there is a good way to do this with QB or should I just build the SELECT without QB.
UPDATE: Apparently I have confused people. I need to do something like this:
To get the records to display:
SELECT id, [ several really time intensive calculated fields ]
FROM table
WHERE [some complicated criteria]
LIMIT 100;
To get the total records I would like to re-use the built query or build a 2nd one simultaneously but without the limit and with only the id field:
SELECT id
FROM table
WHERE [some complicated criteria];
Another option; I have learned that if I put the time intensive calculated fields in a sub SELECT, Postgres will not execute it until the row is actually retrieved.
For now I am manually building two queries but I'd like to do it the correct QueryBuilder way.
There is no way to run multiple query in one statement in codeigniter. But you may do as following
$query1 = $this->db->query('SELECT * FROM table_name LIMIT 10');
$query2 = $this->db->query('SELECT * FROM table_name');
But what I do is make a method and call based on parameter like
public function get_rows($table, $num_rows = FALSE){
$this->db->select('*');
$this->db->from($table);
if($num_rows === FALSE){
$this->db->limit(10);
}
$query = $this->db->get();
if($num_rows){
return $query->num_rows();
}
return $query->result_array();
}
Now you may call it as your requirements
//call for only count rows (without limit)
$this->model_name->get_rows('table_name', TRUE);
//call with limit for result rows
$this->model_name->get_rows('table_name');
with count you can do this with query builder I think
look at this code, you may want something like that
if($limit != null){
$this->db->limit($limit, $offset);
}
if($order_by != null){
$this->db->order_by($order_by, $sort);
}
$query = $this->db->get('courses');
$courses['rows'] = $query->result();
$courses['num_rows'] = $this->db->count_all('courses');
return $courses;
I'm trying to merge multiple objects (like Receipts, Reports, etc) with Collection->merge().
This is the code I used:
$receipts = Receipt::all();
$reports = Report::all();
$collection = $receipts->merge($reports);
This is the result:
The above screenshot shows two elements, but the third element is missing because it has the same id (id: "1") as the first one. What I'm trying to achieve is to display all three of them as a collection.
EDIT:
I need the result to be objects (collection) because I also use the code on my view, where I check the class to determine what to display. Also, I use this function to sort the objects in the collection.
$collection->sort(function($a, $b)
{
$a = $a->created_at;
$b = $b->created_at;
if ($a === $b) {
return 0;
}
return ($a > $b) ? 1 : -1;
});
I know that this is an old question, but I will still provide the answer just in case someone comes here from the search like I did.
If you try to merge two different eloquent collections into one and some objects happen to have the same id, one will overwrite the other. I dunno why it does that and if that's a bug or a feature - more research needed. To fix this just use push() method instead or rethink your approach to the problem to avoid that.
Example of a problem:
$cars = Car::all();
$bikes = Bike::all();
$vehicles = $cars->merge($bikes);
// if there is a car and a bike with the same id, one will overwrite the other
A possible solution:
$collection = collect();
$cars = Car::all();
$bikes = Bike::all();
foreach ($cars as $car)
$collection->push($car);
foreach ($bikes as $bike)
$collection->push($bike);
Source: https://medium.com/#tadaspaplauskas/quick-tip-laravel-eloquent-collections-merge-gotcha-moment-e2a56fc95889
I know i'm bumping a 4 years old thread but i came across this and none of the answers were what i was looking for; so, like #Tadas, i'll leave my answer for people who will come across this. After Looking at the laravel 5.5 documentation thoroughly i found that concat was the go-to method.
So, in the OP's case the correct solution would be:
$receipts = Receipt::all();
$reports = Report::all();
$collection = $receipts->concat($reports);
This way every element in the Report collection will be appended to every element in the Receipts collection, event if some fields are identical.
Eventually you could shuffle it to get a more visual appealing result for e.g. a view:
$collection->shuffle();
Another way to go about it is to convert one of your collections to a base collection with toBase() method. You can find it in Illuminate\Support\Collection
Method definition:
/**
* Get a base Support collection instance from this collection.
*
* #return \Illuminate\Support\Collection
*/
public function toBase()
{
return new self($this);
}
Usage:
$receipts = Receipt::all();
$reports = Report::all();
$collection = $receipts->toBase()->merge($reports);
You could put all collections in an array and use this. Depends on what you want to do with the collection.
$list = array();
$list = array_merge($list, Receipt::all()->toArray());
$list = array_merge($list, Report::all()->toArray());
Is there any smart way of reusing current query. I want to use same query EXCEPT from one join. See example below:
$this->db->distinct();
$this->db->select('outfit_main.*');
$this->db->join('outfit_products', 'outfit_main.id = outfit_products.outfit_id');
$this->db->join('product', 'outfit_products.product_id = product.id');
$this->db->join('favorite', 'favorite.product_id = outfit_products.product_id');
$check = $this->db->get('outfit_main')->result('OutfitModel');
if ($check === false) {
//How to - use above query
//EXCEPT from
//$this->db->join('favorite', 'favorite.product_id = outfit_products.product_id');
$check = $this->db->get('outfit_main')->result('OutfitModel');
}
With smart way I mean that I wouldn't have to reset whole query and create a whole new query.
I'm using codeigniter and the pagination class. This is such a basic question, but I need to make sure I'm not missing something. In order to get the config items necessary to paginate results getting them from a MySQL database it's basically necessary to run the query twice is that right?
In other words, you have to run the query to determine the total number of records before you can paginate. So I'm doing it like:
Do this query to get number of results
$this->db->where('something', $something);
$query = $this->db->get('the_table_name');
$num_rows = $query->num_rows();
Then I'll have to do it again to get the results with the limit and offset. Something like:
$this->db->where('something', $something);
$this->db->limit($limit, $offset);
$query = $this->db->get('the_table_name');
if($query->num_rows()){
foreach($query->result_array() as $row){
## get the results here
}
}
I just wonder if I'm actually doing this right in that the query always needs to be run twice? The queries I'm using are much more complex than what is shown above.
Unfortunately, in order to paginate you must know how many elements you are breaking up into pages.
You could always cache the result for the total number of elements if it is too computationally expensive.
Yeah, you have to run two queries, but $this->db->count_all('table_name'); is one & line much cleaner.
Pagination requires reading a record set twice:
Once to read the whole set so that it can count the total number records
Then to read a window of records to display
Here's an example I used for a project. The 'banner' table has a list of banners, which I want to show on a paginated screen:
Using a public class property to store the total records (public $total_records)
Using a private function to build the query (that is common for both activities). The parameter ($isCount) we pass to this function reduces the amount of data the query generate, because for the row count we only need one field but when we read the data window we need all required fields.
The get_list() function first calls the database to find the total and stores it in $total_records and then reads a data window to return to the caller.
Remember we cannot access $total_records without first calling the get_list() method !
class Banner_model extends CI_Model {
public $total_records; //holds total records for get_list()
public function get_list($count = 10, $start = 0) {
$this->build_query();
$query = $this->db->get();
$result = $query->result();
$this->total_records = count($result); //store the count
$this->build_query();
$this->db->limit($count, $start);
$query = $this->db->get();
$result = $query->result();
return $result;
}
private function build_query($isCount = FALSE) {
$this->db->select('*, b.id as banner_id, b.status as banner_status');
if ($isCount) {
$this->db->select('b.id');
}
$this->db->from('banner b');
$this->db->join('company c', 'c.id = b.company_id');
$this->db->order_by("b.id", "desc"); //latest ones first
}
And now from the controller we call:
$data['banner_list'] = $this->banner_model->get_list();
$config['total_rows'] = $this->banner_model->total_records;
Things get complicated when you start using JOINs, like in my example where you want to show banners from a particular company! You may read my blog post on this issue further:
http://www.azmeer.info/pagination-hitting-the-database-twise/