Currently, I have a lot of Joins and SubJoins with Group By etc.
To make it short: I cannot use paginate, because it is too slow (because of Group by). That is why I need to use pagination manually, however, I need to limit the total number of rows as well. With rows way above 10k, I want to have the maximum of a total of 300!
My current approach works suboptimal and I don't really know how to fix it:
return \DB::table('offers')
->alotOfQueries/joins
->...
->limit(300) //we show only a maximum of 300 offers
->get();
After the return, I'm manually creating the pagination.
I'll just leave the code as well here:
protected function paginate(Collection $offerings): LengthAwarePaginator
{
$page = request()->page ?? 1;
$perPage = $this->config->take;
$offset = ($page * $perPage) - $perPage;
return new LengthAwarePaginator(
array_slice($offerings->all(), $offset, $perPage, true),
$offerings->count(),
$perPage,
$page,
[
'path' => request()->url(),
'query' => request()->query()
]
);
}
This current approach is much faster than using paginate.
But my problem is it is still loading 300 rows.
My goal is to load only 18 rows per page. And calculate the correct total until it hits 300. It won't go above.
The numbers are taken from Airbnb, so I can show you this picture, so you maybe know what I'm after at:
Airbnb is always limiting it's max total to 300. Because 18 rows are loaded, we get 17 pages.
Any idea how to solve this?
Just for you, readers in the future. I'm lucky in that sense that I could limit my sub join ->limit(300)
With that I could now use paginate. Now it is not slow anymore at all!
I tested it against manual pagination like with:
return $offersQuery->offset($this->offset)
->limit($this->perPage)
->get();
and it does not matter what you are going to use.
Related
Why are the two statement below behaving differentlY? The first returns 3 the second returns 1 for $progress. I thought that the first aggregates on the DB and the second on the server. So they should still return the same value.
$progress = $this->user->userActivities()->select('date')
->groupBy('date')
->get()->count();
$progress = $this->user->userActivities()->select('date')
->groupBy('date')
->count();
->get()->count() will load Eloquent model objects into memory and then will count those.
->count() will use DB aggregate function, so it will definitely be more efficient:
select count(*) as aggregate ...
It's quite late answer but I had faced the same issue. These two examples are returning two different counts because of groupBy you have used.
$progress = $this->user->userActivities()->select('date')
->groupBy('date')
->get()->count();
This takes the rows first, then count the row numbers and returns the count.
$progress = $this->user->userActivities()->select('date')
->groupBy('date')
->count();
Whereas this counts the rows by its group in the DB and returns the first group's row record count only from the list.
the solution I used instead of get()->count() for my case is like
$progress = $this->user->userActivities()
->distinct('date')
->count('date');
Use ->get()->count() is wrong. I used ->get()->count() in my web app and when my database records have more than 80000 records I get error 500 in my app.
The first counts the number of returned records, the second counts the records and returns the number. If you are using a paging of 10, for example, then the first will yield 10 and the second the actual number if the number of mathcing elements is greater than 10.
$count = Model::all()->count();
return view('home',compact('count'));
this will function for laravel 6;
Both of them return the same result. But as said for preventing to crash or decrease the speed of response it's better to use count() instead of get()->count()
I have the following eloquent query
$raw = Model::select('out', 'in')->orderBy('created_at', 'DESC')->first();
That returns a collection of a single item, where Out = 0.0 and In = 90.0.
If I then do this:
$sumO = $raw->sum('out');
$sumI = $raw->sum('in');
I get $sumO = 13,651.41 and $sumI = 13371.69
I don't understand, because those sums don't even equal the sum of my entire table for those colums.
But it seems like->sum() is being called on the entire table/query instead of just the first result like I thought it would.
Now, I know sum of a single row is weird, and I'm not actually doing this in production. I just want to know what it is doing.
Shouldn't it still just sum the 1 number to equal itself?
It's just one row when using ->first(), so there's no need to use ->sum() just use $raw->in and $raw->out.
Also, ->sum() used with a single column at a time.
I'm showing 50 records per page. If 30 records have been deleted (or will not show for any other reasons) 1st page end index will 80 right? But second page index will be created 50 from pagination. So records between 50-80 indexes are shown both 1st and 2nd page.
How can I achieve this problem? Any ideas?
Feed the $config['total_rows'] variable in your pagination config the actual rows you want to show in view and it will calculate automatically, In other words count the total rows what you want to show, change the function of counting.
You must to recalculate current page based on max pages allowed after deletes:
Example, if your current page is $page, starting in 1:
$rowsByPage = 20;
$nrRows = ...count()...;
$maxPage = intval($nrRows / $rowsByPage) + 1;
then, adjust your page:
if($page > $maxPage) {
$page = $maxPage;
}
I am looking for a way to be able to get approx values of sales volume (basically how much money a store is generating) upon which I would like to base my rough calculations to show user some more data.
(Option A) I know I can directly query the database and then cache the results, but is Magento already doing that & caching it somewhere which I can just use? (Option B), like Magento saves the value of lifetime sales and average sales value somewhere.
Any data like daily/weekly/monthly average of sales or average order amount & daily/weekly/monthly average of orders would suffice here.
If the option A is only way to go here, how should I be querying this data? I have seen examples doing it like the following:
Mage::getResourceModel('sales/order_collection')
Mage::getModel('sales/order')->getCollection()
Which is faster & more lightweight for my needs? And any link where can I actually see detailed or well explained example of using them so that I can understand what parameters are available for querying in each case?
Update: I still don't understand clearly whats the difference between the above 2 methods, but my guess is one is more abstracted than the other & probably uses the other one internally but I am not sure. Anyways, I have this code snippet pulling in the data but I have a problem with filtering the data for a date range:
<?php
require_once 'app/Mage.php';
#umask(0);
Mage::app('default');
$orderTotals = Mage::getModel( 'sales/order' )->getCollection()
->addAttributeToFilter( 'status', Mage_Sales_Model_Order::STATE_COMPLETE )
->addAttributeToFilter( 'status', 'complete' )
//->addAttributeToFilter( 'created_at', array( 'from' => date( 'Y-m-d', strtotime( '-100 days' ) ) ) )
//->addAttributeToFilter( 'created_at', array( 'from' => date( 'Y-m-d', strtotime( '-100 days' ) ), 'to' => date( 'Y-m-d' ) ) )
->addAttributeToSelect( 'grand_total' )
->getColumnValues( 'grand_total' )
;
$totalSum = array_sum( $orderTotals );
$totalSum = Mage::helper( 'core' )->currency( $totalSum, true, false );
echo $totalSum . "\n";
Update:
This code snippet is working now. I had orders in the range but their order status was "processing", so I couldn't see them. I got the snippet from this question.
I am still looking for an explanation to what differs in the above 2 methods and which one is good to use?
In case someone comes here via Google like I did, this question motivated me to make my report page I've been wanting. I typically do not use the built in MVC (more below!) and instead will plop pages in var/export or something. I eventually came up with the below query after going through all the sales_flat_order* and sales_flat_invoice* tables finding a solution which represented the products we sold and shipped.
Sales report for this week:
SELECT sfi.grand_total as 'total',sfii.base_row_total as 'sales', sfii.sku as 'sku', sfii.qty as 'qty', sfo.increment_id as 'order'
FROM sales_flat_invoice_item sfii
LEFT JOIN sales_flat_invoice sfi ON sfii.parent_id = sfi.entity_id
LEFT JOIN sales_flat_order sfo ON sfi.order_id = sfo.entity_id
WHERE sfo.status = 'complete'
AND sfi.updated_at > "2013-06-01"
The date is this past Friday night at midnight. Note: if you add sfi.* to the SELECT you can do the math to double check shipping and taxes against the totals. The problem with sales_flat_order is that table doesn't represent cash in your checking account. We're not on accrual (and probably few small businesses are), so this is better for me understanding cash accounting basis.
For my report script, I also build a table of values and use Google Charts to make it pretty. The below page loads almost instantly:
Edited to add: While creating this above query, I discovered that a previous order querying tool built USING the proper means of collections, etc, misses a vast number of orders without any reason we could uncover. More and more, I'm finding the only way to get a job done correctly is to do it myself -- and do it the hard way.
Did you take a look # Admin -> Report -> Sales
/app/code/core/Mage/Adminhtml/controllers/Report/SalesController.php
/app/code/core/Mage/Adminhtml/Block/Report/Sales/Sales/Grid.php
Collection: 'sales/report_order_collection'
Tables:
sales_bestsellers_aggregated_*
sales_order_aggregated_*
report_viewed_product_*
I am using LINQ to write a query - one query shows all active customers , and another shows all active as well as inactive customers.
if(showall)
{
var prod = Dataclass.Customers.Where(multiple factors ) (all inactive + active)
}
else
{
var prod = Dataclass.Customers.Where(multiple factors & active=true) (only active)
}
Can I do this using only one query? The issue is that, multiple factors are repeated in both the queries
thanks
var customers = Dataclass.Customers.Where(multiple factors);
var activeCust = customers.Where(x => x.active);
I really don't understand the question either. I wouldn't want to make this a one-liner because it would make the code unreadable
I'm assuming you are trying to minimze the number of roundtrips?
If "multiple factors" is the same, you can just filter for active users after your first query:
var onlyActive = prod.Where(p => p.active == true);
Wouldn't you just use your first query to return all customers?? If not you'd be returning the active users twice.
Options I'd consider
Bring all customers once, order by 'status' column so you can easily split them into two sets
Focus on minimizing DB roundtrips. Whatever you do in the front end costs an order of magnitude less than going to the DB.
Revise user requirements. For ex. consider paging on results - it's unlikely that end user will need all customers at once.