I'm having an issue where I have to display a table rendered client side, but the returned number of rows is about 2.5k entries. The model has a many-to-many relationship with a Sentence model through a bridging/pivot table with about 7k rows:
public function sentences()
{
return $this->belongsToMany(Sentence::class)->withTimestamps();
}
I'm trying to preload the Sentences via $entries = Entry:::with('sentences')->get(); but this generates a query with 2.5k ids in it where entry_sentence.entry_id in (1, 2, 3, 4, 5, 7, ..., 2462, 2463, 2464).
This query generated runs in about 0.06s on my local machine. But it takes about 8 seconds to generate the collection, which I think is due to hydration? Without preloading its very fast, but then I run into n+1 issues looping over my rows.
Am I forced into either "raw" DB::table() queries without Eloquent or pagination (and thus server side filtering ect.) ? What is the reasonable limit on eager loaded relationships? I can't seem to find any advice on this anywhere. I'm using Datatables in frontend and their rule of thumb was doing AJAX until about 5k rows then you should consider going server side pagination instead.
You can customize the related query:
$entries = Entry:::with('sentences' => function($query) {
return $query->limit(5); // however many is useful
}])->get();
Related
I really tried to understand the difference between the with() method and the load() method, but couldn't really understand.
As I see it, using the with() method is "better" since I eager load the relation. It seems that if I use load() I load the relation just as if I would use the hasMany() (or any other method that relates to the relation between objects).
Do I get it wrong?
Both accomplish the same end results—eager loading a related model onto the first. In fact, they both run exactly the same two queries. The key difference is that with() eager loads the related model up front, immediately after the initial query (all(), first(), or find(x), for example); when using load(), you run the initial query first, and then eager load the relation at some later point.
"Eager" here means that we're associating all the related models for a particular result set using just one query, as opposed to having to run n queries, where n is the number of items in the initial set.
Eager loading using with()
If we eager load using with(), for example:
$users = User::with('comments')->get();
...if we have 5 users, the following two queries get run immediately:
select * from `users`
select * from `comments` where `comments`.`user_id` in (1, 2, 3, 4, 5)
...and we end up with a collection of models that have the comments attached to the user model, so we can do something like $users->comments->first()->body.
"Lazy" eager loading using load()
Or, we can separate the two queries, first by getting the initial result:
$users = User::all();
which runs:
select * from `users`
And later, if we decide that we need the related comments for all these users, we can eager load them after the fact:
$users = $users->load('comments');
which runs the 2nd query:
select * from `comments` where `comments`.`user_id` in (1, 2, 3, 4, 5)
...and we end up with the same result, just split into two steps. Again, we can call $users->comments->first()->body to get to the related model for any item.
Why use load() vs. with()? load() gives you the option of deciding later, based on some dynamic condition, whether or not you need to run the 2nd query. If, however, there's no question that you'll need to access all the related items, use with().
The alternative to either of these would be looping through the initial result set and querying a hasMany() relation for each item. This would end up running n+1 queries, or 6 in this example. Eager loading, regardless of whether it's done up-front with with() or later with load(), only runs 2 queries.
As #damiani said, Both accomplish the same end results—eager loading a related model onto the first. In fact, they both run exactly the same two queries. The key difference is that with() eager loads the related model up front, immediately after the initial query (all(), first(), or find(x), for example); when using load(), you run the initial query first, and then eager load the relation at some later point.
There is one more difference between With() & load(), you can put the conditions when using with() but you can't do the same in case of load()
For example:
ProductCategory::with('children')
->with(['products' => function ($q) use($SpecificID) {
$q->whereHas('types', function($q) use($SpecificID) {
$q->where('types.id', $SpecificID)
});
}])
->get();
#damiani Explanied difference between load() and with() as well but he said load() is not cacheable so I wanna say couple words about it.
Let assume we have a blog post and related with comments. And we're fetching together and caching it.
$post = Cache::remember("post.".$slug,720,function()use($slug){
return Post::whereSlug($slug)->with("comments")->first();
});
But if there is a new comment and we want to display it immediately, we have to clear post cache and fetch post and comments together again. And that causes unnecessary queries. Lets think there are another queries for tags, media, contributors of the post etc. it will increase amount of resource usage..
public function load($relations)
{
$query = $this->newQueryWithoutRelationships()->with(
is_string($relations) ? func_get_args() : $relations
);
$query->eagerLoadRelations([$this]);
return $this;
}
As you can see above when we use the method it loads given relation and returns model with fetched relation. So you can return it outside of a callback.
$post = Cache::remember("post.".$slug,720,function()use($slug){
return Post::whereSlug($slug)->first();
});
$post = Cache::remember("post.relation.images.".$slug,720,function()use($post){
return $post->load("images");
});
$post = Cache::remember("post.relation.comments".$slug,720,function()use($post){
return $post->load("comments");
});
So if we load them seperatly, next time when some of them updated all you need to do clear specific relation cache and fetch it again. No need to fetch post, tags, images etc. over and over.
My concern is that while orderBy is applied to the query, I'm not sure how the sortBy is applied?
The reason for using sortBy in my case is because I get the collection via the model (i.e. $user->houses->sortBy('created_at')).
I'm just concerned about the performance: is sortBy simply looping each object and sorting them?, or is Laravel smart enough to simply transform the sortBy into an orderBy executed within the original query?
You need orderBy in order to perform a SQL order.
$user->houses()->orderBy('created_at')->get()
You can also eager load the houses in the right order to avoid N+1 queries.
$users = User::with(['houses' => function ($query) {
return $query->orderBy('created_at');
}])->get();
$orderedHouses = $users->first()->houses;
The sortBy method is applied to the Collection so indeed, it will looping each objects.
The orderBy() method is much more efficient than the sortBy() method when querying databases of a non-trivial size / at least 1000+ rows. This is because the orderBy() method is essentially planning out an SQL query that has not yet run whereas the sortBy() method will sort the result of a query.
For reference, it is important to understand the difference between a Collection object and a Builder object in Laravel.
A builder object is, essentially, an SQL query that has not been run. In contrast, a collection is essentially an array with some extra functionality/methods added. Sorting an array is much less efficient than pulling the data from the DB in the correct format on the actual query.
example code :
<?php
// Plan out a query to retrieve the posts alphabetized Z-A
// This is still a query and has not actually run
$posts = Posts::select('id', 'created_at', 'title')->orderBy('title', 'desc');
// Now the query has actually run. $posts is now a collection.
$posts = $posts->get();
// If you want to then sort this collection object to be ordered by the created_at
timestamp, you *could* do this.
// This will run quickly with a small number or rows in the result,
// but will be essentially unusable/so slow that your server will throw 500 errors
// if the collection contains hundreds or thousands or objects.
$posts = $posts->sortBy('created_at');
I have a problem with a query in Laravel with many records, because it is so slow.
I have a table users that has 4934 records.
I have table of relations that for example is named user_relation_values that has 17482 records.
I have a table values that has 20495 records.
Now I receive,from front end, N id of users, so I could receive 1 id or 4934 ids or 2000 ids or 1000 ids, so I don't know how many ids I will receive.
I have to return, starting by ids received, the relations of model users with id.
So in My backend I have a function like this:
$users= $request->input('users');
$usersValues = array();
foreach ($users as $user){
$o = User::find($owner['id']);
$o->values;
$owersProperties[] = $o;
}
I have the relations in My User model:
public function values(){
return $this->belongsToMany('App\Models\Values');
}
I have put The various indexes on tables but the query Is so slow if I receive the all id.
If I receive all 4934 ids the query takes more than 20 seconds, but I read that someone make query for milions of records in just over 5 seconds.
How can I optimize my table or my query?
Looking at your example code, you’re query users in a loop (n queries), and then also querying values for each user. That’s going to issue an exponential number of queries. Instead, you should query the users and eager-load the values relationship.
If you’re getting an array of user IDs as input, then you can pass that to Eloquent’s find() method:
$users = User::find($request->input('users'));
You should also eager-load the values relationship if you plan on using it in a loop:
$users = User::with('values')->find($request->input('users'));
This should dramatically reduce the number of queries you issue.
I have tables students, profiles, subjects and pivot table profile_subject
-students---------{id,profile_id,year}
-profiles---------{id}
-profile_subject--{profile_id,subject_id,year}
-subjects---------{id}
I want to select a student with id 5, and eager load profile and subjects for the students year.
Something like this:
$student = Student::with('profile','profile.subjects')->find(5);
But I also have to insert the condition
->wherePivot('year','=','students.year')
there somewhere. How to do that?
This query will not do the job cos it will search for records which year is "students.year" literary
Use lazy eager loading. This code will not create any additional queries, it'll create the same amount of queries as with() would:
$student = Student::find(5);
$sudent->load(['profile', 'profile.subjects' => function ($q) use ($student) {
$q->wherePivot('year', $student->year);
}]);
I really tried to understand the difference between the with() method and the load() method, but couldn't really understand.
As I see it, using the with() method is "better" since I eager load the relation. It seems that if I use load() I load the relation just as if I would use the hasMany() (or any other method that relates to the relation between objects).
Do I get it wrong?
Both accomplish the same end results—eager loading a related model onto the first. In fact, they both run exactly the same two queries. The key difference is that with() eager loads the related model up front, immediately after the initial query (all(), first(), or find(x), for example); when using load(), you run the initial query first, and then eager load the relation at some later point.
"Eager" here means that we're associating all the related models for a particular result set using just one query, as opposed to having to run n queries, where n is the number of items in the initial set.
Eager loading using with()
If we eager load using with(), for example:
$users = User::with('comments')->get();
...if we have 5 users, the following two queries get run immediately:
select * from `users`
select * from `comments` where `comments`.`user_id` in (1, 2, 3, 4, 5)
...and we end up with a collection of models that have the comments attached to the user model, so we can do something like $users->comments->first()->body.
"Lazy" eager loading using load()
Or, we can separate the two queries, first by getting the initial result:
$users = User::all();
which runs:
select * from `users`
And later, if we decide that we need the related comments for all these users, we can eager load them after the fact:
$users = $users->load('comments');
which runs the 2nd query:
select * from `comments` where `comments`.`user_id` in (1, 2, 3, 4, 5)
...and we end up with the same result, just split into two steps. Again, we can call $users->comments->first()->body to get to the related model for any item.
Why use load() vs. with()? load() gives you the option of deciding later, based on some dynamic condition, whether or not you need to run the 2nd query. If, however, there's no question that you'll need to access all the related items, use with().
The alternative to either of these would be looping through the initial result set and querying a hasMany() relation for each item. This would end up running n+1 queries, or 6 in this example. Eager loading, regardless of whether it's done up-front with with() or later with load(), only runs 2 queries.
As #damiani said, Both accomplish the same end results—eager loading a related model onto the first. In fact, they both run exactly the same two queries. The key difference is that with() eager loads the related model up front, immediately after the initial query (all(), first(), or find(x), for example); when using load(), you run the initial query first, and then eager load the relation at some later point.
There is one more difference between With() & load(), you can put the conditions when using with() but you can't do the same in case of load()
For example:
ProductCategory::with('children')
->with(['products' => function ($q) use($SpecificID) {
$q->whereHas('types', function($q) use($SpecificID) {
$q->where('types.id', $SpecificID)
});
}])
->get();
#damiani Explanied difference between load() and with() as well but he said load() is not cacheable so I wanna say couple words about it.
Let assume we have a blog post and related with comments. And we're fetching together and caching it.
$post = Cache::remember("post.".$slug,720,function()use($slug){
return Post::whereSlug($slug)->with("comments")->first();
});
But if there is a new comment and we want to display it immediately, we have to clear post cache and fetch post and comments together again. And that causes unnecessary queries. Lets think there are another queries for tags, media, contributors of the post etc. it will increase amount of resource usage..
public function load($relations)
{
$query = $this->newQueryWithoutRelationships()->with(
is_string($relations) ? func_get_args() : $relations
);
$query->eagerLoadRelations([$this]);
return $this;
}
As you can see above when we use the method it loads given relation and returns model with fetched relation. So you can return it outside of a callback.
$post = Cache::remember("post.".$slug,720,function()use($slug){
return Post::whereSlug($slug)->first();
});
$post = Cache::remember("post.relation.images.".$slug,720,function()use($post){
return $post->load("images");
});
$post = Cache::remember("post.relation.comments".$slug,720,function()use($post){
return $post->load("comments");
});
So if we load them seperatly, next time when some of them updated all you need to do clear specific relation cache and fetch it again. No need to fetch post, tags, images etc. over and over.