I am using laravel eloquent to get the query results. I have two tables below:
users table:
| id | department_id
| 1 | 1
| 2 | 3
| 3 | 2
department table:
| id | name
| 1 | A
| 2 | B
| 3 | C
| 4 | D
| 5 | E
How to get one unassigned ID, not existing department ID, into the users table? Example, 4 & 5 are not yet existing in users table, so how can I get 4 or 5 using an eloquent?
I am thinking of this but this is not correct.
Department::select('department.id as id')
->leftJoin('users', 'users.department_id' ,'department.id')
->pluck('id');
Does anybody know?
Try this
//here you first got all the department which is assigned to user
$assigned_dept = Users::pluck('department_id')->toArray();
$department = array_values($assigned_dept); //output:['1','3','2']
//here you can select department which is not assigned to user with limit
$user = Department::whereNotIn('id',$department)
->limit(1)->get();
hope it works for you..
You can do it like this:
Department::whereNotIn('id', User::pluck('department_id'))->get();
I believe below code will work for you :
Department::select('department.id as id')
->whereNotIn('id', User::whereNotNull('department_id')->pluck('department_id'))
->pluck('id');
From the answers others, there is a problem with the array if department_id is NULL. So, I added whereNotNull and also last() and then the problem is solved. Let me post the answer here:
Department::select('department.id as id')
->whereNotIn('id', User::whereNotNull('department_id')->pluck('department_id'))
->pluck('id')
->last(); // since I only need one row
Related
I have a large database table (~1 million records) which I need to purge the duplicate records. The table structure is as follows:
|----|-------------|-----|-----|--------------------|
| id | relation_id | foo | bar | timestamp |
|----|-------------|-----|-----|--------------------|
| 1 | 1 |14.20|0.22 |2019-10-21 14:00:01 |
| 2 | 1 |14.20|0.22 |2019-10-21 14:00:01 |
| 3 | 1 |14.20|0.22 |2019-10-21 14:00:01 |
| 4 | 2 |10.36|0.75 |2019-10-21 14:00:01 |
| 5 | 2 |10.36|0.75 |2019-10-21 14:00:01 |
| 6 | 2 |10.36|0.75 |2019-10-21 14:00:01 |
|----|-------------|-----|-----|--------------------|
As per the example above, there are a lot of records that have the exact same combination of values relation_id, foo, bar and timestamp. I need to create a script that will run to identify the unique values and then delete and duplicate references. So I would end up with something like:
|----|-------------|-----|-----|--------------------|
| id | relation_id | foo | bar | timestamp |
|----|-------------|-----|-----|--------------------|
| 1 | 1 |14.20|0.22 |2019-10-21 14:00:01 |
| 4 | 2 |10.36|0.75 |2019-10-21 14:00:01 |
|----|-------------|-----|-----|--------------------|
I have tested looping through the relation_id (as there are only 20 unique values) and then running something like this to create a collection of the unique records:
$unique = collect([]);
$collection = Model::where('relation_id', $relation_id)->chunk(100, function($items) use ($unique) {
$unique->push($items->unique()->values()->all());
});
From that, I had planned to loop through all of the Model records and delete if the item was not within the $unique collection. Something like this:
Model::chunk(100, function($items) {
foreach ($items as $item) {
if(!$unique->contains('id', $item->id)){
$item->delete;
}
}
});
My problem is as the database table is so large, I cannot test if this logic works. Running the first part of the above script (to populate $unique) for a single $relation_id ran in tinker for 30 minutes without yielding results.
I'm relatively confident this isn't the best approach to delete duplicate records as my approach requires multiple queries which I assume could be optimised (which is critical when dealing with such a large table).
So what is the most efficient way to query a database table to check for unique records (based on multiple columns) and delete the duplicate records?
You can allow the database to do the heaving lifting here. You can query the database using GROUP BY, then remove everything that doesn't match your query.
$ids = Model::groupBy(['relation_id', 'foo', 'bar', 'timestamp'])
->get(['id'])
->all();
This translates to the following SQL:
SELECT id FROM models GROUP BY relation_id, foo, bar, timestamp;
So now $ids is an array of IDs where the other columns are unique ([1, 4]). So you can execute the following to remove all other rows from the DB:
Model::whereNotIn('id', $ids)->delete();
However, since $ids is probably huge, you are likely to hit some upper limit constraints. In that case, you can try using array_chunk() to add multiple whereNotIn clauses to the query:
$query = Model::query();
foreach(array_chunk($ids, 500) as $chunk) {
$query->whereNotIn('id', $chunk);
}
$query->delete();
I created an SQL Fiddle where you can test this out.
For anyone experiencing a similar issue here, I opted to use MySQL to handle the tidy up of the database as it was far more efficient than loading it into memory using Eloquent.
This is the script I used:
DELETE table_name FROM table_name LEFT OUTER JOIN (
SELECT MIN(ID) AS minID FROM table_name GROUP BY table_name.relation_id, table_name.timestamp
) AS keepRowTable ON table_name.ID = keepRowTable.minID
WHERE keepRowTable.minID IS NULL
I accepted #Vince's answer because his approach works using Laravel but we ran into issues when trying to process such a large dataset. Plus he's a hero for being so responsive in the comments!
I have the following tables:
parts:
id | name
1 | Lock
2 | Light
shelves:
id | name
1 | 1A
2 | 1B
part_shelf
id | part_id | shelf_id | stock
1 | 1 | 1 | 5
2 | 1 | 2 | 10
3 | 2 | 1 | 4
The relations:
In Part model:
public function shelves()
{
return $this->belongsToMany( Shelf::class )->withPivot( 'stock' );
}
In shelf model:
public function parts()
{
return $this->belongsToMany(Part::class)->withPivot('stock');
}
What I would like to do is order all the parts based on the total stock of the pivot table.
What I have tried is the following:
$parts = Part::all(); // This is inside controller
#foreach($parts->shelves()->orderBy('pivot_stock', 'desc') as $part)
#foreach($parts->shelves()->orderBy('part_shelf.stock', 'desc') as $part)
I do know that I can get the total stock for individual parts with:
$part->shelves()->sum('stock')
I just don't know how to approach it for all parts in the foreach-loop.
Also looked at other questions but that didn't really help:
How to order by pivot table data in Laravel's Eloquent ORM
Order by pivot table created_at in Laravel
Laravel 5.4 Eloquent SUM and ORDER BY pivot table column I think is the closest to what I'm looking for
I had similar problem. I had three tables. Players (in my code "Igrac"), tournaments(in my code "Turnir_Piramida"), and players_tournaments pivot table(in my code "Nastup_piramida"). In pivot table was also column point for points for each player in each tournament. And i needed to summarize points from each player and then sort table by sum of points.
I manage to do that with third link you posted. You can see my solution on this link
I will have a table that is full of information that involves other tables (relations). Most of the information in this table will only have the ID's of the referencing related table. If I were to use "products" as an example for this table it may look like this for some of the columns:
id | name | type_id | price_id | location_id | sale_id
----------------------------------------------------------------
1 | prod1 | 1 | 1 | 2 | 4
2 | prod2 | 2 | 1 | 1 | 1
3 | prod3 | 3 | 2 | 6 | 2
4 | prod4 | 1 | 2 | 3 | 4
I'm trying to take this "products" table and dump it out into a list. I would need to look up all of the items in these columns as I dump it out (the relation). I know how to do belongsToMany and hasMany, but I'm not sure how I can do this in one shot with an Eloquent model if I have a "products" model? Should I just make the products table just a pivot table? Can I do it with an Eloquent model or should I use query builder directly? I think if I were to use withPivot it would return the extra columns but the raw ID value from the column. I would need the value lookup from their respective table (the relation).
Tried something like this:
public function productItems(){
return $this->belongsToMany(Product::class)->withPivot(["type_id","price_id",...]);
}
As suggested by #BagusTesa, you should eager load your relations:
$products = Product::with(['type', 'price', 'location'])->get();
That will query for the related models allowing you to access them as model properties:
foreach ($products as $product){
// $product->type;
// $product->price;
// $product->location;
}
Background - I'm creating a system where administrators can create arbitrary fields, which are then combined into a form. Users then complete this form, and the values input against each field are stored in a table. However, rather than overwrite the previous value, I plan on keeping each past value as individual rows in the table. I then want to be able to display the contents submitted in each form, but only the most recently submitted value.
Problem
I have a model, Service, that features a belongsToMany relationship with another model, Field. This relationship is defined as:
public function fields()
{
return $this->belongsToMany('App\Field')->withPivot('id', 'value', 'date')->withTimestamps();
}
The intermediary table has 3 values I wish to retrieve, id, value and date.
A Service may have 1 or more Fields, and for each field it may also have more than 1 pivot row. That is, a single Service/Field pairing may have multiple entries in the pivot table with different pivot values. For example:
Table field_service
id | service_id | field_id | value | created_at
------------------------------------------------------
1 | 1 | 1 | lorem | 2018-02-01
2 | 1 | 1 | ipsum | 2018-01-01
3 | 1 | 1 | dolor | 2017-12-01
4 | 1 | 2 | est | 2018-03-10
5 | 1 | 2 | sicum | 2018-03-09
6 | 1 | 2 | hoci | 2018-03-08
What I want is to get either:
A specific row from the pivot table for each Field associated with the Service, or
A specific value from the pivot table for each Field associated with the Service.
For example - in the table above, I would like the Service with ID 1 to have 2 Fields in the relationship, with each Field containing an attribute for the corresponding pivot value. The Fields attached would be specified by the corresponding pivot table entry having the most recent date. Something akin to:
$service->fields()[0]->value = "lorem"
$service->fields()[1]->value = "est"
I feel there's an obvious, 'Laravel'ly solution out there, but it eludes me...
Update
Somewhat unbelievably this is another case of me not understanding windowing functions. I asked a question 7 years ago that is basically this exactly problem, but with raw MySQL. The following raw MySQL basically gives me what I want, I just don't know how to Laravelise it:
SELECT services.name, fields.name, field_service.value, field_service.created_at, field_service.field_id
FROM field_service
INNER JOIN
(SELECT field_id, max(created_at) as ts
FROM field_service
WHERE service_id = X
GROUP BY field_id) maxt
ON (field_service.field_id = maxt.field_id and field_service.created_at = maxt.ts)
JOIN fields ON fields.id = field_service.field_id
JOIN services ON services.id = field_service.service_id
Try this:
public function fields()
{
$join = DB::table('field_service')
->select('field_id')->selectRaw('max(`created_at`) as `ts`')
->where('service_id', DB::raw($this->id))->groupBy('field_id');
$sql = '(' . $join->toSql() . ') `maxt`';
return $this->belongsToMany(Field::class)->withPivot('id', 'value', 'created_at')
->join(DB::raw($sql), function($join) {
$join->on('field_service.field_id', '=', 'maxt.field_id')
->on('field_service.created_at', '=', 'maxt.ts');
});
}
Then use it like this:
$service->fields[0]->pivot->value // "lorem"
$service->fields[1]->pivot->value // "est"
I am using the following LINQ query for my profile page:
var userData = from u in db.Users
.Include("UserSkills.Skill")
.Include("UserIdeas.IdeaThings")
.Include("UserInterests.Interest")
.Include("UserMessengers.Messenger")
.Include("UserFriends.User.UserSkills.Skill")
.Include("UserFriends1.User1.UserSkills.Skill")
.Include("UserFriends.User.UserIdeas")
.Include("UserFriends1.User1.UserIdeas")
where u.UserId == userId
select u;
It has a long object graph and uses many Includes. It is running perfect right now, but when the site has many users, will it impact performance much?
Should I do it in some other way?
A query with includes returns a single result set and the number of includes affect how big data set is transfered from the database server to the web server. Example:
Suppose we have an entity Customer (Id, Name, Address) and an entity Order (Id, CustomerId, Date). Now we want to query a customer with her orders:
var customer = context.Customers
.Include("Orders")
.SingleOrDefault(c => c.Id == 1);
The resulting data set will have the following structure:
Id | Name | Address | OrderId | CustomerId | Date
---------------------------------------------------
1 | A | XYZ | 1 | 1 | 1.1.
1 | A | XYZ | 2 | 1 | 2.1.
It means that Cutomers data are repeated for each Order. Now lets extend the example with another entities - 'OrderLine (Id, OrderId, ProductId, Quantity)andProduct (Id, Name)`. Now we want to query a customer with her orders, order lines and products:
var customer = context.Customers
.Include("Orders.OrderLines.Product")
.SingleOrDefault(c => c.Id == 1);
The resulting data set will have the following structure:
Id | Name | Address | OrderId | CustomerId | Date | OrderLineId | LOrderId | LProductId | Quantity | ProductId | ProductName
------------------------------------------------------------------------------------------------------------------------------
1 | A | XYZ | 1 | 1 | 1.1. | 1 | 1 | 1 | 5 | 1 | AA
1 | A | XYZ | 1 | 1 | 1.1. | 2 | 1 | 2 | 2 | 2 | BB
1 | A | XYZ | 2 | 1 | 2.1. | 3 | 2 | 1 | 4 | 1 | AA
1 | A | XYZ | 2 | 1 | 2.1. | 4 | 2 | 3 | 6 | 3 | CC
As you can see data become quite a lot duplicated. Generaly each include to a reference navigation propery (Product in the example) will add new columns and each include to a collection navigation property (Orders and OrderLines in the example) will add new columns and duplicate already created rows for each row in the included collection.
It means that your example can easily have hundreds of columns and thousands of rows which is a lot of data to transfer. The correct approach is creating performance tests and if the result will not satisfy your expectations, you can modify your query and load navigation properties separately by their own queries or by LoadProperty method.
Example of separate queries:
var customer = context.Customers
.Include("Orders")
.SingleOrDefault(c => c.Id == 1);
var orderLines = context.OrderLines
.Include("Product")
.Where(l => l.Order.Customer.Id == 1)
.ToList();
Example of LoadProperty:
var customer = context.Customers
.SingleOrDefault(c => c.Id == 1);
context.LoadProperty(customer, c => c.Orders);
Also you should always load only data you really need.
Edit: I just created proposal on Data UserVoice to support additional eager loading strategy where eager loaded data would be passed in additional result set (created by separate query within the same database roundtrip). If you find this improvement interesting don't forget to vote for the proposal.
(You can improve performance of many includes by creating 2 or more small data request from data base like below.
According to my experience,Only can give maximum 2 includes per query like below.More than that will give really bad performance.
var userData = from u in db.Users
.Include("UserSkills.Skill")
.Include("UserIdeas.IdeaThings")
.FirstOrDefault();
userData = from u in db.Users
.Include("UserFriends.User.UserSkills.Skill")
.Include("UserFriends1.User1.UserSkills.Skill")
.FirstOrDefault();
Above will bring small data set from database by using more travels to the database.
Yes it will. Avoid using Include if it expands multiple detail rows on a master table row.
I believe EF converts the query into one large join instead of several queries. Therefore, you'll end up duplicating your master table data over every row of the details table.
For example: Master -> Details. Say, master has 100 rows, Details has 5000 rows (50 for each master).
If you lazy-load the details, you return 100 rows (size: master) + 5000 rows (size: details).
If you use .Include("Details"), you return 5000 rows (size: master + details). Essentially, the master portion is duplicated over 50 times.
It multiplies upwards if you include multiple tables.
Check the SQL generated by EF.
I would recommend you to perform load tests and measure the performance of the site under stress. If you are performing complex queries on each request you may consider caching some results.
The result of include may change: it depend by the entity that call the include method.
Like the example proposed from Ladislav Mrnka, suppose that we have an entity
Customer (Id, Name, Address)
that map to this table:
Id | Name | Address
-----------------------
C1 | Paul | XYZ
and an entity Order (Id, CustomerId, Total)
that map to this table:
Id | CustomerId | Total
-----------------------
O1 | C1 | 10.00
O2 | C1 | 13.00
The relation is one Customer to many Orders
Esample 1: Customer => Orders
var customer = context.Customers
.Include("Orders")
.SingleOrDefault(c => c.Id == "C1");
Linq will be translated in a very complex sql query.
In this case the query will produce two record and the informations about the customer will be replicated.
Customer.Id | Customer.Name | Order.Id | Order.Total
-----------------------------------------------------------
C1 | Paul | O1 | 10.00
C1 | Paul | O2 | 13.00
Esample 2: Order => Customer
var order = context.Orders
.Include("Customers")
.SingleOrDefault(c => c.Id == "O1");
Linq will be translated in a simple sql Join.
In this case the query will produce only one record with no duplication of informations:
Order.Id | Order.Total | Customer.Id | Customer.Name
-----------------------------------------------------------
O1 | 10.00 | C1 | Paul