Laravel update chunked result skips rows - laravel

I'm trying to convert our database from ID to UUID. When I run the following code to update the database is skips random rows.
AppUser::select('id')->orderBy('created_at')->chunk(1000, function ($appUsers) {
foreach ($appUsers as $appUser) {
$uuid = Str::orderedUuid();
DB::table('files')->where('fileable_type', AppUserInfo::class)->where('fileable_id', $appUser->id)->update([
'fileable_id' => $uuid
]);
DB::table('app_users')->where('id', $appUser->id)->update(['id' => $uuid]);
}
});
Last time i checked ~290 were skipped out of 236196 total.
I've tried to used chunkById, but the same thing happened.
The update function is always returning true, so I must assume that Laravel thinks every row is updated when executed.

There's a big warning in the Laravel documentation on chunking:
When updating or deleting records inside the chunk callback, any changes to the primary key or foreign keys could affect the chunk query. This could potentially result in records not being included in the chunked results.
You'll need to find another way to update your keys in batches. I've used the technique described in an answer to this question: How to chunk results from a custom query in Laravel when I could not use the callback required by the chunk method, although in that case it was not for an update query, only a select.

This is what i ended up doing
$appUsers = AppUser::select('id')->get();
$chunkSize = 1000;
$numberOfChunks = ceil($appUsers->count() / $chunkSize);
$chunks = $appUsers->split($numberOfChunks);
foreach($chunks as $chunk) {
foreach($chunk as $appUser) {
$uuid = Str::orderedUuid();
DB::table('files')->where('fileable_type', AppUserInfo::class)->where('fileable_id', $appUser->id)->update([
'fileable_id' => $uuid
]);
DB::table('app_users')->where('id', $appUser->id)->update(['id' => $uuid]);
}
}

Related

laravel - update the model(!) without selecting first

I have the following ResearchModel (eloquent):
$research = new ResearchModel();
$research->id = 2;
$research->name = "test";
$research->save();
I am expecting Laravel to run an update statement (because I set the id), but it runs an insert statement instead.
$research->update(); wont do anything.
I don't want to use array in this case because I need the eloquent model events to be triggered.
I also don't want to run ResearchModel::find(2); before, this will cause significant performance problems in my use-case.
Is there any way to tell Laravel to update by the id?
Thanks
If you don't want to "select" first. Then you can update directly. This will run only one query.
$research = ResearchModel::where('id',2)->update([
'name' => 'test',
]);
A dirty solution:
// Create an instance without insert to database.
$instance = new ResearchModel;
$instance->id = 2;
$instance->name = "test";
// Set the instance status directly.
$instance->exists = true;
// Update database without model.
ResearchModel::where('id', $instance->id)
->update(['name' => $instance->name]);
// Manually trigger event.
event('eloquent.updating: App\ResearchModel', $instance);
The premise of this method is that you already have the required data. If the data required for the event must be read from the DB, then this method is useless.
Find by id
$research = ResearchModel::find(2);
Then you can run update function
$research->update([ 'name' => "test"]);
and if u don't want to use
ResearchModel::find(2);
then u have to use route binding
https://laravel.com/docs/6.x/routing#implicit-binding

Laravel - Collection with relations take a lot of time

We are developing an API with LUMEN.
Today we had a confused problem with getting the collection of our "TimeLog"-model.
We just wanted to get all time logs with additional informationen from the board model and task model.
In one row of time log we had a board_id and a task_id. It is a 1:1 relation on both.
This was our first code for getting the whole data. This took a lot of time and sometimes we got a timeout:
BillingController.php
public function byYear() {
$timeLog = TimeLog::get();
$resp = array();
foreach($timeLog->toArray() as $key => $value) {
if(($timeLog[$key]->board_id && $timeLog[$key]->task_id) > 0 ) {
array_push($resp, array(
'board_title' => isset($timeLog[$key]->board->title) ? $timeLog[$key]->board->title : null,
'task_title' => isset($timeLog[$key]->task->title) ? $timeLog[$key]->task->title : null,
'id' => $timeLog[$key]->id
));
}
}
return response()->json($resp);
}
The TimeLog.php where the relation has been made.
public function board()
{
return $this->belongsTo('App\Board', 'board_id', 'id');
}
public function task()
{
return $this->belongsTo('App\Task', 'task_id', 'id');
}
Our new way is like this:
BillingController.php
public function byYear() {
$timeLog = TimeLog::
join('oc_boards', 'oc_boards.id', '=', 'oc_time_logs.board_id')
->join('oc_tasks', 'oc_tasks.id', '=', 'oc_time_logs.task_id')
->join('oc_users', 'oc_users.id', '=', 'oc_time_logs.user_id')
->select('oc_boards.title AS board_title', 'oc_tasks.title AS task_title','oc_time_logs.id','oc_time_logs.time_used_sec','oc_users.id AS user_id')
->getQuery()
->get();
return response()->json($timeLog);
}
We deleted the relation in TimeLog.php, cause we don't need it anymore. Now we have a load time about 1 sec, which is fine!
There are about 20k entries in the time log table.
My questions are:
Why is the first method out of range (what causes the timeout?)
What does getQuery(); exactly do?
If you need more information just ask me.
--First Question--
One of the issues you might be facing is having all those huge amount of data in memory, i.e:
$timeLog = TimeLog::get();
This is already enormous. Then when you are trying to convert the collection to array:
There is a loop through the collection.
Using the $timeLog->toArray() while initializing the loop based on my understanding is not efficient (I might not be entirely correct about this though)
Thousands of queries are made to retrieve the related models
So what I would propose are five methods (one which saves you from hundreds of query), and the last which is efficient in returning the result as customized:
Since you have many data, then chunk the result ref: Laravel chunk so you have this instead:
$timeLog = TimeLog::chunk(1000, function($logs){
foreach ($logs as $log) {
// Do the stuff here
}
});
Other way is using cursor (runs only one query where the conditions match) the internal operation of cursor as understood is using Generators.
foreach (TimeLog::where([['board_id','>',0],['task_id', '>', 0]])->cursor() as $timelog) {
//do the other stuffs here
}
This looks like the first but instead you have already narrowed your query down to what you need:
TimeLog::where([['board_id','>',0],['task_id', '>', 0]])->get()
Eager Loading would already present the relationship you need on the fly but might lead to more data in memory too. So possibly the chunk method would make things more easier to manage (even though you eagerload related models)
TimeLog::with(['board','task'], function ($query) {
$query->where([['board_id','>',0],['task_id', '>', 0]]);
}])->get();
You can simply use Transformer
With transformer, you can load related model, in elegant, clean and more controlled methods even if the size is huge, and one greater benefit is you can transform the result without having to worry about how to loop round it
You can simply refer to this answer in order to perform a simple use of it. However incase you don't need to transform your response then you can take other options.
Although this might not entirely solve the problem, but because the main issues you face is based on memory management, so the above methods should be useful.
--Second question--
Based on Laravel API here You could see that:
It simply returns the underlying query builder instance. To my observation, it is not needed based on your example.
UPDATE
For question 1, since it seems you want to simply return the result as response, truthfully, its more efficient to paginate this result. Laravel offers pagination The easiest of which is SimplePaginate which is good. The only thing is that it makes some few more queries on the database, but keeps a check on the last index; I guess it uses cursor as well but not sure. I guess finally this might be more ideal, having:
return TimeLog::paginate(1000);
I have faced a similar problem. The main issue here is that Elloquent is really slow doing massive task cause it fetch all the results at the same time so the short answer would be to fetch it row by row using PDO fetch.
Short example:
$db = DB::connection()->getPdo();
$query_sql = TimeLog::join('oc_boards', 'oc_boards.id', '=', 'oc_time_logs.board_id')
->join('oc_tasks', 'oc_tasks.id', '=', 'oc_time_logs.task_id')
->join('oc_users', 'oc_users.id', '=', 'oc_time_logs.user_id')
->select('oc_boards.title AS board_title', 'oc_tasks.title AS task_title','oc_time_logs.id','oc_time_logs.time_used_sec','oc_users.id AS user_id')
->toSql();
$query = $db->prepare($query->sql);
$query->execute();
$logs = array();
while ($log = $query->fetch()) {
$log_filled = new TimeLog();
//fill your model and push it into an array to parse it to json in future
array_push($logs,$log_filled);
}
return response()->json($logs);

Update an array of objects in Laravel 5.4

I use Laravel as an API on an Angular app. One of the controllers has to update an array of objects.
This array, coming from Angular, might have
The same objects but with different attributes
One of the objects might have been deleted
New objects
So, I cannot just update the objects I have, since I need to delete the records that are not on the array any more and also create the new records.
At the moment, I have a not-so-nice solution to delete all the previous records and create new based on the array. Like this:
Sample::where('contest_id', $request->get('contest_id'))
->where('type', '0')
->delete();
$samples = $request->get('samples');
foreach ( $samples as $sample ) {
Sample::create($sample);
}
However, I want to add an activity logger to keep track of changes, but the above solution doesn't help. The activity logger works like this:
activity()
->causedBy($user)
->performedOn($sample)
->withProperties($properties)
->log('update'); //or new or delete
Where $properties is this:
$properties = [
'property' => [
'old' => $old_sample, // empty on creating new record
'new' => $sample // empty on deleting old record
],
];
Anything you could suggest?
Can't you have your frontend also send the sample_id when you need to update the records? If you're able to do that, you could play with Collections and make something like this:
// Retrieves all samples and turn them into a Collection
$samples = collect($request->get('samples'));
// Gets only the sample_ids that are not null and greater than zero
$changed_ids = $samples->pluck('sample_id')->filter()->all();
// These samples are new - create them
$new_samples = $samples->whereStrict('sample_id', null);
// These samples were changed - update them
$changed_samples = $samples->whereIn('sample_id', $changed_ids);
// These samples were deleted - remove them
$deleted_samples = Sample::whereNotIn('sample_id', $changed_ids);
Your solution seems like the most efficient, can you not retrieve the records to be deleted, log them as deleted and then actually delete them, then log the creation of the new ones?
In that scenario, your log is actually a better reflection of the activity that's occured.

Yii2- Not able to insert/delete rows using console controller

I am trying to implement email queue in my project which uses CRON.
I am adding a row in one table and deleting a row from another table using CreateCommand of yii2. Database is mysql.
The entire code is executed and it does not display any error. But the rows are not added or deleted from the database.
I have also tried doing the same thing using ActiveRecords but it does not work. Hence I have used CreateCommand.
Below is the code which I am using -
$connection = \Yii::$app->db;
$result = $connection->createCommand()->insert('email_queue_completed', $arrInsert)->execute();
if(!empty($result))
{
$connection->createCommand()
->delete('email_queue', ['id' => $mail->id])
->execute();
}
$arrInsert is an array containing key value pairs of values to be inserted.

CakePHP Pagination sort() on Related Models

I have two models: Plans and PlanDetails.
Relationship is: PlanDetails hasMany Plans. Plans belongTo PlanDetails.
In the PlanDetails view.ctp, I am pulling in related Plans.
I am trying to sort the Plans by ANY field (I've tried them all), and I cannot get it working. I assume I am overlooking something very simple.
Here is my base code:
PlanDetail >> view.ctp:
...foreach ($planDetail['Plan'] as $plan_edit) :
$class = null;
if ($i++ % 2 == 0) {
$class = ' class="altrow"';
}...
<?php echo $this->Paginator->sort('Plan ID', 'Plan.id'); ?>...
...<?php echo $plan_edit['id']; ?>
plan_details_controller.php:
...function view($id = null) {
if (!$id) {
$this->Session->setFlash(__('Invalid plan detail', true));
$this->redirect(array('action' => 'index'));
}
$this->PlanDetail->recursive = 2; // To run the editable form deeper.
$this->set('planDetail', $this->PlanDetail->read(null, $id));
$this->set('plan', $this->paginate('Plan'));
}...
I should add, no errors are being thrown and the sort() arrows on the ID field are showing as expected, but the sort order DOES not change when clicked either way.
Sorry, I'm not able to comment on the question itself, but I've noticed that in your action, you set planDetail to be the PlanDetail record you read (with recursive set to 2), and then you set plan to be the result of the paginate call.
Then, in your view template, you're iterating over $planDetail's contained Plan association, like this:
foreach ($planDetail['Plan'] as $plan_edit):
But in order to get the sorting and pagination done, you need to be displaying the results of the paginate call i.e. iterate over the records contained in $plan.
Do a debug($plan) in your view template to see what results you get there and to see if the records' ordering changes when you sort by different fields.
Also, perhaps you're using syntax I'm not aware of, but if you simply call $this->paginate('Plan') in your controller, I don't know that you're going to get only the related Plan records for your particular PlanDetail record. (There's nothing tying the $id passed into your view action with the Plan records.) You might need to add some conditions to the paginate call, like so:
$this->paginate['Plan'] = array('conditions' => array('Plan.plan_detail_id' => $id));
$this->set('plans', $this->paginate('Plan'));
Here is what I did to solve this. Based on some helpful direction from johnp & tokes.
plan_details/view.ctp:
...$i = 0;
foreach ($plan as $plan_edit) : // note $plan here.
}...
In my plan_details_controller.php view action:
$conditions = array("Plan.plan_detail_id" => "$id");
$this->set('plan', $this->paginate('Plan', $conditions)); // note "plan" in the first argument of set (this is required to pass the results back to the view). Also. The $condition is set to return ONLY plans that match the plan_detail_id of id (on the plan_details table).
And in my view, in order to get my results (because I changed the array name), I had to change the way I was getting the values to:
$plan_edit['Plan']['modified'] // note I placed ['Plan'] in front of these calls as the array called for this to get the data...
Well until the next problem! SOLVED.

Resources