I have potentially 500,000 records to either insert or update into my database.
I'm using the updateOrCreate function in Laravel but it's still taking a very long time.
I'm current using a foreach loop wrapped in a database transaction but is there a better solution?
DB::transaction(function() use ($items, $client) {
foreach($items as $item) {
$data = array(
'external_id' => $item->external_id,
'comment' => $item->comment,
'code_id' => $item->code_id,
'client_id' => $client->id
);
Registration::updateOrCreate(
[
'user_id' => $item->user_id,
'date' => Carbon::parse($item->date)->format('Y-m-d'),
'session' => $item->session
],
$data
);
}
});
Well since you have so many records its inevitable for it to take a long time, my suggestion is to chunk the data you are getting like so
foreach($items->chunk(1000) as $chunk) {
foreach($chunk as $item) {
...
}
}
The above method will go over 1000 (or as many as you want) items at a time, and should theoretically decrease the load time by a bit. But still I really don't think you can make it a lot faster.
I think you should here use ShouldQueue approach of Laravel
and instead of updateOrCreate method use Query builder to update single row using where('id',$id) this will ma
Related
I have this simple eloquent query, works fine, but with few records.
When database increments until 100000 records become very slow.
I read should be use chunk instead of get. How can I implement it for this query?
$collection = Contact::with('shop');
$collection = $collection->orderBy('created_at', 'DESC');
$collection = $collection->get();
$json = $collection->map(function ($contact) {
return [
'id' => $contact->id,
'name' => $contact->name,
...about 50 columns more.
'shop' => [
'id' => optional($contact->shop)->id,
'name' => optional($contact->shop)>name
],
...about 6 relations more.
];
});
$json = $json->paginate(50);
return response()->json(['contacts' => $json], 200);
You are converting getting all the data like 1M or how many records it has. Then you are mapping it and paginate it and getting only 50. There is huge performance problem with your code.
You can directly call like this:
return response()->json(['contacts' => Contact::with('shop')->orderBy('created_at', 'DESC')->paginate(50)], 200);
If you only need id and name for contacts:
return response()->json(['contacts' => Contact::select('id', 'name', 'created_at')->orderBy('created_at', 'DESC')->paginate(50)], 200);
I am trying to increment a column and updateorcreate a record in one query by doing so:
$model = Model::updateOrCreate([
'email' => $email
], ['received_at' => $received_at])->incremet('received_count');
This does the job as I wanted it too. It updates or create a record and increments the received_count column.
But after the query, I wanted to get the updated/created row, but when I log $model, it only logs 0 or 1. I can confirm that this is because of the ->increment().
But to be honest, I don't know any way how to increment the received_count column other than how I currently did it.
How do I achieve so that it updates or create a record, at the same time increments the received_count column, and after all of this, returns the updated/created object?
As much as posssible, I want this all in one query. Getting the model should be a memory.
$model = Model::updateOrCreate(['email' => $email], ['received_at' => $received_at]);
$model->increment('received_count');
$model->refresh(); // this will refresh all information for your model.
or you can simply:
$model = Model::updateOrCreate(['email' => $email], ['received_at' => $received_at]);
tap($model)->increment('id'); // this will return refreshed model.
just fresh model after incremet:
$model = Model::updateOrCreate([
'email' => $email
], ['received_at' => $received_at]);
$model->incremet('received_count');
$model->fresh();
hi i wrote this code and it works just fine but i think its not the best way to do it!
i want to get all the jobs for 1 company.
each company can have many addresses and each address can have many jobs
here is my code:
$company = Company::find($id)->with('addresses.jobDetails.job')->first();
$jobs = [];
foreach ($company->addresses as $address) {
foreach ($address->jobDetails as $detail) {
array_push($jobs, [
'id' => $detail->job->id,
'title' => $detail->job->title,
'country' => $detail->job->country,
'city' => $detail->job->city,
'type' => $detail->job->type,
'work_types' => JobType::where('job_id',$detail->job->id)->pluck('title'),
'income' => $detail->income,
]);
}
}
return $jobs;
can anyone help me to change this to better code please
thank you in advance
You do the opposite and start with JobDetails
$jobDetails = JobDetail::whereHas('address.company', function($companyQuery) use($id) {
$companyQuery->where('id', $id);
})->whereHas('jobs', function($jobQuery) {
$jobQuery->where('is_active', 1);
})->with('jobs')->get();
foreach ($jobDetails as $detail) {
array_push($jobs, [
'id' => $detail->job->id,
'title' => $detail->job->title,
'country' => $detail->job->country,
'city' => $detail->job->city,
'type' => $detail->job->type,
'work_types' => JobType::where('job_id',$detail->job->id)->pluck('title'),
'income' => $detail->income,
]);
}
return $jobs;
EDIT:
In your query
Company::find($id)->with('addresses.jobDetails.job')->first();
You run 4 queries with eager loading. one for each model. You can check in the result that you got that all the data is present in the variable $company.
The example I gave you it runs only two queries, the first one (job_details) will use joins to filter the Job results by the id of the companies table (you can make it faster by using the field company_id in the addresses table)
The second one is for the jobs relation using eager loading.
I'm executing a query like this as well as some others and returning them via response()->json().
$transactions = Transaction::where('created_at', '>=',
now()->firstOfYear())->get()->groupBy(function ($transaction)
{
return Carbon::parse($transaction->created_at)->format('M');
});
return response()->json([
'user' => $user->toArray(),
'transactions' => $transactions->toArray()
]);
However, while transactions is an Array in php, when it goes through response()->json it gets turned into an Object. I was hoping someone could tell me how I can prevent this and keep it as an array so I can iterate over it?
Thanks.
Picture of transactions output as requested. (Had to blur a lot of stuff due to sensitive info.)
Your array is keyed with month names, meaning it is an associative array. If you want the JSON to be an array, you will need your PHP array to be indexed numerically.
One option you can do is this (untested):
$userArray = [];
foreach ($user as $key => $value) {
$userArray[] = (object) [
'month' => $key,
'data' => $value,
];
}
return response()->json([
'user' => $userArray,
'transactions' => $transactions->toArray()
]);
That will make it a numerically indexed array of objects with the month being a property on the object and another property containing the rest of the data.
A solution I used before is to check on the frontend whether the data is array or an object, and if it is an object just convert it to array. Associative arrays will get converted to objects in javascript unless its keys start from 0 and increment like a normal arrays index.
An example of doing this:
window.axios.post('api/endpoint', data)
.then(res => {
const transactions = Array.isArray(res.data.transactions)
? res.data.transactions
: Object.keys.(res.data.transactions).map(key => response.data.transactions[key]);
})
$this->assertDatabaseHas() not working with JSON/JSONb columns.
So how can I tests these types of columns in Laravel?
Currently, I have a store action. How can I perform an assertion, that a specific column with pre-defined values was saved.
Something like
['options->language', 'en']
is NOT an option, cause I have an extensive JSON with meta stuff.
How can I check the JSON in DB at once?
UPD
Now can be done like that.
I have solved it with this one-liner (adjust it to your models/fields)
$this->assertEquals($store->settings, Store::find($store->id)->settings);
Laravel 7+
Not sure how far back this solution works.
I found out the solution. Ignore some of the data label, Everything is accessible, i was just play around with my tests to figure it out.
/**
* #test
*/
public function canUpdate()
{
$authUser = UserFactory::createDefault();
$this->actingAs($authUser);
$generator = GeneratorFactory::createDefault();
$request = [
'json_field_one' => [
'array-data',
['more-data' => 'cool'],
'data' => 'some-data',
'collection' => [
['key' => 'value'],
'data' => 'some-more-data'
],
],
'json_field_two' => [],
];
$response = $this->putJson("/api/generators/{$generator->id}", $request);
$response->assertOk();
$this->assertDatabaseHas('generators', [
'id' => $generator->id,
'generator_set_id' => $generator->generatorSet->id,
// Testing for json requires arrows for accessing the data
// For Collection data, you should use numbers to access the indexes
// Note: Mysql dose not guarantee array order if i recall. Dont quote me on that but i'm pretty sure i read that somewhere. But for testing this works
'json_field_one->0' => 'array-data',
'json_field_one->1->more-data' => 'cool',
// to access properties just arrow over to the property name
'json_field_one->data' => 'some-data',
'json_field_one->collection->data' => 'some-more-data',
// Nested Collection
'json_field_one->collection->0->key' => 'value',
// Janky way to test for empty array
// Not really testing for empty
// only that the 0 index is not set
'json_field_two->0' => null,
]);
}
Note: The below solution is tested on Laravel Version: 9.x and Postgres version: 12.x
and the solution might not work on lower version of laravel
There would be two condition to assert json column into database.
1. Object
Consider Object is in json column in database as shown below:
"properties" => "{"attributes":{"id":1}}"
It can assert as
$this->assertDatabaseHas("table_name",[
"properties->attributes->id"=>1
]);
2. Array
Consider array is in json column as shown below:
"properties" => "[{"id":1},{"id":2}]"
It can assert as
$this->assertDatabaseHas("table_name",[
"properties->0->id"=>1,
"properties->1->id"=>2,
]);
Using json_encode on the value worked for me:
$this->assertDatabaseHas('users', [
'name' => 'Gaurav',
'attributes' => json_encode([
'gender' => 'Male',
'nationality' => 'Indian',
]),
]);