How to insert big data on the laravel? - laravel

Question

I am using laravel 5.6
My script to insert big data is like this :
...
$insert_data = [];
foreach ($json['value'] as $value) {
$posting_date = Carbon::parse($value['Posting_Date']);
$posting_date = $posting_date->format('Y-m-d');
$data = [
'item_no' => $value['Item_No'],
'entry_no' => $value['Entry_No'],
'document_no' => $value['Document_No'],
'posting_date' => $posting_date,
....
];
$insert_data[] = $data;
}
\DB::table('items_details')->insert($insert_data);
I have tried to insert 100 record with the script, it works. It successfully insert data
But if I try to insert 50000 record with the script, it becomes very slow. I've waited about 10 minutes and it did not work. There exist error like this :
504 Gateway Time-out
How can I solve this problem?

Answer 1

As it was stated, chunks won't really help you in this case if it is a time execution problem. I think that bulk insert you are trying to use cannot handle that amount of data , so I see 2 options:
1 - Reorganise your code to properly use chunks, this will look something like this:
$insert_data = [];
foreach ($json['value'] as $value) {
$posting_date = Carbon::parse($value['Posting_Date']);
$posting_date = $posting_date->format('Y-m-d');
$data = [
'item_no' => $value['Item_No'],
'entry_no' => $value['Entry_No'],
'document_no' => $value['Document_No'],
'posting_date' => $posting_date,
....
];
$insert_data[] = $data;
}
$insert_data = collect($insert_data); // Make a collection to use the chunk method
// it will chunk the dataset in smaller collections containing 500 values each.
// Play with the value to get best result
$chunks = $insert_data->chunk(500);
foreach ($chunks as $chunk)
{
\DB::table('items_details')->insert($chunk->toArray());
}
This way your bulk insert will contain less data, and be able to process it in a rather quick way.
2 - In case your host supports runtime overloads, you can add a directive right before the code starts to execute :
ini_set('max_execution_time', 120 ) ; // time in seconds
$insert_data = [];
foreach ($json['value'] as $value)
{
...
}
To read more go to the official docs

Answer 2

It makes no sense to use an array and then convert it to a collection.
We can get rid of arrays.
$insert_data = collect();
foreach ($json['value'] as $value) {
$posting_date = Carbon::parse($value['Posting_Date']);
$posting_date = $posting_date->format('Y-m-d');
$insert_data->push([
'item_no' => $value['Item_No'],
'entry_no' => $value['Entry_No'],
'document_no' => $value['Document_No'],
'posting_date' => $posting_date,
....
]);
}
foreach ($insert_data->chunk(500) as $chunk)
{
\DB::table('items_details')->insert($chunk->toArray());
}

Answer 3

Here is very good and very Fast Insert data solution
$no_of_data = 1000000;
$test_data = array();
for ($i = 0; $i < $no_of_data; $i++){
$test_data[$i]['number'] = "1234567890";
$test_data[$i]['message'] = "Test Data";
$test_data[$i]['status'] = "Delivered";
}
$chunk_data = array_chunk($test_data, 1000);
if (isset($chunk_data) && !empty($chunk_data)) {
foreach ($chunk_data as $chunk_data_val) {
DB::table('messages')->insert($chunk_data_val);
}
}

Answer 4

I used the code below to check the update or insert data of 11 thousand rows. I hope it useful for you.
$insert_data = [];
for ($i=0; $i < 11000; $i++) {
$data = [
'id' =>'user_'.$i,
'fullname' => 'Pixs Nguyen',
'username' => 'abc#gmail.com',
'timestamp' => '2020-03-23 08:12:00',
];
$insert_data[] = $data;
}
$insert_data = collect($insert_data); // Make a collection to use the chunk method
// it will chunk the dataset in smaller collections containing 500 values each.
// Play with the value to get best result
$accounts = $insert_data->chunk(500);
// In the case of updating or inserting you will take about 35 seconds to execute the code below
for ($i=0; $i < count($accounts); $i++) {
foreach ($accounts[$i] as $key => $account)
{
DB::table('yourTable')->updateOrInsert(['id'=>$account['id']],$account);
}
}
// In the case of inserting you only take about 0.35 seconds to execute the code below
foreach ($accounts as $key => $account)
{
DB::table('yourTable')->insert($account->toArray());
}