Import large csv file with laravel excel and postgresql - laravel

Question

Hi I have a large csv file with 130.000 rows
I use laravel excel 3.1 and lavaravel 5.8
Import class:
<?php
namespace App\Imports;
use App\UsoSuelo;
use Maatwebsite\Excel\Concerns\ToModel;
use Maatwebsite\Excel\Concerns\WithHeadingRow;
use Maatwebsite\Excel\Concerns\WithChunkReading;
use Maatwebsite\Excel\Concerns\WithBatchInserts;
class UsoSueloImport implements ToModel, WithHeadingRow, WithChunkReading, WithBatchInserts
{
/**
* #param array $row
*
* #return \Illuminate\Database\Eloquent\Model|null
*/
public function model(array $row)
{
return new UsoSuelo([
'cod_pais' => $row['cod_pais'],
'cod_fundo' => $row['cod_fundo'],
'nom_fundo' => $row['nom_fundo'],
]);
}
public function batchSize(): int
{
return 1000;
}
public function chunkSize(): int
{
return 1000;
}
}
And I use a trait class from my controller:
trait storeTrait{
public function storeUsoSuelo($archivo) {
Excel::import(new UsoSueloImport,$archivo);
}
public function storeFundo($archivo) {
Excel::import(new FundosImport,$archivo);
}
public function storeFundoGrilla($archivo) {
Excel::import(new FundosGrillasImport,$archivo);
}
public function storeCuadrante($archivo) {
Excel::import(new CuadrantesImport,$archivo);
}
}
This is my ImportController
class ImportController extends Controller
{
use storeTrait {
storeUsoSuelo as storeUsoSuelos;
storeFundo as storeFundos;
storeFundoGrilla as storeFundoGrillas;
storeCuadrante as storeCuadrantes;
}
public function store(Request $request)
{
$usoSuelo = 'uso_suelo.csv';
$this->storeUsoSuelos($usoSuelo);
$cuadrante = 'cuadrantes.csv';
$this->storeCuadrantes($cuadrante);
$fundo = 'mv_qav_fundos.csv';
$this->storeFundos($fundo);
$fundoGrilla = 'fundos_grilla.csv';
$this->storeFundoGrillas($fundoGrilla);
}
}
I have done tests and my code works with a csv of less than 100 rows
but when I try with the 130,000 rows it takes too long, and I end up getting the following error:
"Maximum execution time of 60 seconds exceeded"
And after 1 minute only 4000 rows have been inserted in the database (postgresql)

Answer 1

I put these 2 lines in my controller, at the beginning of the script:
ini_set ('max_execution_time', 3600);
ini_set ('memory_limit', '2048M');
With this I solved it, I also changed the chunk from 1000 to 5000, but it still takes too long, at least 5 minutes

Answer 2

Have you tried this package league/csv from The League of Extraordinary Packages.
If you only need support for CSV that is, and not xlsx or other excel types.
From what I read this package might give you a better performance, this comment from one of the Maatwebsite team members (the package you are currently using) also confirms this.