Use Laravel seed and sql files to populate database - laravel-4

I got several .sql files of countries, states and cities of the world from github. How can I run them with Laravel's seed files to populate those tables in my database?

Add DB::unprepared() to the run method of DatabaseSeeder.
Run php artisan db:seed at the command line.
class DatabaseSeeder extends Seeder {
public function run()
{
Eloquent::unguard();
$this->call('UserTableSeeder');
$this->command->info('User table seeded!');
$path = 'app/developer_docs/countries.sql';
DB::unprepared(file_get_contents($path));
$this->command->info('Country table seeded!');
}
}

I found a package that creates seed files from database tables and rows. It currently supports Laravel 4, 5, 6, 7, 8 and 9:
https://github.com/orangehill/iseed
In the end, it's basically as easy as this:
php artisan iseed my_table
or for multiple occasions:
php artisan iseed my_table,another_table

As used by other answers, DB::unprepared does not work with more complex SQL files.
Another better solution would be to use the MySQL cli directly inside a process:
$process = new Process([
'mysql',
'-h',
DB::getConfig('host'),
'-u',
DB::getConfig('username'),
'-p' . DB::getConfig('password'),
DB::getConfig('database'),
'-e',
"source path/to/schema.sql"
]);
$process->mustRun();

2022 Simplified answer from Andrew Koper :
class WhateverSeeder extends Seeder
{
/**
* Run the database seeds.
*
* #return void
*/
public function run()
{
$file_path = resource_path('sql/whatever.sql');
\DB::unprepared(
file_get_contents($file_path)
);
}
}
Fun fact: 60,000 rows took me 50s to import from JSON file where this is 400ms.

#Andre Koper solutions is understandable, but sadly it doesn't work for me.
This one is a bit confusing but atleast works for me.
So instead of using DB::unprepared, I use this:
// DatabaseSeeder.php
class DatabaseSeeder extends Seeder {
public function run()
{
// Set the path of your .sql file
$sql = storage_path('a_id_territory.sql');
// You must change this one, its depend on your mysql bin.
$db_bin = "C:\wamp64\bin\mariadb\mariadb10.3.14\bin";
// PDO Credentials
$db = [
'username' => env('DB_USERNAME'),
'password' => env('DB_PASSWORD'),
'host' => env('DB_HOST'),
'database' => env('DB_DATABASE')
];
exec("{$db_bin}\mysql --user={$db['username']} --password={$db['password']} --host={$db['host']} --database {$db['database']} < $sql");
}
}
Then while migrating database just add --seed
php artisan migrate:refresh --seed
or
php artisan migrate:fresh --seed
Tested on Laravel 7.0.x

Related

Testing Laravel Cached Routes

I ran into an issue where Laravel's cached routes were not the same as uncached, so I am attempting to write a unit test which compares the two.
/** #test */
public function cached_and_uncached_are_identical()
{
Artisan::call('route:clear');
Artisan::call('route:list', ['--compact' => true]);
$uncached = Artisan::output();
Artisan::call('route:cache');
Artisan::call('route:list', ['--compact' => true]);
$cached = Artisan::output();
$this->assertSame($cached, $uncached);
}
The call to route:list after route:cache throws a LogicException: "Route is not bound."
I don't get that error when executing those commands on the command line. My searches haven't turned up any good hints for tracking down the issue.
How do I find the issue and fix it?

Laravel migration package publishing order

I'm building a Laravel Package.
I need to publish my custom migrations. One migration need another one to be run because it has a foreign key constraint.
In my service provider I'm doing this:
public function boot(Filesystem $filesystem)
{
$this->loadMigrationsFrom(dirname(__DIR__).'/migrations');
$this->publishMigrations($filesystem);
}
private function publishMigrations(Filesystem $filesystem)
{
if (! class_exists('CreateReferencedTable')) {
$this->publishes([
__DIR__ . '/../database/migrations/create_referenced_table.php.stub' => $this->getMigrationFileName($filesystem, 'referenced')
], 'migrations');
}
if (! class_exists('CreateExampleTable')) {
$this->publishes([
__DIR__ . '/../database/migrations/create_example_table.php.stub' => $this->getMigrationFileName($filesystem, 'example')
], 'migrations');
}
}
protected function getMigrationFileName(Filesystem $filesystem, string $table_name): string
{
$timestamp = date('Y_m_d_His');
return Collection::make($this->app->databasePath().DIRECTORY_SEPARATOR.'migrations'.DIRECTORY_SEPARATOR)
->flatMap(function ($path, $table_name) use ($filesystem) {
return $filesystem->glob($path.'*_create_'.$table_name.'_table.php');
})->push($this->app->databasePath()."/migrations/{$timestamp}_create_{$table_name}_table.php")
->first();
}
And then I run
php artisan vendor:publish --provider="FedericoArona\MyPackage\Providers\MyServiceProvider" --tag="migrations"
The problem is that the tables are being created with the exact same timestamp so they will be ordered by their name. This means that "Example" migration will be created before "Referenced" migration.
Obviously this means that when I run my migrations, they will fail because "Example" migration need to be run after "Referenced".
How can I solve? Thanks.
The only approach I've made work is a hack: adjust the timestamps. I put the number of migrations into a property in the provider
$this->migrationCount
and then decrement the timestamp (so if the tick counter increments in the middle of execution it still is different):
private function migratePath(string $file): string
{
$timeKludge = date('Y_m_d_His', time() - --$this->migrationCount);
return database_path(
'migrations/' . $timeKludge . "_$file.php"
);
}
It's not pretty but it works.

Laravel 5.5 Queues & Jobs : job is never executed

I'm fairly new to Lumen (Laravel), and I'm currently digging the concept of Queues, Jobs & Scheduled Tasks.
I would like to store in a Queue some Eloquent models, when I receive them from API calls. Then, I want to create a scheduled task which runs a daily Job, to fetch all those models in Queue and then send a report email.
I went the database storing way for Queues.
So I added the QUEUE_DRIVER=database key in my .env file and my queue.php config file looks like this :
<?php
return [
'database' => [
'connection' => 'my_db_connection',
'driver' => 'database',
'table' => 'jobs',
'queue' => 'default',
'retry_after' => 90
]
];
I ran the php artisan queue:table command to create the migration file and ran migration to create my table. I have a table jobs, with the right intended fields.
I created a class, implementing ShouldQueue :
class ProcessDeliveryOrders implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
protected $deliveryOrder;
public function __construct($deliveryOrder)
{
$this->deliveryOrder = $deliveryOrder;
}
public function handle()
{
// $this->deliveryOrder->save(); ?
}
}
As sub-question here is about how this class is working : Can it stores automatically the Eloquent model when it receives it in it's constructor ? to me the handle method is called when the object is retrieved by the Job, but I can be wrong and it's used to store the object ?
Then, I created a MailingJob : it's goal is to aggregate all the ProcessDeliveryOrders stored daily, and create an email from them :
class SendDeliveryEmailJob extends Job
{
public function __construct()
{
//
}
public function handle()
{
// DeliveryOrder is my Eloquent Model
$deliveryOrders = DeliveryOrder::query()
->whereBetween('createdAt', [strtotime('today midnight'), strtotime('today midnight')])
->get();
$mail = Mail::to(config('admin.emails'));
$mail->send(
new DeliveryOrderReportMailTemplate([
'deliveryOrders' => $deliveryOrders
])
);
}
}
And in my Kernel.php, I have added the following line in function schedule() :
// everyMinute for test purposes only
$schedule->job(new SendDeliveryEmailJob())->everyMinute();
As I'm using Lumen, I don't have the exact same process as in pure Laravel, so when I receive a delivery order to call my dispatch, I tried both ways exposed by the Lumen doc (forced to create an instance of my Job) :
dispatch(new ProcessDeliveryOrders($deliveryOrder));
// OR
Queue::push(new ProcessDeliveryOrders($deliveryOrder));
After all this setup, I tried few commands like php artisan queue:listen or php artisan queue:work, and the command line seems to be stuck.
If I run php artisan queue:listen database I get the following looped error :
In QueueManager.php line 172:
No connector for []
I checked documentation twice, and tried the new key QUEUE_CONNECTION=database instead of QUEUE_DRIVER but it's apparently only since 5.7, and didn't worked either. Any chance you spot something I'm missing ? Thanks a lot
EDIT: When I put a logging in the SendDeliveryEmailJob constructor, and I run php artisan queue:listen, I see the echo output every 2 or 3 seconds. I've also put a log into the handle function but I never see this one called.
EDIT 2: I noticed that when I try to execute my scheduled tasks with php artisan scheduled:run, it throws an error :
Running scheduled command: App\Jobs\SendDeliveryEmailJob
In Schedule.php line 87:
Call to a member function onQueue() on null
I guess from this message, that my Job is not instanciated, but I see the constructor message displayed to it's kinda weird..
#alex QUEUE_CONNECTION=database put this in your env and your queue.php is
'sync' => [
'driver' => 'sync',
],
'database' => [
'driver' => 'database',
'table' => 'jobs',
'queue' => 'default',
'retry_after' => 90,
],

Laravel4 seed specific environment

We're are developing multiple applications based on Laravel 4. These applications run on the same webserver.
The Laravel4 environment detection is based on the hostname which sucks because we have multiple applications on the same machine.
We created a work-around in the detection area so that it will set the environment based on the url.
We run the artisan --env=my_env migrate command when we update the applications DB. The problem is in the seeding, the seeding command doesn't have a env option so it will try to seed the db based on the hostname wich will not be correct.
I'm trying all day to find a solution but I can't find any on the Internet and my attempts to build a new command is just taking too much time and energy.
Does someone knows how to set the environment when seeding?
PS: I run the commands on the server through Grunt and I know the environment -inject it into the command-.
You pointed it very well, Laravel environment guessing sucks the way we use to use it, but you can change that:
This is how I do set my environment flawlessly, so I don't have to deal with hostnames and still don't get my local environment conflict with staging and production.
Create a .environment file in the root of your application and define your environment and add your sensitive information to it:
<?php
return array(
'APPLICATION_ENV' => 'development', /// this is where you will set your environment
'DB_HOST' => 'localhost',
'DB_DATABASE_NAME' => 'laraveldatabase',
'DB_DATABASE_USER' => 'laraveluser',
'DB_DATABASE_PASSWORD' => '!Bassw0rT',
);
Add it to your .gitignore file, so you don't risk having your passwords sent to Github or any other of your servers.
Right before $app->detectEnvironment, in the file bootstrap/start.php, load your .environment file to PHP environment:
foreach(require __DIR__.'/../.environment' as $key => $value)
{
putenv(sprintf('%s=%s', $key, $value));
}
And then you just have to use it:
$env = $app->detectEnvironment(function () {
return getenv('APPLICATION_ENV'); // your environment name is in that file!
});
And it will work everywhere, so you don't need to have separate dirs for development and production anymore:
<?php
return array(
'connections' => array(
'postgresql' => array(
'driver' => 'pgsql',
'host' => getenv('DB_HOST'),
'database' => getenv('DB_DATABASE_NAME'),
'username' => getenv('DB_DATABASE_USER'),
'password' => getenv('DB_DATABASE_PASSWORD'),
'charset' => 'utf8',
'prefix' => '',
'schema' => 'public',
),
),
);
Note that I don't set a fallback:
return getenv('APPLICATION_ENV') ?: 'local';
Because I want it to fail on every server I deploy my app to, to never forget configuring my environment on them.
Then you just have to select the environment in your DatabaseSeeder class:
public function run()
{
if( App::environment() === 'development' )
{
$this->call('UserTableSeeder');
}
}
I thought about the situation some days and got to the conclusion that what we're trying to do isn't the correct way.
The correct way would be that every application has his own config files -with the different envs-. This way the resolve function of Laravel works fine.
The situation now is that we have multiple clients within one application and strore does -clients- configuration files within one application. In this case the hostname resolve will return the one client's -every time the same client- config beacuse the clients applications run on the same machine.
Our solution
We are going to write a deployment script for the different clients so that every client has his own application with their configs only (copy application, copy/overwrite client config into app).
Work-around
The answer of #Antonio Carlos Ribeiro works offcourse but had to much impact on our application. We use the different environments and with this solution we had to use the same user/pass info on all environments or provide a different .environment file.
I wrote an Artisan command to make our deployment work for the moment. This command can seed a database with the configuration of the provided environment (php artisan db:seed_env my_env).
use Illuminate\Console\Command;
use Illuminate\Config\Repository;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Input\InputArgument;
class SeedEnvironmentDb extends Command {
/**
* The console command name.
*
* #var string
*/
protected $name = 'db:seed_env';
/**
* The console command description.
*
* #var string
*/
protected $description = 'Seed a database with the configuration of the environment';
/**
* Create a new command instance.
*
* #return void
*/
public function __construct()
{
parent::__construct();
}
/**
* Execute the console command.
*
* #return mixed
*/
public function fire()
{
$cmd = $this;
$app = App::make('app');
// force the environment to the given one.
$env = $app->detectEnvironment(function() use ($cmd) {
return $cmd->argument('environment');
});
// create new config with the correct environment and overwrite the current one.
$app->instance('config', $config = new Repository(
$app->getConfigLoader(), $env
));
// trigger the db seed (now with the correct environment)
$this->call('db:seed');
}
/**
* Get the console command arguments.
*
* #return array
*/
protected function getArguments()
{
return array(
array('environment', InputArgument::REQUIRED, 'The environment to seed.'),
);
}
/**
* Get the console command options.
*
* #return array
*/
protected function getOptions()
{
return array();
}
}

Can I import a mysql dump to a laravel migration?

I have a complete database and need to create migration. I guess there must be a way to do it from a dump but not sure. Is there any way automatically or at least easier to do this task?
You can import dumps in Laravel like this:
DB::unprepared(file_get_contents('full/path/to/dump.sql'));
If I were to refactor an existing app, though, I'd take the time to write migrations from scratch, import the dump into different tables (or a different db, if table names are the same) then import the content to the new structure via seeds.
Laravel can't do that, but I think this will help: Laravel migration generator
It generate migrations based on existing tables.
This question is answered already, but recently in a project, the provided answers did not satisfy my needs any longer. It also does not import a whole database dump, but one (large) table. I felt I should share that with you.
The problem was, I wanted to import a quite large table (list of zipcodes) during my artisan:migrate operation. The solution with DB::unprepared($dump) took way to long and I found an alternative which is MUCH faster.
Just export your table as CSV and use the following Code in your migration's up() function.
// i had to str_replace the backslash on windows dev system... but works on linux, too
$filename = str_replace("\\", "/", storage_path('path/in/storage/to/your/file.csv'));
$query = "LOAD DATA LOCAL INFILE '".$filename."' INTO TABLE yourtable
FIELDS TERMINATED BY '\t'
ENCLOSED BY ''
LINES TERMINATED BY '\n'
IGNORE 0 LINES
(col1,col2,...);";
DB::unprepared($query);
Just update the query as you need. And of course, you should make sure, that the table with the cols
'col1', 'col2' etc... exists. I created it just before the importing of the file. with Schema::create()...
If you run into following error message:
PDO::exec(): LOAD DATA LOCAL INFILE forbidden
There is a way you can get rid of this message: Although it's not really documented you can just add an 'options' key to your config/database.php file. For example mine looks like that:
'mysql' => [
'driver' => 'mysql',
'host' => env('DB_HOST', 'localhost'),
'database' => env('DB_DATABASE', 'forge'),
'username' => env('DB_USERNAME', 'forge'),
'password' => env('DB_PASSWORD', ''),
'charset' => 'utf8',
'collation' => 'utf8_unicode_ci',
'prefix' => '',
'strict' => false,
'options' => array(
PDO::MYSQL_ATTR_LOCAL_INFILE => true,
)
Note: i'm currently using laravel 5 but it should work with laravel 4, too.
I have a complete database and need to create migration. I guess there must be a way to do it from a dump but not sure. Is there any way automatically or at least easier to do this task?
Not automatically, but we run dumps in a migration using DB::unprepared(). You could use file_get_contents to import from a .sql file and thus not have to worry about escaping the entire dump's " marks...
<?php
use Illuminate\Database\Migrations\Migration;
class ImportDump extends Migration {
/**
* Run the migrations.
*
* #return void
*/
public function up()
{
DB::unprepared("YOUR SQL DUMP HERE");
}
/**
* Reverse the migrations.
*
* #return void
*/
public function down()
{
}
}
Another alternative is using the PDO directly:
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\File;
$sql_dump = File::get('/path/to/file.sql');
DB::connection()->getPdo()->exec($sql_dump);
I am writing my answer as this might help to someone who is using new laravel 8.
In laravel 8, dumping SQL and run migration using SQL(.dump) file is possible. Please refer below link for more detail.
https://laravel.com/docs/8.x/migrations#squashing-migrations
php artisan schema:dump
// Dump the current database schema and prune all existing migrations...
php artisan schema:dump --prune
schema:dump will create new directory under database > schema and SQL dump will stored there.
After that when you try to migrate first it will run dump file from schema and then any pending migration.
another solution work for me in Laravel 5.2:
DB::unprepared(File::get('full/path/to/dump.sql'));
Simple solution provided by Laravel Article for generating migration file from an existing database table.
Try: https://laravelarticle.com/laravel-migration-generator-online
If you can dump to a CSV:
An alternative for some generic data tables (Countries, States, Postal Codes), not via migrations but via seeders. Although you could do it the same way in a migration file.
In your seeder file:
public function run()
{
$this->insertFromCsvFile('countries', 'path/to/countries.csv');
$this->insertFromCsvFile('states', 'path/to/states.csv');
$this->insertFromCsvFile('postal_codes', 'path/to/postal_codes.csv');
}
private function insertFromCsvFile($tableName, $filePath)
{
if( !file_exists($filePath) ){
echo 'File Not Found: '.$filePath."\r\n";
return;
}
$headers = $rows = [];
$file = fopen( $filePath, 'r' );
while( ( $line = fgetcsv( $file ) ) !== false ){
// The first row should be header values that match column names.
if( empty( $headers ) ){
$headers = explode( ',', implode( ',', $line ) );
continue;
}
$row = array_combine( $headers, $line );
foreach( $row as &$val ) if( $val === 'NULL' ) $val = null;
$rows[] = $row;
// Adjust based on memory constraints.
if( count($rows) === 500 ){
DB::table( $tableName )->insert($rows);
$rows = [];
}
}
fclose( $filePath );
if( count($rows) ) DB::table( $tableName )->insert($rows);
}
Run the seeder: php artisan db:seed --class=GenericTableSeeder
You can create laravel migration and models directly from database using https://github.com/XCMer/larry-four-generator
Execute the following code after installing the package
php artisan larry:fromdb
i recently standing in front of the same problem. i didn't want to install a package specially for that, so i decided to write a little tool to help me and others ;)
Here is the link:
http://laravel.stonelab.ch/sql-seeder-converter/
And here you can comment it, if you have any improvement proposals or questions:
http://www.stonelab.ch/en/sql-to-laravel-seeder-converter/
You can use Raahul/Larryfour Package, A model and migration generator for Laravel 4
Raahul/Larryfour Package
After insallation you can use a command line to create a migration from existed database like this:
php artisan raahul:fromdb --only yourdatabase
And you will find the migration in app/migrations/ folder

Resources