Laravel job runs multiple times when executing long task - laravel

I'm currently trying to convert a video into multiple .ts segments (for HTTP video streaming). This is a long task so I'm doing this in a Laravel job.
The problem is that the job run multiple times after a few minutes and because of that, the video is processed multiple times. Here you can see my Job class:
<?php
namespace App\Jobs;
use App\Models\Media\MediaFile;
use FFMpeg\Filters\Video\ResizeFilter;
use FFMpeg\Format\Video\X264;
use Illuminate\Bus\Queueable;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Str;
use FFMpeg;
class ConvertVideoForStreaming implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
public $mediaFile;
public $deleteWhenMissingModels = true;
public $timeout = 3600;
public $retryAfter = 4000;
public $tries = 3;
/**
* Create a new job instance.
*
* #return void
*/
public function __construct(MediaFile $mediaFile)
{
$this->mediaFile = $mediaFile;
}
/**
* Execute the job.
*
* #return void
*/
public function handle()
{
$this->mediaFile->update([
'status' => 'Processing',
'status_message' => null,
]);
$convertingId = Str::random(6);
$lowBitrateFormat = (new X264('aac', 'libx264'))
->setKiloBitrate(500)
->setAudioKiloBitrate(128);
$midBitrateFormat = (new X264('aac', 'libx264'))
->setKiloBitrate(1500)
->setAudioKiloBitrate(192);
$highBitrateFormat = (new X264('aac', 'libx264'))
->setKiloBitrate(3000)
->setAudioKiloBitrate(256);
\Log::info('Started a new converting process with convertingID: ' . $convertingId);
FFMpeg::fromDisk('public')
->open($this->mediaFile->file)
->exportForHLS()
->setSegmentLength(4) // optional
->addFormat($lowBitrateFormat, function($media) {
$media->addFilter(function ($filters) {
$filters->resize(new \FFMpeg\Coordinate\Dimension(640, 480), ResizeFilter::RESIZEMODE_INSET);
});
})
->addFormat($midBitrateFormat, function($media) {
$media->addFilter(function ($filters) {
$filters->resize(new \FFMpeg\Coordinate\Dimension(1280, 720), ResizeFilter::RESIZEMODE_INSET);
});
})
->addFormat($highBitrateFormat, function($media) {
$media->addFilter(function ($filters) {
$filters->resize(new \FFMpeg\Coordinate\Dimension(1920, 1080), ResizeFilter::RESIZEMODE_INSET);
});
})
->onProgress(function ($percentage) use($convertingId) {
\Log::info($this->mediaFile->name . " (ConvertingID: $convertingId) - $percentage % transcoded");
\Redis::set('video-transcoded-' . $this->mediaFile->id, $percentage);
})
->toDisk('public')
->save('livestream/' . Str::slug($this->mediaFile->name) . '/playlist.m3u8');
$this->mediaFile->update([
'status' => 'Active',
'status_message' => null,
]);
}
public function failed(\Exception $exception)
{
$this->mediaFile->update([
'status' => 'Failed',
'status_message' => $exception->getMessage(),
]);
}
}
How can I solve my problem?

Laravel offers a 'lock' feature for scheduled commands. See the 'Prevent Task Overlaps' in this documentation.
In case you'd like to have non-scheduled jobs being processed only once I'd advise you to look into the Symfony Lock Component. This component offers you to lock a certain task for a period of time or until it's unlocked. This way you can do something along the lines of:
At the start of your handle() method
a. check if a lock already exists, otherwise skip this job
b. If the lock does not exist, create it
Execute your long running task
At the end of your handle() logic release the lock

Related

How can I queue the laravel excel import, on traditional way, and dispatch the job again after the first chunk reading

My code: (controller)
public function formatCSV(Request $request){
$filename = $request->input('filename');
$exported_filename = pathinfo($filename, PATHINFO_FILENAME).".csv";
$export_filepath = 'files/scrubber/output/'.$exported_filename;
$data = [
'file_name' => $filename,
'header_row' => 10,
'start_row' => 10,
'unit_sqft_col' => null,
'file_path' => storage_path('app/files/scrubber/') . $filename,
'export_path' => $export_filepath,
];
$scrubberJob = (new ScrubberJob($data))->delay(Carbon::now()->addSeconds(3));
dispatch($scrubberJob);
return redirect()->route('scrubbers.index')->with('toast.success','Scrubber job is put into queue, please wait for a while..');
}
On the above controller, I am dispatching ScrubberJob.
ScrubberJob
<?php
namespace App\Jobs;
use App\Imports\ChunkScrubberImport;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldBeUnique;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Excel;
class ScrubberJob implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
protected $data;
/**
* Create a new job instance.
*
* #return void
*/
public function __construct($data)
{
$this->data = $data;
}
/**
* Execute the job.
*
* #return void
*/
public function handle()
{
$import = new ChunkScrubberImport($this->data);
Excel::import($import, $this->data['file_path']);
}
}
This job is triggering ChunkScrubberImport
ChunkScrubberImport
<?php
namespace App\Imports;
use App\Exports\ChunkScrubberExport;
use Illuminate\Support\Collection;
use Maatwebsite\Excel\Concerns\ToArray;
use Maatwebsite\Excel\Concerns\ToCollection;
use Maatwebsite\Excel\Concerns\WithStartRow;
use Maatwebsite\Excel\Concerns\WithLimit;
use Maatwebsite\Excel\Concerns\WithStrictNullComparison;
use Illuminate\Support\Facades\Storage;
use Excel;
class ChunkScrubberImport implements ToArray, WithLimit, WithStartRow, WithStrictNullComparison
{
protected $data;
public $scrubbed_data;
public function __construct($data, $scrubbed_data = [])
{
$this->data = $data;
$this->scrubbed_data = $scrubbed_data;
}
public function array(Array $rows)
{
$this->scrubbed_data = $rows;
return Excel::store(new ChunkScrubberExport($this->data,$this->scrubbed_data), $this->data['export_path']);
}
public function startRow(): int
{
return $this->data['start_row'];
}
public function limit(): int
{
return 1000;
}
}
Now, this import is triggering ChunkScrubberExport
ChunkScrubberExport
<?php
namespace App\Exports;
use App\Helpers\APRHelper;
use App\Models\Scrubber;
use Maatwebsite\Excel\Concerns\FromArray;
use Maatwebsite\Excel\Concerns\FromCollection;
use Maatwebsite\Excel\Concerns\RegistersEventListeners;
use Maatwebsite\Excel\Events\AfterSheet;
use Maatwebsite\Excel\Concerns\WithStrictNullComparison;
class ChunkScrubberExport implements FromArray,WithStrictNullComparison
{
use RegistersEventListeners;
protected $data;
protected $scrubbed_data;
public function __construct($data, $scrubbed_data = [])
{
$this->data = $data;
$this->scrubbed_data = $scrubbed_data;
}
public function array(): array
{
$arr = $this->scrubbed_data;
$rec_arr = $empty_col = array();
$empty_col_checked = false;
foreach ($arr as $ak => $av){
//only if row is not empty (or filled with null), will get inside if condition
if(count(array_filter($av)) != 0){
if(!$empty_col_checked){
foreach($av as $k => $v){
if($v == ''){
$empty_col[] = $k;
}
}
$empty_col_checked = true;
}
$rec_arr[] = $av;
}
}
foreach($empty_col as $ek => $ev){
//get all values from a columns, which don't have value in header row
//and check if all the values from this particular column is empty
//if empty unset the columns
if(empty( array_filter(array_column($rec_arr,$ev))) )
{
//unset array keys from whole array
foreach($rec_arr as &$item) {
unset($item[$ev]);
}
unset($item);
}
}
foreach ($rec_arr as $ak => $av) {
foreach ($av as $k => $v) {
//other stuff
} //end foreach $av
} //endforeach $rec_arr
return $rec_arr;
}
public static function afterSheet(AfterSheet $event)
{
$active_sheet = $event->sheet->getDelegate();
$centered_text = [
'alignment' => [
'horizontal' => \PhpOffice\PhpSpreadsheet\Style\Alignment::HORIZONTAL_CENTER, 'vertical' => \PhpOffice\PhpSpreadsheet\Style\Alignment::VERTICAL_CENTER
]
];
$active_sheet->getParent()->getDefaultStyle()->applyFromArray($centered_text);
}
}
this is working fine for 1 cycle or fetch, now I have the following questions.
After the first cycle, I want to update the start_row to $data['start_row'] + 1000 since 1000 is the limit, but, I don't know what might be the best place to put this one. After 1 cycle I want to trigger, scrubberjob written below
but with updated start_row
$scrubberJob = (new ScrubberJob($data))->delay(Carbon::now()->addSeconds(3));
dispatch($scrubberJob);
Since, on first cycle, the file would already be created since I already know the filename, which is stored in $data['export_path']. So, instead of overwriting the old file, I want to append rows to the already existing file from the second cycle.
How to detect the end of the file, so that no more ScrubberJob is dispatched.
By the way, I have also seen about ShouldQueue in docs but I am not sure if it could provide all the needs I have mentioned here. Or am I just complicating things and should go with shouldQueue instead? But still, that might need the solution of my question number 2

How to modify fortify CreatesNewUsers.php interface?

I need to modify /vendor/laravel/fortify/src/Contracts/CreatesNewUsers.php interface
and to add 1 more bool parameter, as using CreateNewUser in different places of the app
validations rules are different, say in some places password is not filled on user creation, but must be separate function.
So I copied file /project/resources/fortify/CreatesNewUsers.php with content :
<?php
namespace Laravel\Fortify\Contracts;
interface CreatesNewUsers
{
public function create(array $input, bool $makeValidation);
}
and in app/Actions/Fortify/CreateNewUser.php I modified :
<?php
namespace App\Actions\Fortify;
use App\Models\User;
use Illuminate\Support\Facades\Hash;
use Illuminate\Support\Facades\Validator;
//use Laravel\Fortify\Contracts\CreatesNewUsers;
use Resources\Fortify\CreatesNewUsers; // Reference to my interface
use Laravel\Jetstream\Jetstream;
class CreateNewUser implements CreatesNewUsers
{
use PasswordValidationRules;
public function create(array $input, bool $makeValidation)
{
...
But trying to use this class I got error
Interface "Resources\Fortify\CreatesNewUsers" not found
Which is the valid way ?
Thanks!
I moved interface at file app/Actions/Fortify/CreatesNewUsers.php :
<?php
namespace App\Actions\Fortify;
interface CreatesNewUsers
{
public function create(array $input, bool $make_validation, array $hasPermissions);
}
and modified app/Actions/Fortify/CreateNewUser.php :
<?php
namespace App\Actions\Fortify;
use App\Models\User;
use Illuminate\Database\QueryException;
use Illuminate\Support\Facades\Hash;
use Illuminate\Support\Facades\Validator;
use DB;
use App\Actions\Fortify\CreatesNewUsers;
use Laravel\Jetstream\Jetstream;
use Spatie\Permission\Models\Permission;
class CreateNewUser implements CreatesNewUsers
{
use PasswordValidationRules;
/**
* Validate and create a newly registered user.
*
* #param array $input
*
* #return \App\Models\User
*/
public function create(array $input, bool $make_validation, array $hasPermissions)
{
if ($make_validation) {
$userValidationRulesArray = User::getUserValidationRulesArray(null, '', []);
if (\App::runningInConsole()) {
unset($userValidationRulesArray['password_2']);
}
$validator = Validator::make($input, $userValidationRulesArray);//->validate();
if ($validator->fails()) {
$errorMsg = $validator->getMessageBag();
if (\App::runningInConsole()) {
echo '::$errorMsg::' . print_r($errorMsg, true) . '</pre>';
}
return $errorMsg;
}
} // if($make_validation) {
$newUserData = [
'name' => $input['name'],
'email' => $input['email'],
'account_type' => $input['account_type'],
'phone' => $input['phone'],
'website' => $input['website'],
'notes' => $input['notes'],
'first_name' => $input['first_name'],
'last_name' => $input['last_name'],
];
if (isset($input['password'])) {
$newUserData['password'] = Hash::make($input['password']);
}
if (isset($input['status'])) {
$newUserData['status'] = $input['status'];
}
if (isset($input['activated_at'])) {
$newUserData['activated_at'] = $input['activated_at'];
}
if (isset($input['avatar'])) {
$newUserData['avatar'] = $input['avatar'];
}
try {
DB::beginTransaction();
$newUser = User::create($newUserData);
foreach ($hasPermissions as $nextHasPermission) {
$appAdminPermission = Permission::findByName($nextHasPermission);
if ($appAdminPermission) {
$newUser->givePermissionTo($appAdminPermission);
}
}
DB::commit();
return $newUser;
} catch (QueryException $e) {
DB::rollBack();
if (\App::runningInConsole()) {
echo '::$e->getMessage()::' . print_r($e->getMessage(), true) . '</pre>';
}
}
return false;
}
}
It allows me to use CreateNewUser from different parts of app, like seeders, adminarea, user registration
with different behaviour. For me it seems good way of using fortify and CreateNewUser...

Log table creation using Laravel Events

I want to create a log each time when insertion happens to the files table. That is, everytime when an insertion to the files table happens, an event has to trigger to create a log for it automatically. Log table needs to have 3 columns id, fileid - primary key of files table & logDetails ,where the logDetails column store a msg like 1 file inserted.
i have read this - https://laravel.com/docs/8.x/events and also searched many more , but that couldn't help me to find where should i start. I have created an event page and a listeners page for this. But i don't know what to write in that.The below Controller & model does the insertion to the files table well. And where i need ur help is to code for creating a log for this insertion. Any help is much appreciated.
Controller
<?php
namespace App\Http\Controllers;
use Illuminate\Http\Request;
use App\Models\FileLogs;
use Illuminate\Support\Facades\Validator;
use App\Events\InsertFileLog;
class FileLogController extends Controller
{
public function insert(Request $request) // insertion
{
$validator = Validator::make(
$request->all(),
[
'orderId' => 'required|integer', //id of orders table
'fileId' => 'required|integer', //id of file_type table
'status' => 'required|string'
]
);
if ($validator->fails()) {
return response()->json($validator->errors(), 400);
}
$obj = new FileLogs();
$obj->orderId=$request->orderId;
$obj->fileId=$request->fileId;
$obj->status=$request->status;
$obj->save();
//dd($obj->id);
if($obj->id!=''){
InsertFileLog::dispatch($order);
return response()->json(['status'=>'success','StatusCode'=> 200, 'message'=>'Successfully Inserted','data'=>$obj]);
}
else{
return response()->json(['status'=>'Failed','message'=>'Insertion Failed'],400);
}
}
Model
class FileLogs extends Model
{
use HasFactory;
use SoftDeletes;
protected $table='files';
protected $fillable = [
'orderId',
'fileId',
'status'
];
}
Event
<?php
namespace App\Events;
use Illuminate\Broadcasting\Channel;
use Illuminate\Broadcasting\InteractsWithSockets;
use Illuminate\Broadcasting\PresenceChannel;
use Illuminate\Broadcasting\PrivateChannel;
use Illuminate\Contracts\Broadcasting\ShouldBroadcast;
use Illuminate\Foundation\Events\Dispatchable;
use Illuminate\Queue\SerializesModels;
use App\Models\FileLogs;
class InsertFileLog
{
use Dispatchable, InteractsWithSockets, SerializesModels;
public $order;
public function __construct(FileLogs $order)
{
$this->order = $order;
}
}
Listener
<?php
namespace App\Listeners;
use App\Events\InsertFileLog;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Queue\InteractsWithQueue;
use App\Models\Logs;
class FileLogListener
{
public function __construct()
{
//
}
public function handle(InsertFileLog $event)
{
//
}
}
EventServiceProvider.php
class EventServiceProvider extends ServiceProvider
{
protected $listen = [
Registered::class => [
SendEmailVerificationNotification::class,
],
];
public function boot()
{
//
}
}
The App\Providers\EventServiceProvider offers a way to register Events and Listeners. The key is the Event, the values are one or multiple Listeners. You should update your code to contain the following:
EventServiceProvider
class EventServiceProvider extends ServiceProvider
{
protected $listen = [
Registered::class => [
SendEmailVerificationNotification::class,
],
InsertFileLog::class => [
// Listeners in this array will be executed when InsertFileLog was fired.
FileLogListener::class
],
];
public function boot()
{
//
}
}
The FileLogListener will only be triggered after the event InsertFileLog was fired. To do that, you could do the following, by adding: InsertFileLog::dispatch($order); line before return success response in your insert function.
FileLogController
class FileLogController extends Controller
{
public function insert(Request $request) // insertion
{
$validator = Validator::make(
$request->all(),
[
'orderId' => 'required|integer', //id of orders table
'fileId' => 'required|integer', //id of file_type table
'status' => 'required|string'
]
);
if ($validator->fails()) {
return response()->json($validator->errors(), 400);
}
$obj = new FileLogs();
$obj->orderId=$request->orderId;
$obj->fileId=$request->fileId;
$obj->status=$request->status;
$obj->save();
//dd($obj->id);
if($obj->id!=''){
InsertFileLog::dispatch($obj);
// alternatively, the event helper can be used.
// event(new InsertFileLog($obj));
return response()->json(['status'=>'success','StatusCode'=> 200, 'message'=>'Successfully Inserted','data'=>$obj]);
}
else{
return response()->json(['status'=>'Failed','message'=>'Insertion Failed'],400);
}
}
}
FileLogListener
<?php
namespace App\Listeners;
use App\Events\InsertFileLog;
use App\Models\Log;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Queue\InteractsWithQueue;
class FileLogListener
{
public function __construct()
{
//
}
public function handle(InsertFileLog $event)
{
// $event->order contains the order.
// In your question, you have request an example of creating an insertion to the Log table
Log::create([
'fileId' => $event->order->fileId,
'logDetails' => '1 file inserted',
]);
}
}
Model observers
Alternatively, Model observers provide a solution without having to manually register events/listeners, as they listen to Eloquent events of the observed model like: creating, created, updated, deleted, etc.

Laravel Notification doesn't work when queued

I'm using notifications to send emails to the admin when a new note added by the user. If I don't use the ShouldQueue, all work fine.. When I use queue I got an error
ErrorException: Undefined property: App\Notifications\NewKidNote::$note
what might be the reason?
here is my notification code
<?php
namespace App\Notifications;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Notifications\Messages\MailMessage;
use Illuminate\Notifications\Notification;
use App\Models\KidNote;
class NewKidNote extends Notification
{
use Queueable;
protected $note;
protected $kidname;
protected $userfullname;
protected $color;
protected $sentnote;
protected $kidid;
public function __construct($note)
{
$this->note = $note;
$this->kidname = $note->kid->name;
$this->userfullname = $note->user->fullname;
$this->color = $note->color;
$this->sentnote = $note->note;
$this->kidid = $note->kid_id;
}
public function via($notifiable)
{
return ['mail','database'];
}
public function toMail($notifiable)
{
return (new MailMessage)
->subject('New Note added')
->greeting('Hello ')
->line('New Note has been added for '. $this->kidname. ' by '.$this->userfullname)
->line('Note Color: '.$this->color)
->line('Note')
->line($this->sentnote)
->action('You can also see the note here', route('admin.children.show',$this->kidid));
}
public function toDatabase($notifiable)
{
return [
'icon' => 'notepad',
'color' => $this->color,
'message' => 'New note added for '. $this->kidname ,
'link' => route('admin.children.show',$this->kidid)
];
}
}
Note to self.. Make sure you clean the cache and restart the queue :)) then it works fine!! This code runs perfectly. thanks

Laravel Guzzle not perforning any action as requested in Cron Job

I am developing a web application using Laravel-5.8. Also, I am using guzzlehttp/guzzle-6.3 to consume an external api and save it in my local database.
travelupdate.php
<?php
namespace App\Console\Commands;
use Illuminate\Console\Command;
use Carbon\Carbon;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Log;
use GuzzleHttp\Exception\GuzzleException;
use GuzzleHttp;
use GuzzleHttp\Client;
use App\User;
use App\Activity;
use Avatar;
use Storage;
use App\Travel;
class travelsupdate extends Command {
protected $signature = 'command:travelsupdate';
protected $description = 'travelsupdate';
public function __construct() {
parent::__construct();
}
public function handle()
{
$client = new Client();
$res = $client->request('GET','https://api.abcdef.net/travels/v4/sample');
$trips = json_decode($res->getBody(), true);
foreach($trips as $trip) {
Trip::updateOrCreate([
'trip_id' => $trip->trip_id
],
[
'trip_number' => $trip->trip_no,
'truck_no' => $trip->t_no,
'truck_reg_no' => $trip->reg_no,
'trailer_no' => $trip->trailer_no,
'contract_no' => $trip->contract_no,
'contract' => $trip->contract_name,
'driver_id' => $trip->driver_id,
'driver_name' => $trip->driver_name,
'loading_date' => date_format($trip->loading_date, "Y-m-d"),
'loading_from' => $trip->loading_from
]);
}
}
}
app\Console\Kernel.php
<?php
namespace App\Console;
use Illuminate\Console\Scheduling\Schedule;
use Illuminate\Foundation\Console\Kernel as ConsoleKernel;
use Illuminate\Support\Facades\Cache;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Log;
use App\User;
use App\Activity;
use Avatar;
use Storage;
use Mail;
use App\Travel;
use App\Audit;
use Carbon\Carbon;
// use \Carbon\Carbon;
class Kernel extends ConsoleKernel
{
protected $commands = [
'App\Console\Commands\travelsupdate',
];
protected function schedule(Schedule $schedule)
{
$schedule->command('command:travelsupdate')
->hourly();
}
/**
* Register the commands for the application.
*
* #return void
*/
protected function commands()
{
// $this->load(__DIR__.'/Commands');
require base_path('routes/console.php');
}
}
What I want to achieve is that consume the external api using Guzzle GET request. Then save it into the local database. If data already exists (using trip_id), it updates. I set the cron job to every one hour.
I observe that nothing is happening, and no data is being saved to the local database. When I tested on POSTMAN, it displays the data from the api.
However, Laravel log file is not showing me any error.
How do I resolve this?
NOTE: There is no security setting on the external api.
You are retreiving array from api and then you are using it as an object.
Solution 1:
do not decode it as an array
$trips = json_decode($res->getBody());
Solution 2:
use decoded value as an array
foreach($trips as $trip) {
Trip::updateOrCreate([
'trip_id' => $trip['trip_id']
],
[
'trip_number' => $trip['trip_no'],
'truck_no' => $trip['t_no'],
'truck_reg_no' => $trip['reg_no'],
'trailer_no' => $trip['trailer_no'],
'contract_no' => $trip['contract_no'],
'contract' => $trip['contract_name'],
'driver_id' => $trip['driver_id'],
'driver_name' => $trip['driver_name'],
'loading_date' => date_format($trip['loading_date'], "Y-m-d"),
'loading_from' => $trip['loading_from']
]);
}
Everything else seems to be ok.
If you want to display errors in your log you can do it manually:
try {
...
} catch (\Exception $e){
\Log::error($e->getMessage());
}
Not sure if you are misusing the terminology but you don't set the "cron job" for every hour. You set the "cron job" that calls the Laravel Scheduler to run every minute. The scheduler then decides every time it is ran what needs to be run based on how you setup the calls in the scheduler.
To test this you can adjust your scheduled command to run every minute or 5 minutes lets say and manually call the scheduler yourself, php artisan schedule:run, from the command line.

Resources