I made a command and registered it in Kernel.php named "offers:update".
This command should do some changes on "offers" table depending on current status of each offer daily at 00:00.
My "offers" table might have above 100,000 rows and i want to do changes in the the most optimized way.
I've read the documentation (v 5.8) and found chunk() method.
Is it enough or there is better idea?
Related
I am using Laravel with the project I am currently working, the question above is one of the necessary thing that needs to be implement on the project.
So there should be a column that needs to reset everyday for all users and this project may contain hundreds or thousands of users and what should be the best way to do it that will not cause performance issues or server overloading.
I wanted to use Laravel's own scheduling but I not sure if this is quite the right thing to do.
Please help :)
You should create a job and schedule a command for this. So, with command even if you have to reset the column manually you can just run this command.
So, what you would do is:
php artisan make:command ResetColumnCommand
and, to generate a job:
php artisan make:job ResetColumnJob
and then, inside the job write the logic which would be similar to this:
$query = SomeModel::query();
$query->chunk(100, function($data) {
$data->update([
// Set the column to null
]);
});
Since, you're sure that there can be alot of records, you should definitely use chunk in order to reduce your memory usage.
NOTE: If you're applying any condition before chunk on a column. e.g: You want only records where that column is not already null then you should use chunkById instead of chunk because there are some issues that can be encountered with chunk and unexpected results.
I'm using Algolia, via Laravel
I'm frequently updating some entries in my database (every 5 mins), e.g. $object->update(['field'=>$new_value])
I would NOT like these changes to trigger updates / "operations" in Algolia, as this field is not relevant for Algolia's search indexing, and it would otherwise generate too many operations (which I'm then billed for).
Other changes should trigger updates in Algolia.
How can I do this?
if you using Laravel scout. you could do something like
$modal->unsearchable();
//update the changes
$modal->save()
for more info have a look at this https://laravel.com/docs/5.7/scout
cheers!
I am to generate a report from the database with 1000's of records. This report is to be generated on monthly basis and at times the user might want to get a report spanning like 3 months. Already as per the current records, a month's data set can reach to like 5000.
I am currently using vue-excel to which makes an api call to laravel api and there api returns the resource which is now exported by vue-excel. The resource does not only return the model data but there are related data sets I also need to fetch.
This for smaller data sets works fine that is when I am fetching like 3000 records but for anything larger than this, the server times out.
I have also tried to use laravel excel with the query concern actually timed them and both take same amount of time because laravel excel was also mapping to get me the relations.
So basically, my question is: is there some better way to do this so as get this data faster and avoid the timeouts
just put this on start of the function
ini_set(max_execution_time, 84000); //84000 is in seconds
this will override the laravel inbuild script runtime max value.
Looking for some guidance on best architecture to accomplish what I am trying to do. I occasionally get spreadsheets that will have a column of data that will need to be translated. There could be anywhere from 200 to 10,000 rows in that column. What I want to do is pull all rows and add them to a redis queue. I am thinking Redis will be best as I can throttle the queue which is necessary as the api I am calling for translation has throttle limits. Once the translation is done I will put the translations into a new column and return the user a new spreadsheet with the additional column.
If anyone has ideas for best setup I am open but I want to stick with laravel as that is what the application is already running. I am just not sure if I should create one queue job and that queue process will just open the file and start doing the translations. Or do I add a queue for each row of text. Or lastly do I add all of the rows of text to a table in my database and then have a task scheduler running every minute that will check that table for any untranslated rows and process x amount of them each time is checks. Not sure about cron job running so frequently when this happens maybe twice a month.
I can see a lot of ways of doing it but looking for an ideal setup as what I don't want to happen is I hit throttle limits and lose potential translations I have done as it could error out.
Thanks for any advice
I use PHP and Oracle, with crontab executing the PHP scripts at scheduled times. My current logging/auditing solution involves simple log files. I'd like to save my cron execution logs to a database instead.
Right now I'm trying to design the process so that when a cron job starts I create a record in an CronExecution table. Then every time I want to log something for that cron I'll put a record in a CronEvent table which will have a foreign key to the CronExecution table.
I plan to log all events using a PRAGMA AUTONOMOUS pl/sql procedure. With this procedure I will be able to log events inside of other pl/sql procedures and also from my PHP code in a consistent manner.
To make it more robust, my plan is to have a fallback to log errors to a file in the event that the database log calls fail.
Has anybody else written something similar? What suggestions do you have based on your experience?
Yep, I've done this several times.
The CronExecution table is a good idea. However, I don't know that you really need to create the CronEvent table. Instead, just give yourself a "status" column and update that column.
You'll find that makes failing over to file much easier. As you build lots of these CronExecutions, you'll probably have less interest in CronEvents and more interest in the full status of the execution.
Wrap all of the update calls in stored procedures. You definitely have that correct.
Including a 'schedule' in your CronExecution will prove handy. It's very easy to have a lot of cron jobs and not be able to connect the dots on "how long did this take?" and "when is this supposed to run". Including a "next scheduled run" on completion of a job will make your life much easier.
You will want to read the documentation on DBMS_SCHEDULER.
It's a little bit of work to implement (what isn't?), but it will allow you to control scheduled and one-time jobs from within the database. That includes OS-level jobs.