I want to use queue job to update data by time create to database
Example . i have table job_history
I will insert multiple record into job_history table by status (status = 0 not finish)
I want to handle all record with status = 0 by queue (time create ascending)
(It mean after processing the record 1 completed (update record with status = 1 finished or timeout by 60s )
then automatically next to the record 2,3... until the end (update status=1)
And when i create new record into job_history table then queue always listen to continue handle with status =0
=> I can handle by cronjob (but cron job only configured at least once a minute => That interrupts the work) ,
I want to handle if record 1 finished will continue 2 ( not wating 1 minute by cronjob)
I don't know laravel queue can do this job not ?
and how to setup it ?
In production in order tu consume queue jobs you can use supervisor.
You can read this doc : https://laravel.com/docs/7.x/queues#supervisor-configuration
queue:work handles a job at a time, sequentially, as long as there are jobs to do in the queue. You don't have to do anything special to achieve what you're asking for.
Also, you don't need (strictly) a supervisor to run queues, those can be handled with a cron and a relatively simple script to check if the worker is running and launching or restarting it as needed.
Yes, you can handle this situation by excuting this command:
queue:work --stop-when-empty
With the cron, it will check every minute if the queue is not empty. Then it will do all existing jobs in the queue. Then terminate.
Related
Should we always be just creating sync jobs as our general rule of thumb in Upsolver SQLake?
In most cases, yes, you want to use sync jobs.The only case that you don't want to use sync job is when you have an input to table that you don't want to wait.
Example: you have 5 jobs that write to a table and some jobs that read from that table. If you don't want the entire pipeline to stuck if one of the 5 jobs is stuck, then your pipeline needs to be unsync (or at least this specific job that you think it may stuck to be unsync)
Note: unsync is not a keyword. If create JOB by default creates unsync job. CREATE SYNC job creates SYNC job.
In my application I need to log when each job is dispatched, retried and executed. The reason why I need this log is to detect whether on a failure the job is re-dispatched or not.
Also I need to keep 1 month long the job dispatch.
In another words I need to log:
Time dispatched,
Time redispatched,
Time finished
error that caused re-dispatch
where the job has been redispatched.
Is there some sort of trigger in laravel 5.7 that allows you to do custom logs during job dispatch?
So far I have seen that there are the jobs and failed_jobs tables that contains only ephemeral information only during the job execution. After job excecution records are removed.
If you want to build it yourself, then Job events
If you want to have a dashboard and logs for everything you mentioned, then Laravel Horizon
I have 1 application as a producer(scheduler) of jobs and N applications as an consumers (executors).
Is there any way how to prioritize execution of particular job using Quartz?
f.e.: at one moment i have 1000 jobs in queue and i need to execute 1 particular job with the highest priority
One possible solution could be set up different group name for the primary jobs. But is it possible to set up for 1 of my consumer apps to execute only jobs with particular group name?
Thanks in advance for reply.
Background Job A
Picks a record and update the status as picked
It will process sequence of steps
Once everything is done, status is changed to complete
When there is an error encountered during the steps, status is changed to error
Another Job B does the following
Picks a record where the status is picked or error status
Process the sequence of steps
Job B - Reprocessing Job which handles the exception and system crash use case
Example,
Job A - When the application crashes some of the records will be left with status Picked
so inorder to handle the crash scenario and error scenario
Job B - looks for both picked and error status record
Issue:
Job A should take only the fresh records
Job A should handle the exception case and error case. Job B should not fetch the fresh records.
How to handle this situation?
You can use the time stamp here or check the date
Like
Job B should be only take yesterday or like previous hour timestamp record
So the fresh record are taken by job A and the previous hours which would be termed as old job would be taken by Job B
I am working on a use case where I have a cron job scheduled (via quartz) which reads certain entries from db and process them.
Now in each schedule, I can get thousands of records which need to be processed. Processing each record takes time (in seconds/minutes). Currently all those records are getting processed on single node (node elected by quartz). Now my challenge is to parallelize these records processing. Please help me in solving below concerns :
How I can distribute these records/tasks to a cluster of machines
If any machine fails after processing few records then remaining records should be processed by healthy nodes in cluster
Get a signal that all record processing is finished.
Create cron jobs to run separately on each host at the desired frequency. You will need some form of lock on each record or some form of range lock on the record set to ensure that servers process mutually exclusive set of records.
e.g. : You can add following new field to all records:
Locked By Server:
Locked for Duration (or lock expiration time):
On each run, each cron picks a set of records that have expired or empty locks and then it aquires the lock on a small set of records by putting these two entries. Then it proceeds to process them. If it crashes or gets stuck the lock expires, otherwise it is released on completion.