I'm doing a migration of my Laravel 8 app to Cloud Run. But I have problem with my schedulers. My Laravel app using Laravel Scheduling so I got 5 tasks :
protected function schedule(Schedule $schedule) {
$schedule->command(Commands\CmdOne::class)->monthlyOn(1, '02:10');
$schedule->command(Commands\CmdTwo::class)->dailyAt('04:00');
$schedule->command(Commands\CmdThree::class)->dailyAt('04:00');
$schedule->command(Commands\CmdFour::class)->dailyAt('05:00');
$schedule->command('activations:clean')->daily();
}
But I think it's risky to place the cron inside the container because Cloud Run can run multiple container instances of my app and I fear about to run the tasks multiple times because my tasks send email to my customers and I want to run them just once.
e.g: if Cloud Run create 5 instances of my container at 05:00Am so the command $schedule->command(Commands\CmdFour::class)->dailyAt('05:00'); will be executed 5 times and I don't want this.
So I see Google Cloud Scheduler and I can expose a web service to run my tasks. But I don't know if it's the good way ? Or there is another way to execute my tasks ? I don't know if removing Laravel Scheduler is the right way.
So if I'm using Cloud Scheduler now, I have to create 5 crons in Cloud Scheduler. I think it's ok for one application but if I have 10 apps (with the same code base but different Cloud run service) it will be hard to manager all these crons because I'll get 5 crons per apps. So in this case 50 crons.
Do you have a better way to manager this ?
If you have the right cache setup (shared by all servers) then you can use the onOneServer() method.
See https://laravel.com/docs/9.x/scheduling#running-tasks-on-one-server
Related
I have a nginx loadbalancer in front of two tomcat instances each contains a spring boot application. Each spring boot application executes a batch that writes data in a database.
The batch executes every day at 1am.
The problem is that both instances execute the batch simultaniously which i don't want.
Is there a way to keep the batchs deployed in two instances and tell tomcat or nginx to start the batch in master server (and the slave server doesn't run the batch).
If one of the servers stops, the second server could start the batch on his behalf.
Is there a tool in nginx or tomcat (or some other technology) to do that ?
thank you in advance.
Here is a simplistic design approach.
Since you have two scheduled methods in the 2 VMs triggered at same time, add a random delay to both. This answer has many options on how to delay the trigger for a random duration. Spring #Scheduled annotation random delay
Inside the method run the job only if it is NOT already started (by the other VM). This could be done with a new table to track this.
Here is the pseudo code for this design:
#Scheduled(cron = "schedule expression")
public void batchUpdateMethod() {
//Check database for signs of job running now.
if (job is not running){
//update database table to indicate job is running
//Run the batch job
//update database table to indicate job is finished
}
}
The database, or some common file location, should be used as a lock to sync between the two runs, since the two VMs are independent of each other.
For a more robust design, consider Spring Batch
Spring Batch uses a database for its jobs (JobsRepository). By default an in memory datasource is used to keep track of running jobs and their status. In your setup, the 2 instances are (most likely) using their own in memory database.
Multiple instances of Spring Batch can coordinate with each other as a cluster and one can run jobs, while the other actasa backup, if the jobsRepository database is shared.
For this you need to configure the 2 instances to use a common datasource.
Here are some docs:
https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#jobrepository
https://docs.spring.io/spring-batch/docs/current/reference/html/job.html#configuringJobRepository
If you design two app server instances to run the same job at the same time, then by design, one will succeed to create a job instance and the other will fail (and this failure can be ignored). See Javadoc of JobRepository. This is one of the roles of the job repository: to act as a safeguard against duplicate job executions in a clustered environment.
If one of the servers stops, the second server could start the batch on his behalf. Is there a tool in nginx or tomcat (or some other technology) to do that ?
I believe there is no need for such tool or technology. If one of the servers is down at the time of the schedule, the other will be able to take things over and succeed in launching the job.
I did implement a simple BCM Server functionality, where all servers do register(create a Server-table entry) with their unique IP. The Servers need to register within a defined time(e.g. 10 sec). If a Server does not register within time(last update timestamp > 10 sec), then the Server gets de-registered(delete Server-table entry) by the Server, which do register.
At the end I do have a table with ordered Server entries and can define the task uniquely to the registered Servers.
The Implementation is very simple and Works perfectly.
Before I did also have in mind the Spring Batch Job Sharing functionality, but I wanted zu have a more lightweight and more flexible Solution.
Currently I use it in all my projects where I need to have Batch-Processing implemented.
I am trying to execute an api in laravel every minute.
The api's method is GET. However I could not specify the method in the cron.yaml file. Could I use DELETE method here and how? The code should be deployed on google cloud.
I have created a cron.yaml file that has the following format:
cron:
- description: "every minutes job"
url: /deletestories
schedule: every 1 mins
retry_parameters:
min_backoff_seconds: 2.5
max_doublings: 5
I also created the api deletestories that delete rows under specific conditions.
However this isn't working, and when I open google cloud console I could not found any error or any cron job executed.
This cron.yaml file appears to be a Google App Engine cron configuration. If this is correct then only the GET method is supported, you cannot use DELETE.
The GAE cron service itself consists simply of scheduled GET requests that your app needs to handle. From Scheduling Tasks With Cron for Python (the same applies to other languages and to the flexible environment cron as well):
A cron job makes an HTTP GET request to a URL as scheduled. The
handler for that URL executes the logic when it is called.
You also need to deploy your cron.yaml file for it to be effective. You should be able to see the deployed cron configuration in the developer console's Cron Jobs tab under the Task Queues Menu (where you can also manually trigger any of the cron jobs). The performed GET requests for the respective cron jobs should appear in your app's request logs as well, when executed.
I have been trying to schedule spring cloud task via PCF scheduler, however I can't create a job from the app/task (following this documentation on the site - http://docs.pivotal.io/pcf-scheduler/1-1/using-jobs.html)
$ cf apps
name requested state instances memory disk urls
cloud-task stopped 0/1 750M 1G
$ cf services
name service plan bound apps
last operation
my-scheduler scheduler-for-pcf standard cloud-task
create succeeded
$ cf create-job cloud-task my-task-job ".java-buildpa
ck/open_jdk_jre/bin/java org.springframework.boot.loader.JarLauncher"
Creating job ←[33;1mmy-task-job←[0m for ←[33;1mcloud-task←[0m with command ←[33;1m.java-buildpack/open_jdk_jre/bin/java org.springframework.boot.loade
r.JarLauncher←[0m in org ←[33;1mglobal-sales-marketing-customer-experience←[0m / space ←[33;1m141349-dev←[0m as ←[33;1mzzh1bb←[0m
←[31;1mFAILED←[0m
The requested resource was not found.
Not Found
You must create an instance of the scheduler service in this space to use the scheduler service.
Not sure why the job creation command is not able to find the instance of scheduler service - Am I missing something here ?
Also, wondering if there is anything in spring-clould-data-flow that can schedule tasks.
From the output you should be able to create a job in that org/space.
Does the user (zzh1bb?) have SpaceDeveloper privileges? SpaceAdmin should also be sufficient.
Does a cf task execute successfully using:
cf run-task cloud-task ".java-buildpack/open_jdk_jre/bin/java org.springframework.boot.loader.JarLauncher"
And seeing results with:
cf tasks cloud-task
Another diagnostic step might be to check the output of the api calls described here:
http://docs.pivotal.io/pcf-scheduler/1-1/api/#create-job
What version of PCF are you using and what version of the Scheduler for PCF are you using? There were significant changes in the cloud controller api between 1.10, 1.11 and 1.12 that prevent the scheduler service from working across all of those versions.
As far as scheduling SCDF, the Scheduler for PCF service can be used in conjunction with SCDF to allow you to call the task execution endpoint from a Schedeler for PCF call (https://docs.pivotal.io/pcf-scheduler/1-1/using-calls.html).
Call SCDF using the execution endpoint
http://...scdf server.../tasks/executions?name=taskA
doc'ed here:
https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#_launching_a_task_2
This is very useful and convenient especially when creating the SCDF service and the Scheduler for PCF service in the same space.
I have two apps running on the same server.
Now it seems like when adding withoutOverlapping() to the scheduler job and managing the base cronjob via cron itself, these 2 apps are blocking each other in execution.
Could that be?
Yes, withoutOverlapping only works per application.
Laravel creates a file in the storage folder with a hash of the job. This way, if the file exists, Laravel knows the job is still running. The one application cannot possibly know if the other one is currently running a job because it does not have access to the storage folder of the other application.
If your code looks like the following
$schedule->command('process:queue 0')->everyMinute()->withoutOverlapping();
$schedule->command('process:queue 1')->everyMinute()->withoutOverlapping();
It is because same commands with different parameters might bc considered overlapping.
I.e. the hash of the job will consider only the command signature.
I've just started using wercker and I'd like a job to run regularly (e.g. daily, hourly). I realize this may be an anti-pattern, but is it possible? My intent is not to keep the container running indefinitely, just that my workflow is executed on a particular interval.
You can use a call to the Wercker API to trigger a build for any project which is set up already in Wercker.
So maybe set up a cron job somewhere that uses curl to make the right API call?