I have a Flask webserver that interacts with a POSTGREsql database to fetch and insert data from the website.
I also have an APScheduler Background Scheduler task that checks the database regularly, looks for certain objects with certain attributes like 5 minutes past creation, column x is over 10 etc, then sends that information in an email.
The problem I'm running into right now is that,
when run in the local environment, only one task is executed, and it runs every (30) seconds like I want it to.
However, if I deploy it on Heroku, two of the same tasks is executed within milliseconds of each other, and it causes problems.
The code responsible for running the background task would be this.
app = Flask(__name__)
#app.debug = True
app.debug = False
app.config.from_pyfile('config.py')
db = models.db
db.init_app(app)
migrate = Migrate(app, db)
manager = Manager(app)
manager.add_command('db', MigrateCommand)
job_defaults = {
'coalesce': False,
'max_instances': 1
}
sched = BackgroundScheduler(job_defaults=job_defaults)
sched.start()
#sched.scheduled_job('interval', seconds=30)
def my_job():
with app.app_context():
my_class = OutageCheck(db)
my_class.checkOutage()
It can be a reloader which runs 2 instances of flask on remote server.
You can try this
app.run(debug=True, use_reloader=False)
Consult this question How to stop Flask from initialising twice in Debug Mode?
Related
I'm using machinery/V1 and broker and result_backend -> redis
This is my architecture :
2 ECS services:
-> has a machinery server instance and adds the tasks (IgnoreWhenTaskNotRegistered = true)
-> service to process the tasks , has a worker instance with this config (concurrency:1 ) and all the tasks are registered here.
Note : In 1st ECS I'm not registering any tasks when i'm sending the tasks to machinery server.
2 services use same redis instance for broker and result_backend
When IgnoreWhenTaskNotRegistered = false, everything works fine and tasks are picked by worker (if an unregistered task is sent, it is infinitely retried as expected).
Issue : 1. When IgnoreWhenTaskNotRegistered = true, Worker (2nd ECS) is not picking every task that is sent by 1st ECS. Strange thing is that it is picking some tasks (randomly)
My concern is that even when tasks are registered by worker, they are being ignored (with this variable , IgnoreWhenTaskNotRegistered = true)
why can't I use IgnoreWhenTaskNotRegistered = false and be happy about it ?
There are other devs who are not aware of this and may try to add a task unregistered and there will be like 15 GB of logs in 6 hrs(task not registered , retrying again)
I try to run a task withoutOverlapping but the task still called every minute even if i add
->withoutOverlapping()
Here is my task i have a sleep(240) to force the task to be longer than 1 min but the mails is still sent every minutes
$schedule->call(function() {
$user = User::find(4);
Mail::to('john.doe#gmail.com')->queue(new AccountConfirmation($user));
sleep(240);
})->name('update_game')->withoutOverlapping(20);
I use heroku with a scheduler here are the logs : the task "update_game" run every minutes even if the task takes more than 1 minute (with the sleep(240)) i wonder why
The Heroku scheduler starts a brand new dyno (basically, a server) every time it runs.
->withoutOverlapping() only applies to the current server, so it's not doing anything, because the next minute another server is running.
You'll want to call ->onOneServer() too, but to do that, you'll need to move off the file driver onto something like redis for your caching system, or you'll have the same problem - one server not knowing anything about the other, because they each have their own set of files.
Sometimes it is preferred and/or required to host dozens of applications on a single server. Not saying this is "right" or "wrong," I'm only saying that it happens.
A downside to this configuration is the error message Waiting for the script in task [TASK ID] to finish as this script requires that no other Octopus scripts are executing on this target at the same time appears whenever more than one deployment to the same machine is running. It seems like Octopus Deploy is fighting itself.
How can I configure Octopus Deploy to wait for one deployment to completely finish before the next one is started?
Before diving into the answer, it is important to understand why that message is appearing in the first place. Each time a step is run on a deployment target, the tentacle will create a "Mutex" to prevent others projects from interfering with it. An early use case for this was updating the IIS metabase during a deployment. In certain cases, concurrent updates would cause random errors.
Option 1: Disable the Mutex
We've seen cases where the mutex is the cause of the delay. The mutex is applied per step, not per deployment. It is common to see a situation where it looks like Octopus is "jumping" between deployments. Depending on the number of concurrent deployments, that can slow down the deployment. The natural thought is to disable the mutex altogether.
It is possible to disable the mutex by adding the variable OctopusBypassDeploymentMutex and setting it to True. That variable can exist in either a specific project or in a variable set.
More details on what that variable does can be found in this document. If you do disable the mutex please test it and monitor for any failures. For the most part, we don't see issues disabling the mutex, but it has happened from time to time. It depends on a host of other factors such as application type and Windows version.
Option 2: Leverage Deploy a Release Step
Another option is to coordinate the projects using the deploy a release step. Typically this works best when the projects being deployed are part of the same application suite. In the example screenshot below I have five "deployment" projects:
Azure Worker IaC
Database Worker IaC
Kubernetes Worker IaC
Script Worker IaC
OctoStudy
The project Unleash the Kraken coordinates deployments for those projects.
It does this by using the Deploy a Release step. First it spins up all the infrastructure, then it deploys the application.
This won't work as well if the server is hosting 50 disparate applications.
Option 3: Leverage the API to check for running deployments
The final option is to include a step at the start of each project which hits the API to check for active releases to the deployment targets for the deployment target. If an active deployment is found then wait until it is done.
You can do this by hitting the endpoint https://[YOUR URL]/api/[SPACE ID]/machines/[Machine Id]/tasks?skip=0&name=Deploy&states=Executing%2CCancelling&spaces=[SPACE ID]&includeSystem=false. That will tell you all the active tasks being run for a specific machine.
You can get Machine Id by pulling the value from Octopus.Deployment.Machines. You can get Space Id by pulling the value from Octopus.Space.Id.
The pseudo code for this approach could look like this (I'm not including the actual code as your requirements could be very different).
activeDeployments = true
while (activeDeployments)
{
activeDeployments = false
foreach(machineId in Octopus.Deployment.Machines)
{
activeTasks = https://[YOUR URL]/api/[Octopus.Space.Id]/machines/[Machine Id]/tasks?skip=0&name=Deploy&states=Executing%2CCancelling&spaces=[Octopus.Space.Id]&includeSystem=false
if (activeTasks.Count > 0)
{
activeDeployments = true
}
}
if (activeDeployments = true)
{
Sleep for 5 seconds
}
}
I had this message hit me because I hit the Task Cap on the Octopus Server.
In Octopus\Configuration\Nodes change the task cap to 1 to have 1 deployment at a time even with agents on different servers. The message will display constantly
Or simply increase this value to prevent the message from occurring at all.
I have a nginx loadbalancer in front of two tomcat instances each contains a spring boot application. Each spring boot application executes a batch that writes data in a database.
The batch executes every day at 1am.
The problem is that both instances execute the batch simultaniously which i don't want.
Is there a way to keep the batchs deployed in two instances and tell tomcat or nginx to start the batch in master server (and the slave server doesn't run the batch).
If one of the servers stops, the second server could start the batch on his behalf.
Is there a tool in nginx or tomcat (or some other technology) to do that ?
thank you in advance.
Here is a simplistic design approach.
Since you have two scheduled methods in the 2 VMs triggered at same time, add a random delay to both. This answer has many options on how to delay the trigger for a random duration. Spring #Scheduled annotation random delay
Inside the method run the job only if it is NOT already started (by the other VM). This could be done with a new table to track this.
Here is the pseudo code for this design:
#Scheduled(cron = "schedule expression")
public void batchUpdateMethod() {
//Check database for signs of job running now.
if (job is not running){
//update database table to indicate job is running
//Run the batch job
//update database table to indicate job is finished
}
}
The database, or some common file location, should be used as a lock to sync between the two runs, since the two VMs are independent of each other.
For a more robust design, consider Spring Batch
Spring Batch uses a database for its jobs (JobsRepository). By default an in memory datasource is used to keep track of running jobs and their status. In your setup, the 2 instances are (most likely) using their own in memory database.
Multiple instances of Spring Batch can coordinate with each other as a cluster and one can run jobs, while the other actasa backup, if the jobsRepository database is shared.
For this you need to configure the 2 instances to use a common datasource.
Here are some docs:
https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#jobrepository
https://docs.spring.io/spring-batch/docs/current/reference/html/job.html#configuringJobRepository
If you design two app server instances to run the same job at the same time, then by design, one will succeed to create a job instance and the other will fail (and this failure can be ignored). See Javadoc of JobRepository. This is one of the roles of the job repository: to act as a safeguard against duplicate job executions in a clustered environment.
If one of the servers stops, the second server could start the batch on his behalf. Is there a tool in nginx or tomcat (or some other technology) to do that ?
I believe there is no need for such tool or technology. If one of the servers is down at the time of the schedule, the other will be able to take things over and succeed in launching the job.
I did implement a simple BCM Server functionality, where all servers do register(create a Server-table entry) with their unique IP. The Servers need to register within a defined time(e.g. 10 sec). If a Server does not register within time(last update timestamp > 10 sec), then the Server gets de-registered(delete Server-table entry) by the Server, which do register.
At the end I do have a table with ordered Server entries and can define the task uniquely to the registered Servers.
The Implementation is very simple and Works perfectly.
Before I did also have in mind the Spring Batch Job Sharing functionality, but I wanted zu have a more lightweight and more flexible Solution.
Currently I use it in all my projects where I need to have Batch-Processing implemented.
We have our application deployed on Websphere application server. The application is running on clustered environment with 6 Nodes. EJB timer service is configured using custom scheduler with datasource pointing to Oracle database.So when the application is deployed on cluster it triggers the Ejb timer service on Node1 which is given in the Oracle database.
Some times the value in oracle database changes automatically to some
other nodes like node2 or node3 because of which EJB timer is getting
stopped.Any Suggestions or advice on why it gets changed automatically.
EJB timer configuration
Server(0).components.ApplicationServer(1).components.EJBContainer(1).timerSettings.EJBTimer(0).datasourceJNDIName = jdbc/cdb_db
Server(0).components.ApplicationServer(1).components.EJBContainer(1).timerSettings.EJBTimer(0).nonPersistentTimerRetryCount = -1
Server(0).components.ApplicationServer(1).components.EJBContainer(1).timerSettings.EJBTimer(0).nonPersistentTimerRetryInterval = 300
Server(0).components.ApplicationServer(1).components.EJBContainer(1).timerSettings.EJBTimer(0).numAlarmThreads = 1
Server(0).components.ApplicationServer(1).components.EJBContainer(1).timerSettings.EJBTimer(0).numNPTimerThreads = 1
Server(0).components.ApplicationServer(1).components.EJBContainer(1).timerSettings.EJBTimer(0).pollInterval = 300
Server(0).components.ApplicationServer(1).components.EJBContainer(1).timerSettings.EJBTimer(0).tablePrefix = EJBTIMER_
Server(0).components.ApplicationServer(1).components.EJBContainer(1).timerSettings.EJBTimer(0).uniqueTimerManagerForNP = false
As the first comment added to this question points out, it is the designed behavior of EJB Persistent Timers/Scheduler to have any one member run all of the tasks until that member isn't available or cannot respond quickly enough, in which case another member takes over.
If you don't like this behavior and want to change it so that your timer tasks can only run on a single member, you can accomplish that by stopping the scheduler poll daemon on all members except for the one that you want to run the tasks. Here is a knowledge center document which describes how to do that:
https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.5.5/com.ibm.websphere.nd.multiplatform.doc/scheduler/xmp/xsch_stopstart.html
Just be aware that if you do this, you will be losing out on the ability for the scheduler to automatically start running tasks on a different member should the member that you have designated to run them go down. In this case, tasks will not run at all until either of
1) the member that is allowed to run them comes back up, or
2) you manually use the aforementioned WASScheduler MBean to start the scheduler poll daemon on a different member, thus allowing tasks to run there