I am working on a project where the objective is to keep track of whether a client has uploaded data within their expected time-window or not. These windows can occur daily, weekly, bi-weekly, bi-monthly, quarterly... etc. I'm currently using the CRON syntax to describe these schedules which kind-of works, but it falls short in some scenarios, for example the bi-weekly one.
I was wondering if there is some industry standard way to efficiently describe and store schedules that I don't know of. I tried to educate myself via Google, but when I put the word schedule into the mix, it always assumes I'm trying to schedule some task, but I only want to calculate dates based on these schedule definitions and compare the incoming file dates with them.
Thank you
Related
Users are able to set up a marketing email send time within my app for random dates as they need them. It is crucial that the emails start to go out exactly when they are scheduled. So, the app needs to create something that fires one time for a specific group of emails at a specific date and time down to the minute. There will be many more dates for other sends in the future, but they are all distinct (so need something other than 'run every xxx date')
I could have a Scheduler task that runs every minute that looks at the dates of any pending tasks in the database and moves to the command that sends those that need sending. But I'm running a multi-tenanted app -- likely not overlap, but seems like a huge hit to multiple databases every minute for every tenant to search through potentially thousands of records per tenant.
What I need (I think) is a way to programmatically set a schedule for each send date, and then remove it. I liked this answer or perhaps using ->between($startMin, $endMin), without the every xxx instruction, or even using the cron function within Scheduler to set up one single time for each group that needs to be sent. Not sure I'm using this the way it was intended though?
Would this even work? I programmatically created a schedule from within a test method, and it was added to the schedule queue based on the dump of the $schedule I created, showing all schedules - but it did not show up via this method found from this answer:
$schedule = app()->make(\Illuminate\Console\Scheduling\Schedule::class);
$events = collect($schedule->events())->filter(function (\Illuminate\Console\Scheduling\Event $event) {
return stripos($event->command, 'YourCommandHere');
});
It also did not output anything, so I'm wondering if programmatically creating a schedule outside of Kernel.php is not the way to go.
Is there another way? I don't know scheduler well enough to know if these one-off schedules permanently remain somewhere, are they deleted after their intended single use, taking up memory, etc?
Really appreciate any help or guidance.
I'm creating a new service, and for that I have database entries (Mongo) that have a state field, which I need to update based on a current time, so, for instance, the start time was set to two hours from now, I need to change state from CREATED -> STARTED in database, and there can be multiple such states.
Approaches I've thought of:
Keep querying database entries that are <= current time and then change their states accordingly. This causes extra reads for no reason and half the time empty reads, and it will get complicated fast with more states coming in.
I write a job scheduler (I am using go, so that'd be not so hard), and schedule all the jobs, but I might lose queue data in case of a panic/crash.
I use some products like celery, have found a go implementation for it https://github.com/gocelery/gocelery
Another task scheduler I've found is on Google Cloud https://cloud.google.com/solutions/reliable-task-scheduling-compute-engine, but I don't want to get stuck in proprietary technologies.
I wanted to use some PubSub service for this, but I couldn't find one that has delayed messages (if that's a thing). My problem is mainly not being able to find an actual name for this problem, to be able to search for it properly, I've even tried searching Microsoft docs. If someone can point me in the right direction or if any of the approaches I've written are the ones I should use, please let me know, that would be a great help!
UPDATE:
Found one more solution by Netflix, for the same problem
https://medium.com/netflix-techblog/distributed-delay-queues-based-on-dynomite-6b31eca37fbc
I think you are right in that the problem you are trying to solve is the job or task scheduling problem.
One approach that many companies use is the system you are proposing: jobs are inserted into a datastore with a time to execute at and then that datastore can be polled for jobs to be run. There are optimizations that prevent extra reads like polling the database at a regular interval and using exponential back-off. The advantage of this system is that it is tolerant to node failure and the disadvantage is added complexity to the system.
Looking around, in addition to the one you linked (https://github.com/gocelery/gocelery) there are other implementations of this model (https://github.com/ajvb/kala or https://github.com/rakanalh/scheduler were ones I found after a quick search).
The other approach you described "schedule jobs in process" is very simple in go because goroutines which are parked are extremely cheap. It's simple to just spawn a goroutine for your work cheaply. This is simple but the downside is that if the process dies, the job is lost.
go func() {
<-time.After(expirationTime.Sub(time.Now()))
// do work here.
}()
A final approach that I have seen but wouldn't recommend is the callback model (something like https://gitlab.com/andreynech/dsched). This is where your service calls to another service (over http, grpc, etc.) and schedules a callback for a specific time. The advantage is that if you have multiple services in different languages, they can use the same scheduler.
Overall, before you decide on a solution, I would consider some trade-offs:
How acceptable is job loss? If it's ok that some jobs are lost a small percentage of the time, maybe an in-process solution is acceptable.
How long will jobs be waiting? If it's longer than the shutdown period of your host, maybe a datastore based solution is better.
Will you need to distribute job load across multiple machines? If you need to distribute the load, sharding and scheduling are tricky things and you might want to consider using a more off-the-shelf solution.
Good luck! Hope that helps.
Capturing data in Keen.io is pretty straight-forward when it's driven by an event. (User-logged-in, Customer-bought-stuff, ...). That's very nicely explained in the docs.
But what is the best approach when I want insight on a state of my app at a given time in the past?
For instance: how many active licenses were there this time last year?
Our first thought is to run a cronjob every hour, to get those data from te DB, and store them in a custom collection. Is this the right approach, or are there better solutions?
I am trying to understand how to use the FHIR Questionnaire resource, and have a specific question regarding this.
My project is specifically regarding how a citizen in our country could be responding to Questionnaires via a web app, which are then submitted to the FHIR server as QuestionnaireAnswers, to be read/analyzed by a health professional.
A FHIR-based system could have lots of Questionnaires (Qs), groups of Qs or even specific Qs would be targeted towards certain users or groups of users. The display of the questionnare to the citizen could also be based on a Care-plan of a sort, for example certain Questionnaires needing filling-in in the weeks after surgery. The Questionnaires could also be regular ones that need to be filled in every day or week permanently, to support data collection on the state of a chronic disease.
What I'm wondering is if FHIR has a resource which fits into organizing the 'logistics' of displaying the right form to the right person. I can see CarePlan, which seems to partly fit. Or is this something that would typically be handled out-of-FHIR-scope by specific server implementations?
So, to summarize:
Which resource or mechanism would a health professional use to set up that a patient should answer certain Questionnaires, either regularly or as part of for example a follow-up after a surgery. So this would include setting up the schedule for the form(s) to be filled in, and possibly configure what would happen if the form wasn't filled in as required.
Which resource (possibly the same) or mechanism would be used for the patient's web app to retrieve the relevant Questionnaire(s) at a given point in time?
At the moment, the best resource for saying "please capture data of type X on schedule Y" would be DiagnosticOrder, though the description probably doesn't make that clear. (If you'd be willing to click the "Propose a change" link and submit a change request for us to clarify, that'd be great.) If you wanted to order multiple questionnaires, then CarePlan would be a way to group that.
The process of taking a complex schedule (or set of schedules) and turning that into a simple list of "do this now" requests that might be more suitable for a mobile application to deal with is scheduled for DSTU 2.1. Until then, you have a few options for the mobile app:
- have it look at the CarePlan and complex DiagnosticOrder schedule and figure things out itself
- have a server generate a List of mini 1-time DiagnosticOrders and/or Orders identifying the specific "answer" times
- roll your own mechanism using the Other/Basic resource
Depending on your timelines, you might want to stay tuned to discussions by the Patient Care and Orders and Observations work groups as they start dealing with the issues around workflow management starting next month in Atlanta.
Windows Phone allows you to create and register system alarms with the ScheduledActionService. The options for recurrenceType are daily, weekly, etc.
I am trying to get alarms set for specific days. This is not possible using the recurrenceType enum. The only way I can see to achieve this is to create an alarm for each day specified. This way, for a single alarm that is created I will have to create a copy of it on every day I wish it to recur.
The only issue with this is displaying them. They cannot be given a 'Name' property and so there is no way to logically group these alarms in to a single "master" alarm.
has anyone encountered this issue? Is there a way to set alarms without using this scheduler service? I cannot think of a way around this and would appreciate any guidance.
The only example I'm aware of where someone was in a similar situation resulted in them not using the scheduler at all.
As they required a complex scheduling pattern they instead chose to not use any of the built in functionality but rather to create individual alarms as appropriate. They already had an existing BackgroundAgent and added functionality there to ensure that the next two alarms were always queued up.
Your exact requirements may make this possible.
I'd think carefully about adding an agent just for this though.