I have a django application in heroku and one thing I need to do sometimes that take a little bit of time is sending emails.
This is a typical use case of using workers. Heroku offers support for workers, but I have to leave them running all the time (or start and stop them manually), which is annoying.
I would like to use a one-off process to send every email. One possibility I first thought of was using IronWorker, since I thought that I could simply add the job to ironworker's queue and it would be exectuted with a mex of 15 min delay, which is ok for me.
The problem is that with ironworker, I need to put in a zip file all the modules and their dependencies in order to run the job, so in my email use case, as I use "EmailMultiAlternatives" from "django.core.mail.message", I would need to include all the django framework in my zip file in order to be able to use it.
According to this link, it's possible to add/remove workers from the app. Is it possible to start one-off processes from the app?
Does anyone has a better solution?
Thanks in advance
Related
I need to implement a service for my web server that refreshes an access token from some outside rest-api because that token has a 10 minute expiration time. (This is not an accesstoken that my server produces, it is
a token I receive from an outside api that allows me to use their services for a limited time)
For implementing timed functions in Go I've come across both cronjobs and functions using time.Ticker, however I havent come across any posts on the advantages/disadvantages of using one over the other and would like to one which would possibly be a better use for my situation.
If there is an optional route I'd be open to exploring it as well.
Thank you
time.Ticker is included with the Go standard library. No "cron" library is. So you reduce your external dependencies by using time.Ticker.
Cron is designed to run jobs on a specified schedule. Usually these jobs are run outside the Go program by the operating system. This isn't quite what you want. There are other job runners, and libraries called "cron" which are actually job runners, but again they're third party libraries.
A time.Ticker inside a goroutine is very simple and you can just have a nice infinite loop that fetches an API token every few minutes and sends it down a channel to wherever it's needed. That's maybe eight lines of code.
So in my API there are a few places where it is running a process / report that is either hitting a timeout or simply just taking WAY too long. I'd like to defer these jobs off to a queue and instead return a key in my response. The front end would then ping a service using that key to determine the status of its particular job in the queue. This way we don't have hanging ajax calls for 2 - 3 minutes. Maybe I could even create a queue viewer that would allow you to review the jobs in it and even cancel some etc.
Does Laravel have something built in or is there a package for this already? Are there other better options for dealing with this kind of issue?
this is what you are lokking for laravels queues
I don't believe this existed when I first posted this quesiton. However, Laravel now has this built for it: https://laravel.com/docs/5.6/horizon which is everything I was looking for.
Here's the situation:
I have a Laravel 4.2 application that retrieves (from a third party API) an asset. This is a long-lived asset (it only changes once every 12-24 hours) and is kind of time consuming (a large image file). I do cache the asset, so the impact has been more or less minimized, but there's still the case where the first person who logs in to my application in the morning has to wait while the application loads the asset for the first time.
I have set up a job which will be queued up and will run every eight hours. This ought to ensure that the asset in the cache is always fresh. It works by re-enqueueing the job for eight hours later after it runs.
The problem is this: I'm about to deploy this job system to production & I'm not sure how to start this thing running for the first time.
Ideally, I'd like to have an administration option where I have a button which says "Click here to submit the job", but I'd like to make it as foolproof as possible & prevent people (I'm not the only administrator) from submitting the job multiple times. To do this, however, the application would need to check & see if the job is already in the queue. I can't find a way to do that in an implementation-independent way (I'm using redis, but that may change in the future).
Another option would be to add an artisan command to run the initial process. That way I could deploy the application, run an artisan command, and forget about it.
So, to recap, I have two questions:
Is there a way to check a queue to see what jobs are in there?
Is there a better way to do this?
Thanks
When a job is in laravel queue, it will be saved in jobs table, so you can check by DB.
If it's guaranteed to be the only thing ever in the queue, you could use something like:
if (Queue::size() === 0) {
Queue::push(...);
}
You would need to run the php artisanqueue:listen in the terminal.
Here is the complete documentation if you want to learn more about:
https://laravel.com/docs/5.2/queues#running-the-queue-listener
You can use the Laravel Telescope package.
Laravel Telescope is an elegant debug assistant for the Laravel framework. Telescope provides insight into the requests coming into your application, exceptions, log entries, database queries, queued jobs, mail, notifications, cache operations, scheduled tasks, variable dumps and more. Telescope makes a wonderful companion to your local Laravel development environment.
(Source: https://laravel.com/docs/7.x/telescope)
I'm developing an app that needs to fetch a POP3 account every 5-15 minutes to check for new email and process it. I have written all the code except for the part where it automatically runs every 5-15 minutes.
I'm using Sinatra, DataMapper and hosting on Heroku which means cron jobs are out of the question, because Heroku only provides hourly cron jobs at best.
I have looked into Delayed::Job which doesn't natively support Sinatra nor DataMapper but there are workarounds for both. Since my Ruby knowledge is limited I couldn't find a way to merge these two forks into one working Delayed::Job for Sinatra/DataMapper solution.
Initially I used Mailman to check for emails which has built-in polling and runs continuously, but since it's not Rack-based it doesn't run on Heroku.
Any pointers on where to go next? Before you say: a different webhost, I should add I really prefer to stick with Heroku because of its ease of use (except of course, for the above issue).
Heroku supports CloudMailin
A simple trick is to write your code contained in a loop, then sleep at the bottom of it for however long you want:
Untested sample code...
loop do
do_something_way_cool()
sleep 5 * 60 # it's in minutes
end
If it has to be contained in the main body of the app then use a Thread to wrap it so the thread does the work. You'll need to figure out your shared data structures to transfer the data out of the loop. Queue is your friend there.
Why not use the Windows scheduler?
I have several applications that have to run at certain times according to business rules not the typical every weekday at 1pm.
I also need a way for the applications to provide feedback of their progress so that I can have rules that notify me when the applications are running slow or aren't even running anymore.
What Windows API should I be looking into? (like, a time version of the FileWatcher apis)
What's the best way to have the application notify the scheduler of its progress (files, sockets, windows messages, ???)?
For Vista/Win2k8, there's the nice Task Scheduler 2.0 API: http://msdn.microsoft.com/en-us/library/aa384138(VS.85).aspx. Previous version have the Task Scheduler 1.0 API, but I've never used it.
AppControls has a CronJob component that you can use to create scheduled events. This saves your program from having to wake up every minute and check the schedule itself. Instead, just schedule the job and indicate a callback method.
I have used this component for scheduling jobs myself and have been very happy with the way that it works.
I think what you really want is a common framework for your applications that report to something (you or the system messages or tracing or perfmon, event log, whatever) and also to receive via some inter process protocol a way to receive messages and respond.
based on the reporting you can change the scheduling or make changes, etc.
So, there is some monitor app, and then each of your other apps does common reporting.
events I can think of:
- started
- stopped
- error
- normal log messages
- and of course specific things your apps do.
I think there are probably existing classes/framework that do this - you'll have to check around.
If it were me, I would make a service that could talk to all the other apps and perhaps was even an http server. It would be able to route messages to particular apps and start stop those processes and query them.
There are lots of ways to do what you want though. those were just off the top of my head.
Alternatively you might just be able to get these to be services and they handle messages sent to them. Their normal processing does nothing until they are "woken up" with some task command.
You have more questions in one. Normally you should split them. But let's overlook this and try to answer.
To schedule certain events (including running an application): Use TJvScheduledEvents from JVCL. IMHO JVCL is the best Delphi open source library around with extensive number of components, developers & support. TJvScheduledEvents is quite neat, uses threads for event scheduling and also you have in JVCL a detailed editor for your events (it needs a small hack to use it though).
To provide 'feedback' from your applications to a (remote) central point: A very very very good solution (if your requirements permit) is to log the progress of your applications in a table (let's call it LOG) on a Firebird server. In LOG you can have the following fields: COMPUTER, USERNAME, APPNAME, MSG, LOGDATE (etc. etc.). In the After Insert trigger of the LOG table you can fire an event (let's call it NEW_LOG). In your console app you can register the interest for this event and so, your application will be automatically updated with everything which happens in any of your applications, so you can do log analysis, graphs etc. Of course you can do it with IB, but IB costs.
...going on Windows API route you need headers (which probably aren't translated), you'll encounter our dearest Pointers/PChars etc. etc. Of course, building from scratch everything isn't worthwhile but when this is already done in a Delphi way, why don't use it?
Use service with a timer that is fired regulary (for example each minute). It reads the schedule and looks if some are due before the next iteration. If so, you can execute them.
You can add an interface that shows all running apps. For the feedback and query that using a desktop application.