Let's say I have a MyJob class with perform method, and I want to enqueue it different queues in Resque. How can I achieve it?
The documentation states that job's queue is determined by queue class method, which means that the only way to enqueue the same class to different queues is to dynamically create a sub-class for each queue, before enqueing.
Is there a simpler way that I'm missing?
To anyone who might stumble on this, here's what I was looking for:
Resque.enqueue_to(queue, class_name, *params)
have you tried enqueuing the jobs using:
Resque::Job.create(queue, klass, *args)
Related
Is there a good way to orchestrate lambda functionality that changes based on a queue message? I was thinking about taking a similar approach described in the strategy pattern.
The lambda function is polling an SQS queue. The queue message would contain some context that is passed into a lambda telling it what workflow needs to be executed. Based on this message, the lambda would execute some corresponding script.
The idea behind this is that I can write code for different ad hoc jobs and use the same queue + lambda function for these jobs but have it delegate the work. This way, I can track unsuccessful jobs in a dead letter queue. Are there any red flags here or potential pitfalls I should be aware of when you hear this? Any advice would be appreciated. TIA!
EDIT: For some additional context, this different workflows triggered by this lambda will vary in compute resources needed. An example is ingesting a large dataset from an api call and doing some custom schematization on the contents before making an api call.
This is indeed possible, but there's a variety of approaches you may take. These depend on what type of workflow/processing you require.
As you highlight, Lambda could be used for this. It's worth noting that Lambda functions do not work well for computationally-intensive tasks.
If you were looking to perform a workflow with some complexity, you should consider AWS Step Functions. Suppose you had three "tasks" to choose from, you could define a Step Function for each, then use Lambda to (1.) receive the message & work out which task is required, then (2.) start an execution for the desired Step Function.
FYI, you don't need to make your Lambda function poll the SQS queue, instead, you can set up SQS to automatically trigger Lambda once a new message is added to the queue. See AWS Docs - Configuring a queue to trigger an AWS Lambda function.
If you edit your question with more info on what you're looking to do (processing-wise) with each message, people will be able to better help with your use-case.
Best of luck! :)
When it comes to testing jobs, it seems the solution is simply to instantiate the job and call it's handle method.
As per the Laravel Queue Documentation, you can type hint your dependencies on the Job's handle() method. I'm going to assume you don't do it on the constructor, because that's how the Job's data get's passed in, and the queue workers aren't setup to inject dependencies when they process the job.
So when testing, what is the best way to pass dependencies to the handle method. For now I'm doing the following:
$jobFoo = new jobFoo($bar);
$jobFoo->handle(
$this->app->make(DependencyExample::class)
);
Is ther a better way to pass dependencies, or perhaps the whole approach to testing jobs this way is flawed.
You can use app to invoke the method and resolve dependencies automatically:
$this->app->call([$jobFoo, 'handle]);
What changes if I call
schedule->call(function() { .. do something ... })->daily()
and if I call
schedule->job(... my job class where handle do the same things... )->daily()
?
The job() method is there specifically for adding a Laravel job (i.e. a class that implements the Illuminate\Contracts\Queue\ShouldQueue interface) to a queue. Whereas the call() method would run a Closure or an invokable object synchronously at the scheduled time, the job() method would add the given job to a queue at the scheduled time. The queue worker would then process the job when it gets to the front of the queue which might not be immediately.
The $schedule->job(new Heartbeat)->everyFiveMinutes(); is then basically a shortcut for:
$schedule->call(function () {
Heartbeat::dispatch();
})->everyFiveMinutes();
It's practically the same thing.
$schedule->job() requires you to create a job class and add the logic there. This is an easy way to schedule a job without having to write the execution code yourself.
Where as $schedule->call() allows you to do any action directly in a callback. This removes the need to create a job class for simple actions.
job()
The job method may be used to schedule a queued job. This method
provides a convenient way to schedule jobs without using the call
method to manually create Closures to queue the job.
source :https://laravel.com/docs/5.8/scheduling#scheduling-queued-jobs
Basically, there is no significant difference for you. But writing complex code in the call method can become messy quite easily.
The main difference is that one has the code in-line and the other separates it out into its own class.
The in-line option is appropriate if the code is extremely short, readable, and in context.
The separate class option is appropriate if the code is more than a couple of lines long.
In general writing clean code to the SOLID principles would dictate using the separate class option in almost every case.
I am trying to write a consumer for an existing queue.
RabbbitMQ is running in a separate instance and queue named "org-queue" is already created and binded to an exchange. org-queue is a durable queue and it has some additional properties as well.
Now I need to receive messages from this queue.
I have use the below code to get instance of the queue
conn = Bunny.new
conn.start
ch = conn.create_channel
q = ch.queue("org-queue")
It throws me an error stating different durable property. It seems by default the Bunny uses durable = false. So I've added durable true as parameter. Now it states the difference between other parameters. Do I need to specify all the parameters, to connect to it? As rabbitMQ is maintained by different environment, it is hard for me to get all the properties.
Is there a way to get list of queues and listening to the required queue in client instead of connecting to a queue by all parameters.
Have you tried the :passive=true parameter on queue()? A real example is the rabbitmq plugin of logstash. :passive means to only check queue existence rather than to declare it when consuming messages from it.
Based on the documentation here http://reference.rubybunny.info/Bunny/Queue.html and
http://reference.rubybunny.info/Bunny/Channel.html
Using the ch.queues() method you could get a hash of all the queues on that channel. Then once you find the instance of the queue you are wanting to connect to you could use the q.options() method to find out what options are on that rabbitmq queue.
Seems like a round about way to do it but might work. I haven't tested this as I don't have a rabbitmq server up at the moment.
Maybe there is way to get it with rabbitmqctl or the admin tool (I have forgotten the name), so the info about queue. Even if so, I would not bother.
There are two possible solutions that come to my mind.
First solution:
In general if you want to declare an already existing queue, it has to be with ALL correct parameters. So what I'm doing is having a helper function for declaring a specific queue (I'm using c++ client, so the API may be different but I'm sure concept is the same). For example, if I have 10 subscribers that are consuming queue1, and each of them needs to declare the queue in the same way, I will simply write a util that declares this queue and that's that.
Before the second solution a little something: Maybe here is the case in which we come to a misconception that happens too often :)
You don't really need a specific queue to get the messages from that queue. What you need is a queue and the correct binding. When sending a message, you are not really sending to the queue, but to the exchange, sometimes with routing key, sometimes without one - let's say with. On the receiving end you need a queue to consume a message, so naturally you declare one, and bind it to an exchange with a routing key. You don't need even need the name of the queue explicitly, server will provide a generic one for you, so that you can use it when binding.
Second solution:
relies on the fact that
It is perfectly legal to bind multiple queues with the same binding
key
(found here https://www.rabbitmq.com/tutorials/tutorial-four-java.html)
So each of your subscribers can delcare a queue in whatever way they want, as long as they do the binding correctly. Of course these would be different queues with different names.
I would not recommend this. This implies that every message goes to two queues for example and most likely a message (I am assuming the use case here needs to be processed only once by one subscriber).
Laravel 4 has a great list of features in terms of Queues. This question is with respect to the Queue method Queue.later() API Documentation the first parameter being the delay.
A Cron is basically made use of to execute recurring tasks.
If the below snippet were to be made more generic with the time being configurable as well can:
This be used as an alternative to CRON jobs?
Would this be fail safe approach to use assuming we use IronMQ
-
class SendEmail {
public function fire($job, $data)
{
//Connect to SMTP and send email
$job->delete();
//Recall the queue with a delay
Queue::later(60,'SendEmail#send', array('message' => $message));
}
}
//app/events/MailHandler.php
public class MailHandler(){
public function onMailListenerStarted(){
Queue::push('SendEmail#send', array('message' => $message));
}
}
You have to keep in mind that Queuing and Cron-tasking are 2 different things.
A cron job will start (depending on how you configure it) every exact minute.
A queue job will run after the delay time is over AND it is his turn to be processed.
So to compare this to your definition of a Cron "execute recurring tasks", the Queue does nothing like that. Jobs will simply wait in the queue, and they don't do anything. Delayed jobs will give you the advantage that it will at least wait till the time is there to send it, but it won't try and send all emails in one go. The downside if this is that it might take to long before it's send, but to prevent that, you can simply use more workers to process the queue.
And you need a script that processes the queue, which you'll most likely want to start with a cron.
Another problem i see with the approach in the code snippet, is that, if something goes wrong with adding the job back to the queue, the job will be lost, and will never be added back to the queue.
So to answer your questions:
No, Queues are not an alternative to jobs, but Queues do make data processing in cron-scripts easier
In theory this won't be fail safe approach, no matter how good your queue provider is. But it's possible to create some scripts that check if the queue is still doing everything it should, but this does require some logging (e.g. Save when the job has been run for the last time).
..2 years later..
Would this be fail safe approach?
Laravel Forge has made adding and supervising queue workers very reliable and much less painful, it's worth checking out.
So yeah don't know if fail-safe or not, but definitely more reliable than it used to be.