I have a job within a queue that has a higher priority than other jobs in that queue. Is there a mechanism in Resque by which I can move this job to the top of the queue?
If it's an adhoc bump you can use redis from a ruby/rails console as
class_looking_for = "MyWorker"
args_looking_for = [1234, {"foo" => "bar"}]
queue = "foo"
redis = Resque.redis
key = "queue:#{queue}"
redis.llen(key).times do |n|
value = redis.lindex(key, n)
job = Resque.decode(value)
if job["class"] == class_looking_for && job["args"] == args_looking_for
redis.lrem(key, value)
redis.lpush(value)
break
end
end
This will loop through the keys in the redis list used by redis to store the given queue and check if it matches given class_looking_for and args_looking_for, adjust the filter to your needs (the job looks like this {"class"=>"PieCutWorker", "args"=>[42529, "heart shape"]}), once the right job is found, it will be removed from the queue and then appended to it. Then it's ready to picked up by the next available worker.
But, if you need this as a system rule, as job "X" always needs to run right away, then you are better off creating a new queue for that job and either initialize redis workers as
$ QUEUE=important,something,bulk rake resque:work
as the QUEUE param uses the order of the comma separated strings to define priority.
Related
I am exploring ChronicleQueue to save events generated in one of my application.I would like to publish the saved events to a different system in its original order of occurrence after some processing.I have multiple instances of my application and each of the instance could run a single threaded appender to append events to ChronicleQueue.Although ordering across instances is a necessity,I would like to understand these 2 questions.
1)How would the index of the read index for my events be saved so that I don't end up reading and publishing the same message from chronicle queue multiple times.
In the below code(picked from the example in github) the index is saved till we reach the end of the queue while we restarted the application.The moment we reach the end of the queue,we end up reading all the messages again from the start.I want to make sure for a particular consumer identified by a tailer Id, the messages are read only once.Do i need to save the read index in another queue and use that to achieve what I need here.
String file = "myPath";
try (ChronicleQueue cq = SingleChronicleQueueBuilder.binary(file).build()) {
for(int i = 0 ;i<10;i++){
cq.acquireAppender().writeText("test"+i);
}
}
try (ChronicleQueue cq = SingleChronicleQueueBuilder.binary(file).build()) {
ExcerptTailer atailer = cq.createTailer("a");
System.out.println(atailer.readText());
System.out.println(atailer.readText());
System.out.println(atailer.readText());
}
2)Also need some suggestion if there is a way to preserve ordering of events across instances.
Using a named tailer should ensure that the tailer only reads a message once. If you have an example where this doesn't happen can you create a test to reproduce it?
The order of entries in a queue are fixed when writing and all tailer see the same messages in the same order, there isn't any option.
I am working on a feature where I need to check the job status in beanstalkd queues. I have tried a few things but I am not getting the jobs that are reserved for queues other than the default queue
$pheanstalk = \Illuminate\Support\Facades\Queue::getPheanstalk();
$pheanstalk->useTube('import-live');
$pheanstalk->watch('import-live');
while ($job = $pheanstalk->reserve(0)) {
var_dump(json_decode($job->getData(), true));
}
This is what I have tried. But I still get the data for the default queue. Anyone has any idea on how to get the data for the import-live queue as well? Or any other queues I have running in my system. Basically I want to get the data on all the queues in the system.
First of all - make sure that there are jobs in the other queues.
Then, if you don't want to get jobs from the 'default' queue for a particular run, you can ignore a it.
$job = $pheanstalk
->watch('import-live')
->watch('import-other')
->ignore('default')
->reserve();
->useTube('..') is only used when put()-ing messages into a queue.
I have a script that collects data from a database, filters and puts into list for further processing. I've split entries in the database between several processes to make the filtering faster. Here's the snippet:
def get_entry(pN,q,entries_indicies):
##collecting and filtering data
q.put((address,page_text,))
print("Process %d finished!" % pN)
def main():
#getting entries
data = []
procs = []
for i in range(MAX_PROCESSES):
q = Queue()
p = Process(target=get_entry,args=(i,q,entries_indicies[i::MAX_PROCESSES],))
procs += [(p,q,)]
p.start()
for i in procs:
i[0].join()
while not i[1].empty():
#process returns a tuple (address,full data,)
data += [i[1].get()]
print("Finished processing database!")
#More tasks
#................
I've run it on Linux (Ubuntu 14.04) and it went totally fine. The problems start when I run it on Windows 7. The script gets stuck on i[0].join() for 11th process out of 16 (which looks totally random to me). No error messages, nothing, just freezes there. At the same time, the print("Process %d finished!" % pN) is displayed for all processes, which means they all come to an end, so there should be no problems with the code of get_entry
I tried to comment the q.put line in the process function, and it all went through fine (well, of course, data ended up empty).
Does it mean that Queue here is to blame? Why does it make join() stuck? Is it because of internal Lock within Queue? And if so, and if Queue renders my script unusable on Windows, is there some other way to pass data collected by processes to data list in the main process?
Came up with an answer to my last question.
I use Manager instead
def get_entry(pN,q,entries_indicies):
#processing
# assignment to manager list in another process doesn't work, but appending does.
q += result
def main():
#blahbalh
#getting entries
data = []
procs = []
for i in range(MAX_PROCESSES):
manager = Manager()
q = manager.list()
p = Process(target=get_entry,args=(i,q,entries_indicies[i::MAX_PROCESSES],))
procs += [(p,q,)]
p.start()
# input("Press enter when all processes finish")
for i in procs:
i[0].join()
data += i[1]
print("data", data)#debug
print("Finished processing database!")
#more stuff
The nature of freezing in Windows on join() due to presence of Queue still remains a mystery. So the question is still open.
As the docs says,
Warning As mentioned above, if a child process has put items on a queue (and it has not used JoinableQueue.cancel_join_thread), then that process will not terminate until all buffered items have been flushed to the pipe.
This means that if you try joining that process you may get a deadlock unless you are sure that all items which have been put on the queue have been consumed. Similarly, if the child process is non-daemonic then the parent process may hang on exit when it tries to join all its non-daemonic children.
Note that a queue created using a manager does not have this issue. See Programming guidelines.
So, since the multiprocessing.Queue is a kind of Pipe, when you call .join(), there are some items in the queue, and you should consume then or simply .get() them to make the empty. Then call .close() and .join_thread() for each queue.
You can also refer to this answer.
I am trying to create queues with variable queue names.
queue_name = "guide_" + guide['id'].to_s
Sidekiq::Client.push({
'class' => GuidePdfWorker,
'queue' => queue_name,
'args' => [key],
'backtrace' => true
})
I know that I am supposed to add them to config/sidekiq.yml, but I can't, since I don't know the value of queue_name.
When I log Sidekiq::Client.registered_queues() I can see my queues, but they are never processed.
The Sidekiq Dynamic Queues gem will probably help you.
Just to provide a more complete, updated answer: there are plugins and extensions to Sidekiq that can do things like this, but Sidekiq is not designed to operate this way.
I don't recommend having more than a handful of queues ... and Sidekiq Pro cannot reliably handle multiple queues without polling
https://github.com/mperham/sidekiq/wiki/Advanced-Options#queues
the number of named queues used should be minimized.
https://github.com/mperham/sidekiq/issues/835
Instead, consider having a known, static, queue with a worker that dispatches based on what you'd like to be dynamic:
class GuidePdfWorker
include Sidekiq::Worker
sidekiq_options queue: 'default'
def perform(guide)
# branch on guide['id'], perhaps re-queue in higher or lower priority queue?
end
end
I am building a system in ruby (rabbitmq, parallel gem) that takes a list of jobs, queues them up in rabbit and then has workers pop jobs off the queue to execute them.
It is easy to get a worker to pop a single job off of the queue when it is ready but I would like to have each worker run 5 threads such that when all 5 threads are processing, that worker does not pop any jobs off the queue. When a thread becomes free, the worker accepts a job from the queue.
Using the Parallel gem, the only way I see to create multi-threaded processes is with the following code.
results = Parallel.map(array, :in_processes => MAX_PROCESSES) do |item|
item.process
end
I would like to do something like
while true
cur_threads = Parallel.get_cur_threads
if cur_threads < MAX_PROCESSES
# get another job from queue
# allocate a thread for the job
end
Any ideas??
I know this is really old but I think you'd probably get what you're looking for using the work_queue gem.