I have 5 tasks set up in the windows task scheduler.
Tasks A,B,C and D run at the same time each day in parallel. These all run absolutely fine.
I want task E to run each day once tasks A and B and C and D have all finished. This is the one I am having trouble with.
I have researched and found a number of promising links which show how to set up a custom event trigger:
running a scheduled task after another
trigger event on multiple different previous events success
custom event filter trigger is not triggering
I have tried countless different combinations of XML without success. Either task E does not get triggered, or it gets triggered continuously. I have tried so many combinations I am not sure which ones I should post on here (ie which ones are "closest" and would be helpful). I will post 1 below - if anyone requires other variations then I will post those too.
<QueryList>
<Query Id="0" Path="Microsoft-Windows-TaskScheduler/Operational">
<Select Path="Microsoft-Windows-TaskScheduler/Operational">
*[
#Event[
#EventData[#Name='TaskSuccessEvent'][Data[#Name='TaskName']='\TaskA']
and
#System[#TimeCreated[timediff(#SystemTime) <=3600000]]
]
and
#Event[
#EventData[#Name='TaskSuccessEvent'][Data[#Name='TaskName']='\TaskB']
and
#System[#TimeCreated[timediff(#SystemTime) <=3600000]]
]
and
#Event[
#EventData[#Name='TaskSuccessEvent'][Data[#Name='TaskName']='\TaskC']
and
#System[#TimeCreated[timediff(#SystemTime) <=3600000]]
]
and
#Event[
#EventData[#Name='TaskSuccessEvent'][Data[#Name='TaskName']='\TaskD']
and
#System[#TimeCreated[timediff(#SystemTime) <=3600000]]
]
]
</Select>
</Query>
</QueryList>
Can anyone help?
I played around with this some more and didn't get anywhere, so I decided to solve the problem using a different approach.
I built a simple C# console app which would run tasks A,B,C and D in parallel. This app could then be called by a single task on the windows scheduler.
From there I could set a dependency in the scheduler for task E on the single scheduler task above by following the information on the links provided in my original post.
If someone does figure out a way to set up mutiple "AND" dependencies in the scheduler itself then it would be great to know how.
Related
I have created all the tasks in a for loop first.
loop = asyncio.get_event_loop()
tasks =[]
for i in range(100):
task = loop.create_task(...)
tasks.append(task)
Instead of executing all 100 of them, I am trying to fire just a few, say just 1, at a time so I did something like below
await asyncio.wait([tasks[0]]) # just the very first one in the list
# also tried with `asyncio.gather` instead of `asyncio.wait`.
Expected behavior (at least to me)
Work with the very first task as I am only providing one task
Actual behavior
All 100 tasks are fired. How can I only fire just a few?
How can I only fire just a few?
I think the pattern you are looking for is using a queue and having a controlled number of consumers (workers). See for example
https://docs.python.org/3/library/asyncio-queue.html
Assuming I have a long running update query where I am updating ~200k to 500k, perhaps even more.Why I need to update so many documents is beyond the scope of the question.
Since the client times out (I use the official ES python client), I would like to have a way to check what the status of the bulk update request is, without having to use enormous timeout values.
For a short request, the response of the request can be used, is there a way I can get the response of the request as well or if I can specify a name or id to a request so as to reference it later.
For a request which is running : I can use the tasks API to get the information.
But for other statuses - completed / failed, how do I get it.
If I try to access a task which is already completed, I get resource not found .
P.S. I am using update_by_query for the update
With the task id you can look up the task directly:
GET /_tasks/taskId:1
The advantage of this API is that it integrates with
wait_for_completion=false to transparently return the status of
completed tasks. If the task is completed and
wait_for_completion=false was set on it them it’ll come back with a
results or an error field. The cost of this feature is the document
that wait_for_completion=false creates at .tasks/task/${taskId}. It is
up to you to delete that document.
From here https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html#docs-update-by-query-task-api
My use case went like this, I needed to do an update_by_query and I used painless as the script language. At first I did a reindex (when testing). Then I tried using the update_by_query functionality (they resemble each other a lot). I did a request to the task api (the operation hasn't finished of course) and I saw the task being executed. When it finished I did a query and the data of the fields that I was manipulating had disappeared. The script worked since I used the same script for the reindex api and everything went as it should have. I didn't investigate further because of lack of time, but... yeah, test thoroughly...
I feel GET /_tasks/taskId:1 confusing to understand. It should be
GET http://localhost:9200/_tasks/taskId
A taskId looks something like this NCvmGYS-RsW2X8JxEYumgA:1204320.
Here is my trivial explanation related to this topic.
To check a task, you need to know its taskId.
A task id is a string that consists of node_id, a colon, and a task_sequence_number. An example is taskId = NCvmGYS-RsW2X8JxEYumgA:1204320 where node_id = NCvmGYS-RsW2X8JxEYumgA and task_sequence_number = 1204320. Some people including myself thought taskId = 1204320, but that's not the way how the elasticsearch codebase developers understand it at this moment.
A taskId can be found in two ways.
wait_for_deletion = false. When sending a request to ES, with this parameter, the response will be {"task" : "NCvmGYS-RsW2X8JxEYumgA:1204320"}. Then, you can check a status of that task like this GET http://localhost:9200/_tasks/NCvmGYS-RsW2X8JxEYumgA:1204320
GET http://localhost:9200/_tasks?detailed=false&actions=*/delete/byquery. This example will return you the status of all tasks with action = delete_by_query. If you know there is only one task running on ES, you can find your taskId from the response of all running tasks.
After you know the taskId, you can get the status of a task with this.
GET /_tasks/taskId
Notice you can only check the status of a task when the task is running, or a task is generated with wait_for_deletion == false.
More trivial explanation, wait_for_deletion by default is true. Based on my understanding, tasks with wait_for_deletion = true are "in-memory" only. You can still check the status of a task while it's running. But it's completely gone after it is completed/canceled. Meaning checking the status will return you a 'resouce_not_found_exception'. Tasks with wait_for_deletion = false will be stored in an ES system index .task. You can still check it's status after it finishes. However, you might want to delete this task document from .task index after you are done with it to save some space. The deletion request looks like this
http://localhost:9200/.tasks/task/NCvmGYS-RsW2X8JxEYumgA:1204320
You will receive resouce_not_found_exception if a taskId is not present. (for example, you deleted some task twice, or you are deleting an in-memory task, whose wait_for_deletetion == true).
About this confusing taskId thing, I made a pull request https://github.com/elastic/elasticsearch/pull/31122 to help clarify the Elasticsearch document. Unfortunately, they rejected it. Ugh.
I have a task that triggers on "user session logon". I now want to restrict that task to fire only on weekdays, and being ignored on the weekend.
Is that possible?
Sidenote: I cannot use the trigger on schedule as I do not want to run the task periodically, but only on logon, and only on weekdays.
click Weekly (NOT Daily)
Choose the days you want
As far as I understand, this is not possible using the task scheduler alone.
You could use a piece of VBScript to achieve this.
Set up a file, e.g. mytask.vbs, like this:
If DatePart("w", Date, vbMonday) < 6 Then
Set Shell = CreateObject("WScript.Shell")
WScript.Quit(Shell.Run("C:\Windows\System32\notepad.exe", 10, True))
End If
Replace notepad by the task you actually want to run. What this things does is: It checks whether the current day is Mo-Fr (this is done by specifying the start of the week as Monday, so DatePart will return values from 1=Monday to 7=Sunday, and then we're checking if it's below 6), and if yes, it runs a certain program, waits for it to finish and forwards its exit code. (The magic number 10 here means that it will respect whatever setting for window display (normal, maximized, minimzed) was passed by the task scheduler, if any, and also forward it to the program.)
Then you can create a scheduled task with a logon trigger only, which runs wscript.exe /e:vbscript c:\path\to\your\mytask.vbs. That's it!
I need to have a Job with multiple tasks, being run on different machines, one after another (not simultaneously), and while the current job is running, another same job can arrive to the queue, but should not be started until the previous one has finished. So I came up with this 'solution' which might not be the best but it gets the job done :). I just have one problem.
I figured out I would need a JobQueue (either MongoDb or Redis) with the following structure:
{
hostname: 'host where to execute the task',
running:FALSE,
task: 'current task number',
tasks:{
[task_id:1, commands:'run these ecommands', hostname:'aaa'],
[task_id:2,commands:'another command', hostname:'bbb']
}
}
Hosts:
search for the jobs with same hostname, and running==FALSE
execute the task that is set in that job
upon finish, host sets running=FALSE, checks if there are any other tasks to perform and increases task number + sets the hostname to the next machine from the next task
Because jobs can accumulate, imagine situation when jobs are queued for one host like this: A,B,A
Since I have to run all the jobs for the specified machine how do I not start the 3rd A (first A is still running)?
{
_id : ObjectId("xxxx"), // unique, generated by MongoDB, indexed, sortable
hostname: 'host where to execute the task',
running:FALSE,
task: 'current task number',
tasks:{
[task_id:1, commands:'run these ecommands', hostname:'aaa'],
[task_id:2,commands:'another command', hostname:'bbb']
}
}
The question is how would the next available "worker" know whether it's safe for it to start the next job on a particular host.
You probably need to have some sort of a sortable (indexed) field to indicate the arrival order of the jobs. If you are using MongoDB, then you can let it generate _id which will already be unique, indexed and in time-order since its first four bytes are timestamp.
You can now query to see if there is a job to run for a particular host like so:
// pseudo code - shell syntax, not actual code
var jobToRun = db.queue.findOne({hostname:<myHostName>},{},{sort:{_id:1}});
if (jobToRun.running == FALSE) {
myJob = db.queue.findAndModify({query:{_id:jobToRun._id, running:FALSE},update:{$set:{running:TRUE}}});
if (myJob == null) print("Someone else already grabbed it");
else {
/* now we know that we updated this and we can run it */
}
} else { /* sleep and try again */ }
What this does is checks for the oldest/earliest job for specific host. It then looks to see if that job is running. If yes then do nothing (sleep and try again?) otherwise try to "lock" it up by doing findAndModify on _id and running FALSE and setting running to TRUE. If that document is returned, it means this process succeeded with the update and can now start the work. Since two threads can be both trying to do this at the same time, if you get back null it means that this document already was changed to be running by another thread and we wait and start again.
I would advise using a timestamp somewhere to indicate when a job started "running" so that if a worker dies without completing a task it can be "found" - otherwise it will be "blocking" all the jobs behind it for the same host.
What I described works for a queue where you would remove the job when it was finished rather than setting running back to FALSE - if you set running to FALSE so that other "tasks" can be done, then you will probably also be updating the tasks array to indicate what's been done.
I'm writing an rails 3 application which requires performing small tasks on a custom schedule for each user. The scheduled tasks will be defined dynamically. Right now my plan is to use resque scheduler with redis.
Once I set the schedule for a specify task (for eg. run task A every 48 hours) I would like to run that task indefinitely. So I would like to store those schedules in a db or something so in case an app crashes when it restarts it would load queue those task again.
Is this something Resque supports by default by storing it in redis or do I need to write my own custom thing? I was also looking at ruby-taskr (http://code.google.com/p/ruby-taskr/). I am not sure if taskr supports storing it in a database and registering it on start?
Also it would be helpful if there are applications/demo that I can look at it.
Thanks
I have a similar setup for batch jobs. The user adds them on a web dashboard and they get run however often is specified.
I use active-record to store the scheduling definitions, use resque for execution and a single cron entry for enqueueing using a rake task.
so then in the rake task:
to_run = Report.daily
to_run += Report.weekly if Time.now.monday?
to_run += Report.monthly if Time.now.day == 1
to_run.each{|r| r.enqueue!}
where daily, weekly, monthly are named scopes on the model:
class Report < ActiveRecord::Base
scope :daily, where(:when_to_run => 'daily')
scope :weekly, where(:when_to_run => 'weekly')
scope :monthly, where(:when_to_run => 'monthly')
end
This is a little hacky, but it works well and I stay within the stack nicely. Hope that is useful