Does anyone know of a way to configure the built in SLURM GUI sview so that it will show recent job history? I'd like to have it show something like the last 6 hours of jobs or at least display the jobs a little longer before they disappear after completion.
The motivation for this is that user of the cluster often run array jobs wth thousands of sub-jobs. Some of these sub-jobs often fail due to something in the data that needs to be investigated. Having a GUI for this has been an imense help. We used to have a custom scheduler and GUI for this but have moved to more opensource software.
Related
Does anyone have better ways of managing / scheduling console apps, without the use of Windows Scheduler?
eg. Console app to pickup records in the database that requires a set of actions.
*** Experience in the past, when number of tasks increase over time, it get's bit messy and difficult to maintain when moving servers.
There are third party apps you could look at, some of which are suggested in Cron-like system for Windows?.
However, if Windows Scheduler does provide the functionality you need, but just not the ideal management, you could look at using the CLI schtasks.exe
You can define all your tasks in an XML schema meaning they will more easily port across machines.
I have a program written in C++11. On the current input it takes too long to run. Luckily, the data can be safely split into chunks for parallel processing, which makes it a good candidate for, say, a Map/Reduce service.
AWS EMR could be a possible solution. However, since my code uses many modern libraries, it's quite a pain to compile it on the instances that are assigned for Apache Hadoop clusters. For example, I want to use soci (not available at all), boost 1.58+ (1.53 is there), etc etc. I also need a modern C++ compiler.
Obviously, all libraries and compilers can be manually upgraded (and the process scripted), but this sounds like a lot of manual work. And what about slave nodes - will they get all the libraries? Somehow I'm not sure. And the whole process of initializing the environment can now take very long time - thus killing a lot of performance advantage that distributing the jobs was supposed to bring in to begin with.
On the other hand, I don't really need all the advanced functionality that Apache Hadoop provides. And I don't want to set up a personal permanent cluster with my own installation of Hadoop or similar, because I will need to run the tasks only periodically and most of the time the servers will be idle, wasting money.
So, what would be the best product (or overall strategy) that could do the following:
Grab the given binaries + set of input files
Run the binaries on a predefined number of instances, using a recent Linux, ideally Ubuntu 15.10
Put the resulting files in a predefined location (S3 bucket?)
Shut everything down
I am sure I could write a number of scripts using the aws tool to achieve that manually, but I really don't want to reinvent the wheel. Any thoughts?
Thanks in advance!
Honestly that would be pretty easy to script, and you'll need to probably use scripting to grab the latest code on the servers when they start up anyway. I would suggest looking into defining an AutoScaling group with scheduled scaling policies. Alternatively you could have a Lambda function scheduled to run and issue the API command to create your instances.
You could either have a startup script on the server AMI, or simply pass a user-data script when you create the instances, that pulls down the binaries and input files and runs the command. The final step of the script could be to copy results to S3 and shutdown the server.
The (relatively new) AWS Batch is made for this purpose specifically.
I am alloted 'Automation of ETL processing using Scripting Technologies' as my Engineering Final Year project. However I don't have any idea about exactly what is this project supposed to do. I konw the basic concepts of ETL. But can anyone help me about what is meant by automation of these processes.
I am not asking for the implemetation. Just an overview of what needs to be done
If you mean automation of initialization, it could be done by one of several things: You could kick off the process with the task scheduler, you could have a folder watcher that starts a process when it find a file in the folder it's watching, you could have a switch that watches an object's name an when the name is "on" it knows to run and rename the object to "off" when it's done, or it could be kicked off by a database trigger, etc. Or are we talking about using the scripts to automate the steps necessary to complete the ETL? Sounds like you need to talk to the prof.
ETL tools generally have schedulers
and OS's have schedulers
ie Unix with contrab
all ETL processes should be idempotent.
are there any time based triggers in DB2?
You don't have to. There is usually a OS function available for that. If I am not mistaken it is at in unix and the scheduled jobs under windows (there is also a command line interface for windows).
However, I do remember that there was a scheduler in MS SQL server. After asking google, I learned that DB2 has something called task center. Not sure what that thing does or how you configure it (my UI does not open for some strange reason).
Without further details, I can't give you a specific answer. I'm guessing you're asking for "a trigger that is fired at a specific time" and not "a trigger that has different timing options, like fire before event or fire after event".
The answer to the former is no, triggers are not fired by external events like the time of day. Triggers are features specifically designed to respond to monitor for changes in tables. What you are looking for is some form of task/job scheduling, and there are many approaches you can use to accomplish this.
I can't really detail any options without knowing what platform you're running on (Unix, Linux, Mainframe, Windows?), so if you could clarify I might be able to help further. The version of DB2 you're running would also help.
Examples
If you're running DB2 on IBM's iSeries you would probably want to use the OS's native job scheduler (see http://publib.boulder.ibm.com/infocenter/iseries/v6r1m0/topic/rzaks/sc415324.pdf).
On Unix/Linux the popular option is cron (see http://www.adminschoice.com/crontab-quick-reference)
The typical solution for DB2 on windows, z/OS, and others is the Task Center utility (depending on your DB2 version, see http://publib.boulder.ibm.com/infocenter/db2luw/v8/index.jsp?topic=/com.ibm.db2.udb.taskcenter.doc/taskcenter/abouttaskcenter.htm)
Using Process Explorer (procexp.exe), especially with Google Chrome, child processes are called a Job. Same with Internet Explorer 8, but I noticed it first with Chrome.
What is a Job
What should I know about these things?
Why would (you|one) use them?
What scenarios should they be used?
What APIs are used.
I know the questions is a bit clumsy, please try and look past that. Thanks in advance.
I'm using WinXP by the way.
A Job under Process Explorer refers to Win32 Jobs. More information about this feature can be found here.
So,
1. What is a Job?
As above.
2. What should I know about these things?
If a job fails or becomes unstable, all processes it manages will become unstable or crash immediately.
3. Why would (you|one) use them?
They are interesting tools if my application/system fires up several processes. I can centralize certain tasks in one job and attach all processes to it. Like gracefully terminate all processes, manage their working sets, etc.
4. What scenarios should they be used?
Never did anything worthed using them. But as above. In applications or complex systems that fire up several processes. Under Chrome, for instance (since this is where you are seeing a job), it is quite possible the job is managing each process that is fired when you open a new tab.
5. What APIs are used?
The Win32 API
Ad 1/2. A job is a process with a job object assigned. They're used to manage groups of processes. One job object can have multiple process, but a process can be assigned to only one job object. You can also set several limits for the jobs, documented here.
Ad 5. CreateJobObject, AssignProcessToJobObject, SetInformationJobObject, TerminateJobObject, and few more, listed here.