it's a very hard dynamic programming question, and I want to share with you and we can discuss a little bit toward its solution：
You will put your new application to cloud server; you have to schedule your job in order to get lowest cost. you don't need to care about the number of jobs running at the same time on the same server. every job k is given by a release time sk, a deadline fk, and a duration dk with dk ≤ fk - sk. This job needs to be scheduled for an interval of dk consecutive minutes between time sk and fk. server company would charges per minute per server. You only need one virtual server and you can save money moving jobs from sk to fk around to maximize the amount of time without running any jobs or, in other word, to minimize the amount of time running one or more jobs. using dynamic programming to solve problem. Your algorithm should be polynomial in n, the number of jobs.
This is the problem of minimizing busy time.
See Theorem 17 of this paper:
Rohit Khandekar, Baruch Schieber, Hadas Shachnai, and Tami Tamir. Minimizing busy time in multiple machine real-time
scheduling. In Proceedings of the 30th Annual Conference on Foundations of Software Technology and Theoretical Computer
Science (FSTTCS), pages 169 – 180, 2010
For a description of a polynomial time algorithm.
The key is:
To realize there are only certain interesting times that need to be considered (if you have a schedule, consider delaying each busy interval until you hit a deadline for one of the jobs being processed)
To consider when the longest duration job is done. This splits the problem into two pieces; before and after, which can be solved independently in the normal dynamic programming fashion.
Related
In Multilevel Feedback Scheduling at the base level queue, the processes circulate in round robin fashion until they complete and leave the system. Processes in the base level queue can also be scheduled on a first come first served basis.
Why can't they be scheduled on Shortest Job First (SJF) algorithm instead of First Come First Serve (FCFS) algorithm which seems to improve average performance of the algorithm.
One simple reason:
The processes fall in the base level queue after they fail to finish in the time quantum alloted to them in the higher level queues. If you implement SJF algorithm in the base level queue, you may starve a process because shorter job may keep coming before a longer executing process ever gets the CPU.
The SJF algorithm gives more througput, only when processes differ a lot in their burst time. However its not always the case that it will perform better than FCFS. Take a loot at this answer.
Since in Multilevel Feedback Scheduling algorithm, all the processes that are unable to complete execution within defined time quantum of first 2 queues, are put to the last queue having FCFS, its very likely that they all have large CPU bursts and therefore wont differ much in their burst time. Hence, its preferred to have FCFS, scheduling for the last queue.
I'm studying task-based parallel computing and got interested in a variation of the old project management problem -- the critical path of an activity-on-vertex (AOV) project network, which can be calculated using the topological sorting algorithm if there's no deadlock cycle. The total time of those activities on a critical path gives the minimum completion time of the project.
But this is assuming we always have enough workers simultaneously finishing the activities with no dependence on each other. If the number of workers (processors/cores) available is finite, certain activities can wait not because some activities they depend on have not yet been finished, but simply because all workers are now busy doing other activities. This is a simplified model for today's multi-core parallel computing. If there's only one worker who has to do all the activities, the project completion time is the total time of all activities. We are back to single-core serial computing that way.
Is there an efficient algorithm that gives the minimum completion time of an AOV network given a finite number of workers available? How should we wisely choose which activities to do first when the doable activities is more than the number of workers so as to minimize the idling time of workers later on? The minimum time should be somewhere in between the critical path time (infinite workers) and the total time of all activities (one worker). It should also be greater than equal to the total time divided by the number of workers (no idling). Is there an algorithm to get that minimum time?
I found a C++ conference video called "work stealing" that almost answers my question. At 18:40, the problem is said on the slide to be NP-hard if activities cannot be paused, further divided, or transferred from worker to worker. Such restrictions make decisions of which workers to finish which jobs (activities) too hard to make. Work stealing is therefore introduced to avoid making such difficult decisions beforehand. Instead, it makes such decions no longer crucial so long as certain apparent greedy rules are followed. The whole project will be always finished as soon as possible under the constraint of either the critical path or the no-idling time of the finite number of workers or both. The video then goes on talking about how to make the procedure of "work stealing" between different workers (processors) more efficient by making the implementation distributed and cache-friendly, etc.
According to the video, future C++ shared-memory parallel coding will be task-based rather than loop-based. To solve a problem, the programmer defines a bunch of tasks to finish and their dependence relations to respect, and then the coding language will automatically schedule the tasks on multiple cores at run time in a flexible way. This "event-driven"-like way of implementing a flexible code by a distributed task queuing system will become very useful in parallel computing.
When an optimization problem is NP-hard, the best way to solve it is to find ways to avoid it.
There's a large set of objects. Set is dynamic: objects can be added or deleted any time. Let's call the total number of objects N.
Each object has two properties: mass (M) and time (T) of last update.
Every X minutes a small batch of those should be selected for processing, which updates their T to current time. Total M of all objects in a batch is limited: not more than L.
I am looking to solve three tasks here:
find a next batch object picking algorithm;
introduce object classes: simple, priority (granted fit into at least each n-th batch) and frequent (fit into each batch);
forecast system capacity exhaust (time to add next server = increase L).
What kind of model best describes such a system?
The whole thing is about a service that processes the "objects" in time intervals. Each object should be "measured" each N hours. N can vary in a range. X is fixed.
Objects are added/deleted by humans. N grows exponentially, rather slow, with some spikes caused by publications. Of course forecast can't be precise, just some estimate. M varies from 0 to 1E7 with exponential distribution, most are closer to 0.
I see there can be several strategies here:
A. full throttle - pack each batch as much as close to 100%. As N grows, average interval a particular object gets a hit will grow.
B. equal temperament :) - try to keep an average interval around some value. A batch fill level will be growing from some low level. When it reaches closer to 100% – time to get more servers.
C. - ?
Here is a pretty complete design for your problem.
Your question does not optimally match your description of the system this is for. So I'll assume that the description is accurate.
When you schedule a measurement you should pass an object, a first time it can be measured, and when you want the measurement to happen by. The object should have a weight attribute and a measured method. When the measurement happens, the measured method will be called, and the difference between your classes is whether, and with what parameters, they will reschedule themselves.
Internally you will need a couple of priority queues. See http://en.wikipedia.org/wiki/Heap_(data_structure) for details on how to implement one.
The first queue is by time the measurement can happen, all of the objects that can't be measured yet. Every time you schedule a batch you will use that to find all of the new measurements that can happen.
The second queue is of measurements that are ready to go now, and is organized by which scheduling period they should happen by, and then weight. I would make them both ascending. You can schedule a batch by pulling items off of that queue until you've got enough to send off.
Now you need to know how much to put in each batch. Given the system that you have described, a spike of events can be put in manually, but over time you'd like those spikes to smooth out. Therefore I would recommend option B, equal temperament. So to do this, as you put each object into the "ready now" queue, you can calculate its "average work weight" as its weight divided by the number of periods until it is supposed to happen. Store that with the object, and keep a running total of what run rate you should be at. Every period I would suggest that you keep adding to the batch until one of three conditions has been met:
You run out of objects.
You hit your maximum batch capacity.
You exceed 1.1 times your running total of your average work weight. The extra 10% is because it is better to use a bit more capacity now than to run out of capacity later.
And finally, capacity planning.
For this you need to use some heuristic. Here is a reasonable one which may need some tweaking for your system. Maintain an array of your past 10 measurements of running total of average work weight. Maintain an "exponentially damped average of your high water mark." Do that by updating each time according to the formula:
average_high_water_mark
= 0.95 * average_high_water_mark
+ 0.5 * max(last 10 running work weight)
If average_high_water_mark ever gets within, say, 2 servers of your maximum capacity, then add more servers. (The idea is that a server should be able to die without leaving you hosed.)
I think answer A is good. Bin packing is to maximize or minimize and you have only one batch. Sort the objects by m and n.
I'm doing some administration work for an aviation transport company. They build aircraft containers and such here. One of the things they want me to code is a order optimization script that the guys on the floor can use to get the most out of the given material. To give a simple overview: say we order a certain amount beams that are 10 meters per unit. We need beam chunks of 5x 6m, 10x 3.5m, 4x 3m, which are acquired by cutting the 10m in smaller parts. What would be the minimum amount of 10m beams we need to order?
There are some parallels with the multiprocessor job scheduling problem (one beam is a processor, each chunk a job), although that focusses on minimizing the time required to perform all jobs instead of minimizing the amount of processors needed to perform all jobs within a pre-set time. The multiprocessor job scheduling problem is in NP-complete, but I wonder if my variation of the problem is too. Does anybody know similar problems and methods for solving them?
This problem is exactly: http://en.wikipedia.org/wiki/Cutting_stock_problem (more generally http://en.wikipedia.org/wiki/Bin_packing_problem). You can use any old ILP solver. I like http://lpsolve.sourceforge.net/5.5/, its quite friendly to use.
I have a scheduling problem where new jobs (sets of tasks whose execution is sequentially connected) arrive every few seconds or so.
Each job requires some resources to be allocated at known intervals.
For example:
Job j1 is a set of tasks for which we reserve resources {r1, r2, r3}
on a known scheduling pattern:
r1:[t0 .. t1=t0+td1],
r2:[t2=t1+td2+i2 .. t3=t2+td3]
t0 being the start time of execution
td1 is the length of the resource allocation for r1
t1 being the end time of the resource allocation for r1
i1 is length of the waiting perioid between r1, r2 and so on.
In the example, a new job j2 is being scheduled right after j1 execution has started.
The earliest start time for j2 is t1.
A job may take some minutes of execution most of which consists of waiting.
I have a scheduler that looks at the current reservation table and decides which is the earliest possible starting moment for a new job with fixed allocation times and waiting periods and makes the reservations accordingly.
(But in reality, the waiting period doesn't really need to be fixed - but within some percentage (maybe 5%) and there may be alternatives to resource usage, for example, if resource r3.1 is booked, then 3.2 may be used as such to achieve the same thing.)
However, if the scheduler is required (yes, it's been suggested) to be able to dynamically adjust all the schedule allocations when a new job arrives to maximize the total work done (in a day) by taking advantage of the fact that the waiting times need not be exactly as given and the possibility to parallel execution with some resrouce duplicates (3.1/3.2), then I'd be looking at a completely different scheduling scheme (than my current start-as-soon-as-possible approach).
What scheduling scheme would you call that then?
Any suggestions on approaching the (new) problem?
As for your question regarding "alternatives to resource usage":
The pattern most commonly implemented to tackle that sort of problem is the Object Pool Pattern
The most widely known example for this probably is the ThreadPool
I suggest you implement a ResourcePool class with a int GetResource(ResourceType type, int durationInSeconds) method.
The return value indicates when the next resource of the given ResourceType will be available
You could be dealing with the RCPSP (Resource Constrained Project Scheduling Problem). Solution techniques range from integer programming and constraint programming to various heuristics. The technique depends the details such as planning horizon, how tasks/jobs use/share resources, how fast you need an solution schedule, etc.
see:
https://developers.google.com/optimization/scheduling/job_shop
http://www.laas.fr/files/ROC/2014-Presentations/MILP-RCPSP-PMS2014.pdf