I'm working on a project with many unknowns like moving the app from one platform to another.
My original estimations are way off and there is no way I can really know for sure when this will end.
How can i deal with the inability to estimate such a project. It's not that I'm adding a button to a screen or designing a web site, or creating and app or even fixing bugs. These are not methods with bugs, these are assumptions made in the overall code, which are not correct anymore and are found step by step and each analyzed and mitigated with many more unknowns.
I happened to write a master thesis about software-estimation and there are lessons I've learned:
-1st Count, 2nd compute, 3rd judge - this means: first try to identify items in your work which are countable e.g files, classes, LOCs, UIs, etc. Then calculate using this data the effort (in person/days). Use judgement as the last ressort.
-Document your estimation! Show numbers. This minimizes your risk, thus you will present results not as your opinion, but as more or less objective figures. (In general, the more paper the cleaner the backside)
-Estimation is not a commitment. Commitment is one number, estimation is always a range - so give your estimation as a range ( use cone of uncertainty to select the range properly http://www.construx.com/Page.aspx?hid=1648 )
-Devide: Use WBS, devide your work in small pieces and estimate them separately. The granulity depents on the entire length, but at most a working-package soultn't be bigger than 10% of entire effort.
-Estimate effort first, then schedule, then costs.
-Consider estimation as support for planing, reestimate on each project phase (s. cone of uncertainty).
I would suggest the book http://www.stevemcconnell.com/est.htm which deals all these points, in particular how to deal with bosses, who try to pull a commitment from you.
Regards,
Valentin Heinitz
There's no really right answer for coming up with an accurate estimation, because there's no way to know it.
as for estimating the work itself, think about how each step can be divided into separate sub-steps, and break those down even smaller, until you can get a fair picture of as much of the work as you can, with chunks small and discreet enough to give sound estimates for. If you can, come up with both an expected time and a worst-case time, to get a range of where you could land.
Another way to approach this is to ignore the old system. It sounds like a headache. Make an estimate of scraping the old system and implementing a new one from scratch, or integrating a 3rd party, off the shelf solution. If there's a case to be made for this, it is worth at least investigating it.
Sounds like a post for postsecret not SO. :)
I would tell him that it will be done when its done, and if thats not good enough, he can learn to program and help you. Then again, I think that you might get fired, but hey that sounds like it might be better.
Tell him more or less what you told us. The project is too volatile too give an accurate estimate and the best you can do is give an estimate for a given task. As long as the number of tasks is unknown so will be the estimate. If he is at all worth his salary he would rather hear this than some made up number. This is not uncommon when dealing with a large legacy code base.
It's not that I'm adding a button to a screen or designing a web site,
or creating and app or even fixing bugs.
That is a real problem. You can not estimate what you don't have experience in. The only thing you can do is pad your estimate until you think it is a reasonable amount of time. The more unknowns you think there are the more you pad. The less you know about it the more you pad.
I read the below book and it spoke at length about accuracy vs precision. Basically you can be accurate but have a very large range. For instance you can be certain the task will be between 1 day and 1 year to complete. That is not very precise but it is really accurate.
Software Estimation Demystifying...
Some tips for estimating
Related
I have some data about programmer actions inside an ide. From this data I am trying to make a good algorithm to calculate a programmers efficiency.
If we consider
efficiency = useful energy out / energy in
I made this rough equation:
energy in = active time(run events x code editing time)
Basically its the the time where stuff is actually being done by the programmer multiplied by run events like
debugging,build etc x the time where the programmer is actually editing code.
useful energy out = energy in - (#unsuccessfulbuilds + abortedtestruns
+ debuggerusetime)
Useful energy out is basically energy in minus things that I consider to be inefficient.
Can anyone see how to improve this, particularly from a mathematical point of view. Maths isn't my strong point and am not sure if I should use some sort of weighting for the equations and how to do this correctly. Also, I'm thinking of how to make it that whats minused from energy in in the useful energy out equation cant end up as less than 0. Can anyone give a hand with these questions ?
Your "algorithm" is completely arbitrary, making judgement of value over things that are inocuous to whatever you called "efficient/inneficient", and will endup with a completely incoherent final value after being calculated. Compilation time? So the first compilation of a C++ plugin that takes 30+ minutes is good? Debugging time is both efficient and inefficient in your proposal.
A programmer that codes for 10 minutes and make 6 consecutive builds with close to no changes will have the same output as the guy that code for 60 minutes.
I suggest you look firts to what is a good use of a programmers time, how other programs contabilize programmers efficiency. Etc.
Just on a side note, to create a model of efficiency of work of a highly technical and creative field, you must understand quite well math, statistics and project management. Thats why good scrum masters are so sought after.
Anyway, what you propose is not an algorithm, but a scoring system, usually algorithms do make use of scoring systems to help their internal rules work out the best solution based on the scoring. The scoring is just a value, while the algorithm is a process to an end.
I am working on a predictor for learning the most likely period for grape harvesting, depending on weather and on the characteristics of grape, namely sugar level, Ph, acidity. I've got two datasets and I am thinking of how to merge them together: one is the pre-harvest analysis data of some Italian vineyards in the 2003-2013 period, the other is the weather on that decade. What I want to do is learning from my samples when to harvest, given a range for the optimal sugar level, Ph and acidity, and given a weather forecast.
I thought that some Reinforcement Learning approach could work. Since the pre-harvest analysis are done about 5 times during the grape maturation period, I thought that those could be states I step in, while the weather conditions could be the "probabilities" of going from a state to another.
Yet I am not sure of what algorithm would be the best as every state and every "probability" depends on several variables. I was told that Hidden Markov Model would work, but it seems to me that my problem doesn't fit the model perfectly.
Do you have any suggestion? Thx in advance
This has nothing to do with the actual algorithm, but the problem you are going to run into here is that weather is extremely local. One vineyard can have completely different weather than another only a mile away from it, believe or not. If you put rain gauges at each vineyard, you will find this out. To get really good results you need to have a mini weather station at each vineyard. Absent this, your best option is to use only vineyards in the immediate vicinity of the weather measurements. For example, if your data is from an airport, only use vineyards right next to the airport.
Reinforcement learning is appropriate when you can control the action. It is like a monkey pushing buttons. You push a button and get shocked, so you don't push that button again. Here you have a passive data set and cannot conduct experimental actions, so reinforcement learning does not apply.
Here you have a complex set of uncontrolled inputs, the weather data, a controlled input (harvest time), and several output parameters, sugar etc. Given that data, you want to predict what harvest time to use for some future, unknown weather pattern.
In general, what you are doing is sensitivity analysis: trying to figure out how your factors affected the outcome that occurred. The tricky part is that the outcomes may be driven by some non-obvious pattern. For example, maybe 3 weeks of drought, followed by 2 weeks of heavy rain implies the best harvest will be 65 days hence, or something like that.
So, what you have to do is featurize the data to characterize it in possible likely ways, then do a sensitivity analysis. If the analysis has a strong correlation, then you have found a solution. If it does not, then you have to find a different way to featurize the data. For example, your featurization might be number of days with rain over 2 inches, or it might be most number of days without rain, or it might be total number of days with bright sunshine. Possibly multiple features might combine to make a solution. The options are limited only by your imagination.
Of course, as I was saying above, the fly in the ointment is that your weather data will only roughly approximate the real and actual weather at the particular vineyard, so there will be noise in the data, possibly so much noise as to make getting a good result impossible.
Why you actually don't care too much about the weather
Getting back to the data, having unreliable weather information is actually not a problem, because you actually don't care too much about the weather. The reason is two-fold. First of all, the question you are trying to answer is not when to harvest the grapes, it is whether to wait to harvest or not. The vintner can always measure the current sugar of the grapes. So, he just has to decide, "Should I harvest the grapes now with sugar X%, or should I wait and possibly get a better sugar Z% later? To answer this question the real data you need is not the weather, it is a series of sugar/acidity readings taken over time. What you want to predict is whether, given a situation, the grapes will get better or whether they will get worse.
Secondly, grapevines have an optimal amount of moisture they like. If the vine gets too dry, that is bad, if it gets too wet that is bad. You cannot predict how moist a vine is from the weather. Some soils hold moisture well, others are sandy. A sandy vineyard will require more rain than a clay vineyard to have the same moisture levels. Also, the vintner can water his vineyards, completely invalidating the rainfall pattern. Therefore, weather is pretty much a non-factor.
I agree with Tyler that from a feasible standpoint weather might harm your analysis. However, I think this is for you to test and find out!- there could be some interesting data that comes out of it.
I'm not sure exactly what your test is, but a simple way to start perhaps is to make this into a classification problem using svm (or even logistic regression since you want probabilities) and use all the data as the input for the algorithm- assuming you know which years were good harvest years or not. You could even test each variable individually and see how it effects your performance. I suggest you go this way if you can just because there's massive amounts of sources on the net and people here on SO that can help you tune your algo.
When you have a handle on this, I would, as you seem to have been suggested before, try the HMM- as it will tell you which day was probably the best for the harvest. This is where the weather might hurt, but you'll come to understand more about your data from the simpler experiments.
The think I've learned about machine learning is that while there are guidelines for when to choose which algorithm its not always set in stone and you can change your question slightly and try a new approach to the problem, depending how much freedom you have to play with the data. Good luck and have fun!
My friend works for a non-profit organization working to stop the illegal exploitation of minors over sites such as craigslist.org, which is one of the more popular mediums. The question is whether or not it is possible, now or in the near future, to develop an algorithm to analyze a photo of a person and return a prediction of their relative age.
It sounds like a mammoth task. My only thought was some sort of Bayesian probability system. I know even people often have trouble judging someone's age but Bayesian spam filters are advertised as being "10 times as accurate as a human" so maybe it's possible?
I am pretty inexperienced though. I would appreciate it if someone else could suggest whether or not this is feasible and if so how and when?
EDIT: Thank you everyone for the responses. Smoore that study was very helpful but I think Hal's solution is the most practical for the time being.
Here's a possible (left-field) solution. Perhaps, you could tie it into some type of a captcha solution for the site itself. Prompt new users with images of other new users with the question: "Is this person over 18?". It's true that a 50% success rate is not a very effective captcha system, but it's a start.
Coupled with some other checks or repetitive checks and it could work. You could display the image to a number of new users, and base the result on a certain threshold. If, 8 out of 10 people flagged a certain image as not a minor, than it's probably pretty safe they are of age.
But, this whole system can be circumvented by simply uploading someone else's image so I'm not sure how effective any of this really is. :)
I expect it would be pretty hard to get right. Consider this set of photos where the same model is made up to look very different ages.
There are algorithm to reliably determine the attractiveness of a face. See acm.org and uni-regensburg.de. It wouldn't be too much of a stretch to imagine an algorithm which could predict age.
Characteristics such as smoothness would probably have a strong correlation with age. It would probably take a great deal of effort to be more reliable than your average carney though.
I think you would need some input from a forensic anthropoligist ( or at least an anatomist).
Differnet parts of the body grow at different rates so it might be possible to do something like size of head vs. shoulder width, arm length vs. body width.
Unfortunately it sounds like he is trying to differentiate between say a 14 year olds and 18 year olds. Which is only a four year difference, variations in genetic makeup and nutitrition would probaly give any system an accuracy of +/- 20% which would equate to three years for this age group.
On the other hand if you had a large sample of photos then you could account for the variance statisticaly and get a pretty good idea whether a site was likely to be exploiting minors systematicaly.
The direct answer to your question is that no, no such algorithm will exist in the near future, and is probably impossible to achieve with any accuracy without strong AI.
That said, a practical solution to your problem is probably the amazon mechanical turk:
http://mturk.com
There, you can pay a small fee to have real people complete a task for you. I'd probably set your task up so that you paid $0.02 to have a person estimate the age of maybe 5 faces at a time. You could double or triple check your results with other workers, particularly for those faces who seemed close to your age limit. This is probably your only practical solution other than hiring minimum wage interns to manually review all submissions.
Use mechanical turk
In this study they tried it by analysing facial geometry and wrinkle features. Problem is this would be affected by shot angle, lighting, etc.
In some theoretical sense it is probably possible. For all practical purposes though, it is currently impossible.
Mammoth is an understatement I think. "Giant glacier" or "moon" might be more appropriate.
This isn't to say it wouldn't be worth looking into but I have a feeling you'd be in for a lot of man hours before you came up with something remotely useful.
I don't think it's something that a computer could do with any degree of accuracy. It's even really hard for people to do. I mean, have you been the the liquor store lately, they are supposed to ask for ID from anybody who looks under 25 (drinking age is 19 here). Apparently some 40 year olds don't look old enough. Telling somebody's age just by looking at them is a very hard thing to do. Especially when you get into to erotic picture arena, where they are trying to make models seem younger than they really are.
I think you will also have difficulties with different composited pictures. For instance angles on a face, different lighting, as well as context and probably most of all... image quality/resolution. It's a lot easier to work with a 800x600 pic then it is to work with a 320x240. The algorithm is only as good as the subject.
I cannot see this approach (a software solution to measuring age) being very effective. I like the idea of users flagging images - a human being can discern age many times more effectively then any algorithm.
Practical approach aside, I'd advice against trying to develop anything in that direction for now.
Few reasons:
1. guessing someone's age is not a grateful task
2. "biological" age and "calendar" age of people vary greatly - I know people who are 30 and are still asked for an ID when buying liquor, and some who are barely 18 and already look over 30
3. some people's looks don't change over time - they just have that kind of looks
4. nowadays, everyone's working to look as young as they can - so basically, you've got the whole industry working against you :(
Anyways, to cut long story short, I don't think it's feasible for now.
A neural net is a reasonable approach, you would need a training set of pictures of people with known ages and a bit of image processing to remove hats etc.
edit: Question changed?
You might be ale to classify someone as 20-30 or 40-50 on a CCTV but you aren't going to be ale to tell if a model is 17 or 18 in a posed photo.
Just like nearly all advanced tasks in image classification this topic is still in research. Judging from this paper it is possible to do it but non-trivial, also you have to have a lot of (manually) annotated training data. Without any knowledge of this field and no experience in image processing this task is going to take you several months.
Develop a classification algorithm that bases a heuristic on many values of the pictures, amount of pixels that are dark within the face area (possibly wrinkles), and the color of the hair. These values should fall within a general area of any profile-esque picture, if you want to be fancy, carry weights with these values and develop a type of game tree that would be able to search hundreds of thousands of images quickly, finding where this image "falls" in the tree within an age-specific set of values.
Some Japanese cigarette vending machines do this. Not terribly well by all accounts, but then it probably doesn't matter since, as Hal mentioned, the easiest hack is just to use someone else's image...
Impossible is nothing, Only amount of efforts changes :
I think it would be near impossible if you target one particular feature of face.
you have to consider multiple factor, So decision will be lying in a matrix and you have to feed multiple things and you will get your answer i would enlist some feature :
1) Beard (Detect face , Now detect beard on face , Help full in distinguish male/female
/childern )
2) Hair
3) Wrinkles
4) Size of face
5) Ration between height and breadth of face
It would be a tough assignment but algorithm can be developed.
As of now, this is possible with 90% accuracy. Yes. please refer the following link..
http://www.omron.com/r_d/coretech/vision/okao.html
ok, so i have been working on my chess program for a while and i am beginning to hit a wall. i have done all of the standard optimizations (negascout, iterative deepening, killer moves, history heuristic, quiescent search, pawn position evaluation, some search extensions) and i'm all out of ideas!
i am looking to make it multi-threaded soon, and that should give me a good boost in performance, but aside from that are there any other nifty tricks you guys have come across? i have considered switching to MDF(f), but i have heard it is a hassle and isn't really worth it.
what i would be most interested in is some kind of learning algorithm, but i don't know if anyone has done that effectively with a chess program yet.
also, would switching to a bit board be significant? i currently am using 0x88.
Over the last year of development of my chess engine (www.chessbin.com), much of the time has been spent optimizing my code to allow for better and faster move searching. Over that time I have learned a few tricks that I would like to share with you.
Measuring Performance
Essentially you can improve your performance in two ways:
Evaluate your nodes faster
Search fewer nodes to come up with
the same answer
Your first problem in code optimization will be measurement. How do you know you have really made a difference? In order to help you with this problem you will need to make sure you can record some statistics during your move search. The ones I capture in my chess engine are:
Time it took for the search to
complete.
Number of nodes searched
This will allow you to benchmark and test your changes. The best way to approach testing is to create several save games from the opening position, middle game and the end game. Record the time and number of nodes searched for black and white.
After making any changes I usually perform tests against the above mentioned save games to see if I have made improvements in the above two matrices: number of nodes searched or speed.
To complicate things further, after making a code change you might run your engine 3 times and get 3 different results each time. Let’s say that your chess engine found the best move in 9, 10 and 11 seconds. That is a spread of about 20%. So did you improve your engine by 10%-20% or was it just varied load on your pc. How do you know? To fight this I have added methods that will allow my engine to play against itself, it will make moves for both white and black. This way you can test not just the time variance over one move, but a series of as many as 50 moves over the course of the game. If last time the game took 10 minutes and now it takes 9, you probably improved your engine by 10%. Running the test again should confirm this.
Finding Performance Gains
Now that we know how to measure performance gains lets discuss how to identify potential performance gains.
If you are in a .NET environment then the .NET profiler will be your friend. If you have a Visual Studio for Developers edition it comes built in for free, however there are other third party tools you can use. This tool has saved me hours of work as it will tell you where your engine is spending most of its time and allow you to concentrate on your trouble spots. If you do not have a profiler tool you may have to somehow log the time stamps as your engine goes through different steps. I do not suggest this. In this case a good profiler is worth its weight in gold. Red Gate ANTS Profiler is expensive but the best one I have ever tried. If you can’t afford one, at least use it for their 14 day trial.
Your profiler will surly identify things for you, however here are some small lessons I have learned working with C#:
Make everything private
Whatever you can’t make private, make
it sealed
Make as many methods static as
possible.
Don’t make your methods chatty, one
long method is better than 4 smaller
ones.
Chess board stored as an array [8][8]
is slower then an array of [64]
Replace int with byte where possible.
Return from your methods as early as
possible.
Stacks are better than lists
Arrays are better than stacks and
lists.
If you can define the size of the
list before you populate it.
Casting, boxing, un-boxing is evil.
Further Performance Gains:
I find move generation and ordering is extremely important. However here is the problem as I see it. If you evaluate the score of each move before you sort and run Alpha Beta, you will be able to optimize your move ordering such that you will get extremely quick Alpha Beta cutoffs. This is because you will be able to mostly try the best move first.
However the time you have spent evaluating each move will be wasted. For example you might have evaluated the score on 20 moves, sort your moves try the first 2 and received a cut-off on move number 2. In theory the time you have spent on the other 18 moves was wasted.
On the other hand if you do a lighter and much faster evaluation say just captures, your sort will not be that good and you will have to search more nodes (up to 60% more). On the other hand you would not do a heavy evaluation on every possible move. As a whole this approach is usually faster.
Finding this perfect balance between having enough information for a good sort and not doing extra work on moves you will not use, will allow you to find huge gains in your search algorithm. Furthermore if you choose the poorer sort approach you will want to first to a shallower search say to ply 3, sort your move before you go into the deeper search (this is often called Iterative Deepening). This will significantly improve your sort and allow you to search much fewer moves.
Answering an old question.
Assuming you already have a working transposition table.
Late Move Reduction. That gave my program about 100 elo points and it is very simple to implement.
In my experience, unless your implementation is very inefficient, then the actual board representation (0x88, bitboard, etc.) is not that important.
Although you can criple you chess engine with bad performance, a lightning fast move generator in itself is not going to make a program good.
The search tricks used and the evaluation function are the overwhelming factors determining overall strength.
And the most important parts, by far, of the evaluation are Material, Passed pawns, King Safety and Pawn Structure.
The most important parts of the search are: Null Move Pruning, Check Extension and Late Move reduction.
Your program can come a long, long way, on these simple techniques alone!
Good move ordering!
An old question, but same techniques apply now as for 5 years ago. Aren't we all writing our own chess engines, I have my own called "Norwegian Gambit" that I hope will eventually compete with other Java engines on the CCRL. I as many others use Stockfish for ideas since it is so nicely written and open. Their testing framework Fishtest and it's community also gives a ton of good advice. It is worth comparing your evaluation scores with what Stockfish gets since how to evaluate is probably the biggest unknown in chess-programming still and Stockfish has gone away from many traditional evals which have become urban legends (like the double bishop bonus). The biggest difference however was after I implemented the same techniques as you mention, Negascout, TT, LMR, I started using Stockfish for comparison and I noticed how for the same depth Stockfish had much less moves searched than I got (because of the move ordering).
Move ordering essentials
The one thing that is easily forgotten is good move-ordering. For the Alpha Beta cutoff to be efficient it is essential to get the best moves first. On the other hand it can also be time-consuming so it is essential to do it only as necessary.
Transposition table
Sort promotions and good captures by their gain
Killer moves
Moves that result in check on opponent
History heuristics
Silent moves - sort by PSQT value
The sorting should be done as needed, usually it is enough to sort the captures, and thereafter you could run the more expensive sorting of checks and PSQT only if needed.
About Java/C# vs C/C++/Assembly
Programming techniques are the same for Java as in the excellent answer by Adam Berent who used C#. Additionally to his list I would mention avoiding Object arrays, rather use many arrays of primitives, but contrary to his suggestion of using bytes I find that with 64-bit java there's little to be saved using byte and int instead of 64bit long. I have also gone down the path of rewriting to C/C++/Assembly and I am having no performance gain whatsoever. I used assembly code for bitscan instructions such as LZCNT and POPCNT, but later I found that Java 8 also uses those instead of the methods on the Long object. To my surprise Java is faster, the Java 8 virtual machine seems to do a better job optimizing than a C compiler can do.
I know that one improvement that was talked about at the AI courses in university where having a huge database of finishing moves. So having a precalculated database for games with only a small number of figures left. So that if you hit a near end positioning in your search you stop the search and take a precalculated value that improves your search results like extra deepening that you can do for important/critique moves without much computation time spend. I think it also comes with a change in heuristics in a late game state but I'm not a chess player so I don't know the dynamics of game finishing.
Be warned, getting game search right in a threaded environment can be a royal pain (I've tried it). It can be done, but from some literature searching I did a while back, it's extremely hard to get any speed boost at all out of it.
Its quite an old question, I was just searching questions on chess and found this one unanswered. Well it may not be of any help to you now, but may prove helpful to other users.
I didn't see null move pruning, transposition tables.. are you using them? They would give you a big boost...
One thing that gave me a big boost was minimizing conditional branching... Alot of things can be precomputed. Search for such opportunities.
Most modern PCs have multiple cores so it would be a good idea making it multithreading. You don't necessarily need to go MDF(f) for that.
I wont suggest moving your code to bitboard. Its simply too much work. Even though bitboards could give a boost on 64 bit machines.
Finally and most importantly chess literature dominates any optimizations we may use. optimization is too much work. Look at open source chess engines, particularly crafty and fruit/toga. Fruit used to be open source initially.
Late answer, but this may help someone:
Given all the optimizations you mentioned, 1450 ELO is very low. My guess is that something is very wrong with your code. Did you:
Wrote a perft routine and ran it through a set of positions? All tests should pass, so you know your move generator is free of bugs. If you don't have this, there's no point in talking about ELO.
Wrote a mirrorBoard routine and ran the evaluation code through a set of positions? The result should be the same for the normal and mirrored positions, otherwise you have a bug in your eval.
Do you have a hashtable (aka transposition table)? If not, this is a must. It will help while searching and ordering moves, giving a brutal difference in speed.
How do you implement move ordering? This links back to point 3.
Did you implement the UCI protocol? Is your move parsing function working properly? I had a bug like this in my engine:
/* Parses a uci move string and return a Board object */
Board parseUCIMoves(String moves)// e2e4 c7c5 g1f3 ...{
//...
if (someMove.equals("e1g1") || someMove.equals("e1c1"))
//apply proper castle
//...
}
Sometimes the engine crashed while playing a match, and I thought it was the GUI fault, since all perft tests were ok. It took me one week to find the bug by luck. So, test everything.
For (1) you can search every position to depth 6. I use a file with ~1000 positions. See here https://chessprogramming.wikispaces.com/Perft
For (2) you just need a file with millions of positions (just the FEN string).
Given all the above and a very basic evaluation function (material, piece square tables, passed pawns, king safety) it should play at +-2000 ELO.
As far as tips, I know large gains can be found in optimizing your move generation routines before any eval functions. Making that function as tight as possible can give you 10% or more in nodes/sec improvement.
If you're moving to bitboards, do some digging on rec.games.chess.computer archives for some of Dr. Robert Hyatts old posts about Crafty (pretty sure he doesn't post anymore). Or grab the latest copy from his FTP and start digging. I'm pretty sure it would be a significant shift for you though.
Transposition Table
Opening Book
End Game Table Bases
Improved Static Board Evaluation for Leaf Nodes
Bitboards for Raw Speed
Profile and benchmark. Theoretical optimizations are great, but unless you are measuring the performance impact of every change you make, you won't know whether your work is improving or worsening the speed of the final code.
Try to limit the penalty to yourself for trying different algorithms. Make it easy to test various implementations of algorithms against one another. i.e. Make it easy to build a PVS version of your code as well as a NegaScout version.
Find the hot spots. Refactor. Rewrite in assembly if necessary. Repeat.
Assuming "history heuristic" involves some sort of database of past moves, a learning algorithm isn't going to give you much more unless it plays a lot of games against the same player. You can probably achieve more by classifying a player and tweaking the selection of moves from your historic database.
It's been a long time since I've done any programming on any chess program, but at the time, bit boards did give a real improvement. Other than that I can't give you much advise. Do you only evaluate the position of pawns? Some (slight) bonuses for position or mobility of some key pieces may be in order.
I'm not certain what type of thing you would like it to learn however...
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
Suppose you are a project manager. You can estimate an effort in days for specific task for specific developer. After performing estimation you obtain some min and max values.
After this you delegate a task to developer. Actually you also set up deadline.
Which estimation is better to use when set up deadline: min or max?
As I see min estimation can result in stress for developer, max estimation can result in using all the time which is allocated to developer even if task can be complete faster (so called Student syndrome).
Which other pros and cons of two approaches?
EDIT:
Small clarification: I speak about setting up deadlines for subordinates when delegating the task, NOT for reporting to my boss.
EDIT:
To add one more clarification: I can keep in mind my real estimation, provide to boss slightly larger estimation, to subordinates - slightly smaller.
And this questions touches the following thing: is it good idea to provide to developer underestimation to make him working harder?
You should use the best guess which is a function of the min and max estimates* - not just the simple average -
best_guess = (min * min_weighting + max * max_weighting) / divisor*
* Tom Neyland suggests it should be (min_weighting + max_weighting). Actually I'm not sure whether that is correct, but it's probably more correct than my original divisor of 2.0.
The weighting you give to the min and max values will depend on the complexity of the task, the risks associated with the task, the likelihood of the risks occuring, the skill of the developer, etc. and will vary from organisation to organisation and from project to project. If you keep a record of your previous estimates and the actual time each took you'll be able to refine these estimates over time.
You should also use these values, plus a confidence value, when talking to senior management and customers. While giving the max and delivering early is not the same as giving the min and delivering late, it still shows that you don't have control over your development.
Giving the confidence value and an idea of the risks will also help manage expectations so if there are problems they're not unexpected.
* These min and max estimates will be got by various means - asking the developers, past experience etc. If polling developers then the actual min and max values should be treated as outliers and either discarded or modified in some way. What I mean here are the values you get from phrases like "it'll take 2 weeks if all goes well or a month if we hit some snags". So the values you plug into the formula are not the raw numbers.
Use neither min nor max but something in between.
Erring on the side of overestimation is better. It has much nicer cost behavior in the long term.
To overcome the stress due to underestimation, people may take shortcuts that are not beneficial in the long term. For example, taking extra technical debt thast has to be paid back eventually, and it comes back with an interest. The costs grow exponentially.
The extra cost from inefficiency due to student's syndrome behaves linearly.
Estimates and targets are different. You (or your managers and customers) set the targets you need to achieve. Estimates tell you how likely you are to meet those targets. Deadline is one sort of target. The deadline you choose depends on what kind of confidence level (risk of not meeting the deadline) you are willing to accept. P50 (0.5 probability of meeting the deadline) is commonplace. Sometimes you may want to schedule with P80 or some other confidence level. Note that the probability curve is a long-tailed one and the more confidence you want, the longer you will need to allocate time for the project.
Overall, I wouldn't spend too much time tracking individual tasks. With P50 targets half of them will be late in any case. What matters most is how the aggregate behaves. When composing individual tasks estimates into an aggregate, neither min or max is sensible. It's extremely unlikely that either all tasks complete with minimum time (most likely something like P10 time) or maximum time (e.g. P90 time): for n P10/P90 tasks the probability is 0.1^n.
PERT has some techniques for coming up with reasonable task duration probability distributions and aggregating them to larger wholes. I won't go into the math here. Here's some pointer for further reading:
Steve McConnell: Software Estimation - Demystifying the Black Art. It's quite readable and pragmatic but at least the 1st edition I have has some quirks in its math and otherwise.
Richard D. Stutzke: Estimating Software-Intensive Systems - Projects, Products and Processes. It's a little more academic, harder read but for example explains the math better.
Ask for best, likely and worst case scenario estimates instead. Then use Program Evaluation and Review Technique. However you may want to take a look at some PERT critique first.
For individual tasks or tasks making up the critical path it’s simply not prudent to go for the best case estimates. It’s like saying that the project is absolutely free of any risk and uncertainty. If the actual job turns out to be anything but the best case scenario you’ll end up blowing the schedule. It’s better to end up with some extra time on your hands and fill the time by implementing some nice-to-haves as opposed to having to work nights and weekends.
On the other hand if managers mostly went for the worst case estimates and in software world they can easily be an order of magnitude greater than the best case figures most projects would never make it past the feasibility and planning stage. Not all of the risks going to materialise.
Going for the best case estimate won't help fighting student syndrome. Include interim milestones and deliverables instead, beside being helpful at combating the student syndrome they're pre-requisite for having a trustworthy data on the project progress and uncovering early any potential issues.
If the difference between min and max are big rather than using some black magic formula I think it the best thing to do would be to go back to the developers and ask them to do a finer breakdown and prototyping, which will lead to better estimates where the gap between min and max is not that big.
Note to the question: In my opinion, the estimates should be done by the developers/architects since they have the best technical knowledge to be able to break down into tasks and estimate those tasks.
If you are estimating for a specific developer, and you know your estimates are generally accurate for that developer, then the min value is the logical deadline (initially). In the course of the project you will adjust deadlines according to circumstance.
If you have little experience with a specific developer, one of my fondly regarded previous managers would ask the developer himself to do the estimate and set the initial deadline a third of the distance between that developer's min and max, challenging the developer to beat it.
Something which has been missing in many of these answers (perhaps because it's slightly off-topic) is frequent updates. With younger/newer developers this is even more important - read the code they commit, and/or check in daily to ask them for specific, detailed reports.
This also allows you to set tight deadlines for developers without giving them too much stress, because they will know you're around to help adjust deadlines when needed.
Frequent updates give you the most important tool in setting customer/management expectations - early warning of issues which might delay things, and I prefer having that over any formula.
Is the developer going back into a cave to develop this or is there a good chance of changing requirements over the course of the project? I would think most projects will have a good chance that something won't go smoothly and thus it may be better to try to get the prototype up sooner rather than later.
As for the initial question, I think I'd break this out into a few different outcomes and consider each:
Gross underestimation -> This leads to the problem that there is still a lot of work to do and the manager appears unable to make reasonable estimations.
Minor underestimation -> In this case, either there is an extension, scope gets cut or some bugs are in the release, but this is better than the previous case.
Made the deadline, on time and on budget with quality -> While this may seem optimal as everything worked out, I don't think this is the best result possible.
Minor overestimation -> In this case, there is some breathing room that means either things finish early or some extra work is added. A point here is that this may seem to deliver a slightly better result than the previous case like how some companies will try to beat the earnings estimate by a small amount to do better than expected.
Gross overestimation -> I think this would be the worst case outcome though it is similar to the first in terms of someone being way out of their league in being able to provide a reasonable estimation.
That's just my opinion on each and others may have a different take on it than me.
If you're trying to hold developers to their minimum estimate, that's foolish. No one, in any industry, consistently hits their minimum time estimate for getting something done. Eventually, they'll just learn to pad their minimum estimates significantly, and then they'll never hit the old minimums, because all estimates will be above that.
In Agile/Scrum, you don't set firm deadlines, but set "how many hours left on this task". Every day, you update the amount of time left. You do not track hours spent, but do track estimated hours remaining, and you try and stay honest about it.
If you have lazy developers, this is bad, because they can easily game that system. If you have developers that are worth their salt, this is great. They get better at estimation pretty quickly, and you - as a project manager - learn how reliable their estimates are, and you'll have a much better feel for what estimates to pass up the chain based on the individual developer estimates.
Go slightly towards Agile, fire the bad developers as you discover which are which, reward the good developers for actually giving a damn, and have a more productive, happier team while being able to report more accurate expectations to your superiors.
If in doubt under promise and over deliver: you want to be the person who is delivering more than they were expecting, not less. Based on this always go with the higher of any estimate.
Slightly more complex:
For a given potential delivery, if you plot the delivery times against the chances of them being happening, you're going to get a curve which is a variation of a normal distribution, and you can assume that a developers minimum estimates are going to be somewhere towards the left of the curve and their maximum towards the right.
The area under the curve to the left of the single number you select as your estimate represents the probability of you successfully delivering on or before that estimate. So if you give a number at the very left hand side your chance of hitting is effectively zero, if you give a number at the very right hand side your chance is effectively 100%.
What is less commonly realised is if you give the mean value (assuming your min and max averaged out give something approximating the actual mean) you'll only hit that deadline 50% of the time. Effectively if you use the mean you're going to miss the deadline half the time. I don't know about you but I don't like being seen as the guy whose misses half his deadlines.
So you want a number which is going to give you something you hit, say, 90% of the time. Conveniently 95% represents the mean + two standard deviations but if you can't be arsed to calculate that (and most of us probably don't have the data) my experience says that:
(3 x max + 1 x min) / 4
gives a reasonable result.
Incidentally, what you tell the developer is the deadline is another question entirely. Personally I'd give him somewhere around ((2 x max + 1 x min) / 3) and have the rest as contingency.
What are you using the estimates for? Specifically, why will the developer feel stressed if you normally underestimate?
If you're trying to schedule how long something is likely to take, you go for an intermediate value. Probably on the long side, since people normally underestimate. In any case, you shouldn't be using these estimates as firm objectives for developers, and so they shouldn't be overly stressful.
If you're using these estimates to set up commitments, you need to err on the side of overestimating. Giving developers insufficient time leads to burnout, unmaintainable buggy code that doesn't do quite what the user wants, and low morale and high turnover. Set the commitments to be reachable, and encourage the developers to finish early.
This depends on project.
Some projects may require fast development and there's no alternatives if deadline is already set and there's no good chance to prolong development. Typical issue: marketing campaign resulting in new service. Such deadline can be enough for normal development, but in some organizations it is so close, that developers work in stress and make many errors that are fixed during production stage. That's a kind of project when developers have to work with topmost effectiveness and they'd better get good reward on success.
Some projects are accurately planned and here you can use all analytics you have: history data, some developer's time metrics on subtasks, calculating risks, etc.
But anyway MAX time shouldn't be used: its the most inaccurate measure that usually leads to even more time taken. And here's a simple reason: when developer just gives away this MAX, he almost doesn't measure. He just gives away his intuition that has very little info at the time. But if he'll spend at least half an hour he'll understand specifics of his tasks, he even may split it into subtask and increase his accuracy. So you can give developer some bias like "hey, guys, just think in what time you would provide stable code here" but send him measure himself. It is good for a job, it is good for a programmer himself.
The first mistake most estimators use when setting the deadline is assuming that the dev will be full-time every day on that task which is a disastrous mistake. This can result in not meeting the deadline even when you use the over estimate to figure out the deadline. Being under the hours but past the deadline you told the client is a big problem. People take leave, get sick, have jury duty, have to go to required meetings on some new HR policy, get called over to help on another project when someone is stuck, have to load software on a new computer when their old one breaks, have to research a production problem on code they recently deployed, etc. If you are estimating more than 6 hours a day on the project per person, you are already in trouble on the deadline before the project starts. When I did manpower studies, we used a figure that equated to just slightly more than 6 hours a day of direct work when calculating out how many people were needed for any job. And we did a lot of statistical analysis as the basis for the figure we used.
I think you have to decide which of these to use on a case by case basis. We have some projects that we know the max estimate is still probably a little low (usually when someone in management couldn't face the client with the real estimate), we have others where we are doing something new where we know the estimates are more likely to be off, in these kinds of cases go with the max. But for work you've done before that is well-defined and you know the dev assigned won't be learning new skills, then go closer to the min (but never actually use the min, there are alawys unexpected bumps in the road). ALso the shorter the project, the more likely you will be able to meet the min, it is far easier to get a good estimate for a week-long project than a year-long one.
More importantly is changing the estimate and deadline every time the circumstances change. If the client adds work, the extend the deadline and estimate, don't just do it. If your best dev quits and you have to put someone new on the project, extend the deadline becasue that person has to have time to get up to speed (you may have to eat the hours though, the client may not agree to pay for that time. Critical to this is telling the client right away. They tend to be better about moving a deadline (although not happy) than they are about missing one or making the dealine but the product doesn't work as they expect it to. Too many project managers just like to wish a problem is going away and the won't have to face that conversation with the client. But usually when they do finally have to tell him it is a much worse conversation than the difficult one they tried to avoid.