@Rays
2017-03-27T10:32:05.000000Z
字数 14081
阅读 1608
未分类
摘要:
审校:Shane Hastie
正文:
本文要点
Describe how Monte Carlo planning compares to standard velocity-based planning approaches
Explore two examples which show the benefits of Monte Carlo planning
Implement an Excel spreadsheet with a simple Monte Carlo example
See a cool, quick way to determine the accuracy of human estimates
Learn how to roll out Monte Carlo planning in your organization
In 2010 I helped a startup IPO by creating a custom Monte Carlo planning tool in C# that estimated the likelihood that their new small business product would ship on schedule. The output of this tool -- a probability distribution over possible release dates -- helped the software development teams and management make tradeoff decisions. When the product was released on time, just a few weeks before the IPO, the CTO and founder of the company said that it was the first product in the history of the company that was on time. Prior to Monte Carlo planning, the company had been using standard project management and agile planning approaches.
In addition to improving estimation, Monte Carlo planning also helped to change the culture and thinking around planning. When C-level executives asked for a date the VP of Engineering was now able to have intelligent conversations about the confidence level of that date. The VP of Engineering began to say, “A release is not a date, it’s a probability.” This in turn led to significant changes in what metrics were tracked (from velocity to end-to-end inventory management) and how the product management group thought about risk.
This is one of many examples of Monte Carlo leading to better results and decisions. Monte Carlo planning generate the full probability distribution based on historical data. In contrast, the standard agile planning approaches generate a straight line average case analysis.
Let’s walk through a simple example of Monte Carlo planning to illustrate its benefits.
Consider a Scrum team that has a velocity that ranges from 80 to 120 (uniformly distributed). The Scrum team wants to know how many points will be completed in a six sprint release. If the team simply computes its average velocity it will conclude that it can complete 600 points (6*100) over six sprints. In contrast, here are the results of a Monte Carlo simulation with 20 sample runs:
These results show that over six sprints there is a 10% chance that the velocity will average 92, a full 8% less than what the average case analysis shows. This data immediately leads to useful conversations with the Product Owner: “What will happen if the bottom 8% of the backlog is not completed by the release date? If that is acceptable, can the product be released earlier? If this is not acceptable, what options can we explore?”
How was this probability calculated? Monte Carlo samples the historical distribution, in this case a uniform distribution between 80 and 120. To create the graph above, I sampled the distribution 20 times. Here is one of the samples:
100 | 120 | 118 | 112 | 91 | 110 |
---|
This sample consists of six sprints because the team wants to estimate over a six sprint release. In this particular sample, the velocity ranged from a low of 91 to a high of 120. The average was 108.5. In the chart above this sample is responsible for the bar at ‘108’ on the X axis.
Here is another sample:
87 | 93 | 91 | 100 | 87 | 113 |
---|
In this sample, the velocity ranged from a low of 87 to a high of 113 with an average of 95.
Think of each sample as a ‘possible future.’ Monte Carlo creates these possible futures from historical information and correctly weights them.
One of the great benefits of Monte Carlo is that it works with any distribution, even ones that are unknown. In this simple example the distribution is known and the problem is simple enough that the probability distribution at the end of six sprints has a closed form solution. However, this is typically not the case.
For problems that are encountered in real world situations, either the closed form solution is not possible or is so hard to derive that a Monte Carlo approach is faster. Here is an example based on an actual situation that I encountered at a client and is quite common:
Here are the key aspects of this model:
What is the probability that the project will be on time? Monte Carlo produces a result of approximately 45%.
Given that average case analysis (such as the standard velocity based planning used by many Scrum teams) is easier than Monte Carlo, a natural question is whether or not there is any average case straight line assumption that will approximate the Monte Carlo results. To test this possibility we will see what happens under three velocity assumptions for this more complicated problem:
In all three cases Business Acceptance takes one week. Clearly, at the start of the effort all three estimates are far off. The best case and average assumptions indicate that there is a 100% chance the date will be hit while the worst case assumption indicates that there is a 0% chance that the date will be hit.
Given that the initial estimates are far off, how many sprints does it take for these assumptions to get close to the 45% probability that Monte Carlo produces from the start? Here are the results for 20 sample runs:
The chart shows that:
Is there a way to tweak the straight line analysis to produce better results? Some agile experts suggest creating straight line approximations to produce confidence intervals. For example, taking the lowest three velocities out of the last five sprints produces a ‘pessimistic’ estimate while taking the highest three velocities out of the last five sprints produces an ‘optimistic’ estimate. The problem with these straight line approaches is that the probability that the actual results will be between the low and high straight line estimates changes as a function of time -- it is not constant. So these straight line ‘lower’ and ‘upper’ bounds are incorrect. As mentioned previously, confidence intervals change as the square root of time so no straight line approximation works except over very short time intervals.
Here is why I like Monte Carlo simulations:
Here are common concerns I hear about a Monte Carlo approach to planning:
A final objection to Monte Carlo is that it does not address the lack of trust and alignment that organizations often have when contemplating plans and examining estimates. This lack of trust and alignment then invites management to set targets without regard to historical performance. This is certainly a valid worry and one that I suggest be handled separately. There are many alignment activities, such as handing out ‘value dollars’ and inviting stakeholders to buy features or deadlines, which will expose different assumptions and tradeoffs. Once everyone involved has the same understanding of the current reality and the goals then Monte Carlo planning might be appropriate.
One obstacle to embracing Monte Carlo planning is overconfidence in the accuracy of human estimates.
Here’s a simple technique that I use to demonstrate the accuracy or inaccuracy of human estimates:
More often than not, the result shows that the error is greater than that needed to create a useful estimate. This provides a launching pad into a discussion of the value of Monte Carlo and addresses the concern that it is too difficult to do. At a recent training event at a 400 person company, I ran through these steps in approximately 30 minutes and the error rate was 33% (i.e., the high estimate was twice the low estimate) for a single user story.
Monte Carlo planning may not be suitable in all situations. When it is, here is how I like to introduce it:
Step 1 is often overlooked but is a critical part of introducing this change (and, really, any other change). Thankfully, I have never met a company which enjoyed its planning approach or thought the results were highly accurate. In fact, at a Fortune 500 company where I consulted one of the people in charge of annual planning told me that there was “no relationship between planning and actuals”! In these situations, the primary problem is getting historical planning data and modeling the system of work.
A simple Monte Carlo planning spreadsheet can be created in Excel using the randbetween
and hlookup
functions.
Here is what the formulas look like:
Cells A1, B1, and C1 contain the numbers 1, 2, and 3. Think of these as the last three sprints. Cells A2, B2, and C3 are the velocity the team achieve in each sprint.
Row 4 contains the formula
=HLOOKUP(RANDBETWEEN(1,3),$A$1:$C$2,2)
RANDBETWEEN generates a random number between 1 and 3 (because there are three sprints). Call this random number N. HLOOKUP then returns the velocity of the Nth sprint. So row 4 samples the historical distribution to create another three sprint sequence.
You can repeat row 4 as many times as you want to create histograms (probability distributions) and do other types of analyses. In the two examples I gave at the start of this article I repeated row 4 twenty times to create 20 samples. Most Monte Carlo studies use over 1,000 samples.
Monte Carlo planning samples the historical distribution to give a rigorous, quantitative account of what the future may bring.
It holds significant advantages over standard average case straight line agile approaches and you can start your journey to Monte Carlo with a simple Excel spreadsheet.
If you would like to further your study of Monte Carlo planning, I invite you to examine these additional resources:
Michael de la Maza is a Scrum Alliance Certified Enterprise Coach (CEC). As an Agile consultant, his major engagements have been with Paypal, State Street, edX, Carbonite, Unum, and Symantec. Previously, he was VP of Corporate Strategy at Softricity (acquired by Microsoft in 2006) and co-founder of Inquira (acquired by Oracle in 2011). He is the co-author of Professional Scrum with TFS 2010 and Why Agile Works: The Values Behind The Results. He holds a PhD in Computer Science from MIT. Michael is a Co-Active Coach and is co-organizer of the BayALN, the agile user group in San Francisco. He loves playing, creating, and sharing games and co-organized the 2010 Agile Games Conference, the 2011 Agile Games Conference, and the 2016 Agile Games West Conference. He serves on the Scrum Alliance's Certified Team Coach (CTC) review committee and the Virtual Coaching committee. He enjoys mentoring Scrum Masters and agile coaches who want to deepen their understanding of agile. He also mentors and invests in startups through Techstars Boston.