CHAOS theory

Glen Alleman’s post on the Standish Group CHAOS Reports documents nicely many of the weaknesses of those reports.  Having only three categories doesn’t provide enough granularity, fuzzy definitions for these categories, no access to the base data, small sample sizes, etc. 

You also have to consider that the Standish Group is a commercial concern and that shielding their data and presenting it in a way that is advantageous for them can be expected.  We are not dealing with a not for profit research group subject to peer review.

Just for kicks I built the following chart by grouping all the “failed” and “challenged” together from all the CHAOS report information I could find on the web.  I’m assuming that the CHAOS report methodology has been constant over the years (was it?).  Even with a built in bias you see that both groups flat line over time.

 

My own definition of success is no more that 10% off on cost, no more that 10% off on duration, and no significant scope reduction.  By that definition, and based on my experience, I’d say that the same conclusion holds true; the success rate has flat lined over the years.  Whatever that success rate happens to be.  Why is that so?  Here is my personal CHAOS theory; no butterfly wings required:

• Requirements are fuzzy, often documented out of context and not tied to deliverables. See this previous post. If you don’t know what you are delivering how would you when and for how much?

• All requirements are deemed equal. No consideration is given to a must have/nice to have classification. If a particular requirement addresses a seldom encountered condition or action is it that important? I believe that the Pareto principle applies here and that 20% of requirements cause 80% of the cost. Why not implement only the true essential requirements; saving costs and complexity?

What do you think?  As always questions and comments are welcome.

Connect with me on LinkedIn.  I am a LinkeIn Open Networker (LION); you can use “Friend” to add me to your network, please mention the blog in your request.

Comments

[Glen B Alleman]

Patrick,

The current DoD approach - SecDef Gates - is a 60% solution. This means that 60% of the needed capabilities "now" are much better than 100% (or something near that) later.

[Patrick Richard]

Glen,

Thanks for that feedback. I'll use the DoD 60% approach as an argument with my clients.

[Shim Marom]

Hi Patrick, enjoyed reading your post. I've been struggling to understand the Standish Chaos report for quite some time now. It seems to me that something fundamental is wrong with this report, even without taking the scientific approach used by Glen to explain why this report is lacking from the statistical analysis point of view. From my simplistic perspective things are much clearer. In any human endeavour there will always be a line over which further improvements will require infinite levels of resources to attain. The Standish report (at least as far as I understand the psyche behind it) assumes that we as human being, given our limited resources, can achieve much higher success rates than currently obtained. Given your observation (which I tend to agree with) that the success rate has flat lined over the years, I conclude that we have reached that level beyond which far greater resources will be required in order to get that line any higher. Having said that, the problem with the Standish report is not that it is statistically or methodologically incorrect, the problem with it is that it assumes that we are lacking in our project delivery, whereas the truth is that in the main our delivery methods are valid given our limited capacity to do any better.

Cheers, Shim
www.quantmleap.com

[Patrick Richard]

Shim, thanks for your comment; it is fun to see that my post make it Down Under. Let me say right off the bat that I have never seen a full CHAOS report so I cannot talk to the extent of the information they contain.

That being said I do agree that some things are wrong with the CHAOS report, chiefly the fact that it is not peer reviewed. The Standish Group makes money off these reports and would probably not benefit from too much scrutiny of the underlying data.

When you state “In any human endeavour there will always be a line over which further improvements will require infinite levels of resources to attain.”; you are talking about diminishing returns something that is related to the Pareto principle and again we are in agreement.

I do however disagree with the statement “… it assumes that we are lacking in our project delivery, whereas the truth is that in the main our delivery methods are valid given our limited capacity to do any better.”. I find this defeatist. We can in fact do better as demonstrated by some very complex and often costly projects; for example space exploration. My take is that very complex but successful projects attain success by adhering to detailed rules, enforced relentlessly. Most software projects are run in a somewhat cavalier fashion with the (un)expected (in)evitable results that flow from that approach.

[Shim Marom]

Hi Patrick, thanks for your comments and, yes, we do follow certain blogs here in Australia (although it sometimes takes awhile for the bits to cross the Atlantic).

Back to your comments, having looked at my previous notes I guess I can't blame you for accusing me of being a defeatist (although most times I look at myself as being overly realist).

I guess what I was trying to say (unsuccessfully it seems) is that in the main most projects are not run as space exploration projects, the reason being that in most traditional projects the cost of failure will not necessarily translate to massive explosions or costly loss of human lives. In that respect the point I was trying to make (and I think we actually agree on that) is that in most projects there is an acceptance of potential failure by the virtue limiting budgets. So its not that we can’t do better (as I stated earlier), it is just that in the main we look at our opportunity costs and make a decision to take the risk of failing. Had our level of tolerance been adjusted to that applied to space exploration projects we would not have nearly as many ‘failed projects’ as we seem to experience now.

Cheers, Shim.
www.quantmleap.com

[Patrick Richard]

Hello Shim,
Thanks for commenting further; I do agree with your point. I have a few takeaways from this exchange:

1. Email is not the best mechanism for exchanging ideas, especially coupled when one side (me!) is not a native speaker of the language used.

2. “… there is an acceptance of potential failure by the virtue limiting budgets.” I will keep this in mind; this makes a great argument during negotiations.

Regards,

Patrick