Agile; still not convinced

Today I was listening in on a PMI Agile CoP webinar entitled "Beyond Scope, Schedule, and Cost: Rethinking Performance".  I do try to broaden my horizons...

I must say that I am still not convinced that Agile is that different from other iterative approaches.  Actually, at some point, the speaker adopted RUP terminology and if you had just joined in you'd have been confused. 

Much of the argumentation was anecdotal; for example:

• Claiming that Agile yielded an 89% improvement (decrease) in the number of defects over the "traditional" approach is great but when you mention that the organization was apparently 4 times over the average for its industry you could say that the problem lied somewhere outside of the methodology.
• Claiming that a Gantt chart is not as good as a Parking Lot chart is not comparing apples to apples if when the deliverables are not the same.  Comparing a crappy Gantt to a good Parking Lot is a weak argument.
• The fact that some projects fail due to feature bloat is not news.  Using Agile (or RUP, or something else) does not by itself fix that issue.  The "traditional" approach did not cause that problem either.

Making claims from a position of authority does not win debates, it speaks mostly to those who are already believers and it may create in others the perception that the claims cannot stand on their own.  

Using a different vocabulary to talk about the same or equivalent concepts is not enough to create buy in for Agile.  For example, Story Points vs. $ is at some level similar to $ vs. Euros; all we are talking about here is currency conversion.  What if, from now, all projects Agile or not used goats as their currency?  Would that make Agile better or worse than the "traditional" approach?

I'm still open minded about this subject but still not convinced.  What do you think?  As always comments are welcome.

Connect with me on LinkedIn.  I am a LinkedIn Open Networker (LION); you can use “Friend” to add me to your network, please mention the blog in your request.

Comments

[William Nichols]

I wouldn't brag about being the valedictorian of summer school.

There are a number of features of Agile I recommend. Their metrics are not among those. I'm sorry, story points are NOT a measure, they are an opinion. They are non-interval values treated as interval values.

[Glen B. Alleman]

Here's some comments.

Page 4 is a complete red herring question. "Let's see do you want a nice top sirloin that is at market value, which you'll learn about when I bring it, or do you want dog food shaped like a burger that I know the price of?" Come on folks let's get real. Cause that CIO is a poser.

Page 5, the notion of "conforming to plan" is simply BAD project management. The Plan is a strategy for completing the project. Plans change, hell the weather for large construction projects. You think the Plan stays fixed in the presence of a rain storm when we're pouring concrete?

Page 7 - bogus Standish numbers all over again. Bogus red herring - again - on the measures of success. The well developed and practiced method of Measures of Effectiveness and Measures of Performance baked into all Systems Engineering process should be examined. Another poor understanding of good PM.

Of course the Standish numbers are simply poor stats. The cancellation numbers are not connected with the reason for the cancellation. The Standish number would not get a pass in high school stats. When are we going to stop using this as the basis of decisions.

The conjecture at the bottom of the page have not basis.

Page 8 - no units of measure.

Page 9 - no units on the scale, no domain, no context. Simply bad charting.

Page 10 - this is completely dependent on the funding process.

Page 14 – same projects, same teams, same environment. Correlation is not the same as causation. Might be true, but it’d be nice to see the real numbers. BTW, Highsmith introduced me to another scientific instrument firm that was adapted agile and they went completely in the ditch, had to be bought by a competitor and scrapped the entire process. One point does not a line make.

Page 22. Paul got that picture from my Capabilities Based Planning consulting. Good stuff, but not directly related to agile. That picture is also the same topology as DoD Quadrennial Review’s Capabilities Based Planning process.

Page 25 – this of course is completely dependent on the domain and context. And the same for Page 26.

Page 27. Flickr is equivalent to rolling out SAP to 52 sites in an aerospace firm? Domain, Context PLEASE.

Page 37 – this is called Integrated Master Plan / Integrated Master Schedule Rolling Wave where I work. That’s where there build $600M of flight avionics.

Page 38 – this is the lamest Gantt chart I’ve seen. Those durations are forbidden in DoD and DOE. Please make it stop.

Page 39 – any dependencies here?

Page 40 – all good questions, not unique to Agile.

Page 44 – can’t wait for Mike’s presentation. I posted a response to his original submission. http://herdingcats.typepad.com/my_weblog/2011/02/the-seven-deadly-sins-of-project-management.html

[Patrick Richard]

Hello William,

Thanks for the comment. Part of my problem with Agile is related to the opinion/sect/religion aspects that it's wrapped in.

If it is that much better then there should be non anecdotal, measurable evidence to that effect.

Patrick

[Patrick Richard]

Hey Glen,

It's been a while, hasn't it? I'm glad to note that you like the presentation about as much as I did...

On the Standish group numbers I would add that I disagree that "they are an indicator of systemic failure of our planning and measurement processes". Some project are plain wrong, should never have started, and would not be successful regardless of the methodology used. I call those Swiss army knives because they try to do everything or Frankenstein projects because if you apply enough energy the beast will rise.

Slide 14 states an 83% improvement in defects. If that company had been industry average it would have started around 567 defects and the claimed improvement would have been much lower. I think that the main message on that page is “Overhaul the entire product development process”. How much of the improvement is Agile and how much is the rest of the overhaul?

Slide 25 also assumes that “low business value” iterations are done first. This is not a rule and actually RUP iterations would try to remove risk and feasibility issues which in my opinion have tremendous business value.

Slide 38 is indeed a piece of crap and he uses it to setup slide 39. That’s what triggered my comment about comparing a crappy Gantt to a good Parking Lot being a weak argument. He should not have to rely on this weak argument if he could make a better case for his camp.

Thanks for the comments; I enjoyed reading them.

Patrick

[William Nichols]

Patrickm Salut!, comment allez vous?

I apologize for a snarky reponse, I can do a better.

I have been very frustrated with Agilists making claims supported with appeals to authority and narrative rather than data.

I haven't yet found the specific slide deck to which you refer, though Glen recognized it. Seems to be controlled rather than open?

On defects, normalized density (LOC or FP) in release is a pretty good indicator, as are the density in final test an portion of time in final test.
Many Agile methods focus on achieve improvement by introducing or improving unit test. XP at least includes a form of peer review. None seem to emphasize real design.

Another problem is scaling. You need a common language, including data. Story points are too local to scale. I have other issues, but they can be OK for what they are in a single small team.

The biggest strengths of Agile are team involvement and responsibility and explicit use of incremental strategies.

[Patrick Richard]

William,

I did not find your comments snarky although I must say that I didn't get the "valedictorian of summer school" part. Pardonnez mon anglais...

The slide deck can be found at http://agile.vc.pmi.org/Community/Wikis/tabid/173/loc/getfile/Page/March-2011-Webinar-with-Jim-Highsmith/Default.aspx?File=Beyond_Scope_schedule_Cost_Performance.pdf which would require that you are a PMI member and an Agile CoP member (free).

Interestingly enough I found another version of the deck at http://www.slatteryit.com.au/Agile2010/preso/Agile-Australia-2010-Jim-Highsmith.pdf. If you look at slide 12 of that deck, which did not make it to yesterday’s webinar, you see a much lower defects improvement. Interesting.

Patrick

[William Nichols]

Patrick,

Maxime looked at me funny when I tried to coach a player how to insult another player in French. Ca va.

Summer School is the remedial school to which students who failed the normal school year are sent to complete the school work required to pass their normal school year. It is an alternative to repeating the grade. A clever retort does not advance the discussion.

Thanks for that link.

There are some reliable studies, I would point, for example, to Laurie Williams.

One rather long winded take on much of this is in our submission to a panel at ICSSP, http://www.conference-publishing.com/list.php?Conf=ICSSP11 under Nichols, Kirwan

[Glen B Alleman]

William, How can the defect rate be normalized in the absence of domain and technical adjustments? I know you'll have the answer to this, but it's amazing at times how those quoting numbers are clueless of their naivety. They should be sent a copy of Huff's "How to Lie..." book.

What is the baseline between these two comparisons?

[Glen B. Alleman]

Patrick,

My sense from listening to "agile thought leaders," and participating on panels and conferences is many have a different set of experiences when it comes to projects and project management.

Most of my encounters observe that they manage "agile" style projects - self contained, internal, bounded programs. The complexities of enterprise class projects is missed - hence the completely lame Gantt chart as an example.

It's too bad, since there is lots to say about the benefits of agile development processes, but these types of presentations reduce the credibility of the "thought leaders."

[William Nichols]

Glen,
You ask the pointed question of what the numbers mean. It can a long discussion, so I'll to be brief. One cannot ignore domain. But we can recognize outliers.

We can also look for sensitivity points in the process. Production and defect models need to be calibrated, but the models are pretty simple and robust.

Start with clear and precise definitions. How do you count a defect? It varies by a surprising amount. How do you measure size? (More than once I've heard IBMers tell the story of the quality award for what turned out to be nothing more than inflating the LOC count), Similarly, what are your process steps? How do you record effort? If you are not clear about these, your numbers won't mean much. If I speak at our Symposium, we all use common definitions, anywhere else I am obliged to define what I mean.

Next look at a *range* of results from industry. A given point is at best a very crude indicator. (Much like Body Mass Index, BMI is valid in population studies, but far less valid and reliable individually)

Then look from multiple perspectives, e.g. defects normalized to LOC or FP at specific stages (such as UT, ST, Release) effort ratios test/development, effort ratios, % total defects removed at various stages. The specific values will vary, but the profiles should be internally consistent.

It also appears that some process measures are surprisingly consistent across language and domain, For example, inspection or review rates have an inflection point around 200 LOC/hr. Defects injected in coding typically fall within a range centered around 100 def/KLOC if you count everything, more like 45 /KLOC with modern dev environments. But again, this works for development in traditional languages. Newer environments like .Net can be very different.

Test and review yields vary widely, but we have a good idea of normal ranges. There is no excuse for getting inspection yields lower than 25%, and it's really hard to get above 70%. A test yield of 50% is normal, it can be better or worse. Pretty consistently, better requires low input defect density.

Another simple measure is cycles through final test. I've seen 1, 2 or 3 is excellent, 20 or more common than it should be.

Domain is important. A medical devices company cannot ship until all defects have been addressed. They might spend more time in final test than a web based instruction manual.

With all that, knowing nothing else, some first approximations for *normal* are development time =~ final test time (with wide variation) , released Def/Kloc ~1-5 .

When Hill AFB reported 0.016 Def/KlOC in final test and the software for their unmanned vehicle worked the first time in field test http://www.sei.cmu.edu/tspsymposium/past-proceedings/2010/Keynote_Webb.pdf slide 27, I knew that was very good.

Well,that wasn't short, but it was still too short.

[Patrick Richard]

Glen and William,

Thanks again for your comments and allow me to drag this in a slightly different direction.

Regardless of our capacity to measure the impact of Agile or of grading one approach against another I feel that part of the issue around Agile is deeply related to human relations.

I once heard someone say “lead people, manage tasks”; is it possible that Agile is an act of rebellion against a perceived authority that could be seen as stifling creativity? I’ve heard often the comment that Gantt charts, WBSs, etc. are engineer things (I’m an engineer) that are not aligned with the methods of the creatives (programmers, architects, etc.)

When I hear the term “self directed”, I hear self lead. What about the self managed part? The accountability part? I don’t hear that from creatives but I sure do hear that from my clients. Most of the time clients would rather get an unexciting (meaning predictable) result instead of an exciting (meaning stressful) one. Note that I am not talking quality or cost here; just the fact that knowing where you are going and when you’ll get there is very valuable to most people.

Your thoughts,

Patrick

[William Nichols]

Patrick,

You raise two different points. The project with no drama requires no heroes nor heroics. That is the key to the 40 hour week. Minimize the surprises. That is worth a lot.

Your other point relates to self directed and self managed. (the meanings are not used consistently, act surprised).

Agile has some very explicit roots in rebellion against the "Dilbert Boss". When they think of management and management plans, I think that's their mental image.

A very positive outcome of common agile approaches is that the team takes responsibility for many of the details of planning and tracking. Individuals are accountable to the team daily. The team, as a group, typically assumes joint responsibility for quality and details of assigning and tracking the technical work. In principle, the distributed and detailed local knowledge of the individuals on the team is leveraged to improve matches and timeliness of work assignments.In This frees the manager to worry about the things he (or she) is better qualified and better positioned to do, that is things team members generally cannot do but managers can.

Understand that team planning is within boundaries. They are given goals and required to regularly report on progress against those goals.

In this model, workers have a sense of autonomy that many or most find very satisfying.

Like anything it can be misunderstood, misused, and abused. Also, without a rich measurement framework and organizational support it does not scale well. Moreover, there are assumptions of technical skill and discipline that may not always be justified.