Not Waving But …

I have just read this article about Toyota’s continuous improvement culture. The article is very inspiring, I recommend it. The conclusion highlights a crucial point; one that is, I believe, the source of many of the problems introducing agile approaches.

“People who join Toyota from other companies, it’s a big shift for them,” … “They kind of don’t get it for a while.” They do what all American managers do–they keep trying to make their management objectives. “They’re moving forward, they’re improving, and they’re looking for a plateau. As long as you’re looking for that plateau,it seems like a constant struggle. It’s difficult. If you’re looking for a plateau, you’re going to be frustrated. There is no ’solution.’”

Even working at Toyota, you need that moment of Zen.

“Once you realize that it’s the process itself–that you’re not seeking a plateau–you can relax. Doing the task and doing the task better become one and the same thing,” Shook says. “This is what it means to come to work.”

Experienced agile proponents know that continuous improvement is part of the deal. In order to sell the process they usually sell a “brand” that is rolled out “as is” because they are selling to an organisation that is looking for the solution. The problem is that there is not a solution; not within a single organisation if it is any size, not even within a single team if it exists for any length of time.

There is not, and can never be, a perfect system or process. The world is a complex, ever changing environment so any system needs to continually adapt. Even if the environment did not change much changes in the process generate unforeseen consequences that require further changes to compensate.

When selling an Agile, Lean or whatever process you have to make the practitioners understand that this is simply a starting point for a continuous meta-process of continual reflection and improvement carried out by everyone involved. If they don’t understand that they (and you) are going to fail.

In a post about context switching Dmitri Zimine inadvertently shows how a strong commitment to specific Agile Practices can undermine the agile principles.

The story is how a development iteration was screwed up by the organisation’s attempts to expedite an unexpected request from Sales. The story is hypothetical and is there to make a point so the characters actions seem a little extreme.

I will summarize the post:

1. A two week iteration has just left planning and development is starting
2. Sales Guy comes up with urgent request
3. Sarah the Developer makes a vague, non-committal estimate for two hours
4. Development Manager suggests not doing it because iteration is already planned
5. Project Manager uses the “its just two hours” to push the work
6. Sarah ends up spending three days on the work - iteration is “dead”, developers are demoralized, commitment to agile practices is damaged.

Dmitri’s view of what should have happened: Sarah is left alone. Development Manager offers the Project Manager two options:

1. “No” : Put the request in for the next iteration (effectively waiting 4 weeks for feature).
2. “Drop Everything” : Cancel the whole iteration and re-plan with this on top.

What principles do I feel have been sacrificed here?

• Individuals and interactions over processes and tools
• Customer collaboration over contract negotiation

The Development Manager’s basic problem was that he did not interact and collaborate, he had negotiated a process up front and stuck to that process. He also failed to use any opportunity to gather more data to help him make a better decision.

There are a couple of injections that could have been made.

Developer gives a non-committal two hours estimate:

injection: “Lets time box this, have a look for an hour then come back with something firmer. Ask Cassandra (another experienced, but more pessimistic, developer) to look at it with you)”

outcome: We get a better estimate, hopefully nearer the correct order of magnitude, Sarah has some backup when she makes the re-estimate.

Project Manager uses “It’s just a couple of hours”

injection: We’ll lets see, have you discussed this with the sponsors? They need to know that they may loose something for this. We should have a better estimate in an hour, can you schedule the call for then?

outcome: The business stakeholders who have a better idea of the financial impact will get to make an informed decision.

With those two injections the positions move. If Sarah and Cassandra come back with “two to four days” then the stake holder’s discussion will be more focused. If all the business sponsors still agree this is really more important then a re-plan is not out of the question but everyone will know that it is a significant event.

Context is important:

• Does this Sales Guy do this to you every iteration? Is this the first time in a year?
• Is this totally out of the blue? As Development Manager its good to know what’s hot with the sales team.
• The Project Manager does not seem to have bought in to the process? Why not?

No one likes expediting, it always causes disruption and should be seen as an exception. If it is the norm you really are in trouble. It will happen from time to time, for sound reasons and your process should be able to absorb it.

Any development system is a Complex Adaptive System and one of the requirements for a successful Complex Adaptive System is that it does not violate the Law of Requisite Variety which says: “The larger the variety of actions available to a control system, the larger the variety of perturbations it is able to compensate.”

The development Manager had two reactions to the request: “No” or “Everybody drop everything! we have to re-plan the entire iteration because Sarah has to do a couple of hours on something else”.

These seem unreasonable to me.

However, after the injections we get: “The business sponsors don’t think its worth a two to four day disruption, we’ll do it in the next release” or “Everybody drop everything! we have to re-plan the entire iteration because something really important has come up that is going to take several days.”

If we were not doing Scrum we might even be able to shuffle the plan “in flight” but that’s a different discussion. If the team has a rotating firefighter role can they do it? There are nearly always options.

I have been following JP’s musings on Project Management and Communication focusing on estimation and reporting.

For a successful technology, reality must take precedence over public relations, for nature cannot be fooled. : Richard Feynman

In my last entry I discussed the social reasons why I felt that a lot of projects miss their deadlines. Here is the setup for the scenario outlined there.

Richard Glass in Facts and Fallacies of Softwaring Engineering notes that one of the two most common causes of runaway projects is poor estimation (The other is unstable requirements)

• Estimation is usually done at the wrong time (too early)
• Estimation is usually done by the wrong people (management, architects, senior developers)

To which I would add:

• The delivery rate of any one team/project combination is a unique value that is very different to any other combination.
• No one is any good at estimating large software tasks, especially if someone else is actually going to do the work.
• No one should be estimating a software task if they won’t be involved in carrying it out.
• No one is a born estimator. It takes practice.

Many developers/development managers do not realise that “The Map is not the Territory” and , finally, few development managers really understand the above

So, when the project starts to slip, and the estimates are not being met many development managers feel that someone is doing something wrong now. They don’t realise that an error has already occurred and the results of that error have been captured into a number of documents, Gant charts and public commitments.

Tom DeMarco points out that managers often think they have the power to make people think faster by putting on the pressure. So, given that they have committed to a timetable they feel the best way to get back on track by applying pressure - to make doubly sure they make everyone think longer too by increasing the hours.

Finally, to save embarrassment, they finesse the data until they recover the lost ground…

There are better ways.

In Reflections beyond technology Stephen points out that in projects
“denial trumps rationality with alarming frequency!”. This is undoubtably the case. Reality is cold, hard and obdurate. Once it arrives you have to deal with it.

Some people have a habit of leaving bills unopened (or just throwing them away), why do they do that? Do they expect them to get smaller, do they expect the utility company to forget? and yet…

In a software project there comes a point where many, if not all, the development teams know that they won’t make their targets. Everyone has made their commitments, the project looks healthy from above and senior IT management are feeling good about this one.

In the project meetings though, delivery date poker is played, each manager staying in the game with a “yes, we are still on track”; keeping up the fiction that they will hit the date, throwing a little more of their reputation into the pot while hoping for another player to fold and take the blame for the deadline slip.

The game ends when, finally, someone cannot hide it anymore and admits they will not hit their target. The senior stakeholders have to finally accept that the plan is wrong and the project deadline cannot be met. While all the other technical managers breathe a sigh of relief and quietly adjust their schedules to get back in line with reality.

Lying while expecting to be uncovered at any moment is massively stressful. Believing everyone else’s schedule lies while knowing that your public commitments are lies is terrifying. Which is one of the reasons why IT development is still rated as one of the most stressful careers around.

Many managers deal with it by tacitly expecting their reports to lie to them (stacking up those bills). Insulating themselves from fault though plausible deniability, after all “they thought the project was in good shape”. When reality finally beats denial the blame for the slip cascades down through the organisation until the poor guys who finally admitted that there was no chance they could hit their target get the entire blame.

So, why do projects regularly miss their target dates? because, paradoxically, it is in no ones interest to be the first to admit that they will.

In a (rather good) follow up to his red flag pairing post Obie Fernandez thinks I was a “unnecessarily dramatic”. Probably, but what the hell, its a Blog!

Pairing can be a hard sell to a bunch of experienced, effective and productive developers (same goes for TDD actually, but that’s another story). Throwing in “pair programming is one of the only effective ways that a lot of us have ever witnessed keeping average developers from pissing away 95% of their productivity engaging in non-work” really doesn’t help to get their respect and buy-in.

Do I really think they are just “average”? (I don’t)

Do I really think they “piss away 95% of their productivity”? (I don’t)

Do I think they could do better and have more fun and satisfaction Pairing? Yes, that’s the sell, but they are not hearing that because they are too busy seething at being implicitly labelled as average time wasters.

In his latest posting high profile Thoughtworker Obie Fernandez comments that:

“Truth is, pair programming is one of the only effective ways that a lot of us have ever witnessed keeping average developers from pissing away 95% of their productivity engaging in non-work such as reading and writing blogs, instant messaging, personal email, shopping online and otherwise wasting time on bullshit.”

…

“As far as I’m concerned, all the other benefits you get from pairing like continuous review and knowledge-sharing are just gravy.”

So now you know, All that good stuff about continuously reviewing code, helping each other stay focused, making sure everything is tested, avoiding blocks, solving problems faster and sharing knowledge across the whole team doesn’t really matter. No, pair programming is mainly a management technique that sets up average developers as Thought Police to enforce “good” behaviour. From the earlier comment I assume that “a lot of them” (other Thoughtworks Consultants?) have already tried key loggers, IP sniffing and surveillance cameras?

For the record a team will work hard if they get enough internal satisfaction from doing so. External factors such as threats, rewards or the Thought Police have been demonstrated over and over to fail. How you foster that internal satisfaction is where the “art” of management comes in.

I like Pairing, though it is a hard sell. Frankly, the sell just got harder, its a shame, because for a while there I thought Obie “got it”. I know, from working with them, that this isn’t the view of many Thoughtworks guys, I just hope that this sparks off a “robust” internal debate.

Thanks go to James Robertson for spotting this… In bytegeist Dermot Hogan argues that “Object Oriented Programming has failed”.

His first example of the massive failure of OO is:
“But let me give you two cases from my own experience. First was the company that was going to build a new OOP banking system. Great idea! Just create a banking class library and use it to build banking applications like cars off a production line; or like shelling peas - or whatever inane simile you want to come up with to disguise the lack of any deep analysis. Or if the truth be known, any analysis at all: this is the world of banking. The problem was (and is, and always will be) that processing dosh does not correspond to an OOP paradigm.”

Now, maybe I’m old fashioned, but my guess is that “the lack of any deep analysis. Or if the truth be known, any analysis at all” probably had something to do with it.

His second example is:
“I was working on fixing a bug in an object oriented Perl system. Yes. Perl can be made object oriented; it isn’t elegant (think of camels dancing) but it does work. I noticed the error low down in the class library and like a good programmer fixed it, got it tested and thought no more about it.” and then after causing trouble for a supposedly unrelated user it hit him: “What I’d done was to modify the behaviour of a base class. And this behaviour had then propagated with effortless ease through the whole damned system. But, with financial systems, this is a no-no. In a typical financial system, all responsible users have to sign off on a change before it goes live. After all this is real money we’re taking about here. Especially if it’s the bank’s own. With this particular system, that sign-off involved users in most of the capitals of Europe!”

Amazing! OO programming techniques now manage impact analysis, regression testing and change management! And I thought they were a way of managing complexity in code.

The sad thing is that there are a lot of developers and development managers out there who really do think that some new technique will eliminate the need for analysis, correct change management process, configuration management and all the other “boring” bits. Well, they haven’t; the “boring” bits are the bits that separate the professionals from the gifted, and not so gifted, amateurs. They can be done, or not, whether you are building with OO, procedural, relational and/or functional technologies. You can do them with lots of, or little, ceremony. They do, however, have to be done.

The psychology of a project requires closure, being able to feel it is done and allowing yourself to relax without nagging concern is a necessary part of the mind set. At the same time the idea of an end where everything will be okay means that project managers can call for heroic sacrifices from the team in order to acheive this. This is unlike maintenance, where the work goes on indefinitely. Maintenance requires a different psychology. Maintenance is not heroic, it just goes on. You cannot call for a supreme effort from your team to finish it because everyone knows that it will never be finished. As most software is developed during its maintenance phase the Project mindset may well be detrimental to the success of the process.