Not Waving But …

In his latest posting high profile Thoughtworker Obie Fernandez comments that:

“Truth is, pair programming is one of the only effective ways that a lot of us have ever witnessed keeping average developers from pissing away 95% of their productivity engaging in non-work such as reading and writing blogs, instant messaging, personal email, shopping online and otherwise wasting time on bullshit.”

…

“As far as I’m concerned, all the other benefits you get from pairing like continuous review and knowledge-sharing are just gravy.”

So now you know, All that good stuff about continuously reviewing code, helping each other stay focused, making sure everything is tested, avoiding blocks, solving problems faster and sharing knowledge across the whole team doesn’t really matter. No, pair programming is mainly a management technique that sets up average developers as Thought Police to enforce “good” behaviour. From the earlier comment I assume that “a lot of them” (other Thoughtworks Consultants?) have already tried key loggers, IP sniffing and surveillance cameras?

For the record a team will work hard if they get enough internal satisfaction from doing so. External factors such as threats, rewards or the Thought Police have been demonstrated over and over to fail. How you foster that internal satisfaction is where the “art” of management comes in.

I like Pairing, though it is a hard sell. Frankly, the sell just got harder, its a shame, because for a while there I thought Obie “got it”. I know, from working with them, that this isn’t the view of many Thoughtworks guys, I just hope that this sparks off a “robust” internal debate.

Take a desired system.

1. Break it down by desired features
2. Add the features to a Prioritised list
3. Do the most important ones
4. “Prove” you have implemented them
5. Put it into production
6. Reflect on how that went and tweak the details of how you did it

Repeat until your customer asks you to stop.

There, that’s not so hard … is it?

One of the tenets of many Agile Development approaches is that the system’s Architecture evolves over time. The claim being that the architecture will remain optimised for the systems purpose and will be easy to understand because it doesn’t have a load of hooks and parameterisations to allow for unspecified requirements. Extreme Programming has some aphorisms to emphasise this: DTSTTCPW (“Do the Simplest Thing that Could Possibly Work”) and YAGNI (”You Ain’t Gonna To Need It”) spring to mind. Martin Fowler’s article Is Design Dead is one of the classic descriptions of how it should work.

Agile developments have an imperfect memory, with little historic documentation it is reasonable to view each architectural decision as being made using the current system architecture and requirements as the only inputs. That being the case then the evolution of the architecture can be seen as a Markov Chain or Drunkard’s Walk (at least metaphorically).

What does this mean? The ideal architecture is as described above simple, and focused, it should also be balanced and not restrict you from taking it in any a particular direction. This last desirable trait is invisible, you don’t know you cannot go in a particular direction until you try to go there.

It is inevitable that each change you make to the system takes you off the ideal “balance” point for taking the architecture off in an, as yet unspecified, new direction. You might feel that as each requirement and takes you “off balance” in a random direction then the likelyhood is that, overall, you will stay on balance. This is where the drunkard’s walk comes in, over time a random walk diverges from the centre line, never to return:

Over time your architecture will become less balanced and amenable to change. Getting the balance back will probably require an architectural review of some kind and some kind of more disruptive change than another simple refactoring.

As a software developer I don’t make claims to change society, however in both software development and social engineering the actor is peturbing a complex system made up of empirical processes and that means that lessons in one area may apply to the other.

In The Open Society and its Enemies Karl Popper wrote that “the piecemeal engineer will adopt the method of searching for, and fighting against, the greatest and most urgent evil of society, rather than searching for, and fighting for, its greatest ultimate good.”

In many (most?) areas of software development the developers are implementing a continous stream of new requirements and fixes to existing or emerging problems. The process is never “finished”.

So it came as a pleasant surprise when I found the following table in Conservative Political Philosophy and the Strategy of Economic Transition, by Peter Murrel and I couldn’t resist the parallels between this and a comparison between “traditional” and “agile” software project management methods.

Characteristics of Utopian and Piecemeal Approaches to Policy

Note on empirical processes: Process control theory states that a defined process is predictable; it performs the same every time while an empirical process does not. Empirical Processes are chaotic (in the sense of being extremely sensitive to initial conditions) and effectively unrepeatable, requiring constant monitoring and frequent intervention to ensure the desired result.

I recently attended a presentation in the City of London on a high powered infrastructure component and architecture.  The attendes were the usual bunch of attendees, IT guys at the geekier end of middling to senior management, early adopters of various technologies.

During one of the sessions the presenter was explaining how this technology could enable Service oriented architectures:

 ”I guess most of you are implmenting SOA now.  In fact, how many of you are implementing SOA right now?”

Of the forty attendees one hand was tentatively raised.

For me the crucial difference differentiator of an Agile approach is the obsessive desire for feedback. Basically, if your reaction to an issue is to work out how you can get more, objective feedback faster then you are going towards the agile end of the spectrum; if it isn’t you are not.I will go out on a limb and suggest that the vast majority of system development failures are due to the lack of feedback at various phases of the development process.

This isn’t a new thought, it is as old as Kent Beck’s driving story from the earliest days of XP. It does get lost in all the talk of principles’ and ‘practices’ and ‘levels’ though.

Phil Dawes has an excellent post on problems scaling classic Semantic Web approaches to a moderately sized, messy, real world environment. I have watched him develop his solutions to these problems, the evolution of tag triples and JAM*VAT and it is very impressive. One of the lessons I take away from Phil’s work is that for things to scale in the real world they are necessarily fuzzy. If your system needs everything to work out up-front and will only handle 99.999% of cases then scaling to real world sizes quickly generates far too many problems to cope. As brittle systems (those that require certainty) grow they have to increase their complexity enormously. Building a system that can hold ambiguity without worrying about it until the information is needed means that special cases can be ignored until they have to be dealt with. Which is essentially what Phil has done by leaving the disambiguation to the user of the data rather than the source. What this does mean is that its hard to be certain about the meaning of your data. But that’s just the real world for you.

Thanks go to James Robertson for spotting this… In bytegeist Dermot Hogan argues that “Object Oriented Programming has failed”.

His first example of the massive failure of OO is:
“But let me give you two cases from my own experience. First was the company that was going to build a new OOP banking system. Great idea! Just create a banking class library and use it to build banking applications like cars off a production line; or like shelling peas - or whatever inane simile you want to come up with to disguise the lack of any deep analysis. Or if the truth be known, any analysis at all: this is the world of banking. The problem was (and is, and always will be) that processing dosh does not correspond to an OOP paradigm.”

Now, maybe I’m old fashioned, but my guess is that “the lack of any deep analysis. Or if the truth be known, any analysis at all” probably had something to do with it.

His second example is:
“I was working on fixing a bug in an object oriented Perl system. Yes. Perl can be made object oriented; it isn’t elegant (think of camels dancing) but it does work. I noticed the error low down in the class library and like a good programmer fixed it, got it tested and thought no more about it.” and then after causing trouble for a supposedly unrelated user it hit him: “What I’d done was to modify the behaviour of a base class. And this behaviour had then propagated with effortless ease through the whole damned system. But, with financial systems, this is a no-no. In a typical financial system, all responsible users have to sign off on a change before it goes live. After all this is real money we’re taking about here. Especially if it’s the bank’s own. With this particular system, that sign-off involved users in most of the capitals of Europe!”

Amazing! OO programming techniques now manage impact analysis, regression testing and change management! And I thought they were a way of managing complexity in code.

The sad thing is that there are a lot of developers and development managers out there who really do think that some new technique will eliminate the need for analysis, correct change management process, configuration management and all the other “boring” bits. Well, they haven’t; the “boring” bits are the bits that separate the professionals from the gifted, and not so gifted, amateurs. They can be done, or not, whether you are building with OO, procedural, relational and/or functional technologies. You can do them with lots of, or little, ceremony. They do, however, have to be done.

I’ve been following the ruby on rails FastCGI vs threads scaling debate and was particularly taken by Cameron Purdy’s comments about Database connections. Which brought me to thinking about the worlds people live in. Cameron, I expect, lives mainly in the high end of “enterprise” (because people only really need his product when they are hitting those walls). Most people don’t, most developers live in the world of one or two databases and a few thousand users. Which leads to two implications:

• most people don’t understand how hard it is to live in that world
• most people don’t have to

The psychology of a project requires closure, being able to feel it is done and allowing yourself to relax without nagging concern is a necessary part of the mind set. At the same time the idea of an end where everything will be okay means that project managers can call for heroic sacrifices from the team in order to acheive this. This is unlike maintenance, where the work goes on indefinitely. Maintenance requires a different psychology. Maintenance is not heroic, it just goes on. You cannot call for a supreme effort from your team to finish it because everyone knows that it will never be finished. As most software is developed during its maintenance phase the Project mindset may well be detrimental to the success of the process.