Not Waving But …

Well this is very interesting. A highly concurrent, safely distributed, dynamically updateable, highly reliable platform is just what is needed for a stock exchange or ECN (which is really just a less regulated exchange for other assets anyway).

Of course it will not help in a lot of areas. One of the irritating problems building financial systems is that so much is arbitrary. It evolved in an ad hoc manner with a huge amount of Not Invented Here going on; just look at the number of “Standard” Unique Identifier systems there are for Equities (You have to support them all…). No programming system will fix that, but being able to focus on managing it well because you are building on top of a system that runs phone switches (which are about as reliable as you get; outside Aerospace and Nuclear Installations) will be interesting.

I will be following Joel’s work if I can; and I’ll probably end up buying yet another Pragmatic Programmer title.

I have been feeling a bit dense and struggling with a problem of how to change the state of a running process in Erlang. The problem was that Erlang is a Functional language that only allows single assignment. The solution to holding state in Functional languages is (apparently, to keep the state in function parameters. Thanks to the folk on the Erlang Questions lists, sorry it took so long.

Well, what follows is the outcome of a rusty developer trying to get the hang of state in a functional, Actor based language. It is probably not very “erlangy” but I hope it helps others get out of the mess I was in.
If you are interested the key point for me was to realise that the receive loop happens inside the function (loop()). This means that the loop parameter is already bound and is accessible, but cannot be reassigned. Each set of code can use X but X can only be changed by calling loop() with a new value.

I hope the following makes sense (It does work).
-module(basic).
-export([start/0, rpc/2]).

start() ->

spawn(fun() -> loop(1) end).

rpc(Pid, Msg) ->

Pid ! {self(),Msg},
receive

{Pid, Response} -> Response

after 50 ->

io:format(”no response~n”)

end.

loop(X) ->

receive

{From, cancel} ->

io:format(”~p is exiting~n”, [From]),
From ! {self(), X};

{From, {get_x}} ->

io:format(”X is currently: ~p~n”, [X]),
From ! {self(), X},
loop(X);

{From, {set_x, Dx}} ->

io:format(”Changing X from: ~p to: ~p ~n”, [X, Dx]),
From ! {self(), Dx},
loop(Dx);

DoNotUnderstand ->

io:format(”~p does not understand: ~p ~n”, [self(), DoNotUnderstand]),
loop(X)

after 10000 ->

io:format(”Nothing has happened, X is:~p~n”,[X]),
loop(X)

end.

I have been following Matthiou Rieu’s Raven project using Rake and Gems to improve the Java build experience. Now he and Matthew Foemmel have got together to merge Raven and JRake.

I have high hopes for this and will be watching developments with interest.

I do hate it when I’ve been dense. I hate it even more when I have been smugly dense.

I have always had a thing about applications producing to much logging. Generally a production log line should only be produced if it is needed. That is if someone is going to look at it and take action on it or it is going to be part of an audit trail of some kind. I get quite worked up about the issue (a developer on my team once deployed an application that produced several Gigabytes of logging a day; “just in case”.)

Anyway, a couple of days ago, I had just related to Steve Freeman that in another project a previous developer had developed a rather byzantine Java logging framework that required a logging API to be passed in to all classes. I was sitting back waiting for the inevitable “yeah, what an idiot” response when Steve actually said “sounds like a good idea”.

Huh?

“Well”, continued Steve [this is from memory], “passing in the API allows you to test the log output of the class and, as logging is part of the class API it should be tested. If it is important that that line is produced you should test it. If it isn’t then you shouldn’t produce it.”

DOH! There we are, very succinct and to the point. If your production system produces it you should test it. Completely obvious in retrospect, and it may help solve the over logging problem most teams seem to have too.

(more…)

I have just read this article about Toyota’s continuous improvement culture. The article is very inspiring, I recommend it. The conclusion highlights a crucial point; one that is, I believe, the source of many of the problems introducing agile approaches.

“People who join Toyota from other companies, it’s a big shift for them,” … “They kind of don’t get it for a while.” They do what all American managers do–they keep trying to make their management objectives. “They’re moving forward, they’re improving, and they’re looking for a plateau. As long as you’re looking for that plateau,it seems like a constant struggle. It’s difficult. If you’re looking for a plateau, you’re going to be frustrated. There is no ’solution.’”

Even working at Toyota, you need that moment of Zen.

“Once you realize that it’s the process itself–that you’re not seeking a plateau–you can relax. Doing the task and doing the task better become one and the same thing,” Shook says. “This is what it means to come to work.”

Experienced agile proponents know that continuous improvement is part of the deal. In order to sell the process they usually sell a “brand” that is rolled out “as is” because they are selling to an organisation that is looking for the solution. The problem is that there is not a solution; not within a single organisation if it is any size, not even within a single team if it exists for any length of time.

There is not, and can never be, a perfect system or process. The world is a complex, ever changing environment so any system needs to continually adapt. Even if the environment did not change much changes in the process generate unforeseen consequences that require further changes to compensate.

When selling an Agile, Lean or whatever process you have to make the practitioners understand that this is simply a starting point for a continuous meta-process of continual reflection and improvement carried out by everyone involved. If they don’t understand that they (and you) are going to fail.

In a post about context switching Dmitri Zimine inadvertently shows how a strong commitment to specific Agile Practices can undermine the agile principles.

The story is how a development iteration was screwed up by the organisation’s attempts to expedite an unexpected request from Sales. The story is hypothetical and is there to make a point so the characters actions seem a little extreme.

I will summarize the post:

1. A two week iteration has just left planning and development is starting
2. Sales Guy comes up with urgent request
3. Sarah the Developer makes a vague, non-committal estimate for two hours
4. Development Manager suggests not doing it because iteration is already planned
5. Project Manager uses the “its just two hours” to push the work
6. Sarah ends up spending three days on the work - iteration is “dead”, developers are demoralized, commitment to agile practices is damaged.

Dmitri’s view of what should have happened: Sarah is left alone. Development Manager offers the Project Manager two options:

1. “No” : Put the request in for the next iteration (effectively waiting 4 weeks for feature).
2. “Drop Everything” : Cancel the whole iteration and re-plan with this on top.

What principles do I feel have been sacrificed here?

• Individuals and interactions over processes and tools
• Customer collaboration over contract negotiation

The Development Manager’s basic problem was that he did not interact and collaborate, he had negotiated a process up front and stuck to that process. He also failed to use any opportunity to gather more data to help him make a better decision.

There are a couple of injections that could have been made.

Developer gives a non-committal two hours estimate:

injection: “Lets time box this, have a look for an hour then come back with something firmer. Ask Cassandra (another experienced, but more pessimistic, developer) to look at it with you)”

outcome: We get a better estimate, hopefully nearer the correct order of magnitude, Sarah has some backup when she makes the re-estimate.

Project Manager uses “It’s just a couple of hours”

injection: We’ll lets see, have you discussed this with the sponsors? They need to know that they may loose something for this. We should have a better estimate in an hour, can you schedule the call for then?

outcome: The business stakeholders who have a better idea of the financial impact will get to make an informed decision.

With those two injections the positions move. If Sarah and Cassandra come back with “two to four days” then the stake holder’s discussion will be more focused. If all the business sponsors still agree this is really more important then a re-plan is not out of the question but everyone will know that it is a significant event.

Context is important:

• Does this Sales Guy do this to you every iteration? Is this the first time in a year?
• Is this totally out of the blue? As Development Manager its good to know what’s hot with the sales team.
• The Project Manager does not seem to have bought in to the process? Why not?

No one likes expediting, it always causes disruption and should be seen as an exception. If it is the norm you really are in trouble. It will happen from time to time, for sound reasons and your process should be able to absorb it.

Any development system is a Complex Adaptive System and one of the requirements for a successful Complex Adaptive System is that it does not violate the Law of Requisite Variety which says: “The larger the variety of actions available to a control system, the larger the variety of perturbations it is able to compensate.”

The development Manager had two reactions to the request: “No” or “Everybody drop everything! we have to re-plan the entire iteration because Sarah has to do a couple of hours on something else”.

These seem unreasonable to me.

However, after the injections we get: “The business sponsors don’t think its worth a two to four day disruption, we’ll do it in the next release” or “Everybody drop everything! we have to re-plan the entire iteration because something really important has come up that is going to take several days.”

If we were not doing Scrum we might even be able to shuffle the plan “in flight” but that’s a different discussion. If the team has a rotating firefighter role can they do it? There are nearly always options.

A lot of people ask for repeatable and efficient development processes. One of the biggest arguments against many of the agile practices is that they introduce inefficiencies:

“Refactor the codebase? But it is more efficient to just do it right the first time.

“Write tests while I code? But its more efficient to write them afterwards when I know what to test”

As for Repeatable, that is the Holy Grail of process to many managers. If they can have a repeatable process then they can just write the plan, kick off the project, and watch the software being delivered.

The problem is that in the quest for efficiency and repeatability we can loose the more valuable qualities of effectiveness and reliability.

No amount of efficiency will help if you are developing the wrong thing and, brutally, the only repeatable process most software shops have mastered is when to give up on the specified process and get on with shipping something.

When you are thinking about your process forget “repeatable” and ask yourself: “Will this reliably deliver what we want?”

When think about some practice ease up on how “efficient” you will be and ask yourself: “By doing this will we be more or less effective?”

This is a (partial) list of operational tools and habits that I have found extremely useful in the past:

1. One Click deployment: Each component of the system must be deployable in a fully automated fashion. There must be NO manual steps, a checklist is not good enough. If you can transparently audit it all the better.
2. One Click rollback: Your rollback in case of a failure must be fully automated. Again there must be NO manual steps.
1. The one-click should not require explicit access to the deployment servers.
2. If multiple components are to be deployed then this should be done by a script.
3. NEVER give developers write access to the production servers, an accident WILL happen if you do. Read access is fine and as you have one-click, audited, remote deployment they don’t really need it do they?
4. Everything builds from a tagged source in a source management system or is a third party library of a known version (This may seem obvious but is often not the case).
5. The separate components in a distributed system must be loosely coupled enough that they can be deployed independently. This is not always possible but it is definitely worth the effort. You want to avoid having to rollback multiple systems supported by different groups because of a single failure.
6. If you have to deploy a non-backwardly compatible change to the communication protocol (it happens) then do a protocol release on its own.

The one that generally causes the most fuss is 4; but if your system requires regular access by developers you have a problem.

I have been following JP’s musings on Project Management and Communication focusing on estimation and reporting.

For a successful technology, reality must take precedence over public relations, for nature cannot be fooled. : Richard Feynman

In my last entry I discussed the social reasons why I felt that a lot of projects miss their deadlines. Here is the setup for the scenario outlined there.

Richard Glass in Facts and Fallacies of Softwaring Engineering notes that one of the two most common causes of runaway projects is poor estimation (The other is unstable requirements)

• Estimation is usually done at the wrong time (too early)
• Estimation is usually done by the wrong people (management, architects, senior developers)

To which I would add:

• The delivery rate of any one team/project combination is a unique value that is very different to any other combination.
• No one is any good at estimating large software tasks, especially if someone else is actually going to do the work.
• No one should be estimating a software task if they won’t be involved in carrying it out.
• No one is a born estimator. It takes practice.

Many developers/development managers do not realise that “The Map is not the Territory” and , finally, few development managers really understand the above

So, when the project starts to slip, and the estimates are not being met many development managers feel that someone is doing something wrong now. They don’t realise that an error has already occurred and the results of that error have been captured into a number of documents, Gant charts and public commitments.

Tom DeMarco points out that managers often think they have the power to make people think faster by putting on the pressure. So, given that they have committed to a timetable they feel the best way to get back on track by applying pressure - to make doubly sure they make everyone think longer too by increasing the hours.

Finally, to save embarrassment, they finesse the data until they recover the lost ground…

There are better ways.

In Reflections beyond technology Stephen points out that in projects
“denial trumps rationality with alarming frequency!”. This is undoubtably the case. Reality is cold, hard and obdurate. Once it arrives you have to deal with it.

Some people have a habit of leaving bills unopened (or just throwing them away), why do they do that? Do they expect them to get smaller, do they expect the utility company to forget? and yet…

In a software project there comes a point where many, if not all, the development teams know that they won’t make their targets. Everyone has made their commitments, the project looks healthy from above and senior IT management are feeling good about this one.

In the project meetings though, delivery date poker is played, each manager staying in the game with a “yes, we are still on track”; keeping up the fiction that they will hit the date, throwing a little more of their reputation into the pot while hoping for another player to fold and take the blame for the deadline slip.

The game ends when, finally, someone cannot hide it anymore and admits they will not hit their target. The senior stakeholders have to finally accept that the plan is wrong and the project deadline cannot be met. While all the other technical managers breathe a sigh of relief and quietly adjust their schedules to get back in line with reality.

Lying while expecting to be uncovered at any moment is massively stressful. Believing everyone else’s schedule lies while knowing that your public commitments are lies is terrifying. Which is one of the reasons why IT development is still rated as one of the most stressful careers around.

Many managers deal with it by tacitly expecting their reports to lie to them (stacking up those bills). Insulating themselves from fault though plausible deniability, after all “they thought the project was in good shape”. When reality finally beats denial the blame for the slip cascades down through the organisation until the poor guys who finally admitted that there was no chance they could hit their target get the entire blame.

So, why do projects regularly miss their target dates? because, paradoxically, it is in no ones interest to be the first to admit that they will.