Yesterday it happened again. On my Twitter timeline. Someone angrily posted “developers should finally learn how to estimate better.” Another case of work not being done at the end of a sprint. Or worse, another project that didn’t make its deadlines.
There isn’t a single developer out there who hasn’t run into similar experiences.
After another deadline is not met, another manager argues that we are unprofessional. “I don’t understand that you call yourselves professionals, but that you can’t even properly estimate the time necessary to implement features X, Y, or Z.”
Then they take a short pause for the dramatic effect, followed by the ever-popular “How hard can it be?” and the inevitable “When I take my car to the garage, the mechanics know exactly how long it takes to switch my tires. Developers are much higher educated, so why can’t you estimate your work?”
And as usual, we feel guilty. Because we missed another deadline, that was based on our estimates. Because yes, we are professionals. So why is it we can’t seem to estimate our own work? People in almost every other industry can.
We shrug our shoulders and promise to do better in the next sprint. Or in the next project.
But in all honesty, should we really feel so ashamed that we can’t seem to estimate our work? Well, yes and no. As the Scrum people state: “it depends”. Let me introduce a model that helps to explain why we sometimes can and sometimes can’t. It’s called Cynefin.
Let me try to illustrate it really briefly. The Cynefin framework consists of a number of contexts and describes the relationship between problems in particular contexts, and the proper strategies to approach these problems.
In a clear context, there is an apparent best practice for the problem at hand. You simply apply the best practice. Done. These are the problems you’ve dealt with before. Changing your tires in a garage. Or, building a simple web application. Code you’ve written before. This you can estimate.
In a complicated context, there are multiple good practices. Let’s say you are selecting a solution for identity access management. There are quite a few good ones. There are people, possibly even in your own company, who’ve implemented one before. So, you need to select one or two solutions, try them out, and implement one. This is still plannable, and you should be able to give a fair estimate.
Then it gets harder. When a problem is in a complex or chaotic context, best or even good practices don’t exist. At best, they are evolving. You may find people on the internet who’ve dealt with this before. Or even worse, you can’t find anything at all.
When I suggested to the CEO of an insurance company in early 2014 that a microservices architecture would be the best solution for replacing 30 million lines of COBOL and Java code, nobody had done microservices before. We could not build on other people’s experiences. The only way out was to experiment, and hopefully, let practices emerge. If they don’t, you need to experiment more.
As you can imagine, problems that live in complex or chaotic contexts are impossible to estimate well. No story points here. At best, you might present a ballpark figure. “We hope we can pull this of in three or four months.”
Unfortunately, too many managers won’t be happy with your ballpark figures. And this is why we suck at estimation. We are dealing with problems in complex or chaotic contexts, but are expected to estimate our experiments as if our problems are in complicated or clear contexts.
This is where we need to learn to hold our ground. Explain to your managers that you are in the realm of the unknown-unknown – the things you don’t know that you don’t know. Present them with the Cynefin framework, and explain that you are doing things that nobody you know has done before, and therefore, that you need to experiment. And therefore, the metaphors managers eagerly apply from manufacturing, construction, or any other repetitive industry (where work is at worst in a complicated context), do not apply (yet).
But you can also give them some hope. There is some light at the end of the tunnel. After practices emerge, you might be able to push the problems to a complicated context. Where work becomes more predictable. And where you will be able to estimate better.
So, what about the microservices architecture we started to work on in early 2014. Well, it took us almost two years to discover the practices we needed to apply. We started over with parts of the platform multiple times. But eventually, we got good at it. The program finished after almost five years. Successfully.