The End Of Agile Estimation: Predictability Isn’t A “Good Thing™”

by simbo1905

In the last post I gave an overview of large scale build which used Kanban and that shunned estimates. In this post I will get into why Agile estimation is not aligned to agile values. It is simply a hangover from waterfall days and is a “wagile” concept. 
The agile manifesto says that we should value:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on
the right, we value the items on the left more.

First, let us apply a sniff test to “Agile estimates”. Do they look to be aligned to items on the left or items on the right? Proponents of agile estimation say that it is very important that they should be tracking “velocity”. How? Use the tools and follow the process. Why? Velocity combined with consistent estimation leads to predictability. Otherwise, how do you know that you are spending your budget well? That is just about sticking to a plan. This smells wrong.

Estimates are about predictability and that taken as a Good Thing™. But why do we like items on the left more than the right and does predictability help? We want to be generating transformation, responding to change, collaborating with customers, and have a rapid feedback loop, by constantly shipping working software. Then we can be bold, fail fast, learn fast, adapt fast, and maximise the business outcomes. I have no idea why retrofitting predictability onto all that is seen as either possible or particularly desirable. Systems with feedback are not predictable. A compound pendulum is a simple system with feedback. It swings chaotically in an unpredictable manner as though it is alive.

Predictability can be counterproductive. If we aim for predictability we may either consciously or unconsciously change behaviour to look more predictable. It’s easy to be predictable if you consistently overestimating tickets and then either speed up or slow down depending on how things are going. Yet we really want folks simply working on maximising the outcomes not attempting to optimise their activities.

In engineering, a velocity is a speed with a direction. Yet with digital transformation projects the location we are aiming for is a guess. So is a constant speed better? Not really. The secret sauce of agile is to have a rapid feedback loop. So it is shorter cycle time, not predictability, that should be optimised for. The metaphor of what we need isn’t “velocity” but “acceleration” particularly in changing direction, and back tracking, in response to feedback. Only once you have found the mark with a minimal viable feature should you then iterate on it to improve accuracy.

Predictability actually isn’t a Good Thing™ at all when trying to be agile. It’s something that waterfall tried to manage and optimise. Optimising for predictability is a manufacturing paradigm which is actively harmful when building digital services. It presumes an up-front design and predictable execution are optimal. It encourages optimising activities to steadily reach the preordained outcome in a predictable manner. Digital transformation is about optimising for an unknown outcome through continuous redesign and experimentation. An excellent explanation of why old-fashioned budgeting practices are poor IT strategy is in this book. A business led digital strategy should budget for capabilities, not activities. We simply shouldn’t take it at face value that any organisation needs predictability.

In the next post I will debunk the idea that “story point velocity” is a meaningful measurement at all when the scope of any feature is entirely negotiable.