Agile is Failing (again)

Photo by Michal Pokorný on Unsplash

It’s Wednesday, so it must be time for the internet to decide that agile is failing – again.

The constant cycle between waterfall, agile, lean and back again is currently focused on agile. Yes, it was never going to work, or it’s a scam, or its just not working, or maybe you just need to commit more. Or is it time to move to Kanban, or maybe back to Waterfall?

It’s exhausting.

But where you have some luminaries of the industry such as Allen Holub and Kent Beck raising issues with agile methodologies, maybe now is the time to take notice.

What’s the Problem?

After thirty years of the waterfall method (thank you Winston Royce), the engineers stepped in resulting in the Agile Manifesto. We had enough of this documentation heavy approach. Waterfall had clearly failed, with up to 60% of projects failing, time for the real experts to act.

So, the introverts were advocating for individuals and interactions over processes and tools, responding to change over following a plan, customer collaboration over contract negotiation (honestly).

Sounds about right, well, except for that introverts communicating bit.

Agile started of pretty small, a little bit of Extreme Programming, a little bit of Scrum and that was your lot.

On to 2017, I was doing a lot of consulting on agile transformations, architecture and high performing development teams. One of the talks I regularly delivered was an Introduction to Agile, which was challenging as by that stage I could identify approximately 40 different varieties of agile.

What happened? Wasn’t this supposed to be easier?

Three things happened in my opinion:

First the traditional project managers subsumed agile and made it part of the wider PMBOK approach. Agile wasn’t different, it was just a different flavour. When I’m boring people about systems thinking and it’s implications for the development process, I often discuss homeostasis, the process by which any system regulates itself, as one of the challenges experienced.

In human terms you experience homeostasis all the time… Too hot? your body sweats to cool you down, too cold, your body pops goose bumps and starts to shiver to warm you up. In organisational terms it manifests as resistance to change. This is were project managers found themselves. Agile had the possibility of killing their jobs, so it either had to be killed off, or absorbed, a full change was not an option.

They absorbed, and then transformed agile into Watergile? Agifall? Whatever it was called, what it became was an attempt to bring traditional waterfall style processes and metrics to agile methodologies. The very thing that agile was supposed to be changing was now representing itself as agile.

Second, money. In the storied history of humankind it has never failed to discover new ways of separating people from their assets and money. Agile emerged in around the time of the dot com bubble in the late 1990s where money was being thrown at the new gold rush with a fair degree of abandon.

How does one take something like agile and make money out of it without getting caught up in the messy business of having to prove it’s effectiveness? Particularly challenging when the process itself is fundamentally intended to adapt to the unique circumstances of each company in which it is implemented?

Well, you turn it into a product of some kind, say training for example. And then you sell the training, and where possible, establish a some mechanism that drives people towards that training, like certification for example. Products have an interesting characteristic though, they tend to require commonality to sell in any kind of volume. Yes, the solution then was to standardise, stick it on a shelf and then sell it on.

This may sound like it flies in the face of the agile approach, where adaptation is a core feature, but if you can bake that into the product description it should be fine. If you’re sceptical that this approach can work, well I have some healthy, low fat, low sugar yoghurt that might interest you.

It worked, there are as many agile certifications available as there are agile methodologies (try agile certification as a search term).

Third, that introvert thing. The agile manifesto was devised by seventeen people who met in a ski resort in Utah to talk, ski, relax, and try to find common ground. I’ve been in the software industry for over thirty years, and in my experience developers as a community tend not to gravitate towards a weekend of chatting and sports.

It turns out that the vast majority of developers want to write code, and by that I mean just write code. If you look at the principles behind the agile manifesto you’ll find it right there in black and white in the following steps:

One: Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

Three: Deliver working software frequently…

Seven: Working software is the primary measure of success…

Nine: Continuous attention to technical excellence and good design enhances agility.

Eleven: The best architectures, requirements and designs emerge from self-organising teams.

That’s a full 41% of the principles that directly reference software, development or architecture.

Communication, by comparison is referenced in:

Four: Business people and developers must work together daily throughout the project

Six: The most efficient and effective method of conveying information wo and within a development team is face-to-face conversation.

That’s twice, and arguably point 4 doesn’t really require direct communication. At best 16% and in reality closer to 8%.

Question for the audience, how many times are metrics or reporting mentioned?

Yep, developers don’t like the bits that require anything but code. So they increasingly didn’t do it. In fact, over time all that has happened is that developers have forgone architecture and design (see my post here on this topic…) or any substantial documentation and resist anything that looks like, smells like, feels like or hints at waterfall.

Where we are Now?

So, project managers bad, standards bad and developers bad or lazy?

Harsh, but maybe true if we accept sweeping generalisations.

Right now, one one side we’ve got agile coaches, scrum masters and the more traditional project managers merging into the style of the old project management function, which if you review the results was already failing to manage software projects around 60% of the time. The certification process has given a badge of competency to anyone who can study the material, without investigating the awardee’s direct experience or ability to actually apply agile.

On the other side, developers have moved away from all of that communication, preferring to focus on architecture (sometimes) and code, promising that technical solutions like cloud, microservices and DevOps will fix the problems of delivery whilst all the time getting further and further away from the customer. Process in general has declined, and developers took agile as a license to do less of it.

The business, meanwhile, saw agile as a technical methodology, so they’ve rarely engaged to the level of work together daily as suggested by the principles. The developers didn’t complain, it dropped that communication requirement down to 8% – still a little high, but tolerable.

The certification bodies saw a new opportunity, the product manager. Not a project manager, and not a member of the business, but a new role that was injected between the business and the developers so that neither actually had to talk to each other. Everybody seemed happy.

Finally, there have been a few success stories that showed how agile could work. Ladies and Gentlemen, I give you the Spotify Model. And despite Spotify themselves saying that they don’t follow it religiously, and that they published it as a point of interest for people, not for it to be followed by others, it still became the next holy grail. How bad did this become? Do a search for spotify model certification to see for yourself.

Is the Patient Dead?

In short, no.

You can stop reading now if you like.

The problem today is the same problem that we had twenty years ago, we’d rather pick a point, off the shelf solution which has the comfort of familiarity than deal with the discomfort of the unknown.

Thus, when someone shows up with a ‘solution’, we’d rather buy that than trying to figure it out for ourselves.

We’re herd animals, following rather than standing out. It was true for client server, the web, virtualisation, containerisation, microservices, the cloud and then full serverless and function as a service (which we now don’t like). Similarly, waterfall, agile, lean, and then spotify and back again. We follow whatever we see as the leader, ignoring the obvious indications that ‘they’ are not ‘us’ and what ‘they’ did may not actually work for ‘us’.

Releasing a software product of any size is a result of two interrelated systems: the first is the system used to produce the software, this is the development process and the architecture, the second is the system itself, which is the architecture and the code that implements it. It’s inescapably complex, and, as it involves humans, unique to each circumstance.

So agile is as likely to die as waterfall, which is not at all. In principle, I believe it remains the best approach to delivering working software systems.

Just not in isolation.

Agile has much to offer in terms of structuring a development process, as does waterfall and lean (even if lean is a form of agile). But so too does organisational theory, architecture standards, psychology, leadership, strategy development and so on.

We need to correct the divergence between developers, project managers and the business, all have a role to play in the solution, but no single one of them holds all the pieces.