Agile – Just Not Flexible Enough?

What did Kevin Do?

Did you happen to read “We Need To Talk About Kevin” by Lionel Shriver? The entire book progresses hinting at some great tragedy, but without actually revealing the full extent until close to the end (at least that’s how I remember it). The book, written in the form of letters from Kevin’s mother to his father, is a journey of reflection by Eva (the mother) who didn’t really want kids, and how she is now questioning whether her dislike for Kevin may have precipitated the (later revealed) tragedy and the subsequent events.

I won’t ruin the book by doing some big reveal, you really should just read the book.

By now, once again, you’re no doubt thinking, more clickbait Dave? Really? Have you no shame. Those of you that know me have long since realised that no, I have no shame, but I’m still offended by the clickbait remark.

Many organisations have adopted agile and, particularly with large organisations, it is often over the objections or misgivings of a long established project office (Eva) who then treat the methodology (Kevin) badly until, well, tragedy strikes.

What Is Agile All About Anyway?

For the sake of simplicity, lets say that every project is a software project, and further that the management and delivery of those two projects is either by traditional methods, or by agile methods.

Our traditional methods are the well known waterfall method, while the agile methods can include Lean, Scrum, Kanban, SaFE, DSDM, Scrumban – basically anything that can trace its roots back to the agile manifesto (I include Lean for completeness – those of you from the process control world please don’t get too upset).

Already you may be looking at this and thinking, one side of this comparison seems to have just the one entry, while the other has quite a few.

Let’s make it simpler again, all of the agile methodologies are based on the premise that the customer doesn’t know what they want, whereas the traditional / waterfall methodologies are based on the premise that work can’t start until everything is known.

Don’t like this simplification? From Scrum:

“[The demo / review] …allows them to feedback and discuss with the Product Owner and Scrum team any possible amendments to the Product Backlog which would help to maximise value.”

So the demo is to demonstrate the work, get feedback on its suitability and allow adjustments to be made. If everyone knew what they wanted, why the adjustments?

Still don’t like it? I offer you Buddha:

When you find no solution to the problem, it’s probably not a problem to be solved, but a truth to be accepted.

Buddha

On the other hand, waterfall likes to compartmentalise. Requirements before design, design before detailed design, detailed design before test plan, all of this before coding starts, coding complete before testing. Everything controlled by a master plan, that in turn has a documented process for changing the plan, and that’s in place before any work starts and so on and so on.

For those of you looking to understand the fullest extent of where waterfall could get to, I refer you to the European Space Agency PSS 05 standards, which controlled software engineering standards in the time before time, when we dragged our knuckles across clickity keyboards, we trapped mice and before agile was even a concept.

I understood at the time that when dealing with capital intensive projects, particularly those that could literally blow up, that one might tend towards high levels of control, but this was extreme. I used to jokingly look for the ‘Blow your nose process document’ along with the supporting ‘Tissue change control document’.

Truth be told, I actually used to refer to a similar hygiene related process, involving tissues, but somewhat, ahem, ‘South of the Border’, so to speak. This is too polite an audience for such vulgarity.

Agile advocates will tell you that their focus is on Continuous Value Delivery, or some derivative of that phrase. It’s clever, because it not only creates an impression of customer or business focus but also delivers a nice body short to the waterfall method.

Being from the dawn of time, I was involved in many projects that were controlled by project managers with their PMI / PRINCE2 certifications. In all of those projects I don’t remember it ever being mentioned that some aspect of the project wasn’t going to deliver value.

I know this isn’t what the agile advocates mean, but there’s a lot of practitioners out there that seem to suggest that waterfall somehow didn’t deliver value, and the message has stuck stronger than rumours of election fraud.

Stealth Agile and the Resistance

In the early days, Agile was adopted by small teams of engineers, and often championed by the most organised developer. It was welcomed, and indeed lauded by the software folks because it was light on process, and resulted in less documentation leaving more time for coding.

Agile started to sneak into larger organisations, often in the quiet corners where development teams would meet daily for their ‘stand-up’, but publicly continue to meet weekly for project update meetings with project managers.

And it worked!

Word started to spread, and the success of Agile was plain for all to see. Mostly as a result of the many, many independently authored and unbiased books that recounted the success and transformational merits of the process. The books kept on coming, and the processes, the conferences, the advocates and then the hype machine took off. The last time I looked, which I admit was some time back now, there were over 20 distinct agile methodologies with supporting books, champions, certifications and ecosystems. Agile, it turns out is a great money maker.

The original manifesto had four bullet points and twelve principles, just for reference.

By the time the traditional project management world started to take notice, Agile had taken root, had disrupted the entire development and delivery process, and its cheerleaders were now using terms such as ‘Legacy Methods‘ and ‘Old School‘ to emphasise that those dinosaurs who continued to practice the ‘Old Ways’ certainly weren’t going to be allowed to hang with the cool kids.

That didn’t go down well.

Project Managers and Programme Managers started to resist. Initially as a full frontal assault decrying this loose and undisciplined approach as being unsuitable for ‘real’ projects. When the weight of the hype put them in jeopardy of being dubbed ‘archaic’ or ‘lacking innovation’ they changed tack.

Rather than fight, they adopted the agile methodologies (the venerable Project Management Institute now offers agile certification which they call PMI-ACP). In what I presume was an homage to Point Break (the original movie with Keanu, Patrick, Gary and Lori, not the rubbish reboot) project managers (the FBI) shed their suits and joined the Agile Gang (Patrick and his lot, but curiously with a cameo by The Red Hot Chili Peppers, but I digress).

Through this infiltration, increasing levels of standardisation began to be introduced to the Agile process. In an attempt to make Agile more suitable, and indeed palatable to enterprises, it was gutted. In many respects this suited all of the authors and consultants who could happily sell standardised approaches, but it had actually drifted a long way from the original values that were espoused in the manifesto.

In fairness, it wasn’t hard. Those original champions for agile methods who heard about the agile manifesto and principles often didn’t have the discipline and training in project management techniques, and let’s be honest, didn’t fully understand the problems that those techniques were designed to solve.

Worse yet, there was a problem with agile, which continues to this day. In the forms that currently exist it simply doesn’t work for large scale undertakings. It doesn’t meet the needs of the stakeholders in those projects (I’ll write more about this separately), and it supports practices that are very much at odds with how large organisations work, and yes I do include SAFe, LeSS and similar methodologies in this criticism.

So that’s worrying, and it’s not getting any better. A recent survey suggested that currently 71% of companies are planning to or have adopted agile. So here’s a challenge (for the three of you still reading this), do a search and look for reports of successful large scale agile projects. Be careful, don’t just look for large companies that have had success with small projects or small teams.

You’ll find the HP case, you might find the Bosch case, and the more ambitious will bring up Spotify. You’ll also find a lot of academic articles on how to implement agile at scale, and even more from vendors offering insights and courses on how to implement agile at scale. But the world is not being flooded with tales of the rampent success with Agile on large projects.

Ouch.

The fault, dear Brutus, is not in our stars, but in ourselves…

So here’s the thing (welcome back those of you who skipped to this part), we took two broad approaches that each had their merits, and we butchered both.

Quite simply, there are some projects that we never want to subject to an agile approach. Agile relies on the ‘Sashimi‘ approach, delivering small increments of functionality and adjusting based on how well that works. It makes a huge amount of sense when the customer / business doesn’t really know what it wants, or how to articulate its needs fully.

This approach may be fine for small projects, or perhaps more accurately discrete and reasonably isolated projects, but it’s not how you want to build your aircraft control system for example, or your nuclear power station safety systems, or in fact any mission critical system. There are simply things that can’t go wrong in some cases, so you use the most rigid and most disciplined approach you can – so peoples lives aren’t at risk.

Similarly, in a fast moving market, where opportunity is available for only a limited time, where striking whilst the iron is hot is the only factor in play, there’s no point in spending years in design and planning the ultimate solution before getting something to market. The waterfall approach is simply too laborous and to rigid for these scenarios.

That all seems pretty obvious, so surely we just need to cut our cloth to measure as the saying goes? Sure, but that’s not what we do. The hype engine, internal politics, the desire to find a silver bullet that solves all problems, the promise of a quick win and the simple human condition of seeking shortcuts where possible means that we want one way, one simple way that is universally applicable.

On that basis, the Agile advocates have sold the story that it is the ‘best’ way for all things (with few exceptions) despite its shortcomings in large scale projects and the waterfall advocates, recognising that Agile isn’t the be all and end all look to retrofit structure and standardisation onto Agile, killing its agility (remember Eva?) until the Agile tribe to become more entrenched and more zealous (hello Kevin).

The result, well it’s all those articles you’re reading referring to agile being a failed approach.

Pulling It All Together

In a last ditch effort to keep a few of the remaining readers, I’m going to suggest the factors that could make all of this work.

First, we have to recognise that the important aspect of Agile is the philosophy, not any specific methodology. Agile, at its very core is about building the highest level of understanding between all stakeholders, business and technical to maximise the probability of a positive outcome.

Second, we have to recognise that it’s not always possible to put together a team or a number of teams where every member is familiar with the business domain, technically competent and socially aware enough to work with others. It’s tough enough to get that in one team, which may explain why fundamentally agile won’t scale. So we need to compensate for the fact that teams are both fallible and human. Some of the compensations that apply are technical or architectural but also structural and some are simply about better up front planning à la waterfall.

Third, we need to re-imagine our concept of team. One of the things that I think we do well in InvoiceFair is integration, well beyond the simple collaboration, it’s an ethos of shared accountability. Delivery of business outcomes is a team game, and teams need leaders and players from across the organisation. In the fintech world, like so many other industry disruptors, that team better be a high performance team with a diverse range of skills if it wants to succeed. High performing teams recognise that either everybody wins or everybody loses, individual success only helps if the result is the entire team succeeding.

Fourth, the process adopted by an organisation, whether agile, waterfall or something else, has to permeate the entire organisation, and the trade-offs and implications of this need to not only be recognised, but accepted and supported by the entire organisation. The process isn’t just an IT thing or a development thing.

This leads to the last point, and for me goes to the heart of the processes that need to be built. We need a process that recognises that there is something of a natural order of things, which waterfall deals with extremely well. We also need a process that is comfortable with ambiguity and uncertainty, which agile, and scrum in particular deals with well – not Kanban interestingly, but that’s a separate issue.

Furthermore, we need a process that is transparent and delivers the right information to the right people and places at the right time, which is something that Kanban deals with very well. The process needs to foster communication, understanding and critically demands interdependence over independence and for all that every process claims this, developers are shockingly bad at this, but can be helped with coaching.

So which is the answer?

Hybrid, picking the best of each methodology and combining every time. To be successful you simply must give time to building a process that is tailor made for your organisation that takes into account your people, your market, your environment and the current prevailing commercial conditions. You need to shape this process to meet your production and operating needs, including metrics and required outcomes.

Most importantly, when any of the initial conditions change you change the process. There’s no point in applying a process designed for a 10 person organisation where everyone shares a room to an organisation of 50 people or 100 people who are working remotely, or are geographically dispersed. Similarly there’s no point in picking a single established process and hoping that with a few tweaks here and there that it can be adapted to your unique needs.

You need one of your team to have complete accountability for this process, and that person must be carefully selected so that they have a combination of skills from the various disciplines, but who are not anchored to a specific methodology.

What does that look like? Well InvoiceFair is very much at the early stages of this process; I’ll post some more with the lessons we’ve learned and the pitfalls we’ve encountered as we move through our next gen platform project.

Watch this space…

One thought on “Agile – Just Not Flexible Enough?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.