Backlog Refinement – It’s Exploration, not Negotiation

Establishing closer relationships between the business people and developers is a central principle of agile processes. At the heart of this relationship is the Product Owner role, which carries the responsibility for establishing a bridge between both groups.

One of the critical processes managed by this role is that of backlog refinement. Just to remind ourselves:

Backlog Refinement is the process through which the Product Owner and the Development Team establish a shared understanding of the items in the backlog, produce an estimate for those items and establish the priority in which those items should be handled by the developers.

That’s all fairly straightforward, right? A simple, continuous process where the Product Owner and the Development Team discuss, estimate and prioritise. What could possibly go wrong?

Quite a bit as it turns out.

The Product Owner

Product Owners are a curious bunch. As a role, in my experience, it is populated by individuals largely coming from a technical background who can rapidly assimilate knowledge of the domain in which they are operating. There are a number of behaviours that some Product Owners exhibit that can undermine the backlog refinement process; I’ll look at the three most common that I’ve encountered in the following sections.

Sales Mode

Some Product Owners are drawn from the business side of the organisation due to their extensive domain knowledge. With this background, they can be a fantastic asset as they easily understand the capabilities that the business is requesting.

One of the challenges that Product Owners coming from this background must manage is the natural tendency to fall into ‘Sales Mode’. In sales mode, the Product Owner has stopped listening to the business and has started trying to influence their direction based on their own opinions. Product Owners who have turned into sales people are no longer fulfilling the core aspect of their role which is to represent the needs of the business to development team, instead they are starting to subvert the business.

Guru Mode

Some Product Owners are drawn from the technical side of the organisation, where more often than not they have developed domain knowledge through experience of implementing systems for the organisation. With this background they can also be an exceptional asset, understanding what is already in place and the broad rationale behind the technical systems.

Coming from this background, the Product Owner runs the risk of entering ‘Guru Mode’. In meetings with the business they will undoubtedly be the most technical person in the room, where their opinion will be sought on potential solutions. Product Owners who drift into solutions mode are equally drifting away from the other core aspect of their role which is to build a shared understanding of the backlog with the development team, instead they are starting to undermine the development team by limiting the solution options available to them.

Project Manager Mode

The last type of Product Owner we’ll discuss is those drawn from the programme or project management side of side of the organisation. With their background, they offer exceptional coordination and communications capabilities, which is to the benefit to all sides in getting initiatives delivered.

In many respects those coming from the Project Management background have the greatest challenge. The role calls for their coordination and communications skills, but the risk is that they fall back to full management mode. In management mode, the Product Owner transitions to dictating mode rather than collaborating mode. At the most dysfunctional level, this tends to question estimates, and insist on detailed work breakdowns at backlog refinement. This approach is actually abandoning their role as a Product Owner, and in many cases equally abandoning the entire agile process.

It might seem logical to conclude that all of the issues with backlog refinement rest firmly on the shoulders of Product Owners who have turned to the dark side.

If only life were so simple…

Developers and The Development Team

Developers are very far from the white knights to the evil Product Owner.

So much of the education for developers, both formal and on the job, is focused on producing the smallest, most efficient and best performing code that requires a minimum of maintenance and at best no rework. This is ingrained into the psyche of every developer despite the efforts of agile processes to advocate for the Sashimi approach (which I discussed in my last blog post Agile Errors – Sashimi and Skateboards).

This leads to some behaviours that can derail the backlog refinement process as quckly as a bad Product Owner. In the sections that follow I’ll explore some of these particular issues.

The Superior Intellect / The 10x

Coding is hard regardless of what you might have heard. Software developers are creating something functional and tangible out of nothing, often with just experience and a few guiding principles to lead the way. Unlikely as it may seem, you want your developers to have a something of a superiority complex. They need the confidence, bordering on arrogance, that despite of the myriad of challenges that will be faced that they can craft a working solution.

This self belief is further enhanced by tales of the mystical 10x developer: a developer that is so vastly superior in their skills, and so productive that one is worth ten lowly regular developers. There are many more developers who believe themselves to be 10x developers than could statistically exist. Herein lies the problem.

Many developers adopt behaviours where they convince themselves that they already understand not only the problem but also how it can be solved. Many may even arrive at solution before the problem is discussed, but we’ll save that for another post. Developers who drift into this superiority mode tend not to listen, or only listen selectively, which undermines the core aspect of their role which is also to build a shared understanding. They undermine the role of the Product Owner, which, contrary to any agile principles, results in delivering a solution that does not meet the needs of the customer.

The Engineer

Talented software developers live at the intersection of creativity and engineering. Strong engineers produce robust, resilient, secure and scalable code that results in a reliable system that satisfies the customers needs. A strong engineering discipline lays the foundations for flexibility and growth of the system being developed, solving the right problems at the right time.

Leaning too heavily on their engineering foundations can result in over-engineering, often with more complex solutions than are necessary to solve the customers needs. This is often excused with a rationalisation that this is the ‘right’ way to solve the problem.

The backlog refinement process that results in prioritisation uses many information points, one of which is a time / size estimate. Over-engineered solutions take longer, absorb more resources and are almost always prematurely optimised. This ignores the role of the development team which is to split larger items into smaller items and to ensure that the ‘right thing’ is implemented, which can lead to poor prioritisation of items in the process.

The Coding Zealot

Software developers are not noted for being extroverts. There is a reason that our stereotype of a software developer is one of a person hunched over the keyboard typing strange incantations that instruct the machines in the cloud how to behave. Agile processes have evolved over time to allow developers to maximise the time they spend coding, which is what they are being paid to do.

The overzealous coder can fall into the trap of assuming that everything should be coded, to the exclusion of off the shelf solutions and services. Sometimes described as the ‘not built here’ mentality, it can result in large amounts of work being undertaken, often unchallenged, the outcome of which could have been more efficiently achieved using a third party solution.

Central to the backlog refinement process is the idea of a shared understanding. This can often be interpreted as the Product Owner informing the Development Team, but it is equally important for the Development Team to explain and clarify what it will be doing to the Product Owner.

Maybe, then, the problem is with the Development Team? Sadly, also not so.

At the root of these issues are a number of areas that have been discussed in various posts here in the past. At the root of any of these issues is the culture of your development organisation that allow for these poor behaviours to continue.

Fixing the Dysfunction

These behaviours all result in a process that is more of a negotiation between two opposing sides, rather than an exploration of the options by a single team focussed on an outcome. It’s a cold war that allows friction to grow across the entire group. The solution, in my experience, cannot be solved by the Product Owner or the Development Team alone, it requires leadership.

There are a few simple steps that will dramatically improve the situation. Simple to say, not simple to do.

Define the Outcome

Before undertaking any project, the desired outcome must be clearly defined and communicated.

There’s little point in expecting a quick MVP on the market if you don’t set that as a constraint for the development team. Similarly, there’s no point in anticipating a secure, scalable and resilient solution if you don’t leave the development team the freedom, and time, to design and build those foundations.
Everybody needs to be clear on the goal, and where their responsibility lies in achieving that goal.

Define the Roles

Role ambiguity, in my experience, is one of the most commonly cited reasons for people leaving their job.

Everyone must clearly understand each role in the development process including its responsibilities and its constraints. There must be no questions on authority, responsibility and most importantly accountability in anyone’s mind. This clarity also demands consistency. It’s unreasonable to allow the same role to have different responsibilities in different projects and not expect problems.

Nature, as the saying goes, hates a vacuum. Without clarity and constraints, the opportunity for overreach will exist. Overreach by any party to the process will result in friction and friction will absolutely add risk to the possible success of the project.

Permission and Accountability

People must be allowed to do their job. This sounds like an obvious statement, but all of the challenges detailed to this point represent one or more people denying others exactly this opportunity.

There are two clear examples that will illustrate the point, and unfortunately they will be all too familiar to anyone in the software development field.

First, if the Development Team are being asked to do a fast and dirty MVP to meet a specific business need, then the must be allowed to clean up that code, and sooner rather than later. Developers want to produce quality solutions that they can be happy to stand over; they should not need to enter into a prioritisation discussion to clean up their code after the MVP is available. Developers know the cost of dirty code, and they alone carry its consequences, they must have the space to fix the problems before too much new code is built on top of these shaky foundations.

Second, if Product Owners are to meet the needs of the business they must be presented with all of the available options by the Development Team, even of one of those options will result in the Development Team having to rewrite large swathes of the code at a later stage. If that’s the option that meets the business needs, then that option must be offered for consideration and discussion.

Align

More than any other process in the agile world, backlog refinement requires that all participants are acting as a single team. This is the culture that must be established as early as possible. Each person must be aligned to the desired outcome as defined by the organisation.

Working closely together, a healthy culture will allow for open discussion of all of the viable options, a clear understanding of the risks and consequences of each option and a clear path to mitigating those risks. Everyone should be fully behind the decisions and agreed on the follow on steps that ensure that the long term outcomes are in the best interests of the organisation.

There are no individual winners in this process, but a dysfunctional process almost certainly ensures that the business will lose.

Support Ukraine

The invasion of Ukraine is an act of aggression that we should all oppose and speak out against. I will continue to Stand with Ukraine and its people until peace is restored.