I’m involved in quite a few interviews as InvoiceFair grows its technical team.
Interviews are always interesting; you get a sense of what’s going on with the market, but also what people understand of the software development processes that operate within their own organisation and the subject of software development processes in general.
Almost universally candidates I speak to are operating under some form of agile process – I might write another post on what where those conversations went which is also interesting.
With this understanding established, one area I like to explore during technical interviews is that of architecture and design.
Where Does Architecture Fit in Agile Processes?
It seems like a simple question, but the variety of answers I get are, to say the least, surprising.
Some candidates will simply blank at the question, which is always a bit disconcerting.
Many candidates, often the more junior but some mid level, will start to discuss task breakdown and sprint planning. Still others will start to cite some interpretation of the Spotify model and how this is a challenge best left to the individual squads.
To be honest, I think they all miss the point.
The system architecture and design must start as soon as the overall purpose of the project is defined. It must exist, at a high level, to establish a technical direction for the project, and to provide a conceptual framework for all future technical refinement processes in the development process.
If that all sounds a bit waterfall, you’d be right.
But I think this merely serves to highlight one of the many things left unsaid in agile processes – we need to design before we can properly plan. Equally, when the Spotify model was reaching its crescendo in the hype cycle, the question of architecture and its place in co-ordinating the work of squads was little more than a whisper – but the entire concept, like all matrix organisational structures for technical teams, only worked if there was an overarching technical architecture structuring the development.
Architecture is the language of technical conversations. It is the foundational artefact that not only directs the development process, but also provides the technical vernacular that allows the developers to quickly and accurately share concepts.
Architecture sets the technical vision for the project, and cannot be neglected for fear of slowing down the process.
The Missing Piece of Agile
Whatever your preferred flavour of agile you will find, with some experience, that there are gaps in the process. On of the most glaring issues is that the agile processes are focussed on the operation of the development process, leaving many unanswered questions on what comes before development starts. When you get into the world of SAFe, LeSS or Crystal, all of which are bringing some large scale or enterprise thinking to Agile, Architecture at least gets a mention, but with no great detail.
Interestingly, there are many excellent resources for dealing with requirements discovery and the other work of product owners and business analysts including User Story Mapping by Jeff Patton or the equally excellent Agile Estimating and Planning by Mike Cohen. The popular Design Thinking approach has many positive practices to offer, and The Design Thinking Playbook has become one of my most frequent ‘go to’ books when looking at processes on requirements gathering and project initiation.
On the architecture side, there are a couple of extraordinary books including Clean Architecture by ‘Uncle’ Bob Martin which should be on every developers bookshelf, Domain Driven Design by Eric Evans should be read at least once, and of course something on design patterns whether it be the original Gang of Four book Design Patterns or Head First Design Patterns which I still can’t recommend highly enough.
So all covered right? Unfortunately not…
With all of this material, it’s fascinating to note how few books deal with the architecture and design function in agile processes.
The problem, I would argue is that there is a widespread assumption that this is simply an issue for the technology team. Actually this kind of thinking is seen a lot in agile processes including planning, estimation, unit testing, release planning and so on. Agile, despite its best intentions is still seen as purely a technical process.
Except, the point of agile was to increase the collaboration and interaction between the business, not to throw responsibility over the fence to the technology team.
How To Think About Architecture
Architecture, regardless of development approach, is not a static process. In fact, architecture as a concept is pretty fluid when you think about it. This is particularly true when dealing with Software Systems architecture.
In the realm of physical construction, a lot of raw material can be wasted if the architecture is flawed; the waste is visible – rubble, metal and other raw materials – and the cost implications are both apparent and immediate. Software architecture is no different in terms of the consequence, but the waste is less visible – the cost is paid in time, but the wasted code tends to go unnoticed particularly by those who are not familiar with the technology.
It may actually be time to develop a metric capturing the quantity of code unused or thrown away in any development cycle – but I digress.
Architecture naturally exists in layers, and I would suggest that this is is how it should be captured through the process.
The above diagram, quite apart from highlighting my need to retain the services of a decent graphic designer, represents the layers of architecture that, in my opinion, should exist for every system.
Conceptual Architecture
The Conceptual Architecture is the first architecture that should be developed. It’s the equivalent of the back of the envelope designs that capture the overall design of the system.
In my mind, this isn’t a particularly technical architecture. It is an architecture that allows technical people to share ideas with business people. It helps to establish a shared understanding with the business of the problem and the approach to the solution. It allows us all to speak in common terms which is vital to the success of the project.
At the conceptual level, there should be coarse blocks representing large areas of functionality. For example your conceptual architecture might include a single block for CRM, ACCOUNTS or PAYMENTS. It’s far from detailed but starts to build homes for concepts in the design.
Speed of exploration is important with the conceptual architecture; it helps verify that all of the desired capabilities have been captured. By implication, your conceptual architecture should emerge very early in the evolution of your project, and certainly well before any development starts. It will be fluid for quite a while, until the scope of the project is largely defined.
System Architecture
The System Architecture is the next level of refinement of our architecture. Depending on your preferred approach this is the domain level design, the service level design or perhaps even the microservice level design. It’s intended to map the conceptual design into the technical implementation.
The main purpose of this level is to put some structure on the conceptual architecture, to make it more refined, to crystalise where our data will live, how it will be accessed and most importantly how it will move through the system.
At the system architecture level, there are two types of technical diagrams that should be relied upon: one capturing the business processes allowing the logical flow of information to be documented with the business, and the second detailing components or services and the capabilities they offer. In InvoiceFair, we’ve been using BPMN to build the processes with the various areas of the business, and a simple UML component diagram.
Capturing the interactions between components is a little more subjective. If you map the major elements of the business process directly onto services, then the interaction, at this level, is pretty much captured by your BPMN diagram.
Alternatively activity diagrams, or more detailed sequence diagrams can be used with both being quite effective.
The system architecture is a technical diagram (or diagrams); I often describe it as defining the clothes rail on which we will hang more detailed technical artefacts, including code. This requires a certain logical order on things, where the system design must precede technical design, but it does not mean that a detailed system architecture must be completed before any work can start.
The trick here is not to go too far, or too deep, finding the right balance between a system architecture that accelerates the work of development teams, and one that is taking over the work that properly belongs to the development teams is crucial to the success of the process.
Data Architecture
Having defined which data elements should exist, where they are created and how they move through the system, the actual structure of those data elements must be captured.
Data architecture has a significant dependence on the business, particularly in the modern, data rich world. There is a vast gulf between data and information; failing to recognise this will lead to an architecture that shifts vast quantities of data through the resulting system with no tangible value.
Data Architecture establishes what raw data will exist within the system, where and how it will be stored and the security, access controls and other governance systems that will exist. Information Architecture provides context for that data, defining which aspects of the data are useful in which scenario and to whom (or what) including users, automated workflows and machine learning systems. If the Data Architecture is the ingredients to our recipe, Information Architecture is the meal.
Things get pretty detailed at the Data Architecture level, and certainly there are parts that reside purely in the technical domain, but keeping the business close to the Information Architecture is vital. Similar to the System Architecture, the Data Architecture can emerge iteratively, but only if an overarching data design is in place.
Component / Service Architecture
At the most detailed level, there exists the architecture of the individual components, services and microservices. The focus here is on the development team and what will help them deliver a solution that meets the demands of the system.
The System Architecture along with the Data Architecture informs the development team(s) what their external boundaries should be, leaving a high degree of freedom on the internal architecture.
Finally, with the internal architecture, we get to the intersection of agile and architecture for many technical teams. The architecture at this level can be conducted during one or more of the typical agile ceremonies. This is where the design patterns and all of the other purely technical concepts can be brought to bear without the business.
In architectural terms, the Component / Service Architecture is the most detailed level, and arguably the most likely to be left undocumented, or ignored completely.
It’s almost forgivable, almost, as the internal structure of the code can be pretty volatile during the development process, and maintaining documentation can be a chore. Also, the code ‘should document itself’.
At least these are the excuses that are normally thrown up.
Its not really acceptable of course. It falls into the same bucket as ‘I’m too busy doing my job quickly to do my job properly’ in my opinion, but I have been accused of being difficult on occasion.
The trick to component architecture, for me anyway, is to do just enough to capture the overall structure, and the major internal communications and to do this in such a way that it can travel with the code. This means using a tool like PlantUML (my preferred option), Mermaid or yUML, which allow for diagrams to be coded rather than drawn, allowing for them to be included easily in version control systems.
Of course you can achieve this with other tools that use proprietary file formats, but these require the specific tools to be available (and often licensed) separately.
For this level of design, a simple markdown document or two, combined with the produced diagrams leaves enough breadcrumbs for those developers who will follow to at least grasp the intention of the code.
Wrapping Up
Architecture is the often ignored part of software development, particularly in Agile processes where it is rarely mentioned. If you think about any other process that has an architectural component – building a house or an office block for example – the architecture is central to the discussion, not something left to the construction team.
Software architecture should be brought to the same level, and technical teams that are responsible for developing that architecture should be responsible for its documentation in every software project.
The high level and low level designs that existed in more traditional, waterfall methodologies were far too detailed, but that shouldn’t be used to dismiss their usefulness entirely. The benefits of a documented architecture behoves us to find an approach that delivers those benefits in a timely manner.
The thing to remember here is that the value is in capturing your architecture and publishing it. The approach taken, whether it be UML, C4, Hexagonal or whatever, is entirely useless it is taken out of the shadows and communicated amongst the business and the technical teams.
Support Ukraine
The invasion of Ukraine is an act of aggression that we should all oppose and speak out against. I will continue to Stand with Ukraine and its people until peace is restored.
Bows and Arrows 