Waterfall, agile, kanban, scrum - what are we trying to achieve with these?

We are trying to agree on an abstract concept and then turn that into a non-physical product running on a computer that will fulfill the required need, be adopted by users, and ultimately deliver business value.

Photograph by Jason Goodman¹

Let’s break that down a bit…

As with all good things in software, for project management, there are many options. These options often take on a religious fervor, with online battles constantly erupting and wastelands of text debating why one approach is better than another—or even questioning what a particular approach truly is and whether it is being implemented correctly or sufficiently.

We hear things like, “With 7 core competencies and 21 dimensions that enable business agility”. It sounds like software project management is a solved problem, where all software projects run like clockwork, delivering on a fixed timescale and within budget. (Stop laughing at the back…)

According to a study by Geneca, 75% of business or IT executives feel their projects are usually doomed from the initial phase. A report by the Standish Group states that any new software application is challenged in 52% of cases, successful in only 29%, and fails outright 19% of the time.

Reality doesn’t seem to match the theory. Despite so many processes and options, we appear to be far from solving the challenges of software development.

These processes have grown alongside armies of consultants, proprietary variations, training courses, exams, and so on, all aimed at improving software project outcomes.

What are all these different approaches trying to do? #

The parable of the blind men and the elephant provides some insight Wikipedia link

You may have heard the story: a group of blindfolded men who have never encountered an elephant are brought to one. Each feels a different part of the elephant. They describe the animal based on their limited experience, resulting in vastly different perceptions. In some versions of the parable, they suspect the others are being dishonest and come to blows.

The parable illustrates that subjective experience can be true, but it is inherently limited by its failure to account for other perspectives or a totality of truth. It implies a need for deeper understanding and respect for differing views of the same object.

With software, the product is the elephant. While nobody is blind, the product itself is not visible until it has been built. Until then, everyone is feeling around, trying to work out what the project is. Some people have specific knowledge about parts, and on large projects, it’s unlikely any one person has a complete picture. Even if they do, they must still convey this understanding to other team members.

When collaboration and sharing of their knowledge and views starts, respecting different perspectives on the same object of observation, a shared approach begins to crystallise.

These software management processes aim to help teams move from the initial “feeling-out” phase to delivering a working product. They document structured ways to approach software product development. Essentially, they align the different views of the elephant so the team can move forward and deliver a solution.

Process theatre #

However, the danger with any process is that it becomes the focus. Processes are often easier to understand than the thing they are designed to build. This is where many projects go awry. As people focus on the process, it grows and expands, while the “content” - the product - becomes sidelined in favor of increasingly convoluted processes.

Steve Jobs spoke eloquently about this nearly 30 years ago in The Lost Interview under the term “process theater”

It’s that people get confused. Companies get confused. When they start getting bigger, they want to replicate their initial success. And a lot of them think, well, somehow there’s some magic in the process of how that success was created. So they start to try to institutionalise process across the company. And before very long, people get very confused that the process is the content… The best people are the ones that really understand the content. And they’re a pain in the butt to manage, but you put up with it because they’re so great at the content. That’s what makes great products. It’s not process—it’s content.

We build incredibly complex processes because they are the “easy” part. For example, people will spend hours debating the optimum time length for stand-up meetings, rather than focus on the value of what is being discussed in those meetings and how it contributes to the product being developed.

Meanwhile, software projects continue to fail as attention shifts away from the content. Decades on from Steve highlighting this the pattern continues to repeat itself.

The process exists to drive a project to a successful outcome. #

Beyond this, the process holds no value. The best process is the minimum needed to deliver the desired outcome efficiently. A business only earns money when the product is delivered, customers start using it, and it provides value - not from the process itself. Nobody will stop and ask which process was used to create the product.

This doesn’t mean having no process at all. A balance is needed. The goal should always be the minimum process required. Allowing as much time and focus on the content as possible. In larger teams or corporate environments this focus can get lost, and “process theater” starts to take over.

So much so, one of Agile’s founding fathers, Dave Thomas, has declared that Agile is dead

Dave’s really pointing out that the term has been co-opted into a process industry, where the focus has shifted toward rigidly following predefined processes rather than the focus on the product being developed.

Of course, management likes structured processes with defined timescales to plan and budget. If it were as simple as implementing an off-the-shelf package, estimating work and timescales would be relatively straightforward. However, software development inherently means building something that has never been done before. It’s not a production line producing identical widgets - it requires creativity and innovation.

Having a team focused on the product and delivering business value in short cycles, supported by management that trusts them, is key to optimising the process. Trust only comes when everyone is on the same page, understands what is going on, and feels confident that what is delivered solves the problem.

Small steps help maintain team focus and manage expectations around deliverables.

As software developers, we aim to define, create, and deliver a reliable product that’s never been built before, on a platform that must also be created to run that product. Ultimately, the final result must align with the business’s needs.

Taking a step back: What are we trying to do? #

We are trying to agree on an abstract concept and turn it into a non-physical product running on a computer that fulfills a need, is adopted by users, and delivers business value.

The process is there to assist in getting us from idea to working product as efficiently as possible, nothing more.

There are approaches we can use to improve alignment between team members, management, and the business. But before delving into these, we must consider a few underlying concepts that affect understanding and alignment.

Underlying Concepts #

• Pointing at things
• All on the same page?
• When are grains of sand a pile?
• The coastline paradox