Why you should avoid Scrum, and what to do instead

How to manage software projects

2024.08

Overview of Scrum

Scrum is both a process and a meta process.

It's a process because it has strongly defined roles and activities: the scrum master "SM", the product owner "PO", the daily scrum, the retrospective, the backlog, etc.

It's a meta process because it doesn't define the actually important roles well enough and instead tell you to define your own:

• Who is responsible for architecture?

• What are the different roles for senior, intermediate, and junior programmers?

• What happens if there's disagreement about important architecture decisions?

These are the actually important questions that you need a framework in place in order to answer. "Talking about it as a team" is not a framework.

Scrum pretends that the answers to these questions is a mystery and you can only discover it by doing the Scrum process. But the answer is not a mystery at all, and by avoiding the answer to these important questions, Scrum provides no value at all.

Scrum's idea for project management starts with the "PO" creating a backlog of tickets. On what basis are the tickets created? That's left up to them!

The PO then talks to their team of programmers to select tickets to complete in a "Sprint", which is a time-box where the team "commits" to a specific "goal".

After the sprint is complete, the teams stops to evaluate the situation and adjust accordingly.

Starting on the wrong foot

Quote from the scrum guide:

1. A Product Owner orders the work for a complex problem into a Product Backlog.

Scrum's very first step is already a mistake. You can't just have one non-technical person competently break down the product into a series of tickets.

The PO, according to scrum rules, should not be a technical person. As such, they don't really know what's the best way to break down the problem into a list of tickets. Whatever they come up with is probably not of good quality.

By the time the programmers get to have a look at the backlog, it's too late to fix.

The PO has already been put in too much effort into creating all these elaborate user stories, and they are not going to fundamentally change how the tickets are structured.

The implicit assumption in Scrum is that programmers are incapable of performing the work of breaking down a large goal into a series of small steps, because it's a kind of soft skill that programmers don't have.

This is of course misguided. Many programmers have those skills. The better the programmer is, the more likely it is that they have a deeper understanding of how to best break down a project into a series of steps.

Phases and Milestones

You can't actually start at the level of an individual ticket, or even a user story.

You have to start at a somewhat higher level: breaking it down into a series of phases, each encompassing a meaningful set of features that can be released, either to the public, or to a select group of test users. It should be something that can be completed in a few months.

Each phase is then broken further down into a series of milestones that can be reasonably completed within a few weeks.

You may notice that "few weeks" is vague. This is on purpose. You need some amount of flexibility in terms of scope and time. This will be explained further on the section titled "The Sprint".

Arriving at meaningful phases and milestones requires collaboration between technical leaders and stakeholders.

A tech leader can come up with milestones on their own, but these milestones might not make business sense. Similarly, the business owner can come up with arbitrary milestones, but they might not make sense from a technical perspective. That's why it should be a collaboration between the two sides.

Now, to reduce communication overhead, one or two people from each side would usually suffice. Adding more people to the mix does not usually result in a better understanding of the problem. Instead it would just increase the communication overhead. It would also increase the probability of personal conflicts, as each member tries to prove how valuable they are to the process by suggesting new ideas and thinking of new potential obstacles.

The goal is not to arrive at the perfect plan. We just need something reasonable; a rough plan that allows us to start moving forward.

How to break the project down into a series of milestones and how to prioritize the work is itself an important topic that needs its own treatment.

You have to start by building a shared understanding (between the business side and the tech side) of the high level goals and the vision behind the project.

Based on these goals, you think of a smaller version of the product that aligns with the ultimate goals and vision.

In each phase, you increase the scope of the project so that it comes closer and closer to fulfilling all of its goals.

One common mistake I've seen some managers make, specially when operating under the tacit assumptions of scrum, is the following: they break down the project by slicing it down vertically; for example: their plan is to develop the product one page at a time. Each time, they aim to get all the details for that page fleshed out in terms of graphic design and implementation. If the page is too large to develop in one sprint, they break it further down (again vertically) into sections, each of which they expect to be developed fully in a sprint.

It's as if you plan the construction of a house by building one room at a time, each time everything about that room is fleshed out: all the electric sockets, the windows, the water pipes, etc, but only for that room! And then you break that down to one wall at a time, again with all the windows and electricity sockets and water pipes for that wall!

This is not a good way to manage the project's milestones.

Instead, start with a broad sketch; a skeleton. Then flesh out the details gradually and iteratively.

Start with a rough sketch of the core feature. Use fake data to compensate for lack of real data.

For example, if the the current phase of the project is mainly about booking and managing appointments, start with the calendar page. It doesn't have to look good. It doesn't even have to look correct; it's fine at first to pretend that all months just have 30 days.

What matters at this stage is nailing the feel and interaction.

Since you haven't yet built anything to input the data, you will have to fake it: fake users, fake sessions, fake appointments, etc.

You try various ideas for how to interact with the calendar until you find something that seems to work well enough. Then you can start filling out the other details: getting the month display to be correct, working out the graphic design, the colors, the shadows, etc.

If you have more programmers, you can let them work on sketching out the other more peripheral pages, or do various types of independent UX experiments, etc.

Think of how you would go about putting in screws on a table. You don't put it one screw at a time. You put in all the screws first but keep them loose, then you gradually start tightening them up, one by one, but all together: never tightening one screw too much while leaving others too loose.

Planning the sprint

Quote from the scrum guide:

2. The Scrum Team turns a selection of the work into an Increment of value during a Sprint.

The second step in the scrum process is also deeply misguided. It assumes that a PO, having created a backlog of tickets without much engineering considerations, can just share the top items from the list with his team of programmers, and they can just grab a bunch of tickets that fit into a sprint and just start working on them!

That's not how software development works. You cannot just pickup a ticket and "implement it" as if it's an isolated unit.

A lot of upfront preparatory work is usually required, specially if we assume that most tickets are created such that each can be completed by a single engineer in a couple of days.

At the very least, you need to define the interface between the different components in the system, but more broadly, you need to define the data model and outline the list of processes that need to be implemented.

This is especially required if you have a team of multiple people that are expected to work in tandem on different components of the same feature or related set of features.

Scrum provides no guidance here. Instead, it's often a hinderance, as it primes the PO into thinking that the programmers just need to break down the tickets into new, smaller tickets if necessary, and that's it!

As if features can just be built onto the air, with no foundation in place.

When the programmers manage to convince the PO that they do in fact need to build some foundation first, the PO and the SM would ask the programmers to build the minimum amount of foundation necessary for the next sprint, and nothing more!

To see how absurd this is, imagine a construction crew being asked to build a house in sprints, each sprint they build just one wall, and each sprint they are only allowed to build the foundation for that wall only and nothing else!

Building software is not like building physical buildings, but the analogy here stands: you need to build the lower layers first, you need some foundation to build on, and there are good and bad ways of breaking down the work into stages. Building a house one wall at a time is nonsensical, but if you've never seen how houses are built, you wouldn't know.

How do you to take a series of abstract user requirements, and produce an actionable plan that would bring them to reality? This is one of most important problems in project management, but Scrum provides no guidance here what so ever. It is left as an exercise for the reader.

The PO just creates a list of backlog items. The team just makes a plan to implement an increment for the next sprint.

If you are looking into scrum as a solution to project management, this should be the biggest red flag!

Quote from the Scrum Guide

Topic Three: How will the chosen work get done?

For each selected Product Backlog item, the Developers plan the work necessary to create an Increment that meets the Definition of Done. This is often done by decomposing Product Backlog items into smaller work items of one day or less. How this is done is at the sole discretion of the Developers. No one else tells them how to turn Product Backlog items into Increments of value.

Notice the strong implication that developer's planning stage mostly involves breaking down backlog tickets into small tickets that can be completed in a day or two.

It assumes the creation of the backlog tickets was already done in a way that is appropriate from an engineering standpoint.

The programmers, according to scrum, do not get to tell the PO that the current way the work is broken down makes no sense from a technical point of view.

That is not their job!

They are just to figure out how to make it work! If the PO says we need to build the west facing wall with a window in it that can be opened and closed, you are not allowed to tell the PO that this is not how houses are built. Your job is to figure out how to just build the west facing wall with the window on it. You can break the work down into two stages: first you build the west facing wall with a hole for the window, and then you attach the window with the handles.

The missing Architect

There's no one in Scrum who plays the role of an Architect.

Quote from the Scrum Guide:

Scrum Team

The fundamental unit of Scrum is a small team of people, a Scrum Team. The Scrum Team consists of one Scrum Master, one Product Owner, and Developers. Within a Scrum Team, there are no sub-teams or hierarchies. It is a cohesive unit of professionals focused on one objective at a time, the Product Goal.

Scrum Teams are cross-functional, meaning the members have all the skills necessary to create value each Sprint. They are also self-managing, meaning they internally decide who does what, when, and how.

Scrum purports to be a software development process, yet it has no place and makes no room for an architect. It does not even pretend that it addresses the question of how to handle architectural decisions.

This is by far the worst aspect of this process: the diffusion of leadership roles - other than the SM and the PO.

This can be very appealing to business people, but it's disastrous for a software project.

A well functioning engineering team needs a well defined hierarchy. The senior engineers and junior engineers need to understand their role within the team. Responsibilities must be assigned individually, and individual incentives must be aligned.

Scrum diffuses responsibility, and thus destroys incentives.

Under Scrum, the responsibility for architecture is assigned to "the team". There's might be a de-facto architect: a team member who demonstrates more knowledge and experience to his peers, and maybe gets some tacit recognition as such; but never formal recognition, even if they end up doing the majority of architectural work.

The PO and the SM have well defined roles, so they receive visibility and recognition, while individual developers, even those making critical decisions or making important contributions, often end up underappreciated. This can lead to misaligned incentives, and may open the door for unintentional sabotage amongst team members.

A well functioning company would have a structure where the senior engineers, the architects and the tech leads get the visibility and the recognition they need to align their incentives.

The importance of hierarchy

Leadership roles within a software development team should be clearly defined. If there's no leader, infighting and bitterness ensues, as everyone tries to be a leader.

There has to be a leader who makes decision and everyone follows along. This is not to say that team members have no say in the matter. It's of course important that people can state their opinions and objections, and the leader does have a duty to listen to them, but at the end of the day, the buck stops with him; he makes the call.

Without leadership roles, politics take over: people who want power and can tolerate disagreement and social conflict will often get their way, not because of any merit, but because other people give in to keep the peace.

Without leadership roles, a team composed of mostly junior programmers will block the senior architect from making the required decisions because they disagree with him or don't understand the reasoning behind his decisions.

It gets worse if there are multiple senior or intermediate engineers with overlapping responsibilities, as they can disagree about how to do things or second guess each other's decisions.

Someone has to be responsible for making the decision at the end of the day, and when that role is well defined, most engineers would have no problem deferring to him even if they disagree with the final decision.

Engineering Roles

This is a rough outline of what the engineering roles ought to be and what the responsibilities and limits of each role are.

• Tech Lead / Architect

  • The most senior engineer on the team.

  • Makes technical decisions after consulting with seniors.

  • Produces design documents as needed to ensure everyone understands how the system is supposed to work.

  • Responsible for ensuring a coherent technical design for the product as a whole.

• Senior Engineers

  • Each responsible for a relatively large sub-system within the project.

  • Work with the tech lead to design the system as a whole and the sub-system they are responsible for in particular.

  • Implement the core and most critical features of their respective sub-system.

  • Mentor the junior members

• Junior Engineers

  • Each assigned to a Senior

  • Learn how the system is supposed to work

  • Implement all the peripheral and non-critical features

  • Handle the mundane details to ensure their Seniors can focus on the important aspects of the system

  • Aim to improve at their craft and eventually attain the requisite skill level to be recognized as a senior.

Committing and Reviewing Code

Seniors commit/merge to the master branch without waiting for code review, unless they have doubts and need some help.

Juniors at first must get approval from their Senior before merging their changes into the master branch. After a while they get permission to merge without review.

Code reviews can occur after the merging has already been done. For example, Senior engineers can hold code review sessions with their juniors if they notice a drop in the quality of code being committed.

The Sprint

Scrum requires programmers to commit to completing a fixed amount of work in a fixed amount of time. This is the "core" of scrum. It appeals to managers because the consistent delivery of small improvements in fixed cycles creates the impression of consistent progress. But it's mostly an illusion.

The scrum sprint is very strict. You have to make 100% commitment to completing the work. If you end up only completing 95% of it, Scrum would view this as a failure! During the "Retrospective", the PO and the SM will tell the programmers that they failed to deliver the promised work this sprint, and they need to work on improving their delivery, their estimates, or their ticket refinements. Sometimes, an entire hour, or even two, would be spent discussing why the sprint failed, and the programmers would be required to criticize their own efforts and add new rules to the process to make sure such a travesty (of having completed only 95% of the work instead of 100%) does not happen again.

The reality is, unless you set to yourself very "easy" goals, it's impossible to consistently deliver on the estimated time.

An estimate and a commitment are not the same thing. When you estimate that a thing can be done in three days, you cannot commit to completing it in three days. If you have to commit, you need to add in some slack.

When you do scrum, your team are always spending a week to finish work that could be basically completed in two days, because if they don't, they will have to suffer through the next retrospective again!

Setting fuzzy goals

If one wanted to actually be agile, they would not fix the scope nor the time. To be agile, you need flexibility. You must allow some "fuzziness" into the process: we will aim to get roughly this much work done in roughly this much time. Maybe we'll end up getting 90% of the work done in 90% of the time, but the rest would take much longer than expected, so we stop here.

This is good and normal. A healthy work environment would see it as positive progress towards the goal.

Fuzziness is important because you can't actually know how far ahead you would progress without actually doing the work. By the time you get near the end of milestone, you'll have a better idea about what can be reasonably completed in the remaining time, what would take a bit more time but can still be completed soon, and what would take a lot more than originally anticipated.

You can then decide whether it's worth to extend the milestone a little bit in order to give the engineers the time needed to get more work done.

This kind of flexibility is made impossible in Scrum, but it's essential for agility.

Imagine a predator chasing its prey, but at the very start they set a timer for themselves: I will catch it in 10 seconds! If 10 seconds pass and they haven't caught the prey yet, they halt the chase! Full stop! Even if they were just two seconds away from actually catching the prey! This cannot be described as "agile". In the real world, an agile predator would only halt the chase if they determined it to be futile because the prey has outpaced them and ran too far for them to catch up to.

Closing words

We've outlined some problems with Scrum and introduced a set of alternative ideas for project management. I'm not aware of any name for these set of ideas. They're just the obvious way to produce software, and I've seen them practiced informally in several companies I've worked with.

A part of me feels a little bit of unease about trying to outline these ideas. They feel so natural and obvious.

However, we live in a world were bad ideas like Scrum are being touted as "agile" and being adopted by companies trying to figure out how to manage their software projects, so we have no choice to counter act this except to expose its problems and outline the obvious better way.