Let me begin by stating that while nearly every manager in the software industry advocates Agile development, which is often incorrectly used interchangeably with the Scrum flavor of Agile development, there is no consensus amongst engineers and designers as to whether or not Agile or Scrum is a “good thing”. And, having practiced Scrum myself in seven different teams throughout my career, having attended several Agile Development workshops (all of which were biased towards the Scrum methodology), and having listened in on several conversations about the pros and cons of Scrum, there seems to be some confusion as to what problems the Scrum methodology is actually solving for, and what side effects it has on the overall software development process. In addition, you may have heard about Kanban as well, but aren’t quite sure how it relates to, or differs from, Agile and Scrum. Is Scrum all that it’s hyped up to be? Let’s find out.
Agile Development is like a religion. Everyone does it differently and no one is sure if they’re doing it right – Eric Rowell
The Interface and the Implementation
The terms “Agile Development”, “Scrum”, and “Kanban” are often times used interchangeably which can cause some confusion amongst designers, engineers, managers, and product managers. I’m going to define them as succinctly as possible to clear up any confusion.
Agile Development – a abstract software development concept. It’s not a process. In fact, Agile development is defined by it’s manifesto, found here http://www.agilemanifesto.org/, which basically just emphasizes the importance of team member collaboration, constant collaboration with customers, and an ability to adapt to changing conditions rather than following a plan. This is all goodness and makes a lot of sense.
Scrum – an implementation of Agile Development which imposes a process on the software development pipeline in order to maximize developer throughput, improve predictability as to when units of work will be completed, and guarantee working software milestones. Clearly, this methodology was created with managers and product managers in mind, because these folks are pressured to optimize designers’ and engineers’ time as efficiently as possible in order to produce results as fast as possible, and also do so in a predictable manner so that executives know when the work will be finished. This makes a lot of sense, but it comes at a cost, as we will discuss later.
Kanban – a system for monitoring the progress of work over time. Kanban was originally developed at Toyota to manage assembly lines. With Kanban, managers, designers, and developers get a clear sense of the work passing through different states, such as “not started”, “in progress”, “code complete”, “tested”, and “verified”.
Scrumban – a combination of Scrum and Kanban. Scrumban essentially adds taskflow monitoring to the Scrum process, which is sometimes referred to as the Scrum board.
The Observer Effect
So why do managers and product managers advocate Scrum, while designers and engineers are less enthusiastic? To understand this, I’d like to reference the observer effect. In the field of science, the observer effect refers to changes that the act of observation will make on a phenomenon being observed. A classic example of this is checking the air pressure of an automobile. By measuring the pressure, you inadvertently change the pressure of the tire by letting some of it out. Thus, the act of observing the tire pressure has altered its state.
The same effect applies to the Scrum Methodology. To my knowledge, there are two major factors that impede the velocity of software development when trying to improve predictability with Scrum – meetings and technical debt.
Meetings, Meetings, and More Meetings
First, because Scrum is a process, it requires several meetings every sprint to groom backlogs, plan future sprints, break down tasks, scope tasks, have retrospectives, etc. Although the ratio of Scrum meetings to actual development time is somewhat low, it still has an effect. Typically, in total, Scrum meetings result in a loss of a half day or a full day of development, each sprint, for every engineer on the team.
Take a look at the diagram below. It outlines the components of Scrum methodology that serve the needs of designers, engineers, managers, and product managers.
As you can see, the Scrum Methodology serves the needs of both parties only in regards to task workflow, backlog grooming, and daily standups. The backlog, when prioritized, serves as a plan. The standups and task workflow (Kanban) are useful to managers and product managers because they serve as a heart beat for the progress of the project. On the other hand, they are also useful to designers and engineers because they keep everyone on the same page in regards to what is being done and what is about to be done so that people aren’t working on the same things, and so that units of work can be done in a logical order. This is definitely goodness.
The items on the left of the Venn diagram, unfortunately, do take up quite a bit of time, and provide little to no value to designers or engineers. Blah!
Origins of Technical Debt
Next, because engineers are pressured to complete all of the tasks in a given sprint, quality is often times sacrificed and technical debt is unnecessarily added just to get stuff over the finish line.
Imagine if residential builders used Scrum to build houses. Rather than building a solid foundation first, including plumbing, wiring, and gas, the builders try to produce “working software”, aka livable quarters, by building one room in each sprint. If a builder were to construct the kitchen first, followed by a living room, followed by a garage, etc, without having a foundation in place, the house would be a disaster.
This type of scenario is one of the most common sources of technical debt, where engineers get the forward facing part of a product done as fast as possible and add hacks for foundational pieces that can’t be seen. Any engineer can tell you that technical debt piled up over time is irreversible, usually resulting in an eventual rewrite of entire modules, or a whole project. If that’s not a time killer, I don’t know what is.
Software Development Predictability vs. Speed
Interestingly, spending more time realizing predictability results in less time actually designing and building products. Vice versa, spending more time designing and building products results in less time realizing predictability. Therefore, it follows that predictability and software development speed are inversely related, i.e. you can never have both.
It’s safe to say that you at least need some level of predictability, or else you’ll never have a sense for when things might be completed. On the other hand, it’s also safe to say that you don’t want to spend so much time analyzing throughput and realizing predictability that you are paralyzed and can’t get any work done. Because designers and engineers are more concerned with the creation of high quality products than the degree of accuracy for when their work will be completed, it should come as no surprise that designers and engineers will therefore naturally resist the Scrum Methodology simply because it gets in the way. Thus, it’s really important that as a team or organization, you find the right balance between predictability and software development speed.
Fallacies of the Scrum Methodology
One of the major flaws with Scrum is it’s assumption that software can be designed, built, and tested within a single sprint such that new features are relatively polished and completed at the end. It is theoretically impossible for designers, engineers, and QA to efficiently work in parallel when designing, building, and testing a feature simultaneously. Can you decide what you are going to eat for lunch, eat it, and then feel satisfied with what you ordered simultaneously? Of course not, that’s ridiculous.
What realistically happens is that designers naturally fall out of the sprint cycles and just try and stay a few weeks ahead of the engineers. Engineers start building the new features or fixing bugs, but QA can’t test anything until they are finished, so QA naturally works on improving the tests for the last sprint. As engineers scramble to finish the remaining tasks at the end of the sprint, QA is left high and dry because they didn’t even get a chance to start testing those tasks. This cycle repeats, and thus, there is a constant blurring effect between sprints.
Secondly, shit is going to happen that compromise a Scrum sprint. You can count on that. For example, QA finds a gnarly bug that is going to jeopardize a customer demo, or an escalation is filed by an existing customer that has to be fixed asap. Scrum does not account for anything unexpected that might occur during a sprint. Yet, unfortunately, during all parts of the software development process, one should always expect the unexpected. Despite Scrum’s claim to be agile, it can feel pretty rigid at times, which is the very thing it is supposed to remedy.
Natural Tendencies and the of Scrum Reduction
In just about all of my experiences with Scrum, the overhead associated with scoping and partitioning tasks for each and every sprint, coupled with an inability to accurately predict the time it takes to complete every unit of work that makes up a product, plus the fact that nearly all sprints will be compromised with unforseen challenges, has nearly always resulted in the Scrum process being reduced to a Kanban-like process over time, in which teams fall into a natural pattern of continuing standups and monitoring progress via task workflows and essentially reducing sprints to just “the stuff we are working on next”.
Perhaps then, Kanban is a better process than Scrum for teams that
- have designers and QA
- have to move fast
- don’t necessarily have to know exactly when a product is going to be finished
For teams that are made up of engineers who are barbarically responsible for both design and testing (typical for large IT departments), or for teams who can afford to move at a snail’s pace (again, typical for large IT departments), the fallacies of Scrum will not be as evident, and thus will seem acceptable or even conducive to the process of software development. In my opinion, the positive feelings that these teams may feel towards Scrum are actually misguided because these feelings are likely formed from the value realized by the Kanban components of their process (task workflow and standups).
Kanban does not necessarily care about throughput, delivering working software on specific days of the month, or predicting exactly when everything will be finished. It cares about daily collaboration, keeping in touch, and optimizing workloads depending on changing conditions. This, in spirit, is what Agile Development was meant to be.
Kanban also makes it easier for folks to trade, give, or receive tasks at any moment depending on changing conditions. It allows for shifts in priority and escalations to be handled immediately, rather than waiting for the end of a sprint. Also, because there are no hard sprint deadlines, the process of designing, building, and testing are much more fluid. Designers are working on designs that will be built next week, engineers are building portions of the product this week, and QA is testing the work that engineers built last week.
