Is your IT department still doing waterfall?
Change your ways if you want to improve the quality of your deliverables
Some time ago I published an article talking about why I think companies should not be doing projects anymore.
It was on purpose that I left out the waterfall versus agile discussion. The reason for this was because projects or products are what is done while waterfall and agile are methodologies¹ that help on how it is done.
There is a tendency to associate waterfall with projects and agile with products which makes the distinction sometimes difficult but that’s not necessary correct. Let’s try to see why.
How does waterfall work?
It’s in the name: water, fall.
When we look at a waterfall, in nature, we see gravity doing its work. The water goes from a higher point to a lower point. Sometimes there are layers, creating a cascading effect with small ponds between the two. In these ponds it may be possible to find, for example, small fishes, feeding from what’s coming from the pond above and letting some leftovers go to the fishes in the pond below. But there’s never any communication between the fishes in the different layers. They’re probably not even aware that the other fishes exist as their world is physically limited.
And no mater how many layers there are, the water runs only in one direction. Once a drop of water has left one of the ponds, those fishes will never find it again.
The waterfall methodology works pretty much the same way. Instead of water, the work moves from a top layer to the next, never coming back. Each layer has its own people (instead of fishes), with its own responsibilities. They do their work based on what comes from above, and send the result to the next layer. Many times, not being aware of what is happening there or even knowing who is in the next pond.
There are advantages in this way of working. The structured work is one of them, for example. Being sequential, it’s, theoretically, predictable. People know what to do, focusing on their one job.
On the other hand, what happens if the work gets stuck in one of the layers? It will never be completed. What if there’s a mistake in one of the top layers that is not identified until the work is in another layer? There’s no way to fix it. Even worse, what if it’s not identified and the work gets produced anyway? Specially in bigger pieces of work it might take months from beginning to end, and only at the end someone will realize that the result has nothing to do with what was requested. In the meantime, a lot of time and effort was put into something that will not be of any use.
How does Agile work?
Agile² can have many forms: Kanban, Scrum, Design Thinking, LEAN, eXtreme Programming, etc. .
They are all different. They all have their own place and usage. But they all rely on the same concepts.
In all of them the entire team works together, collaborating for the best outcome. Small delivery cycles, making sure what is selected for delivery will actually be delivered, and not stuck in some process. Constant feedback to make sure the right thing is being built and immediately adjust in case not.
If we continue with the fish metaphor, instead of having separate fishes in different ponds, what we have is a shoal in the ocean, with all the fishes moving together, in a synchronized way, making sure their actions are the best for the shoal.
To summarize, in waterfall you follow a quiet, smooth process, thinking you know what you’ll get at the end of it. Most probably, once you get it, you realize that’s not what you wanted.
In Agile you accept that you don’t know from the beginning what you’ll get. And the journey will be much more chaotic. But you’re sure that whatever it is you’ll get, it is what is needed.
Doesn’t waterfall has it’s place?
In order to answer this question, many times, the Stacey’s matrix is used.
This matrix makes a connection between the understanding of what needs to be done — also known as scope — and the knowledge required to deliver it — usually related with the technology selection — and tries to advice on the methodology to use.
When there are many unknowns, more abstract methods are required. On the other end, when everything is perfectly clear, waterfall is the best option. Without a doubt, when everything is crystal clear then waterfall will, in theory, always win. But let’s face it, when was the last time there was a project where this really happened?
This is where waterfall/project management advocates end up struggling to adapt. There is a tendency to think that a requirement document will be clear and definitive enough. With a couple of “Proof of Concepts” before the beginning of the project the knowledge about the technology will be brought the the required level. Working around the business requirement document and squeezing all the hypotheses, the scope will be clear.
The problem is when reality hits. As Field Marshal Helmuth Karl Bernhard Graf von Moltke said: “No plan survives the first encounter with the enemy”. And when this happens the discussion will shift from solving the problem to who will take the blame and deal with the budget changes.
Still, every time a new endeavor starts some people will insist everything is known — in terms of requirements — and their experience in the technology will ensure all will go smooth. But here the “IT” part of this article’s title kicks in. Maybe this is somehow true, and necessary, in other areas of knowledge but in IT there is only one way, if at all, for everything to be perfectly known: if that specific person has done that specific task, in the same conditions, already in the past, which is very unlikely. The thing is, humans are very bad at executing repetitive tasks. But computers are really good at it. With all the new technologies around robotic processes automation, for example, every repetitive process can be easily automated. There is no excuse to perform the same task twice. If you think you know everything, both from a scope as well from a technical perspective, and you insist on doing waterfall, you’re in for some trouble. Alternatively, maybe you really know everything, and then you are using valuable resources on something that could be performed by a simple script or some other form of automation.
Given this, it’s very difficult to argue that waterfall still has a place in the modern world of information technology.
If we link with my first article and think about the project/waterfall and product/agile binomial, you see it’s really easy to favor product thinking and agile methodologies. But don’t get me wrong, even if I fully advocate the product mindset I agree that some activities can still be seen as projects (within the product). I’m thinking about activities like a datacenter migration or a software upgrade. They are required to be done but don’t add any direct value to the customer. What I would not do is still go with waterfall for that. Most likely a simple form of Kanban would be the best option. If you still insist on using waterfall you’re going to have bad surprises really quick. If not, you could probably have been replaced by a script right at the beginning.
What do you prefer? Strict processes with no communication that end up in finger pointing, waste of time, and wrong deliverables? Or a collaborative environment where people help each other finding the best means to a successful end?
- That is not entirely true, specially for Agile which is not a methodology at all. But for the sake of this article let’s take it as if it was.
- Again, I know Agile is not, and should not be reduced to a methodology and that it’s possible to fully implement one of these methodologies/processes and not “be” agile.
