Three Categories of Infrastructure Work (and why you don’t need it all right now)

I encourage my clients to find and focus on the core value of whatever they’re creating. What is it that doesn’t yet exist in the world but should? Go there first. Don’t defer that value.

“But we need to create the infrastructure for that first, don’t we?” they often reply, by which they mean things like hardware, foundational technology layers, architecture, capabilities like user management, etc.

Well, there’s infrastructure and there’s infrastructure. For the purpose of thinking about what to do when in an Agile approach, I divide infrastructure (broadly defined) into three categories:

1. We can’t start producing value without it
2. We need it for the first increment of value
3. We eventually need it to achieve our vision

Broadly, teams make the mistake of assuming it’s all category 1. In reality, it’s mostly category 3. Let’s break that down.

Category 1 (“We can’t start producing value without it”) is things like a basic computer setup and development tools. If you don’t have this, you really can’t start. You can’t, for example, code with paper and a pencil.

Keep this list as short as possible. Any time we spend ramping up to actually start delays value and learning and costs money.

You don’t need to artificially limit this, though. No need to use a bare-bones text editor if it’s just as easy to install your favorite IDE. A shared continuous integration server can be really useful. Can you set it up quickly? Then go for it. Need to wait on an outside group to do it for you? Then maybe don’t treat this as an impediment to starting but find a fast and cheap alternative for now.

Category 3 (“We eventually need it to achieve our vision”) is all the stuff we would have made documents and diagrams about in the Big Design Up Front days. When we think about the big vision for what we’re going to create, what are the key components, technologies, and architecture that allow us to achieve that vision?

If you lived through the BDUF days, you may remember how often those documents and diagrams were wrong. As we did the work, we learned what was /really/ necessary.

But you may also have lived through the reaction to that, the pendulum swing the other way, where we just started. That didn’t go so well either. We’d back ourselves into corners and waste effort on avoidable rework.

The third way—the way that avoids those pitfalls—is what I call Target Architecture. Target Architecture means thinking about the what could be necessary for your big vision, but instead of treating it as correct and committing to it, sketch it and treat it as a set of options to preserve.

Now, on to Category 2 (“We need it for the first increment of value”). Use something like my Feature Mining technique (in my CSPO and Advanced CSPO courses) to find a meaningful slice of the core value of your big idea. Then ask, “What’s the minimum infrastructure required to create this thing while keeping our options open to evolve towards the Target Architecture?”

Let’s look at a concrete example. Years ago, I led the development of a business intelligence (BI) system for wastewater plant optimization. Cities grow, and utilities need to keep up with that growth without building new plants, so engineers help them get more capacity out of existing plants. We were creating a tool to help these engineers do their work more effectively.

The traditional approach to BI treats everything as Category 3. You choose expensive BI tools and buy them. You design a data warehouse (DW). You design the extract-transform-load (ETL) process to get transactional data into the DW. You build cubes on top of the DW to aggregate the data for faster queries. Then, you finally create reports for the end users. (And discover everything else was wrong in some way, but it’s too expensive to change now, so we don’t talk much about that.) It would typically take 6-12 months before the first report hit production.

We did the opposite.

We asked the engineers, “What questions are hard for you to answer about a wastewater plant right now but would make a huge difference for you in your optimization work?” After some discussion, we found one high-leverage question that would get them quick value and teach all of us a lot about the core hypothesis, that better views into the data would accelerate plant optimization.

We sketched a Target Architecture for the big vision. We needed to be able to evolve to a proper DW, for example. But we didn’t necessarily need to start there.

Then, we asked ourselves: What’s the smallest thing that would allow us to answer that first key question, while leaving our options open to evolve towards the Target Architecture?

It turned out that we didn’t need to even select a BI system. We could create a simple web page with a chart and a table to answer that key question. We didn’t need a proper DW—a flat text file would do the job fine for now. We didn’t need a proper ETL process—a nightly scheduled job to select data from production and write out that flat text file would be sufficient.

Two weeks later, that report was in production, helping engineers do their job and teaching all of us about where to do next.

As we built out our product backlog, we made notes about when we’d need to grow different parts of the Target Architecture. And we gave ourselves buffer to build those in as we added more features.

A few sprints in, we did have a proper relational DW. The original flat file easily migrated. We grew a proper ETL process (using test-driven development, by the way, but that’s a different story). We never needed cubes, so we didn’t spend money building those.

What are some examples of the three categories of infrastructure on your product? How might you avoid building Category 3 right now, while leaving your options open for it in the future?