Technical Debt Isn’t Really a Metaphor
Many years ago, Ward Cunningham posted an excellent video on YouTube regarding refactoring and “debt”. If you haven’t seen it, I have it for you, below. But is “Technical Debt” simply a metaphor, or is he describing a real fiscal debt associated with software development activities?
Design Debt
The form of technical debt that Ward is referring to is noted by developers as poor, brittle design. Since a poor design is one that’s difficult to change or maintain, it will take developers longer to add features, and their changes will more likely result in defects. They will tip-toe slowly and cautiously within (or around) untested code, and even the best developer will inadvertently introduce the occasional defect.
Isn’t developer time expensive? Of course. So, in effect, this Design Debt (my name for this subcategory of technical debt) is real debt. We may not be able to easily quantify this debt in terms of dollars–unless we routinely use Cost of Delay to aid our high-level decision-making process–which makes it even more dangerous.
Quality Debt
Design Debt feeds into another form of debt which is easier for the business to see: Quality Debt. This can be quantified as the number and severity of defects experienced by users. This metric has a fairly direct relationship with the financial impact of lost sales, time spent on support calls, the impact of negative word-of-mouth (I found a book with the bone-chilling title of Satisfied Customers Tell Three Friends, Angry Customers Tell 3,000), developer time spent debugging the product, and so on.
Often teams and whole organizations assume that Quality Debt is “the nature of software development.” That may have been so in the past, but we now have modern development practices that greatly reduce those numbers; if the teams are permitted to learn, use, and discover the whole-team benefits of those practices and skills.
Testing Debt
Unfortunately, what we usually try to do may just make it worse…
The functionality grows, the complexity grows, and it all has to be tested. We hire more testers, or give ourselves longer for the test “phase” of an iteration/sprint. This gated approach has been noted on many “agile” teams. If waterfalls are bad, why are more of them better???
As the stack of manual test-cases grows and grows, the poor test team works more overtime to run all the tests: Point-click-type-point-click-type… And even the most professional testers make occasional mistakes.
Later, the team will selectively run only the most “critical” tests each iteration, and save the whole suite for the pre-release shake-down cruise. But which tests–which product features–are not critical? Plus waiting until the end to do full testing creates a process gate (a la waterfall) resulting in delayed feedback and painful rework.
This is the third form of technical debt, Testing Debt. The anecdotal measurement of Testing Debt is the height of the stack of printed manual test-cases. A modern measure would be the time it takes to run the entire regression suite: How long after a change does it take the team to know whether or not they’ve broken something?
Is Testing Debt real financial debt? Well, there’s testers’ time and salary. Also, Testing Debt again feeds into Quality Debt. If you don’t assure that there are no defects, your customer will find them for you.
Compounded Interest
You know what you get when you add more features to existing defective code? Yeah, more broken features.
What can be Done?
First, in the immortal words of Douglas Adams: Don’t panic.
In 2002, on the University of Michigan Transplant Center’s two-year “OTIS2” rewrite of their aging Organ Transplant Information System; and after bi-weekly reminders that “a mistake could kill a patient”; we would release changes into production with only one day of additional (i.e., gated) exploratory testing.
After 15 years, this team is still active, still making enhancements, and–according to Richard Sheridan of Menlo Innovations–there hasn’t been a software-related emergency requiring developer overtime since 2004!
The majority of testing is built into an automated suite of unit-tests and story-tests that execute with each build. After only two years this team had about 20,000 such tests.
They all ran within 15 minutes.
Because the build/test feedback loop was so fast, multiple developers would confidently commit changes (i.e., in git-speak, “push to master“) every 2-4 hours.
New members of the team start working on code in the first two days on the team. They are told, effectively, “You can make any change to the system, as long as you run all the tests and they all pass. Oh, and you are always working with someone else on the team.”
In the story above, I’ve alluded to quite a few of the Essential Agile Engineering Skills:
Test-Driven Development: Creating that fine-grained safety-net of thorough, comprehensive tests.
Pair-Programming: Working with one other developer to sustain quality, adherence to agreed-upon practices and idioms, enthusiasm, curiosity, and courage.
Continuous Integration: All pairs integrate changes into the release candidate branch (we used master) every 0-4 hours.
Sustainable Pace: We avoid coding when tired. People make mistakes when they are tired, even with Red Bull supplements.
Too Little Too Late?
Most teams I encounter in coaching or technical courses have not been doing these things. Is it too late?
Consider: How does anyone get out of debt?
- Stop denying the existence of the debt.
- Make regular payments.
- Stop accruing more debt.
Examples of how agile teams can pay down technical debt:
Design Debt: When tasking and estimating stories, the team should consider the refactorings necessary to get any new story designed correctly. Those refactorings become explicit tasks for the first story that calls for those refactorings.
Quality Debt: Set aside time within each iteration (a practice I call Creative Slack) to explore and fix high-priority defects, or create a story for a group of related bugs. On a well-running agile project, each reported defect becomes a story of its own.
Testing Debt: Set aside time within each iteration (Creative Slack again) for testers to automate those “critical” tests, or a particularly onerous set of tests. If you’re not ready for that, create a story (or in this case, a “spike”) for the team (testers and developers working together) to explore automation tools, including record/playback tools as well as Behavior Driven Development tools such as Cucumber (available for .Net, Java, JavaScript, and Ruby).
You and your teams will have to slow down the introduction and completion of additional stories/features. Think of this as lowering your interest rate, or paying down principal.
Ways to avoid new debt:
Design Debt: Developers, adopt Test-Driven Development (TDD) and refactor in tiny increments as you identify seemingly insignificant trends towards poor design. Whenever a task or story requires a new variation in your design, first refactor the appropriate piece of the design into one that follows the Open-Closed Principle for that variation. (That’s as concise an intro I can fit into a blog post. Explaining this technique requires UML, at least, and I’d probably rather post a vodcast.)
Quality Debt: Pin down desired behavior immediately with fast, automated tests. Adopt TDD and Behavior Driven Development (BDD) to keep away any new defects. Avoid feeding Quality Debt from the other two forms of debt.
Testing Debt: Stop writing manual test cases, perhaps today. As soon as practical, build test scenarios using Cucumber (or something similarly flexible and readable by your Product folks). It’s also up to the team to find ways to keep all tests running fast. Usually, this requires Test-Doubles.
Related courses and resources:
Wondering where developers and quality engineers can learn more about these critical skills? Here are our related courses:
After all the “Agile-ish” horror stories and doom and gloom we’ve been getting recently this is a really positive, succinct and practical post that I can point naysayers to. Thanks: you’ve brightened my day!
Ron Jeffries talks about refactoring and the XP customer (product champion).
http://xprogramming.com/blog/tech/technical-debt-2.htm
Hi Rob. I just read your Developer Essentials newsletter on this topic. The newsletter does not have a feedback link, so I am responding here instead.
You mention that Ward Cunningham introduced the Tech Debt metaphor around 2009. That is when he posted the YouTube video. In the video, he says that he introduced the metaphor “a while ago” when working a Smalltalk project, which probably means the nineties. And Martin Fowler in Refactoring (1999) uses the same metaphor, as he likens delaying refactoring due to release pressure to making a purchase on a credit card.
As for the ordering of your “core four” practices, I would move collaboration (pairing/mobbing) to #1. Taking a cue from Arlo Belshee, I have recently begun launching my team coaching engagements by facilitating the creation of a pairing/mobbing agreement (starting small, whatever the team can commit to). Once the team recognizes that it is okay to work together and that effectiveness is usually more important than efficiency, the magic begins to happen. Not only are skills (including refactoring and TDD) quickly propagated, but the team can also now apply its collaborative mindset to iterative improvement.
I was recently coaching a team whose manager had been beating the drum for daily check-ins for a year to no avail. Once the team began pairing and then meeting weekly to retrospect their effectiveness and launch experiments, they came up with the daily check-in “idea” on their own!
Hi Daniel,
Thank you for the corrections on the dates! In the absence of data, I think I chose the date of the video. Met Ward in 2001 at PDXP (possibly before then at OOPSLA, but I think I’d remember that!), and he was probably already using the term then, but I couldn’t recall.
Aside (for others reading this comment): We’re talking about the latest newsletter, which references this post. Newsletters become blog posts soon after, but there is a slight delay. So the rest of this is about text that (at this moment) has yet to land on the blog.
I agree with Arlo: The best place to start is with pairing or mobbing. He and I actually had this discussion once. Me: “TDD!” Arlo: “Pairing!” Arlo won. 🙂
So my ordering is not temporal, but more…conceptual? I wanted to lead the reader (particularly readers in leadership positions) through the reasons why these practices are needed.
I’m an old XPer, and on every team where we used XP, we used all 12-13++ practices all at once (though sometimes we tried too hard with Metaphor and that led us to the Dark Side of software design). These days folks keep telling me that Scrum teams can’t handle that much change all at once. I know that’s not true (I have faith in the resilience of software developers!), but I needed to come up with a minimal set. Ergo, the Core Four.
Which, frankly, I’d recommend dropping into a team’s working agreements as soon as possible. If we old-timers could handle going from gated “waterfall” processes to Extreme Programming in mere weeks, I’m certain today’s developers can handle picking up the “Core Four” in a week, and be comfortable with them in a month. I’ve seen it happen dozens of times.
Oh, sure, it doesn’t always work. It seems to depend on organizational culture. Start-ups and smaller, innovative product teams do best. They find the Core Four gives them more freedom and sanity. Larger, IT-ish teams are usually struggling against off-stage non-technical constraints, so the practices seem like more work (or worse: punishment), rather than sanity.