Agile antipattern: Using manual tests

In an agile environment manual testing is fine – except for when it isn’t!  In particular, everyone recognizes manual regression testing takes time.  When using a traditional development process companies use it as an argument for longer release cycles saying something like “It takes 8 weeks of regression testing anyway, so a large release makes sense otherwise the overhead of the testing would be too high.”  It is easy to see how people could buy into that argument.  Because of similar arguments it is hard to get organizations to understand regression testing outside the context of RELEASE regression testing.  The primary cadence of an agile process is the iteration cadence which puts a much different spin on regression testing.

Let me back up for a moment and sum up what most people believe about testing within iterations during an agile project.  Put very simply, teams are led to believe they need to get each user story to the “done” state which matches the team’s definition of done (DoD).  In most cases the DoD says the story has been unit tested, acceptance tested and has been accepted by the Product Champion, Product Owner, Customer or other equivalent role.  Using this definition will lead to having stories which all work in isolation.  The way I put it in my classes is the team has proven there is a finger, a thumb, a knee, etc. but have they proven they are part of the same body and the body works?  Concentrating on a story-centric DoD can lead to significant issues if things don’t work at a system level.  Using a story-centric DoD means it is possible to create “a finger” in one iteration and not know it completely destroyed “a knee” created in an earlier iteration.

Some of you may be thinking “But Bob, you can’t be saying we have to do our regression testing every single iteration!  It takes 8 weeks to do regression testing and if iterations are two weeks long the math won’t work.”  You are right, the math doesn’t work, but not in the way you are thinking.  What I am suggesting is during an agile process you have to have a DoD which is iteration-centric as well as one that is story-centric.  An iteration-centric DoD should include something saying all of the tests from previous iterations still pass.  In other words, tests from prior iterations become regression tests.  There are many great reasons for this, but let me describe just one situation which should make the point.

This chart shows the amount of testing done on each feature if regression testing is done every iteration.  Notice that in 5 iterations feature #1 gets tested 5 times.  Because teams develop software in priority order this means the highest value feature is also the most tested.  The lowest value feature is the least tested.  Common sense tells us this is a good thing!  Wouldn’t it be great if the most important features were the least likely to have bugs?

When using manual regression testing at some point there will be a limit to how much regression testing can be done.  This will directly impact how often the highest priority features get tested prior to release.  If a team can do no regression testing then every feature gets tested once.  If they can regression test just one iteration then almost all features will be tested twice, and so on.  For most teams the lack of regression testing leads to “hardening iterations” prior to release.  This point is very important – if you do not do full regression testing each iteration, you will pay a penalty of both time and money later!

When I present this information to executives they quickly realize an investment in some sort of automated test strategy could save them quite a bit of money.  Automated testing can be done using a variety of free tools so the only investment is in research and learning curve.  I usually recommend tools like FitNesse, Fit, Selenium, Cucumber, Watir and WatiN as starting points.  If you are using Java or a .NET language there are many available frameworks with small learning curves which can deliver value very quickly.  I tend to stay away from tools like HP Quality Test Pro only because it is a heavyweight tool requiring specialized knowledge in order to do automated testing effectively.  Note I am not talking about record/playback automated testing!  I want teams to strive for automation which doesn’t require too much incremental knowledge from what they already know – if it is too big a learning curve they just won’t do it.

If you want to take this to the next level I strongly recommend reading this article.  It takes a holistic view of the defintion of done which really pushes teams to think of everything!

Thank you for reading!  Until next time I’ll continue recommending automated testing frameworks so organizations can continue Making Agile a Reality™.