Michael Feathers posted about “Taking Automated Tests Off The Pedestal”. The post generated quite a discussion, which really revolved around the time it takes to run tests in established teams. Michael talks about how the number of tests grows larger, and that the related time needed to run these tests becomes a “yoke around their neck”. But teams continue to put the effort into writing more tests, because of the perceived cost of not covering features.
Eventually, Michael suggests that we evaluate the feedback value we get from tests, and maybe put a limit on how much quickly we get this feedback. If we’re over the limit, we should sacrifice coverage for the sake of feedback. As Michael said: quite controversial.
I agree.
One of the principles guiding us with Isolator V7 is the idea of immediate feedback. That’s, after all ,what unit tests are all about.
The problem of course, is that not all unit tests are really “unit tests”. If you are experienced enough, they might be. But unit testing is an acquired skill, and as such, you may not write quick tests right at the start. Or you may combine some quick tests with long running tests. These long running tests can add up for a large inexperienced team. And when they get experienced, they are already stuck with the long running test suite. How do you get immediate feedback?
We decided to take the controversial step.
Right at the beginning when we started building our auto-runner we understood that we can’t rely on tests running automatically for a short time. So we decided that if tests take longer than 5 seconds, the runner moves on to the next test, and excludes them from further automatic runs. You want to run them? go to the test and run it manually. Or run all the tests manually. But they are out of the automatic loop.
Like Michael said: controversial.
We know that in order to take the experience of unit testing to the next level, we should maximize the value we get from it. And the value of unit tests is immediate feedback. The rest of the tests are nice to have also, but should not impede our flow. The more unit tests you write, you get better coverage and feedback, which are their value.
Isolator V7 has some more punch in it.
Stay tuned.
