Fragile Tests and the Cost of Maintenance
This post came from reading a question on our forum, as well as reading one on TDD experience. It’s about fragile tests, and the cost of maintenance.
We’re talking about refactoring. And not just for making the code prettier and maintainable. We refactor and then our tests break. Well, sometimes they do, depending on how we wrote them. But if we change functionality, yes they will have to change.
It becomes worse when our tests know too much about the inner logic of our components. If component A uses B in some way, and you change that, it maybe that integration tests for A are still passing, but the ones where you fake B break.
What should we do? Are we doomed? Is this our fate? Is there no way out?
Alas, there is no simple answer. First, remember that the cost of maintaining your tests is largely outweighed by having these tests in place at all. Having no tests at all costs a lot more than having to maintain a few tests. So if someone tries this on you, show them the door.
The second thing is to come to terms with the fact that test code is just that – code, and therefore is subject to the lifetime changes such as production code is. If you are using Isolator or any mocking framework out there, you’re bound to hit this sometime.
But you can minimize the risk of creating fragile tests. Here are a few suggestions:
- Verify less. The Verify APIs (or Mocks in general) tie your test code to internal functionality. These have much more impact on breaking tests on changes than WhenCalled APIs (or Stubs). Use them only when you need to.
- Write focused tests. The more focused you test, meaning testing a small portion of the code, you’ll be less affected when you change other code.
- Minimize usage of NonPublic. This makes a lot of sense, although obviously it’s there because you need it. Still, try.
- Use less WhenCalled statements. Less statements means less knowledge of tests of the tested code. Here’s Isolator biggest advantage when you use recursive fakes. Since you’re changing the behavior of an entire tree of objects (without specifying which objects) in a single line, it minimizes the risk of a broken tests.
- Use proper tools, especially refactoring tools. Resharper, CodeRush and others make changes much safe then by doing them manually. Like I always say: use the proper tools for the right job.
What’s you learned-the-harsh-way lessons for avoiding fragile tests?
