We recently caught up with Doug Thompson to talk about unit testing, Typemock, TDD and Steven Sanderson’s blog post on the differences between good and bad unit tests.
There is a known notion that sometimes TDD practitioners tend to write too many unit tests in their pursuit of 100% coverage.
The question that many may be asking themselves is whether Typemock helps to focus testing efforts on the code that really needs to be tested. Let’s see what Doug Thompson thinks of that.
Please illustrate for us your experiences with difficult test cases where Typemock has been effective.
Typically where Typemock has been most helpful is in fully faking the HttpContext or WebSockets allowing me to test some distant condition, to assert some post conditions or side effects.
Tricky examples are where I only had the DLL to go with and there was no ‘sandbox’ mode native in it. I had to fake it, so as to use it as if ‘in the real world’ where a dependency injection route was not so easy.
The other examples are when taking ownership of legacy projects that have very modest code coverage, where the test cases were really just a method to initialize the database. All very typical when the stakeholders (accounts or marketing department) want a product delivered, and technical debt is not something that appears on a budget. This is common when it’s ‘development by cattle prod’, which often is the result of unrealistic expectations being set during the ‘Bid’ phase.
There are different priorities if you are a software house that has optimistically quoted for a project (developing to a price), the customer having a different expectation for that price (wanting it for free), or the person that has to maintain it as if scaling up from a handful of users to a multi-tenanted cloud application.
This is when there is some distance between the original technical architect (who has possibly left after enhancing his CV with all these cool new technologies) and the operational development team.
All you can do is create a test harness of the as-is system (assuming it is testable), before you refactor something you don’t fully understand, to meet the new business requirements.
So Typemock allowed me to mark as ‘safe’ subsystems that I didn’t fully understand.
Can you give us a few examples in which you used Typemock for testing subsystems?
For the most part I have been working on (Web) APIs connecting into suppliers and consumers of GMS/SIM chips. We have thousands of SIMs in the estates of our clients.
These are typically used in the IoT devices, but also Hybrid Networking (eg. WAN anywhere-type solutions as used on oil rigs to planes, trains and automobiles).
You would buy SIMS from diverse suppliers because your connections need as many paths as possible for a good, reliable, bulletproof network connection.
Each of these suppliers have their own propriety system which has their own naming conventions and data quality issues.
So the case of gaining the usage data of a SIM will vary depending on who you got the SIM from and how they define usage.
A challenge I had was to assume there was an industry standard, so I had to abstract away all these differences, and provide a unified status and usage data no matter where you got the SIM from. I had to establish correctness and I wrote tests cases against this that I could build up from.
Another example, is if you only have the DLL that is your gateway into their system.
This is where you have to decompile that DLL into a project and create test cases for that project, which will give you examples of how to test the bit of interest to you when you sandbox that DLL.
One example is that a DLL I was using had the SQLHelper lib for ADO (EF was too slow for us). It had additional logging (log4net) in there for when you were trying to call a user-stored procedure with differences in the parameters called.
The initial fakes would just test if got a set of results when you called a stored procedure. If you used ReSharper (big fan) you would be left to actually call your test local database, which is fine, but you now don’t need to.
Moreover you can also test the side effects, e.g. that logging was actually called. This can be a long side effect chain that you are testing.
The usual route with these sorts of problems is to use a repository pattern and swap out the test version with the live version. While the repository pattern has value, how many times are you going to swap your architecture from SQL Server to Mongo? With the repository you are not testing down to the metal and in typical TDD you just mock these methods. You test what you can test, but with Typemock you can test further.
For me I needed to know that it actually worked down to the metal. Once I was sure of that I could fake that metal.
These test cases also have the benefit of a library of code examples.
Another example, is testing the SMTP server – everyone has written an emailer, typically for login recovery.
You don’t really want to be sending out test emails to real users, especially if the emails are hostile.
One route is to have an SMTP black-hole (e.g. smtp4dev.exe) so it does not matter how many emails you send to the outside world – nothing gets through once enabled.
Another is the fake the email client, which behaves exactly as in production with all the post-conditions you expect, but sends nothing out over the network.
So I can construct a situation in which an email cannot be sent and gets queued in the Try Again Later queue. I then have to change the system time (NOW) to the Try Again time and arrange things so it can be ‘sent’.
So I can test a multi-threaded email gateway with robust retry behaviour.
I have not yet moved these test into TFS for CI. It’s more about ensuring something is testable, and how easy you can do this over and over.
One big benefit of Typemock is it telling you (in under a second) how many tests currently cover the class or method you are working on, and if any have failed.
Its communication of CodeCoverage is different to that of ReSharper, which is a little better.
I can understand the 100% coverage ‘security blanket’ desire. Like if ‘everything’ is covered all is well, but is it really covered enough?
For me, TDD using MSTest and ReSharper is the norm. I just get the basic Lego building blocks working, then move them into bigger classes, locating/filing developed work into the right name space. My current code coverage is as below and the ReSharper coverage (which can’t run Typemock’s autosuggested tests) is below that.
Typemock and testing subsystems
Typemock is good for testing subsystems. The next layer up from this is TDD, as it looks after what you have done – what is in your ‘keep net’ as it were.
In one of the projects that I picked up, the ‘tests’ were some code that were not part of the main development branch. They used EF CodeFirst to create the local database and populate it with some data, e.g. the admin user. This at least creates a consistent starting point each time.
I am quite happy in having unit tests that have a section for setting up the preconditions, the arrangements doing the actions and asserting the post-conditions, as I was brought up on ‘Programming by Contract’ (I used to be an Eiffel developer).
This route explained why something was done, not just the test failing. The biggest problem in the past was with the ‘oh all the tests failed – they are useless’ approach.
There were two reasons for this – the tests were not maintained, and they did not use fakes or mocking. So the tests were good for a given phase, but circumstances moved beyond that. Typemock will move redundant tests into a grey ‘ignored’ box, and it is also amazingly easy to activate or deactivate tests.
So if tests are hard to maintain because the ‘agile’ / fast moving requirements now render some development obsolete, you have a lot of tests that now fail.
Some of the Typemock suggested tests repeat – like trying to get an Argument Exception by passing in a bad variable. This is fine when you have a param that is a string, where a string can be null, or a sentence or a valid date but in a format (e.g. wrong culture) that cannot be Cast to an SQL Date .
Typemock will keep exploring these routes so you have many similar tests – there comes a point when you want to tell Typemock to say enough already. This is where some of the under-documented ‘Pragmas’ (directives in between []) become useful.
On testing for testing’s sake
I am used to unit tests telling you that you have to use the methods and APIs at hand. Sure you expect the happy path to be covered – often that is not the case.
So in Steven Sanderson blog, he makes a good point about testing for testing’s sake. You’re there to write a product, not test it – what did the customer pay for? What was covered in the bid price?
As long as it passes QA in another department, that is all that matters. They usually follow a paper script, for example creating a user account using friendly data.
They are using the system per the manual, but no-one gave you a manual to use Amazon. You never cease to be amazed at what a user will do and how your system is meant to respond (defensively) .
Testing code that connects to a subsystem such as a database is very normal in business. Connecting to other actors or business partners is very normal – this is usually over an internet connection of some kind.
These are effectively faked in Typemock, which MSTest can’t do (and so you have to use a local test database) and test web service. I have already taken down the test website of one of our partners because the test case killed its API – like a LoadTest.
So this is where you need to differentiate between unit-tests and integration tests. Because they are both run using a test runner and are in an MS Test project they seem the same. So it might be a better habit to have a project called ‘integration tests’ and ‘unit tests’ and move the Typemock one into the ‘integration tests’ project.
At least if this starts off as a template, it clarifies the purpose of both.
