I’ve been thinking a lot about writing tests for web apps lately which gave me an idea for a series of posts.
First up: Enforcing 100% test coverage.
On the project I’m on we’ve been enforcing 100% test coverage for much of the system for the past few years.
What does this mean?
There are probably multiple ways for test frameworks to calculate test coverage, but the one we use (@hapi/lab) considers all lines in all files in your repo imported by your tests. All the files that are not in your tests. Typically everything in lib/ or src/, but not test/. One subtle thing to keep in mind is it only considers files that are used either directly or indirectly by your tests.
This has the effect that all the code that you test has to meet your specified threshold. You can set thresholds other than 100%, but for this discussion I’m going to focus on 100%.
So in our case, all the code has to be executed by our test suite at least once.
One more subtle thing to consider: Every condition has to be met AND not met. So if you have an if statement like if (a) {, then you need a test for the case where a is truthy and also one for when it is falsy. If you have an if statement like if (a && b), then you need test cases for when 1) a is falsey, 2) a is truthy and b is truthy and 3) a is truthy and b is falsy. ie. Every condition has to be tested. Not all coverage calculators are equally good, but this is the ideal.
What I thought it was about
At first, like many, I grumbled about this. It felt overly strict. You would work on a feature for what felt like ages and got all the tests to pass, but your PR build would still fail because you’re missing coverage on a couple of lines. I’d find myself arguing that this line doesn’t matter. It will never happen in production. We can just add a linter comment to ignore this one. etc.
And almost every time I added tests to fill out that last bit of test coverage I would discover a bug. But that’s not even the most valuable thing. I gradually came to realise that test coverage isn’t about testing at all. Let me explain.
100% test coverage is about code quality
Often when I find that a line misses coverage it ends up being a really complicated statement with lots of ands and ors. And the coverage tool just points at the line, so I don’t know which condition it is that misses coverage. So I do the sensible thing and split that line up into multiple if statements so I can at least tell which part is missing coverage.
A common case would be when checking multiple things to determine one boolean. So the easiest is usually to move it all out to a function. Then I’m forced to think of a name for this function which is already a good thing because good names are part of the way to self-documenting code. Since this function is just going to return true/false I can now use early returns. So I can divide and conquer. if (!something) { return false; }. if (somethingElse) { return false; }. And so on, ending finally with a return true;. (or obviously you can approach it the other way around - whatever makes most sense in the moment).
Now I can run the test suite again and the lines with the missing coverage will point to exactly the condition that was never tested. It is usually obvious whether it is the truthy or falsy path depending on whether the code inside the if statement blocks executed or not.
Already we have ended up with better, much more readable code.
If it is a legitimate case where you’re just missing a test, then write the test. But what if you find yourself arguing that “this can never happen!” or what if you try to write tests for it and realise that you cannot ever get that condition to be either true or false? This often means that the code is redundant and should be removed. It could be that you’re testing things you control, it could be that the input was already validated in an earlier scope, it could legitimately be something that could never happen or maybe another check in the same function already covers it. (All good topics for future posts.)
This, to me, is the true benefit of 100% test coverage. Without it there is a good chance that you would never have stopped to think about any of the above. And even if you did it is unlikely that you would have had as much confidence that you could just remove the code.
