I have created a scenario in which I have two functions, A and B, both are tested, but despite the tests I still injected a rather stupid bug and some data is not getting into the database. Note that there is no ORM present. I’ve written my CRUD operations directly, and I’m paying the price therefor.
In this scenario, I have (perl) something like this:
sub A {
... do a bunch of work ...
my $res = B( %data );
... do a bunch more work ...
}
sub B {
my ( %data ) = @_;
... record data to the database and return a result ...
}
What happened, though, is that in my higher-level application and in A itself, I changed the name of one of the fields that goes into %data, but forgot to change it in B. I have a test for A which stubs/mocks out a lot of the things A does (A needs to meet with a goodly refactoring), including B. The stub for B verifies that all of the correct parameters, including the correct field names get passed in, and so the test passes.
BUT the test for B verifies the old field names, the real B actually uses the old field names, and that test passes as well.
And, thus, I put into production some code that, because of this oversight, is putting NULLs into the database where I don’t want NULLs.
- A is clearly too big. To isolate it, I mock out six different functions, each of which could cause a system state change.
- A, and thus the test for A is theoretically unconcerned as to the strategy by which B will store data in the database or even what kind of database B will store data to, so I think the test for A probably should not query the database directly.
- But, I changed the format of %data, and I think that the test for A should have detected that I basically changed the API without propagating the change down.
What is a better way to set up a test to catch this case beforehand?
- Continue with the isolation strategy for A and B, but also write an integration that tests A –all the way to-> the database? This would lead to a proliferation of possible test cases, and I’m already having trouble justifying the existing test case proliferation to management (though I know the coverage to be woefully inadequate).
- Just write the test for A to query the database, possibly through an abstraction?
- Is there something different that would help here?
Ultimately, I took Karl Bielefeldt’s suggestion. I added the data validation code into function B. I also added the integration test that tests from A all the way to DB.
As far as I can tell, I need to have as many integration tests as I possibly can, verifying that the overall state of the system resources change predictably. At the same time, I expect I’ll still need to mock network resources if I cannot actually get a test network environment.
Not every problem can be solved with black box testing. The problem with end-to-end integration testing is that the number of possible paths through the code is much larger than with unit testing. It’s still good to have integration tests, but it’s generally impossible to make them thorough unless your system is very simple. Therefore, it’s likely the next integration failure you have won’t be covered by a test unless you’ve seen it before.
My recommendation is to add some integration tests to make sure you don’t make this same exact mistake again, but also to add some data validation inside B and make unit tests for that validation (see assertion for more info). Check that it isn’t passed in any fields it doesn’t know about, that all the required fields are passed in, etc. If the validation adversely affects performance, you can have a flag to turn them off. So instead of your unit tests catching the bug, the developer who introduces it will see it in his test runs.
0
Continue with the isolation strategy for A and B, but also write an integration that tests A –all the way to-> the database? This would lead to a proliferation of possible test cases, and I’m already having trouble justifying the existing test case proliferation to management (though I know the coverage to be woefully inadequate).
This is your solution.
Having this bug should provide more fodder for management. “If we had integration tests, this would’ve never been a bug.”
There are several things you should do :
- refactor the code. Make your functions A and B smaller, with less functionality, and unit test small pieces
- add missing integration tests. Obviously you missed some tests, since it was possible to trigger a bug.
- add some run-time checks. If certain condition is not met, throw an exception.
The stub for B verifies that all of the correct parameters, including the correct field names get passed in, and so the test passes.
Your best bet for avoiding something like this is getting into the TDD habit. If you had, you’d have realised that you needed a test to fail before you made your change. That test (whether it be a new test or an extension of an existing test) would have been checking your new fields.
2