at company we’re developing quite a big project and we’re arguing at the testing strategy. The question is: should all of the tests be executed in isolation of external services like database or APIs (facebook etc) or just part of them? Of course what I have in mind is using mockups.
We discussed following strategies:
- Write all of the tests in isolation of external services – mock externals everywhere
- Write part of the tests in isolation and create single bigger functional test for every feature that tests it using externals (without mocking anything) – using fixtures
- Run all tests communicating with externals (of course that’s never appliable to unit tests so they’re out of scope in that case)
I know it can start quite a discussion but I think that’s what it is all about, I’d like to find pros and cons that I didn’t think of already.
3
If done right, your second strategy can have the best of both worlds:
Write part of the tests in isolation
Isolating most tests from external services is a good idea because you have no control over that service. The worst-case scenario is to have unit tests fail intermittently, because eventually you’ll start to ignore the failures, at which point the tests are useless. Creating a mock that provides consistent responses to your tests makes it much easier to verify how your code reacts to a given response from the external service.
create single bigger functional test for every feature that tests it using externals
This can provide important insight into the behavior of the system closer to actual operating conditions, allowing you to recognize and handle different issues than you’ll encounter when external services are mocked, such as network issues, or input or output data formatting errors.
sometimes you don’t know how external source might react in some conditions, it’s hard to mock it (for example Solr with complex query)
sometimes you can forget about software-specific case (for example switching from MySQL to Postgres, while using ORM, might return different results and cause some other problems)
If you can dependably reproduce some response from the external service, and you want to verify that your code reacts to it in a certain way, then it makes sense to test it. The important thing is that you don’t allow intermittent failures in your tests that e.g. run with every build. I’ve seen very good developers switch off the tests because a build needed to be made and they didn’t have the time to fix an obscure, complex test. That’s a bad situation to be in, because the tests are no longer serving their purpose, and confidence in them is reduced.
2