Is their only similarity the fact that they are not xUnit (or more precisely, not based on enumerating specific test cases), or is it deeper than that?
Property-based testing (using QuickCheck, ScalaCheck, etc) seem well-suited to a functional programming style where side-effects are avoided. On the other hand, Design by Contract (as implemented in Eiffel) is more suited to OOP languages: you can express post-conditions about the effects of methods, not just their return values.
But both of them involve testing assertions that are true in general (rather than assertions that should be true for a specific test case). And both can be tested using randomly generated inputs (with QuickCheck this is the only way, whereas with Eiffel I believe it is an optional feature of the AutoTest tool).
Is there an umbrella term to encompass both approaches? Or am I imagining a relationship that doesn’t really exist.
Maybe one relationship lies in the concept of Oracle described by Bertrand Meyer, author of Eiffel, in “Seven Principles of Software Testing”.
A test run is only useful if you can unambiguously determine whether
it passed. The criterion is called a test oracle. If you have a few
dozen or perhaps a few hundred tests, you might afford to examine the
results individually, but this does not scale up. The task cries for
automation.Principle 4: Applying oracle
Determining success or failure of tests must be an automatic process.This statement of the principle leaves open the form of oracles.
Often, oracles are specified separately. In research such as ours,
they are built in, as the target software already includes contracts
that the tests use as oracles.Principle 4 (variant): Contracts as oracles
Oracles should be part of the program text, as contracts. Determining test success or failure should be an automatic process consisting of
monitoring contract satisfaction during execution.
-
Based on this, I guess we could say QuickCheck and Eiffel’s Auto Test both use automated test oracles but different oracle forms, the former being external to the production program (properties expressed in separate test suite) and the latter, built into the production program (contracts).
-
From a historical perspective, we can assume that QuickCheck (first released in 1999) and earlier work on property-based testing dating back to the 90’s served as inspiration to Eiffel’s AutoTest (which appeared in 2008), although I couldn’t find any firm evidence of that in Bertrand Meyer’s papers or interviews.
2
The relationship is that the combination of design by contract and the testing methods are attempting to substitute for a correctness proof, which would be the ultimate goal, if it were feasible. Pre-conditions and post-conditions (for return values) are useful even in methods with no side effects, as they limit input domains and output ranges.