Code Exploration (CE) is quite a new term and I wonder if there already any successful examples of implementing this techniques in terms of Continuous Integration principles?
In short, Code Exploration can be described as the process of testing your code using different parameters to achieve high code coverage. The one-and-only tool for .NET world is PEX, but I know there are commercial instruments for code exploration in java world.
This topic is very powerful in conjunction with Design By Contract, because it provides you with the means of some formalism in reasoning about code correctness. For example PEX would show that next code contains error (source code uses Code Contracts and I assume default assembly configuration):
public int Factorial(int number)
{
//postcondition of positive result, precondition is omitted
Contract.Ensures( Contract.Result<int>() > 0 );
return number <= 0 ? 1 : number * Factorial( number - 1 );
}
Because by default overflow is not checked in arithmetic operations the input 17 will give the result of -288522240.
PEX was discussed here several times here:
-
How does Pex work
-
Pex and Code Contracts
But I don’t want to touch internal implementation details, I’m interested in practical aspects of this problem.
Main questions are:
-
How exactly do you integrate Code Exploration with Continuous Integration?
-
Do you reject build if PEX gives errors or you notify team about inconsistencies and contracts violations?
-
Scott Hanselman has point, that PEX should run on separate machine (or multiple machines) continuously testing your code for errors (so called Continuous Exploration). You can listen to the talk Hanselminutes Podcast 93 – Pex. Are there any practical implementations of dedicated machine (would it be virtual or physical), that digs into your codebase in search of errors?
-
How do you treat test coverage of such tools? I think it would be unreasonable to merge code coverage results for unit tests written manually with the results by a code exploration tool.
-
Are there any examples of Parameterized Unit Tests in real projects, that would increase you productivity and unit tests quality?
Even if you have a boundary checking tool generate your unit tests, they’re still just unit tests.
- Do you reject build if PEX gives errors or you notify team about inconsistencies and contracts violations?
Do you have gated check-ins turned on for your other unit tests?
This really isn’t some new, distinct concept that needs us to make new rules to handle it. It’s just being better about making the unit tests we should’ve been writing if we weren’t too lazy or too naive about how corner cases impact code quality.
4