This is a follow up to this question. There I was asking how to do unit testing when you have a library of scientific algorithms. I have a similar problem now but with a different project.
I’m working on a 3D graphics engine framework abstraction for DirectX, OpenGl, WebGl, Silverlight, WPF, and basically whatever 3D API is out there, on C#, called Rendering .NET. The scenario is the following, the project is being developed by a small team of colleagues of mine, all working in the same office. We are all computer scientists, not much into the software engineering community and practices. I even had a hard time to convince to switch from Subversion to Git, and to publish the project on Codeplex. The project is getting big (around 80K lines of C#), and it now covers most of DirectX and OpenGl up to Shader Model 5.0, so its not a one man’s project like the one described in the linked question. We also want to encourage the open source community to collaborate.
So far we’ve been testing by writing small applications that involve initializing all devices and resources, setting up a scene, and drawing. The thing is, I don’t know how to make it different, I mean, how to design small test cases that test specific features of the framework, like resource allocation, primitive tessellation, shader compilation, etc., without having to recreate the entire engine initialization. For some of these cases I can think of useful mocks, but in general, how can I test the correctness of a rendering algorithm when the output is visual (an image, or an animated scene)? Right now the most clever thing we have come up is to render the same scene to a software renderer, DirectX and OpenGl, and compare them pixel by pixel, but tiny variations of the APIs make the resulting images different enough to fail the tests without being actual bugs.
So, what is the “correct” testing approach here?
3
The “correct” testing approach would be to decouple your drawing logic from the DirectX or OpenGL calls, so you can mock the latter out (and as a another use case, use the same business logic either for DirectX or OpenGL).
You don’t want to test if DirectX or OpenGL is working correctly – that is Microsoft’s job. You also want your device initialization code to be written and tested only once (and then reused), so a manual test of that part should be affordable. If you want to test that part automatically, too, use your pixel-by-pixel approach, but with a very simple test drawing. Thus focus on writing automated regression tests for your decoupled drawing logic, that will bring you the most benefit.
So the whole idea is to split your code into really independent components – your DirectX initialization should not be coupled to the drawing logic, and the drawing logic should not depend on DirectX – this makes your program testable.
EDIT: found this former post on unit tests for OpenGL code, there were only a few answers, but perhaps it will bring some additional insights.
5