I primarily develop in .Net, and have been playing around with F#. It’s a nice concise language, but I’m just using it for throwaway code currently. I’ve taken to sitting my tests in the same file as the main code that I’m writing.
I am under the unexamined opinion that this is done so that we don’t have testing code floating around in production, and that we have separate dependencies for testing which we don’t want to go to production.
This has raised the question for me: why do we traditionally separate out test assemblies and not just use conditional compilation?
4
Unit testing can involve a number of different libraries and dependancies than the application that is associated with it.
Thinking of an application I recently wrote as a proof of concept (Java) – a simple CRUD app with unit tests against an in memory database.
WebApp | Unit Test
-------+----------
| JUnit
| Mockito
Tomcat | Jetty
Mysql | HSqldb
So in the web app, I’m just linking against Tomcat and Mysql. Within the Unit testing I’ve got JUnit, Mockito, Jetty, and HSqldb that are being used. So its more than just conditional compilation to try to remove, its also the various testing frameworks that are used that you don’t want linked in with the main application. Mockito, Junit, Jetty and HSqldb have no place at all in the production application – why include them as part of the linking and deployment for it?
When working with Maven (again Java), the directory structure looks like:
my-app
|-- pom.xml
`-- src
|-- main
| `-- java
| | `-- com
| | `-- mycompany
| | `-- app
| | `-- App.java
| `resources
|
`-- test
`-- java
| `-- com
| `-- mycompany
| `-- app
| `-- AppTest.java
`resources
The resources directories are where one has the dependency injection information so that when the test runs, the test resources are used (allowing one to have the connection to the in memory database instead of production).
This structure also prevents AppTest from getting compiled in with the main application. Or any of the resources for testing to get compiled in. The build process doesn’t even see the test directory. Thus, the code that is tested is identical to the code that is deployed.
That last bit is the key – the code that is tested being the same as the code being deployed. I remember my days back in college with some C code that had #ifdef DEBUG scattered through it, and on occasion there was code in there that had a side effect. I’d take the debug off, and it wouldn’t work. Debug on, its fine. The same issue exists with conditional compilation. It can exist with testing too (if you start tinkering in the guts of the module while testing it, but this becomes harder to do bad practice because its easier to do it the right way.
The other bit is even if this code isn’t code that you can invoke easily – having testing code of any sort in deployed code means that a malicious user may be able to find that function to invoke, or entry point to call that now exposes something you shouldn’t let users get at.
It is normal to have a build server with a very well control version of the compiler etc on it that builds the software, runs the tests and if the tests pass creates the installer using the same dlls that were tested.
There is an additional risk of configuration errors if the software is built in a different way for testing then for shipping.