I added a short answer saying that the answer is the same as for the question “do you care if the feature has bugs in it”. A moderator didn’t like that answer and deleted it, so I’ll elaborate.

Anytime you write code there is a chance it will have bugs. To help possibly uncover these bugs, you should test the code. Now, in some cases it doesn’t matter if the code has bugs or not. For example, internal code such as that in the question often carries very low risk – low risk for bugs because it is uncomplicated code, and because the cost to fix bugs is low. Other software, such as for medical equipment, has a very high risk, and bugs can be costly.

To decide if you should test software –any software– you need to ask yourself if it matters if the code has bugs. Will a bug cause irreparable harm, or harm that is expensive to fix? Is a bug simply a minor inconvenience? Where does your software lie on that continuium? Answering that goes a long way toward answering if you should test your code.

Of course, testing won’t guarantee your software is bug free, but even simple tests can raise your level of confidence high enough to know if you can depend on the code.

So, in your specific case, what is the cost of the dynamic re-compiling failing? Will it cause developers to potentially introduce bugs because they are running old code? Or, is it just a minor inconvenience because they can always run a manual step?

I would say this depends on what you call a “development feature”.

• Unit tests or other automatic tests could be called “development features”. Testing them separately does not make sense. In fact, you “test” them every time you run them.
• “Debugging tools” – maybe, if they are important to your team and you think someone could accidentally break them.
• features which are currently only active in your development environment, but could become production features in the future – probably yes.
• code generation tools: this depends on what happens with the generated code. If there is a syntax check for the generated code (for example by a compiler), and a semantic check – for example by unit tests for the generated code- then it may be safe enough not to test the generator separately. One could argue the unit tests for the generated code are also tests for the code generator. But in fact, if a separate test makes sense depends on the complexity of the generator, how often the generator is subject to change, and how hard it gets to find the root cause of potential bugs without separate tests. This former SO question also dealt with the question of how to unit test a code generator.