In an ideal world, dev/ops are working together as one team to release something onto production.
However, there are organizations with limitations so that release engineers/teams are kept completely away from development/project teams.
This not only added the communication pain for releases but also making the production environment almost “invisible” to project team. To some extent, there is “trust crisis” between project and deployment team.
Even we have good and reliable Acceptance/Functional tests in UAT environment but we just not sure if the deployment process has been done properly and if everything works on production exactly as UAT. So the only way we can verify this is through some tests on production environment.
Some simple smoke tests could be helpful to identify connectivity problems.
So my question is : What’s your opinion on applying all tests we had on UAT to production (with configuration to disable the bits that we don’t want to happen on production such as removing data etc.)
If you have done something similar, how do you play data on production environment without having audit trails tainted (one obvious thing would be account based, but let’s assume there isn’t segregation by account).
EDIT: I didn’t mean to test prod environment all the time, but only right after deployment.
Also we will not stop trying solving the problem from the root – dev/op relationship. But at the same time would like to explore other possibilities.
5
Usually, tests are performed outside production servers for three reasons:
-
Safety. If something goes wrong and all data from staging is accidentally removed, you don’t care. If it happens in production, it’s… let’s say annoying.
-
Performance. An extensive suite of tests can easily use 100% CPU on several servers for several minutes. You can’t afford slowing down production servers for a few minutes ten times per hour.
-
Goal. The goal of testing is to prevent issues in production. If you find an issue when the application is already deployed, it’s too late. Moreover, reverting to the previous revision can be complicated. For example, what if the new version created a column in the database, and meanwhile, you have new records where this new column was used?
For those reasons, I would strongly advise you against the usage of production environment for testing. Moreover, I don’t think running tests in production and sending the results back to your team will help to gain the trust of system administrators; they may perceive that as a tentative to circumvent the limitations they set to protect their infrastructure from your team.
Instead, address the original issue, i.e. the lack of trust between developers and system administrators. Until communication and trust issues exist on this level, the testing and deployment process will be painful, no matter what tricks you’ll use to make it easier.
1
Provided you’ve implemented a robust version control system you spread the tests out so that you just need to confirm basic functionality.
As an example the following are done in the four environments (simplified for brevity):
- DEV: Unit tests, some behavioural tests
- FAT: Integration tests, regression tests, installation tests
- UAT: Test of deployment to LIVE (possibly with faux data)
- LIVE: Common functionality, possible environment testing
DEV (Development) environment
The focus is on unit tests to test the new functionality, bug fixes, and the previous tests; some behavioural tests can be performed if required for new/modified functionality.
FAT (Factory Acceptance Test) environment
You test your installers and allow your testers to run through the application and test the new functions, bug fixes, and to ensure the old functionality isn’t broken through regression testing.
UAT (User Acceptance Test) environment
Your client/customers/users test your application to ensure it works according to their expectations. Of note is that UAT should be identical to LIVE so what works here should work in LIVE.
LIVE (Production) environment
You just need to get the update to the latest version done, and recheck basics (smoke-tests): can access data, open reports, view documents, export/import data, etc.
By the time software gets to the LIVE environment is it has passed all the previous environments without issue, so there is no need to retest; if any environment fails for any reason the software release goes back to DEV, changes are made, and the whole process starts again under a new release number e.g. 1.2.3.0 -> 1.2.3.1.
You build trust in your release to LIVE by going through each environment and completing tests so that you don’t need to retest in LIVE.
2