A question that has been vexing me lately has been about how to effectively test (end-to-end) features in a distributed system. Particuarly, how to effectively manage (through time) test data for feature testing.
The system in question is a typical SOA setup. The composition is done in JavaScript when call to several REST APIs. Each service is built as an independent block. Each service has some kind of persistent storage (SQL Server in most cases).
The main issue at the moment is how to approach test data when testing end-to-end features. Functional end-to-end testing occurs through the UI, and it is therefore necessary for test data to be set up before the test run (this could be manual or automated testing). As is typical in a distributed system, identifiers from one service are used as a link in another service. So, some level of synchronization needs to be present in the data to effectively test.
What is the best way to manage and set up this data after a successful deployment to a test environment?
For example, is it better to manage this test data inside each service? Or package it together with the testing suite? Does that testing suite exist as a separate project?
I’m interested in design guidance about how to store and manage this test data as the application features evolve.
I faced a similar question recently. The answer so far has been a mix of both. Some shared data in a single place, other data created by relevant components/tests.
I found shared, centralized data convenient when it reduces common boilerplate, but it’s costlier to maintain because it must be reconciled with all the myriad services.
Specialized data has been more suitable for narrow service tests, and often feature tests (if they don’t span too many services). Individuals focused on developing or testing particular services may be best suited to curate this data.
However, with enough complexity, I believe data can deserve significant specialist attention, specifically a person or team who is at least somewhat dedicated to the task of data generation, data tools and strategy. This person/role(s) would drive especially the shared data story, when data is distributed and use cases span multiple services. If done well, test data and/or data generation tools may be used for several needs — development, testing, product management, and demonstrations. I have seen successful situations where an automation or tools department responsible for test automation tools and infrastructure is also responsible for data and data tools.
Trying to provision for a widely distributed system is a hard thing to do.
Assumption 1: You don’t have control over the databases at the level where you can reset them readily. Institutional test databases tend to be used, often heavily, by a number of separate development/maintenance groups. Hence, you cannot wipe/restore all of the databases without extensive coordination.
Assumption 2: You have support from the data stewards of each repository for necessary testing; hence, you can call on data managers to work with you on data sets. The data manager might allocate a range of test ZIP codes, or usernames, or similar.
Assumption 3: You are able to manage the data sets you are working with in each repository domain. For instance, in a contact database, you might be able to add an address of “1234 Testers Lane”, or, perhaps “Suite CamoTester1, 1234 Birch Street” when ZIP code checking is being done.
Assumption 4: You are able to establish clear mappings between systems that exchange data. While not completely necessary, it will leave holes in your testing if you can’t arrange this.
Set up provisioning sheets (MS Excel is your friend) with worksheets for each repository. You may end up with hundreds of data cases for each repository. Nobody said it would be easy! Now you have to come up with a way for your data not to explode combinatorially on you — see assumption 4. Take data from sheet 1 for repository 1, determine if it is a source or sink for repository 2, whether it needs to coordinate with repository 2. Continue on down the line.
I have experience setting up testing for the connection between two ERP systems. Likely, the size of the problem was similar to what you are facing, although yours is more complex.
If your prod, dev, and test environments are seperated then this should be relatively easy. The database should also be in test, which allows you to import a known dataset into the database for testing. Instead of import, you could restore a db snapshot from backup before starting test.
In the past, I have bundled test data in the application test suite. This is either a set of SQL scripts to insert data or a db backup to be restored. My test suite was a seperate project but was always matched the application release. So, I would not use the v1.7 tests against the v1.9 app, unless I explicitly wanted to simulate an upgrade scenario.
1