I have recently come across a requirement related to our release/SDLC process that seems a little out of the ordinary to me so I am writing this to inquire whether this practice is common and, if not, whether our deployment process should be changed to eliminate a burdensome constraint.
I work for a small software shop that makes enterprise software (so not a household name) and we only have a handful yet very large customers. It is a multi-server system management application running on Linux, DB backend, web front end, lots of async I/O and messaging. We roll out upgrades approx. every 2-3 months and our customers deploy them using deb or rpm (depending on OS) packages we provide them. All the customers deploy the app in a cluster environment with each server instance pointing to multiple database shards.
As of late I have been working on some legacy refactoring that significantly changed our DB schema. To make a long story short, I reorganized how we store some data, split some tables into multiple new tables, reduced exposure to race conditions, and made corresponding code changes.
Just as I was getting ready to submit my changes, I was informed by our systems architect that, “due to the way our customers deploy upgrades” (each server and each database separately, one by one and can take days if not weeks, during which some of the servers must stay up), we cannot have tightly coupled changes and that all code upgrades must be reverse compatible with the database within a one release cycle period. IOW, we cannot remove fields from database tables because in all customer configs all servers point to all database shards and we must enable a server to simultaneously work with one shard that is upgraded along with another that is not.
Now, I was befuddled and troubled by this requirement to say the least. It seemed indicative of problems that the way our customers like to deploy changes (taking a long time), as well as their configuration, necessitates us to provide cumbersome dual functionality in parallel. In my particular case, I was told that I need to keep the refactored fields in the old tables for legacy reads while writing into the new tables and then amalgamate the retrieved data. This seems unnecessarily complex to me to say the least.
My questions are as follows:
-
Is this requirement unusual for the general configuration I described? I have worked here around six months and I never worked for a product company before, always worked on site that made and deployed apps of their own (like a web site etc.). Is this scenario common in software product companies?
-
Is this phenomenon of necessary reverse DB compatibility indicative of problems with the way our customers like to deploy our software? To my knowledge, they apply our upgrades manually server by server, database by database while maintaining the production function. Should they be using something like
PuppetorChefto deploy our upgrades in a much shorted timeframe? -
How should I approach presenting a need to change all of this, it presents a developer inconvenience to say the least having to provide for this bifurcated functionality?
1
No, it’s not an unusual requirement. It’s common to expend significant effort to make upgrades as seamless as possible for the customer, sometimes even going so far as to purposely retain bugs that client software depends on.
I’ve had similar requirements even in environments where upgrades are done quickly with a script, even if the dual code is only in place for 15 minutes while the script runs. It’s very difficult to achieve zero downtime without such requirements.
Your job is to remove complexity for your customers. Embrace that role.