I work on a fairly large software system and over the years it has accumulated a lot of entropy. There is plenty of scope for refactoring but there is always pressures to build the next features upon what’s already there. This adds to more entropy, because design choices for implementing new features are typically made by first accepting what’s there and ‘working-around’ some earlier weaker design that is otherwise ripe for refactoring.
What are some ways to manage this kind of complexity and build functionality without substantially weakening the structure of the system further. I know this is a very broad question and the approaches/solutions depend on the particular software system at hand, but I am still hoping there are some generic ways to manage the problem I am highlighting.
6
You need apply Separation of Concerns principle in order to separate the system in modules which only communicate between each other other via very well defined interfaces. This greatly reduces complexity due to many reasons:
- Some can work on a part of the project without understanding the other parts;
- You can easily improve the system by adding another module;
- Loose coupling – the modules are only connected by interfaces;
- High cohesion – the data from a module is used in almost all the parts of that module.
Ah yes, this problem.
The first thing you have to do is to get management on board with the idea that the software needs to be made evolvable (my term). The idea here is that instead of a system of fixed design and changing requirements, you have to get them on board with the idea of modifying the design as the requirements change. Otherwise you invariably end with monsterous systems that are impossible to maintain.
The second thing (once you have management on board) is that you have to get a commitment from programmers to separate contract from implementation. Then you have to be willing to write unit tests against contracts, not against implementation.
Now if this is a large-scale system (and it sounds like one) the next stage is actually to build a new framework. New code goes on the new framework and eventually the old framework will be retired. You should do this in a way that allows you to preserver old contracts (with external apps) in the new framework for as long as you can.
From there, the solution is what I call “refactor with a chainsaw” and remove well-defined blocks of functionality from the old framework and move it onto the new framework. On a large scale system this can take years to move over everything.
Now if you have unit tests on contracts, then you can refactor your own implementations as you go without worrying as much about breaking things.
You can’t. As you already acknowledge, every new feature built on the weak foundation makes that feature weaker, perpetuating the cycle.
What you need to do is spend the time to refactor things to at least not require hacks for new functionality. Maybe that means a heart-to-heart with management about tech debt and/or maintenance costs. Maybe it means having high estimates to make sure you get the time to do it right. Maybe it means having the pride at your craft to take the time to do it right, even if it doesn’t get done ‘on time’.
But you’re the one writing the code – so you’re the one who is going to have to fix it. How you do that will vary from place to place.
One reason that systems get into this state is that there has not been anyone guiding the changes. Many successful long-term projects have associated steering committees and roadmaps to guide development. These are especially helpful when there are multiple development teams and stakeholders involved.
My suggestion is that your team develop a roadmap to refactor the system. First, define a goal. In your case, define the ideal architecture for the system. Remember, it doesn’t have to be perfect, just workable and better than what you are working with. Take into account software, development tools, hardware platforms, major new features that may be added to the system. Do a cost analysis. If you have the desired architecture, what are the tangible benefits of performing the refactoring? (Benefits would include reduced test time, more reliable software products, more predictable development cycles, etc.) If you can show enough benefits to performing refactoring, you can get management support.
Once you have a roadmap and management support, Develop a series of steps that you need to take to get to this desired state. It may be a good idea to put together a steering committee that maintains the roadmap, trains and educates the development teams on the roadmap, and periodically reviews changes for architectural integrity.
Roadmaps are really useful, and perhaps the 13th question on the Joel Test should be “Do you have a roadmap?”
Here is a video of the first hour of a recent Red Hat Linux roadmap session.
0
I must say I don’t really like the unit testing, and I still see it as a partial patch which doesn’t fix good design, technical debt or any of the multi-threading issues. But, I think that the unit testing might be the only way to fix the issue.
After things get really bad, only the unit tests can help to get stuff under control.
Or a rewrite with skilled developers, but that will loose features, some of which might be bugs that customers perceive as features.