The build server at our company sucks. The build agents (there are a dozen of them, each is a separate VM, and apparently they are all living on their own ESX server) are SLOW. The web interface is SLOW and is running on a Windows VM, also on ESX. Everything is SLOW, SLOW, SLOW. Artifact upload sometimes fails for unknown reasons. Sometimes the web interface reports in the UI that it has run out of memory and it seizes up then crashes. It seems like IT is rebooting it at least once every day. Today they are rolling back a minor update because it has caused problems.
Currently IT owns the hardware that the build servers run on and we have zero visibility into that. We do have admin control over the software. We (the developers) are constantly complaining about it but nothing seems to really improve. Does it make sense for developers to own the build server(s)? IT doesn’t really have as much incentive to make them reliable and fast as developers do. In order to get IT to fix the problems we have to complain to our managers and then they have to talk to their managers who then tell IT to fix it. We can report problems to IT directly but that just doesn’t seem to work to get things fixed because IT has lots of other things they need to deal with. Also, the developers are the ones who are actually using the build system and are probably a lot more familiar with it than IT.
How does it work at other companies?
We are about a 150 person company. So probably half of that is engineering, but a third of that is probably hardware, so we have maybe 40 or so devs.
3
Ideally developers should not have to own the build server or even maintain the build process themselves. Every build in a company should go through common parameterized pipelines that have been defined for building projects at the company. This provides several benefits: First, developers don’t have to worry about the build process, a dedicated team maintains it. Second, new changes to the pipeline can benefit everyone at once, rather than having to roll it out to dozens of individual build processes. Lastly, it allows the organization to derive powerful metrics about the quality of their builds across the board.
This isn’t a pipe dream; we actually do this at my company. 300 engineers build several hundred projects in the same pipeline on a regular basis, and we sometimes have thousands of runs through the pipeline in a single day, all building and automatically deploying to production. It works amazingly well, because we have a strong emphasis on Continuous Delivery principles, as espoused by the canonical book on the subject.
My organization didn’t arrive there overnight. It required buy-in from executives at the top. We were at the same point as most people about two years ago: Individual builds for projects and manual configuration and deployment to environments. Following principles of Continuous Delivery has yielded enormous benefits for developers.
So, like I said, in my mind that’s the ideal builds and deployments strategy. But it sounds like your IT department hasn’t bought into the concept of providing quality build systems and processes yet. If you can’t get support from your IT dept., then my suggestion would be to try to roll your own process for now with your own build servers if you have the organizational freedom to do so. Leaving IT to do your own thing is a surprisingly effective way to get them to offer better support to you. 🙂
1
In the past, I had very positive experiences with having a dedicated build team consisting of just a few developers who reported to the QA department. The build team’s builds were the company’s gold standard. If your build was broken, they wouldn’t accept it. The code that shipped came from their servers only. This provided a lot of incentive for each team to ensure their builds and unit tests were sane. This worked very well in a company of nearly 120 developers and 40 QA all spanning ~15 teams.
In my view, each team is responsible for their own builds but not the build server. Leave the developers to develop! A build team is responsible for the build servers and for collaborating with each respective team to ensure build and test sanity.
You may want to take a “grass roots” approach. If you have the option to set aside your own hardware, I would set up a personal build server and set the example. Once word gets out that you are able to quickly provide successful and repeatable builds, others will want to adopt your strategy. Hopefully, this will encourage buy-in from those who have the power to make the necessary changes.
2