DevProd

Quantifying the Costs of Builds

In this document, I want to provide a model on how to calculate the costs of your builds and the return you get on improving it.

Developers and build engineers are under constant pressure to ship faster and more frequently. This requires fast feedback cycles and thus fast builds. At the same time, cloud, microservices, and mobile have made our software stacks more complex. Together with growing code bases, and without taking any actions, this will slow you down as your builds become slower and build failures are harder to debug.

Improving this is a huge organizational challenge for build engineers and development teams. To trigger organizational change it is important to have quantitative arguments. Inefficient builds do not just affect your ability to ship fast they also waste a lot of your R&D bandwidth.

Some of our customers have more than 100,000 Gradle build executions a day. Most medium to large engineering teams will have at least thousands of builds a day. For many organizations, this is a multi-million dollar developer productivity problem that is right under your nose. And every effort to improve it should start with assessing its impact.

Meet our example team

Our example team consists of 200 engineers with the following parameters:

CM $1 Cost per minute of engineering time
DE 230 days Working days of an engineer per year
CE = DE * CM * 8 * 60 $110,400 Cost per engineering year
BL 2000 Local builds per day
BCI 2000 CI builds per day
DW 250 days Number of days the office is open
BYL = BL * DW 2000 Local builds per year
BYCI = BCI * DW 2000 CI builds per year

The example numbers above and later in the article reflect what we see typically in the wild. But the numbers out there also vary a lot. For some, we have better averages than for others. The example numbers are helpful to get a feeling for the potential magnitude of the hidden costs that come with builds. Your numbers might be similar or very different. You need your own data to get a good understanding what your situation is and what your priorities should be. The purpose of this document is to provide a model with which you can quantify costs based on your own numbers.

The Challenge of the Build Engineer

The role of the build engineer is undergoing a transformation. Build engineers are becoming the key to success for any software producing company. But this transition is not happening without a lot of pain.

Traditionally the responsibility of a build engineer was to create and pass on artifacts from development to production. More and more the focus is shifting to improving developer productivity and enabling fast and frequent shipping of software.

Companies that rely on software to run their business might find themselves at a competitive disadvantage (and even out of business) if they do not aggressively optimize developer productivity. But across the industry, developer productivity is much lower than what it should be.

Software stacks are becoming more complex, and the code bases are growing which, without taking any actions, will slow down your builds and make build failures harder and harder to debug. At the same time, there is great pressure to become more agile and release fast and often, which requires fast feedback cycles and thus fast builds. This friction and frustration start hurting the team. Why?

Over the past several years as founder and CEO of Gradle, I have met with hundreds of build teams. In this post, I want to discuss root causes of this pain — both organizational issues and how technology gaps work against the build engineers and entire development team.

Common organizational anti-patterns

Build teams are understaffed and considered 2nd class engineering

In many organizations build engineers are in a vulnerable position, without the resources to do a good job.

Traditionally build engineering has been considered a less important engineering discipline compared to application development. This function is often a scapegoat for developer productivity problems, often for behavior that is not their responsibility.

Developers often change the behavior of the build and yet the build engineers are made responsible for the result. The build is slow — but did the build engineers introduce the language with a very slow compiler? Who wrote the slow tests and who publishes dependencies that always break downstream teams? Not the build team!

But in this case, the build engineers don’t have a strong standing organizationally and also don’t have sufficient data to explore and show what impacts the build and who is accountable. How the builds behave on local machines is a complete black box.