GDC Presentation Reviews

GDC Europe 2013: "Toolchain Independent Distributed Compilation" by Dietmar Hauser (@rattenhirn?) of Sproing Interactive Media GmbH https://gdcvault.com/play/1019295/Toolchain-Independent-Distributed

This was okay. They created a distributed build system that works for any console SDK. It was built for a small office of 34 people, where each person runs a screensaver on Windows that acts as a compile server.

1/3

Toolchain Independent Distributed Compilation

In 2012, a crack programming team set out on a mission to speed up compile times for C++ projects, utilizing the untapped power of idle workstations. With a lot of experience in multi-platform development, the cumulative pain of years of watching...

Sep 2, 2025 at 4:36amWeb
Show threadGDC Presentation ReviewsSep 2, 2025

They made a few interesting decisions:

1. Rather than installing every SDK on everyone's computer, they just send the compiler along with the sources to compile

2. They use something that sounded like a regex search to find the #includes, and package all these headers up with the file to be compiled. This ended up not working very well because they're not actually running a full preprocessor, so they can't handle things like #ifdefs. (For comparison, distcc runs the preprocessor locally)

2/3

Show threadGDC Presentation ReviewsSep 2, 2025

3. They only targeted code compilation, not data compilation, which I think is mostly of an indication of the size of their studio and the kind of games they make.

Review: 2/10 I wouldn't recommend this because it was made for a very specific use case: code compilation during screensavers of the desktop machines of a small office. I don't think it's general enough that it would make sense to use it as a general model.

Show threadDietmar HauserSep 2, 2025
@GDCPresoReviews Data compilation was mentioned on the "future" slide, but we actually found a better solution: Storing the converted assets in the SCM + building a hash based system to only rebuild assets that had changed.
This way, most people didn't even have to sync the source assets, greatly saving them time and disk space.
Show threadDietmar HauserSep 2, 2025
@GDCPresoReviews The feedback collected at the conference was this:
28% Excellent, 67% Good, 6% Poor, 0% Terrible
It was probably a lot more relevant in 2013, as there are now out of the box solutions for this. I think anyone looking to build a similar solution themselves might still get some use out of it.
Show threadDietmar HauserSep 2, 2025
@GDCPresoReviews Not exactly regex, but a quite simple parser that cached it's result and only had to reparse changed files. This actually came from an earlier usage of the tool to build with other compilers in Visual Studio in the pre MSBuild days, where we simply replaced cl.exe, link.exe, and lib.exe with our own that translated the MSVC command line to and output from the actually used compiler (CodeWarrior and SNC). Since this broke VS's dependency check, we had to roll our own.
Show threadGDC Presentation ReviewsSep 2, 2025

@rattenhirn replacing cl.exe is certainly an interesting way to do it! Does Visual Studio not have a way of saying “use a different compiler instead of the default one”? (It must - the console SDKs can’t take the same approach of replacing the built-in compilers)

I’m not that familiar with Visual Studio’s build infrastructure.

Show threadDietmar HauserSep 3, 2025
@GDCPresoReviews Before VS used the MSBuild build system to build, the compiler was fixed to the built in one. I don't remember exactly when this change occurred, but it must have been around 2010.
Show threadDietmar HauserSep 2, 2025
@GDCPresoReviews It'sa indeed me!
Here are the slides: https://www.slideshare.net/slideshow/toolchain-independent-distributed-compilation/66251001
The presentation was meant for people who want to build this themselves, or maybe to make it a "joint venture".
In the meantime Fastbuild and other solutions have arrived that can do everything this could do.
Toolchain Independent Distributed Compilation

This document discusses toolchain independent distributed compilation. It explores previous solutions like IncrediBuild and distcc, and proposes a plan to analyze source files to find dependencies and distribute compilation across servers while caching inputs and outputs. Key challenges include precompiled headers, preprocessor directives, and debug information storing absolute paths. The author details solutions using sandboxing software. Benchmarking shows speedups of up to 17x for large codebases. Future work could involve distributed linking and unifying with LLVM/Clang. - Download as a PPT, PDF or view online for free

SlideShare
Show threadGDC Presentation ReviewsSep 2, 2025

@rattenhirn right, these other solutions are the main reason I didn’t rate this presentation higher. This is a problem the entire industry has, and making a new solution in 2025 wouldn’t be the right engineering decision when confronted with this problem.

(I rate these presentations in the context of watching them now, in 2025, not in the context of when they were presented.)

Show threadDietmar HauserSep 3, 2025
@GDCPresoReviews I understand that, that's why I didn't complain about the rating.
Back then there was Incredibuild, which only supported MSVC and was very expensive, and distcc, which only supported GCC and, I think, was not available on Windows.
IIRC we looked into improving distcc, but it was too tied up in its eco system (gcc, linux).
We also already had a solution that could run all the compilers we needed locally to make them use all CPU cores (also not a common feature in the naughties).
Show threadDietmar HauserSep 2, 2025
@GDCPresoReviews Oh, one more tidbit, the studio grew to over a 100 people in the next 3 years, using the same system. With more people it only got better, obviously.
Show threadGDC Presentation ReviewsSep 2, 2025

@rattenhirn oh nice!!! That’s reassuring to hear 🙂

My favorite systems are the ones that get *faster* the larger they get, rather than slower 😁

Show threadDietmar HauserSep 3, 2025
@GDCPresoReviews The magic of distributed systems where the few (programmers) can use the resources of the many (everyone). Usually it is the other way round.