Build Configurations and Reproducible Builds

The puzzle

A device returns from the field with a fault address in its crash log. You check out the tagged release, rebuild it, look the address up in the new map file, and it points into the middle of an unrelated function. The source is identical; the binary is not, because the release was built on a colleague’s laptop, in a different directory, on a different day, with a slightly newer compiler. Meanwhile, on your own machine, you edit a header and the build finishes in half a second, having rebuilt nothing, and the board behaves as if the change never happened. Both are build-system problems, not code problems. What makes a build fast, correct and repeatable?

STEP 1

Debug and release

Most projects build the same source in at least two configurations. CMake names the common ones Debug, Release, RelWithDebInfo and MinSizeRel; whatever the tool, the differences come down to three settings:

Setting	Debug	Release
Optimisation	`-O0` or `-Og`: predictable stepping	`-O2` for speed or `-Os` for size
Debug information	`-g`	often still `-g`
Assertions	enabled	often disabled with `-DNDEBUG`

Two of these matter more than they look. Debug information does not change the code: -g adds DWARF sections to the ELF file, which are never loaded into flash, and GCC is designed so that -g leaves the generated instructions unchanged (it even has -fcompare-debug to test that). Keep -g in release builds; you will want it for the crash in the field. Assertions do change the code: with NDEBUG defined, C’s assert expands to nothing (C11 §7.2), so any side effect inside an assert(...) disappears from the release build. Never write assert(init_uart() == 0);.

Optimisation changes the code the most, and a bug that appears only at -O2 is almost always a real bug the optimiser exposed: undefined behaviour, or a missing volatile (unit 2). The fix is in the source, not in the flags.

STEP 2

Incremental builds

A full build of a few hundred files can take minutes, so build tools rebuild only what changed. The rule, from make onwards, is: rerun a step if any of its inputs is newer than its output. The work of an incremental build is then

T \approx \sum_{\text{objects to recompile}} t_{\text{compile}} + t_{\text{link}}

which is fast when a change touches one .c file and slow when it touches a header everything includes.

The rule is only as correct as the list of inputs. A .o depends not just on its .c file but on every header that file includes, directly or indirectly; nobody maintains that list by hand. GCC writes it: -MMD makes the compiler emit a .d file per object, listing the user headers it read, which the build tool includes on the next run. It also depends on the command line: change -O2 to -Os and every object is stale, but the files’ timestamps have not changed. Plain make cannot see that; Ninja records each output’s command line and rebuilds when it changes. And the link depends on the linker script, which a hand-written makefile often forgets to list.

INTERACTIVEWhat an incremental build rebuilds

What changed

Choose a compiler-flags change to compare flag tracking.

Recompile main.o, led.o, then relink.

A build tool reruns a step when one of that step’s inputs is newer than its output. The compiler can write the list of headers each object depends on (gcc -MMD); without those dependency files, a header change rebuilds nothing and leaves stale objects. A change of compiler flags is invisible to plain make unless the build records the flags; Ninja, for example, logs the command line of every output and rebuilds when it changes.

A missing dependency does not produce an error; it produces a stale object and an image that mixes old and new code, which is why “it works after make clean” is a symptom, not a fix.

STEP 3

Reproducible builds

A build is reproducible if the same source, built with the same tools, gives bit-for-bit identical output, whoever builds it and whenever. For firmware that matters in three ways: a crash address from a field unit can be looked up in a rebuild of its release; anyone can verify that a signed release binary really came from the published source (unit 15); and a change in the image hash is a reliable signal that something in the inputs changed.

Builds leak their environment into the output in a handful of standard ways, each with a standard fix:

Time. __DATE__ and __TIME__ embed the build moment. GCC replaces them with the time in SOURCE_DATE_EPOCH when that environment variable is set.
Paths. __FILE__ in assert messages records the source path in the image, and debug information records it in the ELF file. -ffile-prefix-map=/home/ana/fw=. rewrites them.
Order. A build that lists files with a raw directory scan may get them in whatever order the file system returns, and link order changes the layout. Sort the list.
Randomness. Some compiler-generated names use random numbers; -frandom-seed fixes them.
Archive metadata. ar records timestamps, owners and modes unless run in deterministic mode (the D modifier, which is the default when binutils was configured with --enable-deterministic-archives).
Tools. A different compiler version can generate different code. Pin the toolchain version, or build in a container image with a fixed toolchain.

↑ This step uses the figure at the top of the page.

STEP 4

Worked example: how much does a header change cost?

A project has 120 source files, each taking about 0.8 s to compile, and a 2 s link. config.h is included by 40 of them; uart.c includes only its own header.

Full build: $120 \times 0.8 + 2 = 98$ s.

Edit uart.c: one object, $0.8 + 2 = 2.8$ s.

Edit config.h with -MMD dependencies: 40 objects, $40 \times 0.8 + 2 = 34$ s: slower, but correct.

Edit config.h without dependency files: 0 objects recompiled and no relink, about 0 s, and 40 stale objects that still use the old configuration. The fast build is the wrong one.

Change -O2 to -Os in plain make: 0 s and every object stale; with Ninja or another flag-tracking build, all 120 are rebuilt, 98 s.

The lesson generalises: headers that everything includes should change rarely, and the build must know every input, including flags and the linker script, or its speed is bought with correctness.

MYTHS AND FACTS

Common misconceptions

-g makes the firmware bigger

It makes the ELF file bigger. Debug sections are not loaded into flash, and the code is intended to be identical with and without -g.

Release builds should have no debug information

Keep -g in the ELF you archive; strip it only from what you ship, if at all. You need it to decode field crashes.

If it builds without errors, the build is up to date

Missing dependencies produce stale objects silently.

make clean fixes build problems

It hides missing dependencies; the next incremental build will be wrong again.

Same source means same binary

Only if timestamps, paths, input order, randomness, archive metadata and tool versions are all controlled.

Assertions are free in release builds

They are removed, and so is any side effect written inside them.

Check yourself

Answer in your head, then open the card.

A release image is 180 bytes smaller than the debug image built from the same source at the same optimisation level. Both used -g. What is the most likely difference?

Assertions: the release defines NDEBUG, so every assert (and its message strings) is compiled out. -g affects only non-loaded debug sections.

In a project of 200 files at 0.5 s each with a 3 s link, a header included by 25 files changes. How long should the incremental build take, and what happens without -MMD?

With dependencies: $25 \times 0.5 + 3 = 15.5$ s. Without: no recompilation, and the 25 objects remain stale, built against the old header.

Two engineers build the same tag and get different images. The map files are identical, but a byte comparison differs in a string. Name the most likely cause and the fix.

An embedded __DATE__/__TIME__ string: those are always 11 and 8 characters, so they change bytes without changing any size or address. Set SOURCE_DATE_EPOCH for the build. (A different __FILE__ path would usually change string lengths and therefore the map too; -ffile-prefix-map fixes that.)

Why is assert(flash_erase(sector) == OK); dangerous?

With NDEBUG defined, the whole expression is removed, so the release build never erases the sector.

Sources (6)

CMake documentation, CMAKE_BUILD_TYPE and cmake-buildsystem(7) “Build Configurations” — typical build types Debug, Release, RelWithDebInfo and MinSizeRel; single-configuration generators choose the configuration at configure time
GCC manual, “Options Controlling the Preprocessor”: -MD, -MMD; “Environment Variables Affecting GCC”: SOURCE_DATE_EPOCH — -MMD writes a .d dependency file listing the user headers an object depends on; SOURCE_DATE_EPOCH “specifies a UNIX timestamp to be used in replacement of the current date and time in the __DATE__ and __TIME__ macros, so that the embedded timestamps become reproducible”
GCC manual, “Options Controlling the Kind of Output”: -ffile-prefix-map; “GCC Developer Options”: -frandom-seed, -fcompare-debug — -ffile-prefix-map=old=new records paths under old as if under new, which “can be used to make reproducible builds that are location independent”; -frandom-seed replaces random numbers used in symbol names “to produce reproducibly identical object files”; -fcompare-debug checks that -g does not change the generated code
Ninja manual, “Comparison to Make” and “The Ninja log” — “Outputs implicitly depend on the command line that was used to generate them, which means that changing e.g. compilation flags will cause the outputs to rebuild”; header dependencies are discovered at build time
GNU Binutils manual, “ar”: the D (deterministic) modifier — in deterministic mode ar uses zero for UIDs, GIDs and timestamps so that identical inputs create identical archives
ISO/IEC 9899:2011 (C11), committee draft N1570, §7.2 “Diagnostics <assert.h>” — if NDEBUG is defined where <assert.h> is included, assert expands to nothing