Build Speed

A build spent much of its time on work other than compiling. Scheduling steps, analyzing dependencies, checking files before the first step starts, and rerunning steps that had nothing to rebuild all added up. The changes below cut that overhead or ran it in parallel.

Dependency tracking outside the source tree. siso did not record dependencies for outputs stored outside it, so every build rescheduled all 166,000 steps. After the fix, a repeat build finishes with 0 steps.

Commits: build/siso a1fb7a7, build/soong 1e8564b, build/soong 3fced42, external/xos fd1c512, external/xos f428e88, build/siso 607980c, build/siso e53b2fc, build/siso 0275dfb, build/soong e268a42, build/blueprint b753706, build/blueprint 6aec609.

Scheduling latency. siso stat'ed every source file in series before the first step. Running stats in parallel dropped scheduling of 5.4 million nodes from 32 minutes to under two. A separate check of all stored output files also ran before any step could start. Workers now size by throughput and each file defers its check to first use, cutting time to the first step from 68 seconds to 32.

Commits: build/siso aa17e0f, build/siso f3573a1, build/siso e1f5bf6, build/siso 908251d.

Deps log loading. The deps log (645 MB) loaded in 35 seconds because each append copied the whole record list. Factor growth, direct decoding and parallel reads brought it under nine seconds.

Commits: build/siso 709ab3a, build/siso 6ef85ea.

Memory-aware step execution. siso ran one step per CPU. Builds were killed when large steps overlapped in memory. Steps now adjust from free memory and CPU, using per-step cgroups and persisted memory peaks.

Commits: build/siso f3573a1, build/siso 2f88ed2, build/siso 9ef8d29, build/siso bc1f989, build/siso 0d6987d, build/siso 2777eb8, build/siso ca3201c, build/siso bc83110, build/siso 3382840, build/siso aaba837, build/soong ea38e5e, build/soong f0d9121, external/xos 3cbf1d3.

Input prefetching. A prefetch reader now loads inputs for upcoming steps into the page cache, so they are in memory when the step runs.

Commits: build/siso f3573a1.

Soong glob overhead. Soong globs sent 75,000 stats for three tree-wide patterns, one directory at a time. Globs now read each directory once, cache results, and walk in parallel. A single glob dropped from 12 to 1.6 seconds. The check that follows reuses the finder's stats instead of repeating 156,000 of them, dropping from 7.8 to 1.9 seconds.

Commits: build/blueprint 3037722, build/blueprint e5b9a15, build/blueprint 14713a3, build/soong b87b913, build/soong fbdc285.

soong_build memory. soong_build reached 54 GB without a memory limit. A soft limit at half of physical memory keeps it at 31 GB.

Commits: build/soong 6557b4a, build/soong f0d9121, build/blueprint d2a8ef5.

Release-config flag reading. release-config read 1,250 flag files two at a time. 32 readers brought it from 21 seconds to 720 ms. A wildcard BoardConfig.mk lookup dropped lunch dumpvars from 49 to 14 seconds.

Commits: build/soong c959b27, build/make 5bb24fe.

File server serialization. File lookups in one parent directory were serialized by the file server. Strided dispatch cut the finder probe from 10 to 8 seconds and siso source stats from 3.4 to under one second.

Commits: build/soong afa57f0, build/siso 25c0b3d, build/siso 4b868e5.

Kati and Ninja generation. Kati used one CPU and parsed 676 MB of C/C++ export lists that nothing compiled. Parallel writes and numbered export lists shrank the Ninja file from 1,010 MB to 337 MB.

Commits: build/kati 9419e88, build/kati 824367c, build/soong 6ca7aa0, build/make 5527163, build/soong 8780007, build/kati 2c86ec2, external/xos c36e5b6.

Kati stamp checking. The Kati stamp check expanded 11,806 globs in series. Checking on every CPU dropped it from 24 to 6.5 seconds.

Commits: build/kati ff722c6, build/kati 6127cb9, build/make b90873a.

Kati symbol table. Kati's symbol table spent 13% of CPU interning and 7.6% destroying diagnostic messages. An open-addressing table cut about 12%.

Commits: build/kati 1265c82, build/kati c726778.

Kati allocator overhead. A CPU profile of the main Kati run showed about 12 % of its time in the glibc allocator. ckati now links mimalloc. Measured on two guacamole builds with Kati evaluation forced: the main Kati run dropped from 213.3 s to 198.5 s wall time (7 % less) and evaluation from 152.8 s to 138.2 s. Peak memory rose from 19.48 GB to 19.85 GB (1.9 %). Kati runs before the build steps start, so the increase does not add to the memory peak of the build.

Commits: external/xos 5d97bf8.

Kati find-output ordering. Kati regenerated the Ninja file when rewriting a file changed the order in which find listed names, even though the set of names was unchanged. The stamp check now compares find output as a set of words. Adding or removing a file still triggers regeneration.

Sandboxed step reruns. A change to sbox reran 110,000 steps because each overwrote its outputs with a new mtime. Sandboxed rules now pass --write-if-changed and are marked restat.

Commits: build/soong af86ccb.

Jarjar class renaming. Soong renamed classes inside jars with jarjar, a Java tool built on ASM that started a JVM for each step. In a guacamole build, 990 of 1,007 jarjar steps had rule files with only exact class renames, such as aconfig flag classes moved to a hidden package. A Rust tool, rename_classes, now handles those 990 steps; the 17 steps with wildcard rules still run jarjar. Measured on 59 Settings jar shards run sequentially with files in memory: jarjar took 67.7 CPU-seconds and 56.9 s wall time, rename_classes took 0.07 CPU-seconds and 1.2 s wall time. The framework-minus-apex jar also stopped renaming its header jar a second time.

Commits: build/soong 287c5a5, build/soong d4acfa8, build/soong 9cf4211.

Copy and link overhead. Copy steps ran a shell at 3.4 seconds each for a few kilobytes. siso now copies in-process. SDK links depended on the full NDK copy set; they now reference stubs where Soong built them.

Commits: build/soong 598e78d, build/soong be77da2, build/make 2738756, build/siso 33dd0cf.

Linker debug info. Linking soong_build included 11 MB of DWARF in a 62 MB binary. Go programs now link with -w.

Commits: build/blueprint 01670cd.

Kernel rebuild checks. The kernel step took over an hour with nothing to rebuild because git rehashed 78,000 files on the network mount. core.checkStat=minimal brought git status from two minutes to three seconds. Parallel reads cut the input check from nine to under four minutes. The git state step listed all 78,700 kernel-tree files as inputs and ran find over the tree after each run, about 38 seconds each time. It now depends only on the four git files a commit, checkout or tag writes, and kbuild's own depfile catches edited sources.

Commits: build/custom f9b80b1, build/custom 838a9f9, build/custom cb42a29, build/custom 94c29b5.

Signing and packaging. A no-change signing run still checked all 675,000 stored files. It now checks only the delta directory and derives the date from file mtimes. The target-files intermediate used full deflate. It now uses level 0, and the final package stays compressed.

Commits: build/siso b13e27c, build/siso 916a0f1, build/soong f99b51c, build/soong f0d9121, build/make a17e64d, external/xos 26a624e.