bootstrap: doc: document std crates one by one by Fabian-Gruenbichler · Pull Request #157143 · rust-lang/rust

Fabian-Gruenbichler · 2026-05-30T07:01:43Z

Since switching over from individual, per crate cargo rustdoc invocations to a single cargo doc invocation for all crates, the doc output for Std became racy. Dependencies between library/ workspace crates were handled in a flaky fashion, sometimes being emitted as relative references, sometimes as references to the html root.

This is probably only a workaround, and it should actually be fixed in cargo or rustdoc, but I haven't managed to reproduce this issue outside of the Std doc build via bootstrap.

Closes: #156567

Fixes: 58e18dd

rustbot · 2026-05-30T07:01:48Z

r? @Mark-Simulacrum

rustbot has assigned @Mark-Simulacrum.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

Owners of files modified in this PR: bootstrap
bootstrap expanded to 6 candidates
Random selection from Mark-Simulacrum, clubby789

Fabian-Gruenbichler · 2026-06-01T06:47:19Z

force-pushed with clippy fix

Mark-Simulacrum

I'm not sure how I feel about the tradeoff here...

cc @rust-lang/bootstrap in case others have ideas

View changes since this review

Mark-Simulacrum · 2026-06-07T15:55:48Z

+        cargo
+            .arg("--no-deps")
+            .arg("--target-dir")
+            .arg(&*target_dir.to_string_lossy())


It's preexisting, but I think we should be able to just pass target_dir here? Cargo::arg should take Path/OsStr.

Mark-Simulacrum · 2026-06-07T16:05:49Z

-        Kind::Doc,
-    );
+    let mut crates = requested_crates.to_vec();
+    let std_lib_crate_order = STD_PUBLIC_CRATES


I think this is a regression in terms of needing STD_PUBLIC_CRATES again... right?

cc @jyn514 since you wrote #111955

yes, but it seems it is still required to document things in the right order. I am happy to fix this another way, but I haven't yet found a simple reproducer that allows me to figure out what takes a wrong turn where. if somebody has a clue, I am happy to take a closer look there :)

jieyouxu · 2026-06-08T00:05:43Z

I'm a bit hesistant on papering over this in bootstrap 🤔 Is there any particular combo that makes it more likely to trigger this?

EDIT: ah, the discussion on the issue also points to rust-lang/cargo#8487.

Also not super sure on the tradeoffs. Do we know if using this approach can bring back other problems?

Fabian-Gruenbichler · 2026-06-08T07:15:26Z

my reproducer was simply running x.py doc --stage 2 a few times and comparing the generated html files. it sometimes takes up to 10 tries on my machine until it does it wrong, but usually the first "rebuild" already has differences.

Kobzol · 2026-06-08T15:25:19Z

Is there a way to override the order without documenting the crates one by one? Sure would be nice to fix this in Cargo than to add yet another hack to bootstrap :)

Fabian-Gruenbichler · 2026-06-09T07:16:57Z

AFAIK/AFAICT cargo already tries to pick the right order, but somehow still sometimes takes a wrong turn. I tried manually reproducing it with a workspace and crates referencing eachother, but failed to do so. I think the way bootstrap calls cargo doc here is a bit special (always passing --manifest-path pointing at sysroot, which in turn references other crates, but also specifying -p .. for all the documented crates), but even with trying to mimic those I failed to reproduce it with a minimized set of crates.

Kobzol · 2026-06-15T13:05:37Z

Since you have been digging into this, have you tried diffing whether the current invocation vs documenting crates one by one actually produces the same output? In other words, is documenting everything together only sometimes non-deterministic, or does it also produce something different than setting a specific doc build order?

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>

rustbot · 2026-06-16T08:46:35Z

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

Fabian-Gruenbichler · 2026-06-16T08:47:18Z

the html output looks identical, modulo references (both to docs.rs and doc.rust-lang.org instead of relative local ones) and settings.html/help.html mentioning a different "current crate" (std vs std_detect). the generated javascript search index files also have a lot of diff, but I guess that is to be expected?

wdiff for the html files attached, this is with this PR (just rebased/force-pushed) compared to b5d46ecb51c3e4134b82570cfe718f093daa6390

wdiffs.txt

Kobzol

The current crate things looks a bit.. concerning, but the rest looks fine. Left some review comments, but I still feel iffy about this.

View changes since this review

Kobzol · 2026-06-19T06:52:39Z

+        .enumerate()
+        .map(|(i, v)| (v.to_string(), i))
+        .collect::<HashMap<_, _>>();
+    crates.sort_by_key(|c| std_lib_crate_order.get(c).unwrap_or(&usize::MAX));


This is potentially non-deterministic, we should sort by a tuple (crate order [number], crate name).

Kobzol · 2026-06-19T06:56:03Z

-    if builder.config.library_docs_private_items {
-        cargo.rustdocflag("--document-private-items").rustdocflag("--document-hidden-items");
-    }
+        compile::std_cargo(builder, target, &mut cargo, std::slice::from_ref(&krate));


Suggested change

compile::std_cargo(builder, target, &mut cargo, std::slice::from_ref(&krate));

compile::std_cargo(builder, target, &mut cargo, &[krate]);

Mark-Simulacrum · 2026-06-21T18:32:43Z

Would passing -j1 to the build produce a deterministic order? I would expect that Cargo should start builds deterministically in that case? If not, that seems like a bug to be fixed in Cargo.

I think if it does fix this I'd be OK taking a patch that sets it; maybe in the future we could gate that on something like build.reproducible to avoid slowing down builds where the user doesn't really care.

Fabian-Gruenbichler · 2026-06-23T12:03:44Z

Would passing -j1 to the build produce a deterministic order? I would expect that Cargo should start builds deterministically in that case? If not, that seems like a bug to be fixed in Cargo.

I think if it does fix this I'd be OK taking a patch that sets it; maybe in the future we could gate that on something like build.reproducible to avoid slowing down builds where the user doesn't really care.

that's the first thing I tried on the Debian side - we call bootstrap.py -j1 doc ..., and AFAICT this ends up correctly passing -j 1 to the cargo doc invocation:

RUST_BACKTRACE=1 python3 src/bootstrap/bootstrap.py -j1 doc --stage 2 --config debian/config.toml --on-fail env
[..]
running: /build/reproducible-path/rustc-1.95.0+dfsg1/build/bootstrap/debug/bootstrap -j1 doc --stage 2 --config debian/config.toml --on-fail env
[..]
running: cd "/build/reproducible-path/rustc-1.95.0+dfsg1" && env -u MAKEFLAGS [lots of env variables] RUSTDOCFLAGS="-Zannotate-moves --check-cfg=cfg(feature,values(any())) -Zunstable-options -Wrustdoc::invalid_codeblock_attributes --crate-version 1.95.0\t(59807616e\t2026-04-14)\t(built\tfrom\ta\tsource\ttarball) -Zcrate-attr=doc(html_root_url=\"https://doc.rust-lang.org/1.95.0/\") -Zcrate-attr=warn(rust_2018_idioms) --extern-html-root-url std_detect=https://docs.rs/std_detect/latest/ --extern-html-root-takes-precedence --resource-suffix 1.95.0 --markdown-css rust.css --markdown-no-toc --index-page /build/reproducible-path/rustc-1.95.0+dfsg1/src/doc/index.md -Zunstable-options" RUSTDOC_LIBDIR="/build/reproducible-path/rustc-1.95.0+dfsg1/build/x86_64-unknown-linux-gnu/stage1/lib" RUSTDOC_REAL="/build/reproducible-path/rustc-1.95.0+dfsg1/build/x86_64-unknown-linux-gnu/stage1/bin/rustdoc" RUSTFLAGS="-Zannotate-moves --check-cfg=cfg(feature,values(any())) -Zunstable-options -Zmacro-backtrace -Csplit-debuginfo=off -Cprefer-dynamic -Zinline-mir -Zinline-mir-preserve-debug -Zmir_strip_debuginfo=locals-in-tiny-functions -Zcrate-attr=doc(html_root_url=\"https://doc.rust-lang.org/1.95.0/\")" RUST_TEST_THREADS="1" __CARGO_DEFAULT_LIB_METADATA="stablestd1.95.0" "/usr/bin/cargo" "doc" "-Zwarnings" "--target" "x86_64-unknown-linux-gnu" "-Zbinary-dep-depinfo" "-j" "1" "-Zroot-dir=/build/reproducible-path/rustc-1.95.0+dfsg1" "-Zno-embed-metadata" "--verbose" "--profile=dist" "-p" "alloc" "-p" "compiler_builtins" "-p" "core" "-p" "panic_abort" "-p" "panic_unwind" "-p" "proc_macro" "-p" "profiler_builtins" "-p" "rustc-std-workspace-core" "-p" "std" "-p" "std_detect" "-p" "sysroot" "-p" "test" "-p" "unwind" "--features" "backtrace panic-unwind profiler compiler-builtins-c" "--manifest-path" "/build/reproducible-path/rustc-1.95.0+dfsg1/library/sysroot/Cargo.toml" "--no-deps" "--target-dir" "/build/reproducible-path/rustc-1.95.0+dfsg1/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/doc" "-Zskip-rustdoc-fingerprint" "-Zrustdoc-map" (failure_mode=Exit) (created at src/bootstrap/src/core/build_steps/doc.rs:810:21, executed at src/bootstrap/src/core/build_steps/doc.rs:843:22)

and things are still unreproducible: https://reproduce.debian.net/all/forky.html#rust-doc

but I just saw that there are two more cargo doc invocations later on in the build log which are not covered (and get the default parallelism) maybe as part of test or install - I will double check why and whether a plain -j1 is indeed reproducible!

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels May 30, 2026

rustbot assigned Mark-Simulacrum May 30, 2026

This comment has been minimized.

Sign in to view

Fabian-Gruenbichler force-pushed the mr/std-doc-reproducibility branch from fe86a47 to c67b1f5 Compare June 1, 2026 06:46

Mark-Simulacrum reviewed Jun 7, 2026

View reviewed changes

bootstrap: doc: document std crates one by one

94bc438

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>

Fabian-Gruenbichler force-pushed the mr/std-doc-reproducibility branch from c67b1f5 to 94bc438 Compare June 16, 2026 08:46

Kobzol reviewed Jun 19, 2026

View reviewed changes

	compile::std_cargo(builder, target, &mut cargo, std::slice::from_ref(&krate));
	compile::std_cargo(builder, target, &mut cargo, &[krate]);

Uh oh!

Conversation

Fabian-Gruenbichler commented May 30, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented May 30, 2026

Uh oh!

This comment has been minimized.

Fabian-Gruenbichler commented Jun 1, 2026

Uh oh!

Mark-Simulacrum left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mark-Simulacrum Jun 7, 2026

Choose a reason for hiding this comment

Uh oh!

Mark-Simulacrum Jun 7, 2026

Choose a reason for hiding this comment

Uh oh!

Fabian-Gruenbichler Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

jieyouxu commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Fabian-Gruenbichler commented Jun 8, 2026

Uh oh!

Kobzol commented Jun 8, 2026

Uh oh!

Fabian-Gruenbichler commented Jun 9, 2026

Uh oh!

Kobzol commented Jun 15, 2026

Uh oh!

rustbot commented Jun 16, 2026

Uh oh!

Fabian-Gruenbichler commented Jun 16, 2026

Uh oh!

Kobzol left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Kobzol Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

Kobzol Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

Mark-Simulacrum commented Jun 21, 2026

Uh oh!

Fabian-Gruenbichler commented Jun 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Fabian-Gruenbichler commented May 30, 2026 •

edited by rustbot

Loading

Mark-Simulacrum left a comment •

edited by rustbot

Loading

jieyouxu commented Jun 8, 2026 •

edited

Loading

Kobzol left a comment •

edited by rustbot

Loading