Skip to content

build: make the workspace crates packageable for crates.io - #927

Open
LeeroyHannigan wants to merge 1 commit into
mainfrom
build/crates-io-packaging
Open

LeeroyHannigan wants to merge 1 commit into
mainfrom
build/crates-io-packaging

Conversation

@LeeroyHannigan

@LeeroyHannigan LeeroyHannigan commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

What

This change makes all 14 workspace crates pass cargo package, the step before a publish to crates.io. It publishes nothing.

  • Each internal entry of [workspace.dependencies] requires the workspace version exactly: version = "=0.1.14".
  • The 14 internal dev-dependencies that used { workspace = true } now use a path and no version. Cargo removes them from the published manifests.
  • The workspace uses resolver = "3".
  • Each crate has a description, a repository and a homepage. The extenddb crate also gets a README for its crates.io page, keywords and categories.
  • Each crate directory links LICENSE and NOTICE to the root files. I regenerated the four license notices, and they now list the workspace crates under the text of our LICENSE file.
  • Cargo.lock moves spin 0.9.8 to 0.9.9 and chacha20 0.10.1 to 0.10.2. crates.io lists both old versions as yanked.
  • The backup-store README and config.rs drop one S3 server from their examples.
  • cdrs-tokio stays on 9.0.x and cassandra-protocol on 4.0.x. cassandra-protocol 4.1.0 moved its public BigInt to num-bigint 0.5, and the Cassandra backend then fails to compile. Without the bounds, a fresh resolution, such as cargo install without --locked, picks 4.1.0.
  • crates/bin/build.rs reads the packaged commit from .cargo_vcs_info.json. A build from crates.io then reports its commit instead of unknown.
  • The new devtools/check-crate-packages checks the pins, the dev-dependencies, the license files and the metadata. It also checks the field rules and size limits that crates.io applies at upload. It reads the workspace through cargo metadata. Its tests are in devtools/check_crate_packages_test.py.
  • The Test workflow gets a package job, and the required test gate needs it.
  • ADR-0029 records the decision. CONTRIBUTING.md describes the dependency rules, and packaging/README.md adds the pins to the release bump.

Why

Because girls just wanna have fun!!

On main, cargo package --workspace stops at the first crate with an internal dependency:

all dependencies must have a version requirement specified when packaging. dependency extenddb-cache does not specify a version

With a version on each internal dependency, it stops at the dev-dependency cycle between extenddb-engine and extenddb-storage-sqlite:

no matching package named extenddb-engine found

crates.io accepts the extenddb binary only after the 13 crates it depends on, so all 14 must package. The crates change their APIs between patch releases. An exact pin stops a fresh install from mixing crates of two releases.

Without --locked, cargo install resolves every dependency again. On Rust 1.88, our minimum version, that chose uuid 1.29.2, which needs Rust 1.89, and the build failed. With resolver = "3", cargo chose uuid 1.26.1 and the build passed.

Testing done

  • cargo package --workspace --no-verify --locked wrote the 14 packages in 5 s, and devtools/check-crate-packages printed ok.
  • From a cold cache, cargo package --workspace --locked built all 14 packages from their tarballs in 305 s with 12 jobs. It passed again on the final tree. The packaged extenddb build recorded the full commit SHA.
  • cargo metadata --locked gave 530 nodes before and after. Only the spin and chacha20 nodes, the edges of their 3 dependents, and the new edge from extenddb-storage-cassandra to cassandra-protocol differ.
  • I copied only Cargo.toml, Cargo.lock and crates/, as the build stage of Dockerfile.dev does. In that copy the LICENSE and NOTICE links point at nothing. The sqlite,dev-mode build passed on Rust 1.88, and cargo gave no warning.
  • devtools/generate-software-license-notices --check for each backend and devtools/generate-dev-license-notices --check pass.
  • python3 -m unittest discover -s devtools -p '*_test.py' runs 35 tests, and all pass. Each of 48 one-line changes to the check script makes a test fail.
  • With the old requirements, cargo update -p cdrs-tokio moved cassandra-protocol to 4.1.0, and cargo check -p extenddb-storage-cassandra failed with E0308 on BigInt. With the bounds, a fresh lock on Rust 1.88 chose 4.0.1, and the check passed. The bounds change one line of Cargo.lock, the new dependency edge.
  • In CI, the run-tests jobs run the workspace tests on Rust 1.88.0 and on stable, and both pass. The Clippy workflow runs clippy with -D warnings on every crate and target of the workspace, and it passes.
  • With -W clippy::pedantic, clippy reports nothing on the lines that this change touches. It still reports warnings in code outside this change. cargo fmt --all -- --check passes.
  • A probe of the .cargo_vcs_info.json reader ran on Rust 1.88. It reads clean and dirty files in pretty and compact form, and it returns None for malformed input.
  • No CI job in this change compiles the packages from their tarballs. That build takes minutes. The crates.io release workflow, which is follow-up work, has to run it before each upload.

Checklist

  • I have read CONTRIBUTING.md
  • All tests pass (cargo test --workspace)
  • Code is formatted (cargo fmt --check)
  • Clippy is clean (cargo clippy -- -W clippy::pedantic)
  • I have added or updated tests for new functionality
  • I have updated documentation if behavior changed
  • Breaking changes are noted below (if any)
  • If this changes the wire protocol, Storage trait, auth model, on-disk
    format, or public CLI surface, an RFC has been accepted or is linked
    below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: ADR-0029, docs/adr/0029-crates-io-packaging.md in this PR


By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

All 14 workspace crates now pass `cargo package`, the step before a
publish to crates.io. This change publishes nothing.

- Each internal entry of `[workspace.dependencies]` requires the
  workspace version exactly, as in `version = "=0.1.14"`. The crates
  change their APIs between patch releases. An exact pin stops a fresh
  install from mixing crates of two releases.
- Internal dev-dependencies use a path and no version, so cargo removes
  them from the published manifests. Packaging then no longer stops at
  the dev-dependency cycle between extenddb-engine and
  extenddb-storage-sqlite.
- The workspace uses `resolver = "3"`. On Rust 1.88, our minimum
  version, resolver 2 picked uuid 1.29.2 for an install without
  `--locked`. That version needs Rust 1.89. Resolver 3 picks uuid
  1.26.1, and the build passes.
- Each crate has a description, a repository and a homepage. The
  extenddb crate also has a README for its crates.io page, keywords and
  categories.
- Each crate directory links LICENSE and NOTICE to the root files. This
  change regenerates the four license notices, and they list the
  workspace crates.
- Cargo.lock moves spin to 0.9.9 and chacha20 to 0.10.2. crates.io lists
  the old versions as yanked.
- cdrs-tokio stays on 9.0.x and cassandra-protocol on 4.0.x.
  cassandra-protocol 4.1.0 uses num-bigint 0.5 for its public BigInt,
  and the Cassandra backend then fails to compile. To hold the bound,
  extenddb-storage-cassandra now depends on cassandra-protocol directly.
- crates/bin/build.rs reads the packaged commit from
  .cargo_vcs_info.json. A build from crates.io then reports its commit
  instead of "unknown".
- The new devtools/check-crate-packages checks the pins, the
  dev-dependencies, the license files and the metadata. It also checks
  the field rules and size limits that crates.io applies at upload. A
  new `package` job in the Test workflow runs it, and the required
  `test` gate needs that job.
- The backup-store README and config.rs drop one S3 server from their
  examples.
- ADR-0029 records the decision. CONTRIBUTING.md describes the
  dependency rules, and packaging/README.md adds the pins to the release
  bump.
@LeeroyHannigan
LeeroyHannigan force-pushed the build/crates-io-packaging branch from 586e3c7 to 0ce13cf Compare November 10, 2026 13:55
@LeeroyHannigan LeeroyHannigan added v1 Needed for the 1.0 release release-blocker Must ship for its milestone's release labels Nov 10, 2026
@LeeroyHannigan LeeroyHannigan added this to the 1.0 milestone Nov 10, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-blocker Must ship for its milestone's release v1 Needed for the 1.0 release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant