Hakutaku

Hakutaku is an authenticated, random-access resource package designed for offline games and optimized for visual-novel content. Its packing and runtime policies account for many small scripts and UI assets, seekable voice, music, and video streams, large backgrounds and character images, scene-local lookahead, and patch-friendly resource replacement. A release consists of one signed game.haku snapshot and a small set of immutable, content-addressed .taku segments. The wire format remains general enough for other offline games; the reference policy is tuned for visual novels.

The repository is intentionally split by responsibility:

• hakutaku-core: the only crate linked by a game runtime;
• hakutaku-pack: publisher identity, full/incremental bundle construction;
• hakutaku: command-line pack/list/extract/verify tool;
• hakutaku-gui: optional resource, release, and publisher-identity manager.

The v2 wire format has one cryptographic suite: AES-256-GCM for encrypted catalogs, pages, and blocks, Ed25519 for publisher identity, and BLAKE3 for content addresses and signed commitments. It does not contain an algorithm identifier or compatibility branch.

This directory is an independent Cargo workspace so it can be moved into its own repository without inheriting Kēne's dependency graph.

Quick start

# Create once and keep this publisher file private.
cargo run -p hakutaku-cli -- identity create publisher.hakutaku-key

# Export the non-signing material used by runtime/read-only tools.
cargo run -p hakutaku-cli -- identity export-runtime \
  publisher.hakutaku-key game.hakutaku-runtime-key

# Build or incrementally update a complete release directory.
cargo run -p hakutaku-cli -- pack \
  --input path/to/assets \
  --output path/to/release \
  --identity publisher.hakutaku-key \
  --deferred-prefix dlc

# Verify every immutable segment, or inspect the file table.
cargo run -p hakutaku-cli -- verify \
  --package path/to/release --keys game.hakutaku-runtime-key
cargo run -p hakutaku-cli -- list \
  --package path/to/release --keys game.hakutaku-runtime-key
cargo run -p hakutaku-cli -- segments \
  --package path/to/release --keys game.hakutaku-runtime-key

# Inspect source changes, build and verify releases, and manage identities.
cargo run -p hakutaku-gui

Set Assets, Output, and Key once. Use Scan → Build → Verify. The icons show the workflow; change bars show added, changed, removed, and unchanged files. File access/install details and longer explanations appear on hover. Import, replace, search, and reveal remain in the file list.

Reuse blocks is independent of Dev cache. The Strict badge means full source reads and staged byte comparison; this is the default. Opt into Dev cache under Options for local iteration only. Full builds disable it. Details contains build/package statistics; Key tools manages the shared publisher identity. Changing paths clears old results and selections.

Only game.haku, data/*.taku, and the runtime's equivalent embedded key material belong in the shipped game. The *.hakutaku-key publisher identity contains both the content root key and Ed25519 signing private key; never ship or commit it. *.hakutaku-runtime-key contains the content decryption secret and public verification key but cannot sign a release. It is suitable for read-only tooling, not for public distribution as a loose file. The packer and GUI reject either key format as an asset by file magic, regardless of filename.

Incremental packing reuses chunks from the active release. Identical chunks in the same placement class are also deduplicated during the first release, so copied assets with the same access policy are encrypted and stored only once. Scripts, UI files, and small configuration up to 32 KiB are marked Hot. Short voice/SFX audio up to 1 MiB is Transient, while BGM/music, longer audio, and video are Streaming; media classification takes precedence over size. Other assets enter the bounded second-hit Normal cache. The reference packer keeps each availability/access class in its own bounded segment stream: 64 MiB for Hot, 256 MiB for Normal, 128 MiB for Transient, and up to the configured limit (512 MiB by default) for Streaming. These are upper bounds rather than padded target sizes. Deferred content is also isolated; required and deferred blocks never share a segment. See docs/PERFORMANCE.md for the allocation policy and the local Unity- and Unreal-shaped I/O regression baseline.

Runtime and update integration

The runtime opens and authenticates game.haku first. Package::list_segments then exposes the signed ID, byte length, and Required/Deferred availability of every immutable segment. Installers can download only missing content while the game supplies storage through the dependency-free SegmentSource and PositionedFile traits. A missing on-demand segment returns Error::SegmentUnavailable(id); networking and retry policy stay outside the format reader.

Sequential cursors reuse their ciphertext and decompression buffers and keep the current and previous Streaming blocks for short decoder seeks. Engines can schedule Asset::prefetch_range on their existing task pool; its dedicated cache is bounded by ResourceBudget::prefetch_cache_bytes and creates no Hakutaku-owned threads.

Asset::read_at remains the convenient one-shot random-access API. Code that performs repeated reads should retain an AssetCursor for sequential/seekable decoders or an AssetReadSession for offset-based engine APIs; both preserve the active segment handle and decode buffers across calls.

The signed release_sequence prevents undetectable edits but cannot by itself remember a previously accepted version. A launcher should persist its highest accepted sequence and pass OpenPolicy::requiring(sequence) to Package::open or Package::open_directory. Hakutaku then rejects a correctly signed older snapshot without owning platform state. The first trusted install must deliberately pass OpenPolicy::TrustFirstRelease; there is no implicit rollback-permitting open path.

This boundary works with desktop files, Android/iOS asset storage, memory maps, or a platform download manager without adding an HTTP client or async runtime to hakutaku-core. Snapshot and segment reads are positional and safe to share between concurrent cursors.

Publisher writes use synchronized temporary files, verified staging, atomic snapshot replacement, Unix directory synchronization, and post-commit garbage collection. Interrupted .part files are removed only after the exclusive build lock is acquired. If a process is killed while packing, confirm no packer is still running before removing a reported stale .hakutaku.lock.

For local 20–40 GiB VN iteration, incremental builds may opt into --dev-cache. The release-local cache records source size, modification time, available file identity, and authenticated chunk metadata so unchanged voices do not need to be reread merely to rediscover their chunks. It is bound to the project and active release sequence; corrupt or stale caches are ignored. --full cannot be combined with it, and normal/final builds retain complete source rereads and byte-for-byte staged verification. After a successful GUI incremental build, the post-build plan validates the complete path/size/access inventory without immediately rereading every source body.

Full builds preserve and advance the authenticated release sequence even when rebuilding unchanged source or recovering missing old segments. An existing snapshot must authenticate; --full does not reset the launcher's rollback floor.

Hakutaku 0.2.0 reads and writes v2 only. Existing v1 packages require a full source repack into a fresh directory using the same identity; pass --release-sequence-floor N above the last published sequence. Deploy the new runtime with its matching package. There is no legacy format fallback.

See docs/FORMAT.md for the implemented byte layout and performance evidence for the measured design choices. The compact catalog uses 8-byte path slots and 32-byte positional chunk IDs; headers are 256/128 bytes. Audio uses 64 KiB blocks, video retains 256 KiB, and image CDC remains unchanged. Bounded publisher consolidation prevents small patch segments accumulating indefinitely.

Validation

Hakutaku requires Rust 1.99.0. The same version is pinned for local development and CI in rust-toolchain.toml. Local acceptance uses:

cargo fmt --all --check
cargo check --workspace --locked --all-targets --all-features
cargo clippy --workspace --locked --all-targets --all-features -- -D warnings
cargo test --workspace --locked --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --locked -p hakutaku-core -p hakutaku-pack --no-deps
cargo llvm-cov --locked -p hakutaku-core -p hakutaku-pack --fail-under-lines 97.5
cargo llvm-cov --locked -p hakutaku-cli --fail-under-lines 45
cargo bench --locked --workspace

The larger VN benchmark is opt-in because it creates 10k/50k/100k-entry fixtures. A 10k smoke run and the complete matrix are:

HAKUTAKU_VN_BENCH=1 HAKUTAKU_VN_BENCH_COUNTS=10000 \
  cargo bench -p hakutaku-pack --bench vn_runtime
HAKUTAKU_VN_BENCH=1 cargo bench -p hakutaku-pack --bench vn_runtime

Every successful push to main creates the next patch tag and publishes the CLI and GUI builds, together with SHA256SUMS, on GitHub Releases. The first automated tag follows workspace.package.version; later releases increment its patch component. Raise the workspace version explicitly to start a new minor or major line.

Publisher-key storage, CI trust, and platform signing boundaries are documented in docs/SECURITY.md.

Both library crates deny missing public API documentation. CI therefore keeps public rustdoc coverage at 100%. The canonical wire-format parser is held at 100% line coverage; the combined runtime and publisher baseline is 97.5% or higher, and CLI argument handling has a 45% regression floor. The remaining lines are operating-system and cryptographic-provider failure returns that cannot be triggered deterministically without replacing the production backends.

v2 acceptance

The complete 0.2.0 workspace passes Rust 1.99.0 fmt, all-target/all-feature check and Clippy, 83 tests, and public documentation with warnings denied. Fresh coverage targets report Core/Pack 97.63%, wire parser 100%, CLI 53.42%; the existing thresholds remain unchanged. The initial mixed-version coverage target is excluded from acceptance. Actionlint passes.

The 500k-resource, 1,024-update and eight-reader stress cases pass. Real v1/v2 binaries reject each other's wire format; identity reuse and release floors are checked. See integrated results for paired measurements, tradeoffs and raw evidence. Kēne's local integration passes its workspace gates, all 196 Loader tests and 29 publisher tests, both native/FFmpeg media acceptance commands, and the existing dependency policy. The local macOS debug GUI passes direct native checks of the single-workspace flow: shared paths, collapsed identity/advanced/statistics sections, same-size source change detection with preserved mtime, strict incremental build, release verification, and independent cache/reuse controls. The demo build advances release 42 to 43 with 199 reused blocks and one new block; verification succeeds. Full mode disables the development cache and keeps strict validation. Its ad-hoc signature verifies. This covers the local debug app; release artifacts and Windows/Linux GUI acceptance remain separate. Publishing a portable Git dependency and remote release/CI remain separate.

Quality-review baseline (before v2)

The workspace review covers every crate, with runtime and publisher paths as the priority. Scope coverage is not a claim of exhaustive proof.

Module	Reviewed paths and resulting changes
Core	Authenticated format/crypto, positional I/O, pages, segments, budgets, prefetch and cursors; remove quadratic cache insertion and path-validation allocation; share cursor/session block loading and preserve cursor position on read errors
Pack	Source inventory/classification, identities, planning, chunking, dedup, cache parsing, staging, recovery and commit; bound reads and allocations before parsing, handle short reads correctly, use standard file replacement, preserve monotonic release ordering during full reconstruction
CLI	Argument/error paths, identity/runtime-key separation, package open, verification and confined extraction; retain the existing public command contract
GUI	Worker lifecycle, progress, resource filtering/selection, import/replace, release and identity flows; use egui visible-row rendering, cache filtering, throttle progress, handle worker creation failures, and scroll long release/identity pages
Dependencies/CI	Rust/MSRV, direct dependencies, feature graph, coverage, native/mobile build jobs and release gates; pin Rust 1.99.0 and lock resolution, add native platform tests

Direct dependencies now use eframe 0.36.2, rfd 0.17.2, zstd 0.14.2 and FastCDC 5.0.0; the lockfile also advances BLAKE3 to 1.8.7 and zeroize to 1.9.1. The redundant direct egui dependency is removed in favor of eframe's re-export. No new direct crate or runtime framework is introduced. FastCDC keeps the same 32/128/512 KiB parameters; its updated API uses usize. The native zstd version remains 1.5.7. See the FastCDC changelog and eframe release.

On Apple Silicon macOS with Rust 1.99.0, fmt, all-target/all-feature check and Clippy, documentation, and 76 workspace tests pass, including the later full-rebuild sequence regression. Before that follow-up, measured line coverage was 97.59% for Core/Pack, 53.29% for CLI, and 100% for the wire-format parser; coverage was not remeasured after the sequence fix. Existing coverage thresholds are unchanged. Actionlint passes with its optional shellcheck/pyflakes integrations disabled.

The dependency review, before the sequence follow-up, verified that old and new dependency builds read each other's v1 packages byte-for-byte; new strict and development-cache incremental builds preserve an unchanged old snapshot and its segments. The fixture covers compressed text, raw video, empty files and deterministic random data with an odd tail. New full and incremental packages also pass the old dependency build's verifier/extractor. These two CLI builds both contain this review's reader fixes; the comparison isolates dependency compatibility. A separate archive of Kēne 6635d1f, patched only to consume this local Core/Pack, passes all 196 Loader tests. Kēne's live checkout and dependency pins were unchanged during that baseline review.

Native macOS acceptance directly exercised scanning 305 temporary assets, scrolling and filtering the resource list, incremental build, unchanged rebuild, verification and scrolling the complete release report. The local debug app's ad-hoc signature also verifies. This is not acceptance of a release artifact: remote CI, Windows/Linux GUI operation and mobile targets still need their own builds/runs. Dependency-policy boundaries are recorded in SECURITY.md, and measured results in PERFORMANCE.md. Raw local evidence is under the ignored target/quality-review/ directory.

License

Licensed under either the Apache License, Version 2.0 or the MIT license, at your option.