Publish experimental Maven path indexes during version finalization

[gerrod3]

Users upload Maven content as orphans and add it through repository modify. This integration publishes its binary index to S3 during finalization, before core completes the version or runs retention. If publication fails, modify fails and the previous completed version remains served.

The binary engine in #497 is merged. HTML generation optimization is now separate in #500. This draft preserves the existing HTML generator and removes the journal tables, asynchronous publisher, reconciliation schedule and reader overlays.

Behavior

• The path_index=true repository label is the only feature switch. Enabled repositories publish synchronously and serve from prepared views; publication failure fails modify. Background view preparation is fixed at two threads per content process.
• Indexes inherit the domain’s S3 client, bucket, location, credentials, endpoint, region and object options, including encryption. A storage identity fingerprint selects an immutable namespace; credential rotation alone does not require rebuilding. There are no separate index S3 settings.
• Stream an initial baseline when the predecessor has no descriptor; later versions extract changed paths and upload a delta, then an immutable manifest. A roughly 200-byte descriptor in RepositoryVersion.info commits with version completion. No new index-state tables or migrations.
• Consumers refresh scalar IDs and descriptors with a short TTL, never the membership array. Version UUID plus manifest digest identifies the exact view, including same-version compaction and backfill. Cold, invalid or full caches use DB fallback.
• Workers share pod-local index segment downloads, file locks and mmap pages. Cached views have bounded preparation and reader leases.
• Add repository-reserved build and compaction actions plus descriptor status. Delta merging runs during finalization; reaching the segment cap can require a full rebase. Operators can rebase through maintenance ahead of time.
• HTML listings stream through the shared indexed artifact response with HTML content type, inline disposition and redirects disabled. ETag, Last-Modified and Cache-Control support HTTP caching and revalidation; there is no separate local HTML cache.
• Preserve hosted authentication and guards. Use public ArtifactResponse, indexed validators and configurable S3 redirects. No pulpcore changes, auth endpoint or Redis guard cache.

See the rollout and failure guide, including the reset procedure for test deployments of earlier drafts.

Validation

• 222 tests and 4 subtests passed, including real PostgreSQL lifecycle tests, HTTP checks, and all six S3 service tests. Domain S3 round trips cover tenant, default STORAGES, and legacy AWS_* configuration with domain credentials instead of global credentials.
• Covers public modify with 200 orphan uploads, retention, S3 failure, DB failure after upload, no-ops, backfill, descriptor/profile changes, compaction, pinned versions and cache-pressure recovery.
• Four independent processes share segment downloads. HTTP coverage includes validators, HEAD, range streaming, redirects and inline HTML delivery without artifact reads for HEAD/304 responses.
• Ruff lint/format, public pulpcore imports and diff whitespace checks pass.

Review and remaining deployment work

Review the foundation commits in order: 2a7df17 (implementation), 4106eab (tests), and 9041306 (documentation), followed by 810312e (shared inline HTML response) and 3e0d24b (repository-only opt-in, domain S3 configuration, and fixed loader concurrency). The branch is rebased onto main at e5c6d9f. The review-order comment below follows this sequence.

Earlier experimental deployments must rebuild their indexes in the domain namespace before resuming modify tasks; readers use DB fallback during the rebuild.

S3 is a write dependency. Bootstrap and occasional full compaction hold the repository reservation. Build scratch needs disk beyond the cache allowance; automatic index-object collection remains unimplemented. Cold pods can increase fallback DB load. Production-size modify throughput, queue latency, readiness and the actual hosted middleware still need deployment validation.

AI disclosure: implementation, tests, documentation and this description were generated with Codex. Commits include Generated-by: Codex; human review is pending.

[gerrod3]

Suggested review approach (AI-generated guide; human review pending):

Start with the incremental diff above #497 and the rollout guide. The PR's default Files changed view includes the still-unmerged engine work from #497.

Then review the four commits in order:

1. 9c14ce5 — overlay primitives and disk handling. Check that local checkpoint-only updates do not claim immutable version manifests, sorting stays inside builder scratch, and abandoned overlay cleanup runs only while holding the shared fill lock. Existing live mappings must survive eviction/cleanup.
2. 9d11b03 — journal, HTML writer, and tasks. This is the main correctness review. Follow finalization → durable capture → completed-version receipt → retention → confirmation/publication. Check failed/no-op versions, recovery, repository reservations, and UUIDs surviving retention. Then inspect direct-child state, dirty-directory selection, page membership reuse, and empty-directory removal. Directory links intentionally lose descendant size/date metadata.
3. 89b722c — cached reads and HTTP behavior. Trace a removal and a replacement through an unpublished journal suffix. Only a complete view ending at the selected version may answer a miss; gaps and resource failures must use fallback. Check multi-process locking, leased mmap lifetimes, HTML cache failure paths, conditional responses, and the S3 redirect threshold. The expanded lifecycle tests include 200 retained-away versions and four independent content processes.
4. fbfc8a0 — operation and capacity evidence. Review settings, bootstrap/reconciliation/status permissions, rollback, and disk/scratch sizing. The version benchmark measures sequential core/Maven costs; its 1,000-artifact smoke results do not establish production upload capacity.

Before enabling serving in hosted staging, the most useful deployment checks are the current guard/middleware contract, the existing 1.7 GB S3 redirect policy, cold-cache fallback load, and representative large-repository version creation with a burst of 200 API uploads. This draft keeps hosted auth unchanged, uses a process-local serving-descriptor TTL, and leaves remote/journal collection and cache-aware readiness for follow-up work.

Local validation on the assembled branch: 205 tests and 4 subtests passed, including the S3 wire tests; Ruff, migration consistency, public imports, and Maven OpenAPI validation passed.

[gerrod3]

The earlier review instructions are superseded. #497 is merged, and HTML generation has moved to independent #500.

Review 2a7df17 (implementation), 4106eab (tests), and 9041306 (documentation), followed by 810312e (shared inline HTML response) and 3e0d24b (configuration simplification). The branch is rebased onto main at e5c6d9f. Use Files changed for the final combined diff.

For the latest commit, start with path_index/config.py and settings.py: the repository label is the sole switch, S3 comes from domain.get_storage(), and view loading is fixed at two threads. Then check s3.py for domain encryption/object options with engine-owned checksums and conditional writes. Finally read test_path_index_config.py and the three domain-backed service cases in test_path_index_s3_service.py. The full suite passed 222 tests and 4 subtests, including six real S3 service tests.

For the HTML follow-up, check inline headers, redirect suppression, shared validators/HEAD handling, and removal of the HTML cache setting and disk reservation.

Suggested order:

1. path_index/publish.py and the finalizer in models.py: follow metadata/HTML generation, changed-path extraction, segment and manifest uploads, then descriptor attachment. Failure must raise before core completion and retention. No-ops must not publish.
2. path_index/state.py and tasks/path_index.py: review committed descriptor identity, digest/scope/profile checks, and repository reservations for build/compaction. Same-version compaction changes the token without changing content.
3. path_index/content.py and cache.py: check scalar TTL refresh, exact-version selection, shared downloads, reader leases, cache-pressure recovery and DB fallback. No old view may answer for a newly selected version.
4. test_path_index_integration.py: start with S3 failure preserving the previous version and orphans, DB failure after upload, the 200-orphan public modify call, retention, compaction refresh and four-process downloads. Then review HTTP and engine storage tests.
5. viewsets.py and the integration guide/glossary: review permissions, fail-modify policy, bootstrap queueing, disk headroom, occasional full compaction and the reset procedure for earlier drafts.

The implementation does not optimize the immediate maven_deploy deadline or change hosted guards. Large-repository throughput, cold-pod admission and hosted middleware remain deployment validation work. Human review of this AI-generated code is pending.