Skip to content

feat(buffer): harden shared buffer pool core - #708

Open
iaojnh wants to merge 5 commits into
alibaba:mainfrom
iaojnh:codex/buffer-pool-core-pr1
Open

feat(buffer): harden shared buffer pool core#708
iaojnh wants to merge 5 commits into
alibaba:mainfrom
iaojnh:codex/buffer-pool-core-pr1

Conversation

@iaojnh

@iaojnh iaojnh commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR hardens the shared Buffer Pool core for future Buffer Storage integration with IVF, DiskANN, and other indexes. Index integrations, benchmarks, and user documentation are out of scope.

Key changes

  • Unify page buffers, metadata, and external caches under one process-wide memory budget.
  • Harden concurrent page loading, pinning, eviction, and failure recovery.
  • Add admission control, hot-page protection, and priority-aware CLOCK eviction.
  • Support dirty-page tracking, batched writeback, failure recovery, and file extension.
  • Add background reclamation with low-watermark control.
  • Share Buffer Pool state across executables and DSOs.
  • Consolidate io_uring support and reuse its reader types in DiskANN.
  • Add statistics for memory usage, hits, eviction, admission, and writeback.

Verification

  • vector_page_table_test: 66/66 passed.
  • Concurrent load-coalescing test: 2,000 repeated runs passed.
  • CLOCK second-chance test: 1,000 repeated runs passed.
  • clang-format 18 passed.
  • MSVC and Linux Clang compatibility issues were fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant