Skip to content

New option to cache data - #6500

Open
ArbaazKhan1 wants to merge 2 commits into
apache:mainfrom
ArbaazKhan1:accumulo-6023
Open

New option to cache data#6500
ArbaazKhan1 wants to merge 2 commits into
apache:mainfrom
ArbaazKhan1:accumulo-6023

Conversation

@ArbaazKhan1

Copy link
Copy Markdown
Contributor

Closes issue #6023

Implements a new optional compressed block cache for RFile blocks. Currently Accumulo's primary block cache stores uncompressed blocks for tablets with high compression ratios. This means cached data takes up more space on the Heap than the corresponding on-disk footprint, reducing the number of distinct blocks that can be held in memory.

The changes here add an optional compressed data cache to TabletServers and ScanServers. When enabled blocks are compressed before being stored in the secondary cache and transparently decompressed when read. The feature is disabled by default, so existing deployments are unchanged unless the new properties are used.

Changes:

  • Added CompressedBlockCache to compress blocks before storing them and decompressing them on retrieval
  • Added configurable cache sized for Tablet and Scan servers
  • Added a secondary cache lookup path: primary cache → compressed cache → disk.
  • Blocks retrieved from the compressed cache are decompressed and promoted back into the primary cache.
  • Updated the cache provider and TabletServer resource management to support the new cache.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Optionally cache compressed data

1 participant