Repository navigation
Conversation
Reading the last block of a file prefetched block nblocks, which does not exist: blocks are numbered 0..nblocks-1. The empty result of that fetch was added to the LRU on the next read, where it occupied a slot and evicted a live block, and it counted as a miss and as requested bytes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BackgroundBlockCacheprefetches the block after the one a read ends in, and while reading the last block of a file it prefetches blocknblocks— a block that does not exist, since blocks are numbered0..nblocks-1.BaseCache._fetchshort-circuits a request at or past the file size withb"", so the phantom fetch issues no HTTP request, but_fetch_blockstill counts it as a miss and astotal_requested_bytes, and the empty result is added to the LRU by the next read's join — where it occupies a slot, so a live block is evicted and has to be fetched again. Withmaxblocks=2and a 52-byte file in 4-byte blocks, reading block 0, then the last block, then block 0 again leaves the LRU holding block 13:The other two places that answer the same question are consistent with each other and not with this one:
BlockCache._fetchcomputes the end block as(end - 1) // blocksize(so a range ending exactly on a boundary does not cover the next block), andMMapCacheclamps its fetch toself.size.What changed: the prefetch condition is
end_block_plus_1 < self.nblocks, so only blocks that exist are prefetched.Test:
test_background_block_cache_no_prefetch_past_the_last_blockreads the last block, forces the join with a later read, and asserts every cached block number is belownblocksand no cached block is empty. It fails onmasterwith[0, 12, 13]and passes on this branch. The full suite is 2157 passed, 251 skipped, 2 xfailed;ruff checkandruff format(0.14.3) are clean on both files.#2150 is about the same class but a different root cause — blocks being fetched twice — and is already merged; this is the boundary case it did not cover.