Skip to content

Do not prefetch a block past the end of the file in BackgroundBlockCache - #2249

Open
feiiiiii5 wants to merge 1 commit into
fsspec:masterfrom
feiiiiii5:fix/background-cache-prefetch-bound
Open

feiiiiii5 wants to merge 1 commit into
fsspec:masterfrom
feiiiiii5:fix/background-cache-prefetch-bound

Conversation

@feiiiiii5

Copy link
Copy Markdown
Contributor

BackgroundBlockCache prefetches the block after the one a read ends in, and while reading the last block of a file it prefetches block nblocks — a block that does not exist, since blocks are numbered 0..nblocks-1.

BaseCache._fetch short-circuits a request at or past the file size with b"", so the phantom fetch issues no HTTP request, but _fetch_block still counts it as a miss and as total_requested_bytes, and the empty result is added to the LRU by the next read's join — where it occupies a slot, so a live block is evicted and has to be fetched again. With maxblocks=2 and a 52-byte file in 4-byte blocks, reading block 0, then the last block, then block 0 again leaves the LRU holding block 13:

LRU keys: [(13,), (0,)]     # block 13 does not exist

The other two places that answer the same question are consistent with each other and not with this one: BlockCache._fetch computes the end block as (end - 1) // blocksize (so a range ending exactly on a boundary does not cover the next block), and MMapCache clamps its fetch to self.size.

What changed: the prefetch condition is end_block_plus_1 < self.nblocks, so only blocks that exist are prefetched.

Test: test_background_block_cache_no_prefetch_past_the_last_block reads the last block, forces the join with a later read, and asserts every cached block number is below nblocks and no cached block is empty. It fails on master with [0, 12, 13] and passes on this branch. The full suite is 2157 passed, 251 skipped, 2 xfailed; ruff check and ruff format (0.14.3) are clean on both files.

#2150 is about the same class but a different root cause — blocks being fetched twice — and is already merged; this is the boundary case it did not cover.

Reading the last block of a file prefetched block nblocks, which does not
exist: blocks are numbered 0..nblocks-1. The empty result of that fetch was
added to the LRU on the next read, where it occupied a slot and evicted a live
block, and it counted as a miss and as requested bytes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant