[avgvi]

Two fixes for the 0.14.2 release.

Prefix lookup leaves shared memory open. When a dataset starts, the local leader walks the existing shared memory prefixes and opens each locals block to check for a reused local directory. Each probe was opened with the default auto_cleanup=True, which registers an atexit handler that keeps the mapping and its file descriptor until the process exits, and the probe was never closed. A process that creates many datasets kept two descriptors per probe. The probe now opens with auto_cleanup=False, copies the bytes it needs and closes the block straight away.

A failed download leaves the shard stuck in PREPARING. prepare_shard marks a shard PREPARING, releases the cache lock and downloads. If the download raised, the shard stayed PREPARING: every worker waiting for it spun forever, and no later call downloaded it again. The same happened when decompressing a zip-only local shard. The shard now goes back to REMOTE after a failed download and back to LOCAL after a failed decompression, so the next call retries (get_item already retries on a missing file).

Tests

• test_check_and_find_closes_probes: 20 probes of an existing prefix leave the open descriptor count unchanged (40 extra on main).
• test_failed_download_returns_shard_to_remote: a forced download failure leaves the shard REMOTE, and the next call downloads it.
• test_failed_decompression_keeps_shard_local: a forced decompression failure leaves the shard LOCAL, and the next call decompresses it.
• All three fail on main and pass here. Locally (macOS, Python 3.12) the eviction, golden, shared, download and util test files show the same failures as on main apart from these three, all of them environment-dependent (Databricks, gs://, test_forced_race).

CHANGELOG entries under Unreleased.

[read-the-docs-community]

Documentation build overview

📚 streaming-community | 🛠️ Build #35283300 | 📁 Comparing f086d16 against latest (7eb8f89)

  🔍 Preview build  

2 files changed

± _modules/streaming/base/dataset.html
± _modules/streaming/base/shared/prefix.html