[avgvi]

When a downloaded shard file fails its checksum, Stream raises Checksum failure but leaves the file in the cache. Since #54 a failed download returns the shard to REMOTE, so the next prepare_shard call retries; that retry finds the raw file present, skips both the download and the check, and marks the shard LOCAL with the corrupt data. A plain read of a sample in that shard used the file the same way.

This deletes the file that failed (the raw file in _prepare_shard_part, the zip in _decompress_shard_part) before raising, so the next attempt downloads it again. The deletion is skipped when the local directory is the dataset itself (remote is None or equals local), using the same check as safe_keep_zip, so a local-only dataset never loses its only copy.

Tests: test_checksum_failure_downloads_shard_again corrupts the first download of shard 0, uncompressed and zstd, and asserts the first call raises, the file is gone, and the retry downloads it again and reads back correctly. Both cases fail without the change.

This was meant for #54 but missed its merge.

[read-the-docs-community]

Documentation build overview

📚 streaming-community | 🛠️ Build #35284019 | 📁 Comparing c3b594d against latest (bf0b36e)

  🔍 Preview build  

1 file changed

± _modules/streaming/base/stream.html