[avgvi]

With shuffle=True, per_stream, stratified and device_per_stream batching shuffle each stream separately over num_canonical_nodes. A stream with fewer samples than canonical nodes (for example a small stream in a mix, or a small proportion of a short epoch) failed the whole epoch:

ValueError: The number of samples assigned to a canonical node is 0. ...

The partition already repeats such a stream's samples so that each canonical node holds one, so the shuffle has nothing to do there. The stream is now kept in partition order, with a warning, instead of failing. With shuffle=False it already behaved this way.

A related failure: a stream whose share of shuffle_block_size rounds down to zero failed with ValueError: high <= 0 under per_stream and stratified, and with "Samples from stream N are not being used" under device_per_stream. It now gets a shuffle block of one sample.

The default random batching method is unchanged: it shuffles the whole dataset at once, and a dataset smaller than num_canonical_nodes still raises there (test_tiny_dataset_exception).

Tests: test_stream_smaller_than_canonical_nodes and test_stream_share_of_shuffle_block_below_one, each for all three methods. All six fail on main with the errors above. The existing batching tests pass.

[read-the-docs-community]

Documentation build overview

📚 streaming-community | 🛠️ Build #35285571 | 📁 Comparing 989ec90 against latest (bf0b36e)

  🔍 Preview build  

1 file changed

± _modules/streaming/base/stream.html