Conversation
Add equivalents of the Parallel methods that allow flattening the result of a limited parallel traversal. These are helpful in case like big database lookups, where a set of IDs is batched, then each batch is queried and produces a collection of results that needs to be flattened into a larger result.
|
This is excellent but I'd definitely like to push pause until #4451 lands just to make sure we avoid creating more conflicting implementations. I'll try to prioritize getting that over the line. |
|
Alright! Now that #4451 is somewhat on sane footing, we can safely revisit this. These functions are going to need similar bespoke treatment (and extensive tests). Unfortunately, we also missed the boat on inclusion in 3.7.0 (mostly because it took me forever to fix up the other PR), so this will have to wait for 3.8.0 due to our forward compatibility model. |
|
Any interest in fixing this up for the latest on series/3.x? |
Yes, absolutely. Might take until next week though that I get to it. |
|
Sounds good! I have very spotty availability myself so no shade whatsoever. I'll try to review and merge as quickly as I can |
…ersen # Conflicts: # tests/shared/src/test/scala/cats/effect/IOSuite.scala
With the rewrite of parTraverseN's internals for performance, the MiniSemaphore approach is no longer optimal. This extracts the parTraverseN rewrite into a private helper that takes a function that converts the result into its final state, either via `sequence` for the parTraverseN case or via `flatSequence` for `parFlatTraverseN`.
|
@djspiewak The code is ready for review. One |
| assertCompleteAs(test.attempt.void, ()) | ||
| } | ||
|
|
||
| real("parFlatTraverseN - throw when n < 1") { |
There was a problem hiding this comment.
Can we have some success tests for this function?
There was a problem hiding this comment.
Good point.
I added ticked success tests for both new methods, asserting the result and that some parallelism is applied. parTraverseN also has a real test, but that comes at a heavy price in terms of test runtime and both effectively share the implementation, so I kept it simple. Happy to add more / real tests if you think they're needed.
Ensure parFlatTraverseN and parFlatSequenceN yield the correct results and run in the expected time, ensuring parallelism.
2 participants
Add equivalents of the
Parallelmethods that allow flattening the result of a limited parallel traversal. These are helpful in case like big database lookups, where a set of IDs is batched, then each batch is queried and produces a collection of results that needs to be flattened into a larger result.If #4451 is merged, the implementations in this PR should be aligned, either by delegation and flattening afterwards or by using a customised variant of the new code.