From 5570390a156e742107dd067688048034fd977a07 Mon Sep 17 00:00:00 2001 From: Adam Warski Date: Fri, 9 Oct 2026 15:05:32 +0000 Subject: [PATCH 01/11] orca: progress log --- .orca/runs/0a3279c8e3d7.progress.json | 1 + 1 file changed, 1 insertion(+) create mode 100644 .orca/runs/0a3279c8e3d7.progress.json diff --git a/.orca/runs/0a3279c8e3d7.progress.json b/.orca/runs/0a3279c8e3d7.progress.json new file mode 100644 index 00000000..fa12cfff --- /dev/null +++ b/.orca/runs/0a3279c8e3d7.progress.json @@ -0,0 +1 @@ +{"header":{"startingBranch":"zippy-astrodon-94aa2fb6","branch":"zippy-astrodon-94aa2fb6","branchMode":"Reused","userPrompt":"Implement GitHub issue softwaremill/ox#528: \"Flow.fromPublisher: allow releasing elements which are buffered but not emitted\".\n\nProblem: `Flow.fromPublisher` (core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala) buffers elements in a channel. Some of them never reach the consumer:\n- when the publisher fails, `channel.error` drops the buffered elements\n- when the consumer stops early (an exception, an interruption), the buffered elements are never received\n- elements can still arrive (onNext) after the subscription is cancelled\n\nIf elements hold resources (e.g. Netty's reference-counted ByteBufs), nobody releases them, so they leak.\n\nGoal: add an optional callback, e.g. `fromPublisher(p, onDiscard: T => Unit)`, called exactly once for each element received from the publisher but not emitted downstream. Existing call sites without the callback must keep compiling and behaving the same.\n\nSuggested implementation: on publisher error, store the error and call `channel.done()` (so buffered elements are kept and emitted before the error is rethrown). When the flow ends early, cancel the subscription, close the channel, and drain it, calling the callback for each drained element. In `onNext`, call the callback if the channel is already closed. Make sure there are no races where an element is neither emitted nor discarded.\n\nAlso update the variant in flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala (org.reactivestreams Publisher) to accept the same optional callback, and update any docs (doc/ directory) describing fromPublisher.\n\nAcceptance criteria:\n- tests covering: publisher error with buffered elements (elements still emitted, then error); consumer stopping early (exception / interruption) with buffered elements -> callback called for each; elements arriving after cancellation -> callback called; every element is either emitted or discarded, never both or neither.\n- `sbt compile test` passes for affected modules (core, flow-reactive-streams).\n- Code follows existing style in the repo; concise doc comments.\n\nContext: tapir currently tracks unconsumed elements itself using a lock: https://github.com/softwaremill/tapir/pull/5573","flow":{"type":"Catalog","name":"implement.sc"},"startingCommit":"664f01730cda34fd88e8c3db3deb600934eb1d1e"},"entries":[],"stageStarts":[]} \ No newline at end of file From 2aaa78b977a0718aec4076cb5f3d81bbcbb11b2f Mon Sep 17 00:00:00 2001 From: Adam Warski Date: Fri, 9 Oct 2026 15:10:53 +0000 Subject: [PATCH 02/11] stage: Plan --- .orca/runs/0a3279c8e3d7.progress.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.orca/runs/0a3279c8e3d7.progress.json b/.orca/runs/0a3279c8e3d7.progress.json index fa12cfff..58539714 100644 --- a/.orca/runs/0a3279c8e3d7.progress.json +++ b/.orca/runs/0a3279c8e3d7.progress.json @@ -1 +1 @@ -{"header":{"startingBranch":"zippy-astrodon-94aa2fb6","branch":"zippy-astrodon-94aa2fb6","branchMode":"Reused","userPrompt":"Implement GitHub issue softwaremill/ox#528: \"Flow.fromPublisher: allow releasing elements which are buffered but not emitted\".\n\nProblem: `Flow.fromPublisher` (core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala) buffers elements in a channel. Some of them never reach the consumer:\n- when the publisher fails, `channel.error` drops the buffered elements\n- when the consumer stops early (an exception, an interruption), the buffered elements are never received\n- elements can still arrive (onNext) after the subscription is cancelled\n\nIf elements hold resources (e.g. Netty's reference-counted ByteBufs), nobody releases them, so they leak.\n\nGoal: add an optional callback, e.g. `fromPublisher(p, onDiscard: T => Unit)`, called exactly once for each element received from the publisher but not emitted downstream. Existing call sites without the callback must keep compiling and behaving the same.\n\nSuggested implementation: on publisher error, store the error and call `channel.done()` (so buffered elements are kept and emitted before the error is rethrown). When the flow ends early, cancel the subscription, close the channel, and drain it, calling the callback for each drained element. In `onNext`, call the callback if the channel is already closed. Make sure there are no races where an element is neither emitted nor discarded.\n\nAlso update the variant in flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala (org.reactivestreams Publisher) to accept the same optional callback, and update any docs (doc/ directory) describing fromPublisher.\n\nAcceptance criteria:\n- tests covering: publisher error with buffered elements (elements still emitted, then error); consumer stopping early (exception / interruption) with buffered elements -> callback called for each; elements arriving after cancellation -> callback called; every element is either emitted or discarded, never both or neither.\n- `sbt compile test` passes for affected modules (core, flow-reactive-streams).\n- Code follows existing style in the repo; concise doc comments.\n\nContext: tapir currently tracks unconsumed elements itself using a lock: https://github.com/softwaremill/tapir/pull/5573","flow":{"type":"Catalog","name":"implement.sc"},"startingCommit":"664f01730cda34fd88e8c3db3deb600934eb1d1e"},"entries":[],"stageStarts":[]} \ No newline at end of file +{"header":{"startingBranch":"zippy-astrodon-94aa2fb6","branch":"zippy-astrodon-94aa2fb6","branchMode":"Reused","userPrompt":"Implement GitHub issue softwaremill/ox#528: \"Flow.fromPublisher: allow releasing elements which are buffered but not emitted\".\n\nProblem: `Flow.fromPublisher` (core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala) buffers elements in a channel. Some of them never reach the consumer:\n- when the publisher fails, `channel.error` drops the buffered elements\n- when the consumer stops early (an exception, an interruption), the buffered elements are never received\n- elements can still arrive (onNext) after the subscription is cancelled\n\nIf elements hold resources (e.g. Netty's reference-counted ByteBufs), nobody releases them, so they leak.\n\nGoal: add an optional callback, e.g. `fromPublisher(p, onDiscard: T => Unit)`, called exactly once for each element received from the publisher but not emitted downstream. Existing call sites without the callback must keep compiling and behaving the same.\n\nSuggested implementation: on publisher error, store the error and call `channel.done()` (so buffered elements are kept and emitted before the error is rethrown). When the flow ends early, cancel the subscription, close the channel, and drain it, calling the callback for each drained element. In `onNext`, call the callback if the channel is already closed. Make sure there are no races where an element is neither emitted nor discarded.\n\nAlso update the variant in flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala (org.reactivestreams Publisher) to accept the same optional callback, and update any docs (doc/ directory) describing fromPublisher.\n\nAcceptance criteria:\n- tests covering: publisher error with buffered elements (elements still emitted, then error); consumer stopping early (exception / interruption) with buffered elements -> callback called for each; elements arriving after cancellation -> callback called; every element is either emitted or discarded, never both or neither.\n- `sbt compile test` passes for affected modules (core, flow-reactive-streams).\n- Code follows existing style in the repo; concise doc comments.\n\nContext: tapir currently tracks unconsumed elements itself using a lock: https://github.com/softwaremill/tapir/pull/5573","flow":{"type":"Catalog","name":"implement.sc"},"startingCommit":"664f01730cda34fd88e8c3db3deb600934eb1d1e"},"entries":[{"id":[{"name":"Plan","occurrence":0}],"resultJson":{"id":"ox-528-from-publisher-on-discard","description":"`Flow.fromPublisher` buffers elements received from a reactive `Publisher` in a channel. Today, elements that are buffered but never emitted are silently lost: `channel.error` drops the buffer on publisher error, a consumer that stops early (exception, interruption, `take`) never receives what is buffered, and `onNext` calls that arrive after the subscription is cancelled are dropped by `channel.send` throwing. When elements own resources (Netty reference-counted ByteBufs in tapir), this leaks them. Tapir currently works around it with its own lock-based tracking (softwaremill/tapir#5573).\n\nThis epic adds an optional `onDiscard: T => Unit` callback to `Flow.fromPublisher` (core, JDK `Flow.Publisher`) and `FlowReactiveStreams.fromPublisher` (flow-reactive-streams, `org.reactivestreams.Publisher`), with the invariant that every element received from the publisher is either emitted downstream or passed to `onDiscard`, exactly once, never both or neither. Publisher errors no longer drop buffered elements: the error is stored, the channel is completed with `done()`, buffered elements are emitted, and then the error is rethrown. When the flow ends early, the subscription is cancelled, the channel is closed and drained through `onDiscard`, and any late `onNext` goes to `onDiscard` as well. Existing single-argument call sites keep compiling (the new parameter is an overload, which also keeps MiMa happy for `core`). Docs in `doc/streaming/flows.md` describe the new callback.","tasks":[{"title":"Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","description":"In core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala add `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg `fromPublisher` delegate to it with `_ => ()`. Implement per the brief's design decisions: `onNext` uses `sendOrClosed` and discards on closed; `onError` stores the throwable in an AtomicReference and calls `doneOrClosed`; `onComplete` calls `doneOrClosed`; `onSubscribe` cancels immediately if a `cancelled` flag is already set, else requests `capacity`; the consumer loop rethrows the stored error when it sees `Done`; on any Throwable escaping the loop: set `cancelled`, cancel the subscription if present, `doneOrClosed` the channel, drain it calling `onDiscard` per element (keep draining across an InterruptedException and re-interrupt the thread afterwards), rethrow. Preserve the existing batched demand logic. Update the scaladoc of both overloads (exactly-once invariant, callback runs on publisher threads and must not throw, buffered elements are now emitted before a publisher error is rethrown).\n\nTests, in a new core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardTest.scala (AnyFunSuite with Matchers), using a hand-written `ManualPublisher[T]` helper as described in the brief (keep the helper in this file or a sibling file in the same package so Task 2 can reuse it):\n1. Publisher error with buffered elements: push 1,2,3 then `error(boom)` before/while the consumer runs; `runToList` throws boom and all of 1,2,3 were emitted; `onDiscard` never called.\n2. Publisher error with nothing buffered still throws.\n3. Consumer exception: `.map(x => if x == 2 then throw boom else x)`, push 1..5; `runToList` throws boom; discarded == 3,4,5 (order-independent compare), subscription cancelled.\n4. Consumer stops via `take(2)`: result List(1,2), discarded == 3,4,5, cancelled.\n5. Consumer interrupted: run `runForeach(x => { received.release(); gate.acquire() })` in `forkCancellable`, push 1..5, wait for `received`, `cancel()`; discarded == 2,3,4,5, cancelled.\n6. Late onNext after cancellation: after any of the above, call `next(6)` on the publisher; `onDiscard(6)` is called synchronously and nothing throws.\n7. Existing one-arg overload: compiles unchanged and `FlowPublisherPekkoTest` still passes.\nRun `sbt \"core/testOnly ox.flow.reactive.*\"` and `sbt core/scalafmt`; all green."},{"title":"Exactly-once stress test: every element is emitted or discarded, never both or neither","description":"Add a test (same file as Task 1 or a sibling in core/src/test/scala/ox/flow/reactive/) that uses a demand-respecting publisher running on its own thread: it emits 1..N (N around 1000) only while `sent < requested` and not cancelled, calling `onNext` from that thread, then `onComplete`. Compose `Flow.fromPublisher(p, discarded.add)` with a consumer that stops at a random point (e.g. `take(k)` for random k, and separately a `map` that throws at a random element, and separately a `forkCancellable` cancelled after a random short delay). Collect emitted elements via `.tap(emitted.add)` into thread-safe collections. After the flow finishes and the publisher thread is joined, assert: `emitted` and `discarded` are disjoint, their union equals exactly the set of elements the publisher actually sent, and no element appears twice. Repeat each scenario in a loop (e.g. 20 iterations) with a small `given BufferCapacity = BufferCapacity(4)` to make buffer-full races likely. The publisher's late `onNext` calls (after cancellation) must land in `discarded`. If this exposes a race in Task 1's implementation, fix it in FlowCompanionReactiveOps.scala as part of this task. Run `sbt \"core/testOnly ox.flow.reactive.*\"`."},{"title":"Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","description":"In flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala add `def fromPublisher[T](p: org.reactivestreams.Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p), onDiscard)`, keep the one-arg overload, and update its scaladoc to point at the core overload for the callback semantics. Create flow-reactive-streams/src/test/scala/ox/flow/reactive/FlowReactiveStreamsFromPublisherTest.scala (AnyFunSuite with Matchers) with a small hand-written `org.reactivestreams.Publisher[Int]` (same shape as the core ManualPublisher, but implementing the reactive-streams interfaces; it cannot be shared because the module does not depend on core's test classpath) and two tests: elements flow through the one-arg variant; with `.take(1)` and the new overload, remaining pushed elements are passed to `onDiscard` and the subscription is cancelled. Run `sbt flowReactiveStreams/test` and `sbt flowReactiveStreams/scalafmt`."},{"title":"Document the onDiscard callback in doc/streaming/flows.md","description":"In doc/streaming/flows.md, section \"### Publisher -> Flow\" (around lines 211-219), add a short paragraph explaining: elements received from the publisher but not emitted downstream (buffered when the publisher fails, when the consumer stops early via exception, interruption or `take`, or arriving after cancellation) can be released by passing a second argument `onDiscard: T => Unit` to `Flow.fromPublisher` / `FlowReactiveStreams.fromPublisher`; it is called exactly once per such element, possibly from the publisher's thread, and should not throw; typical use is releasing reference-counted buffers. Also state that on publisher error, elements already received are emitted before the error is propagated. Keep the prose as short as the surrounding text. If you add a code example, use a ```` ```scala mdoc:compile-only ```` block and verify with `sbt compileDocumentation`; otherwise no build step is needed. Do not touch generated-doc/out."}],"brief":"## Modules and files\n\n- `core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala` — the only implementation file in core. Trait `FlowCompanionReactiveOps` (mixed into `Flow.type`) defines `def fromPublisher[T](p: java.util.concurrent.Flow.Publisher[T])(using BufferCapacity): Flow[T]`. Read the whole file (80 lines) before changing anything. Current structure: `usingEmitInline: emit => unsupervised { channel = BufferCapacity.newChannel[T]; forkUnsupervised { p.subscribe(Subscriber: onSubscribe -> subscriptionRef.set(s); s.request(capacity); onNext -> channel.send; onError -> channel.error; onComplete -> channel.done) }; repeatWhile { channel.receiveOrClosed() match Done -> false / Error -> throw / t -> emit(t); batched subscription.request } }.tapException(cancel subscription)`. The demand logic (`demandThreshold = ceil(capacity/2)`, `toDemand` counter, lazily cached `subscription` read from `subscriptionRef` after the first element) must be preserved.\n- `flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala` — `object FlowReactiveStreams { def fromPublisher[T](p: org.reactivestreams.Publisher[T])(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p)) }`. Module has no `src/test` dir yet; scalatest is already a test dependency (build.sbt `flowReactiveStreams`), it `dependsOn(core)` (compile only, not test->test).\n- `core/src/test/scala/ox/flow/reactive/` — existing reactive tests: `FlowPublisherPekkoTest.scala` (AnyFunSuite + Matchers, uses Pekko `Source...runWith(Sink.asPublisher)` and `FlowAdapters.toFlowPublisher`) and `FlowPublisherTckTest.scala`. Put new fromPublisher tests here. No test anywhere currently uses a hand-written Publisher; write one (see below).\n- `doc/streaming/flows.md` lines 211-219, section \"### Publisher -> Flow\" — the only user doc mentioning `fromPublisher`. `generated-doc/out/**` is mdoc output regenerated at release time by `updateDocs`; do NOT edit it. Code blocks in docs use ```` ```scala mdoc:compile-only ```` and are compiled by `sbt compileDocumentation` (slow; only needed if you add a code block). Doc dependency snippet convention: `// sbt dependency: \"com.softwaremill.ox\" %% \"flow-reactive-streams\" % \"@VERSION@\"`.\n\n## Key APIs (ox.channels, core/src/main/scala/ox/channels/Channel.scala)\n\n- `Sink.sendOrClosed(t): Unit | ChannelClosed` (non-throwing), `Sink.send(t)` throws `ChannelClosedException` when closed.\n- `Sink.doneOrClosed(): Unit | ChannelClosed` — \"buffered values will be delivered; send operations in progress complete normally when a receiver arrives\"; returns `ChannelClosed` instead of throwing if already closed. `Sink.errorOrClosed(reason)` — drops buffered values (that is the bug). Use the `*OrClosed` variants everywhere in the subscriber: a misbehaving publisher may call onError/onComplete twice.\n- `Source.receiveOrClosed(): T | ChannelClosed` (blocking); match `case ChannelClosed.Done => ...; case e: ChannelClosed.Error => throw e.toThrowable; case t: T @unchecked => ...`. `Source.tryReceiveOrClosed()` is non-blocking but may spuriously return None, so it is not suitable for a drain loop.\n- `Source.foreachOrError(f)` / `drainOrError()` from `SourceDrainOps.scala` loop until Done; usable for the drain, but write the drain inline so you can handle `InterruptedException` (see below).\n- jox semantics (com.softwaremill.jox channels 1.1.2, verified from source): after `done()`, buffered elements stay receivable; a receiver observing Done has already seen every element whose `send` started before `done()` (an in-progress sender is waited for); a `send` racing with `done()` either lands in the buffer or returns closed. Hence \"drain until Done, then `sendOrClosed` returns closed\" gives exactly-once discard with no extra locking.\n- `BufferCapacity`: `given default: BufferCapacity = 16`; `BufferCapacity.newChannel[T]`; `summon[BufferCapacity].toInt` (import `ox.channels.toInt`). Tests can override with `given BufferCapacity = BufferCapacity(4)`.\n- `ox.util`: `t.tapException(f)` runs `f(e)` for any Throwable and rethrows `e` (exceptions from `f` added as suppressed). `ox.pipe`, `ox.discard`, `ox.repeatWhile` already imported in the file.\n- `FlowOps.take(n)` stops upstream by throwing a private `ControlThrowable` (`abortTake`) out of `emit`, caught in `take` itself. So \"consumer stops early\" includes `take`, and the cleanup path must run for every Throwable, not just NonFatal. `tapException`/`catch case e: Throwable` already does this.\n- `unsupervised { ... }` + `forkUnsupervised { ... }` (import `ox.unsupervised`, `ox.forkUnsupervised`): scope end interrupts still-running forks once. `onNext`/`onError`/`onComplete` run on the publisher's threads (or on the `p.subscribe` fork for synchronous publishers), never on the consumer thread.\n- Test helpers: `ox.forkCancellable { ... }` inside `supervised` returns a `CancellableFork` with `cancel(): Either[Throwable, T]` (interrupts once, joins). Interruption inside the flow surfaces as `InterruptedException` thrown from a blocking call (e.g. `Semaphore.acquire`, `channel.receiveOrClosed`).\n\n## Design decisions (apply as-is)\n\n- API: add an overload `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg method delegate with `_ => ()`. Do not add a default parameter and do not change the existing signature: `core` runs MiMa (`enableMimaSettings` in build.sbt) and only additive changes are safe. Same shape in `FlowReactiveStreams`.\n- Subscriber: `onNext(t)` = `channel.sendOrClosed(t)` and if the result is `ChannelClosed` call `onDiscard(t)`. `onError(e)` = store `e` in an `AtomicReference[Throwable]`, then `channel.doneOrClosed().discard`. `onComplete()` = `channel.doneOrClosed().discard`. `onSubscribe(s)` = set `subscriptionRef`, then if a `cancelled: AtomicBoolean` is already set call `s.cancel()` else `s.request(capacity)` (both sides use volatile atomics, so at least one side sees the other; a double `cancel()` is allowed by the spec).\n- Consumer loop: on `ChannelClosed.Done`, read the stored error; if non-null throw it, else finish. Keep the `ChannelClosed.Error` branch for safety.\n- Cleanup, run on any Throwable escaping the loop (the stored publisher error included; cancel after onError is a spec-allowed no-op): set `cancelled`, cancel the subscription if `subscriptionRef` is non-null, `channel.doneOrClosed().discard`, then drain: loop `receiveOrClosed()` until `Done`, calling `onDiscard` for each element. The drain is bounded (channel is done, at most capacity elements plus in-flight sends) but blocking; if `InterruptedException` is thrown mid-drain, catch it, keep draining, and call `Thread.currentThread().interrupt()` once the drain finishes, so no element is lost on a second interrupt. Then rethrow the original exception. An element that was passed to `emit` and whose `emit` threw counts as emitted (downstream owns it): do not discard it.\n- Scaladoc on the new overload: state the exactly-once invariant, that `onDiscard` may be called from publisher threads, that it must not throw, and that on publisher error buffered elements are emitted before the error is rethrown (this is a deliberate behaviour change for the one-arg variant too; mention it in its doc).\n\n## Test publisher helper (write it in the core test, package `ox.flow.reactive`)\n\nA `ManualPublisher[T] extends java.util.concurrent.Flow.Publisher[T]` with: `@volatile var subscriber: Subscriber[? >: T]`, `requested: AtomicLong`, `cancelled: AtomicBoolean`, a `subscribed: CountDownLatch(1)`; `subscribe(s)` stores `s`, calls `s.onSubscribe(new Subscription { request(n) => requested.addAndGet(n); cancel() => cancelled.set(true) })`, counts down the latch. Helper methods `next(t)`, `error(e)`, `complete()` forward to the subscriber from the test thread. For the exactly-once stress test, a second variant that runs its own thread emitting `1..n` while `sent < requested` and not cancelled. Run the flow under test in `supervised { forkCancellable { ... } }` or `fork`, await `subscribed`, then drive the publisher. Collect emitted elements with `.tap(emitted += _)` or `runForeach`, discarded with `onDiscard = discarded += _` (use thread-safe collections such as `ConcurrentLinkedQueue` or `java.util.Collections.synchronizedList`, since `onDiscard` runs on other threads).\n\n## Conventions / build\n\n- Scala 3 indentation syntax, `end X` markers on long definitions, `.discard` on ignored results (compiler warns otherwise), scalafmt configured in `.scalafmt.conf` (140-column lines); run `sbt core/scalafmt flowReactiveStreams/scalafmt` after editing.\n- Comments: short, explain non-obvious reasoning only; never restate code. Scaladoc describes usage and invariants, not implementation.\n- Tests: scalatest 3.2.20; the reactive tests use `AnyFunSuite with Matchers` with `test(\"...\")` blocks and Scala 3 colon syntax. `core` runs tests forked (`Test / fork := true`).\n- Commands (sbt 2.0.10): `sbt \"core/testOnly ox.flow.reactive.*\"`, `sbt flowReactiveStreams/test`, `sbt core/mimaReportBinaryIssues` (only meaningful when a previous stable version resolves), `sbt compileDocumentation` for mdoc. Never commit; the user commits."}}],"stageStarts":[]} \ No newline at end of file From ecb0c90e535ca7046cfd13620c881f7dbcf8d57b Mon Sep 17 00:00:00 2001 From: Adam Warski Date: Fri, 9 Oct 2026 15:17:26 +0000 Subject: [PATCH 03/11] Add onDiscard callback to Flow.fromPublisher and fix error handling --- .orca/runs/0a3279c8e3d7.progress.json | 2 +- .../ox/flow/FlowCompanionReactiveOps.scala | 91 +++++++++++++-- .../FlowFromPublisherDiscardTest.scala | 106 ++++++++++++++++++ .../ox/flow/reactive/ManualPublisher.scala | 36 ++++++ 4 files changed, 222 insertions(+), 13 deletions(-) create mode 100644 core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardTest.scala create mode 100644 core/src/test/scala/ox/flow/reactive/ManualPublisher.scala diff --git a/.orca/runs/0a3279c8e3d7.progress.json b/.orca/runs/0a3279c8e3d7.progress.json index 58539714..8e7a755e 100644 --- a/.orca/runs/0a3279c8e3d7.progress.json +++ b/.orca/runs/0a3279c8e3d7.progress.json @@ -1 +1 @@ -{"header":{"startingBranch":"zippy-astrodon-94aa2fb6","branch":"zippy-astrodon-94aa2fb6","branchMode":"Reused","userPrompt":"Implement GitHub issue softwaremill/ox#528: \"Flow.fromPublisher: allow releasing elements which are buffered but not emitted\".\n\nProblem: `Flow.fromPublisher` (core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala) buffers elements in a channel. Some of them never reach the consumer:\n- when the publisher fails, `channel.error` drops the buffered elements\n- when the consumer stops early (an exception, an interruption), the buffered elements are never received\n- elements can still arrive (onNext) after the subscription is cancelled\n\nIf elements hold resources (e.g. Netty's reference-counted ByteBufs), nobody releases them, so they leak.\n\nGoal: add an optional callback, e.g. `fromPublisher(p, onDiscard: T => Unit)`, called exactly once for each element received from the publisher but not emitted downstream. Existing call sites without the callback must keep compiling and behaving the same.\n\nSuggested implementation: on publisher error, store the error and call `channel.done()` (so buffered elements are kept and emitted before the error is rethrown). When the flow ends early, cancel the subscription, close the channel, and drain it, calling the callback for each drained element. In `onNext`, call the callback if the channel is already closed. Make sure there are no races where an element is neither emitted nor discarded.\n\nAlso update the variant in flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala (org.reactivestreams Publisher) to accept the same optional callback, and update any docs (doc/ directory) describing fromPublisher.\n\nAcceptance criteria:\n- tests covering: publisher error with buffered elements (elements still emitted, then error); consumer stopping early (exception / interruption) with buffered elements -> callback called for each; elements arriving after cancellation -> callback called; every element is either emitted or discarded, never both or neither.\n- `sbt compile test` passes for affected modules (core, flow-reactive-streams).\n- Code follows existing style in the repo; concise doc comments.\n\nContext: tapir currently tracks unconsumed elements itself using a lock: https://github.com/softwaremill/tapir/pull/5573","flow":{"type":"Catalog","name":"implement.sc"},"startingCommit":"664f01730cda34fd88e8c3db3deb600934eb1d1e"},"entries":[{"id":[{"name":"Plan","occurrence":0}],"resultJson":{"id":"ox-528-from-publisher-on-discard","description":"`Flow.fromPublisher` buffers elements received from a reactive `Publisher` in a channel. Today, elements that are buffered but never emitted are silently lost: `channel.error` drops the buffer on publisher error, a consumer that stops early (exception, interruption, `take`) never receives what is buffered, and `onNext` calls that arrive after the subscription is cancelled are dropped by `channel.send` throwing. When elements own resources (Netty reference-counted ByteBufs in tapir), this leaks them. Tapir currently works around it with its own lock-based tracking (softwaremill/tapir#5573).\n\nThis epic adds an optional `onDiscard: T => Unit` callback to `Flow.fromPublisher` (core, JDK `Flow.Publisher`) and `FlowReactiveStreams.fromPublisher` (flow-reactive-streams, `org.reactivestreams.Publisher`), with the invariant that every element received from the publisher is either emitted downstream or passed to `onDiscard`, exactly once, never both or neither. Publisher errors no longer drop buffered elements: the error is stored, the channel is completed with `done()`, buffered elements are emitted, and then the error is rethrown. When the flow ends early, the subscription is cancelled, the channel is closed and drained through `onDiscard`, and any late `onNext` goes to `onDiscard` as well. Existing single-argument call sites keep compiling (the new parameter is an overload, which also keeps MiMa happy for `core`). Docs in `doc/streaming/flows.md` describe the new callback.","tasks":[{"title":"Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","description":"In core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala add `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg `fromPublisher` delegate to it with `_ => ()`. Implement per the brief's design decisions: `onNext` uses `sendOrClosed` and discards on closed; `onError` stores the throwable in an AtomicReference and calls `doneOrClosed`; `onComplete` calls `doneOrClosed`; `onSubscribe` cancels immediately if a `cancelled` flag is already set, else requests `capacity`; the consumer loop rethrows the stored error when it sees `Done`; on any Throwable escaping the loop: set `cancelled`, cancel the subscription if present, `doneOrClosed` the channel, drain it calling `onDiscard` per element (keep draining across an InterruptedException and re-interrupt the thread afterwards), rethrow. Preserve the existing batched demand logic. Update the scaladoc of both overloads (exactly-once invariant, callback runs on publisher threads and must not throw, buffered elements are now emitted before a publisher error is rethrown).\n\nTests, in a new core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardTest.scala (AnyFunSuite with Matchers), using a hand-written `ManualPublisher[T]` helper as described in the brief (keep the helper in this file or a sibling file in the same package so Task 2 can reuse it):\n1. Publisher error with buffered elements: push 1,2,3 then `error(boom)` before/while the consumer runs; `runToList` throws boom and all of 1,2,3 were emitted; `onDiscard` never called.\n2. Publisher error with nothing buffered still throws.\n3. Consumer exception: `.map(x => if x == 2 then throw boom else x)`, push 1..5; `runToList` throws boom; discarded == 3,4,5 (order-independent compare), subscription cancelled.\n4. Consumer stops via `take(2)`: result List(1,2), discarded == 3,4,5, cancelled.\n5. Consumer interrupted: run `runForeach(x => { received.release(); gate.acquire() })` in `forkCancellable`, push 1..5, wait for `received`, `cancel()`; discarded == 2,3,4,5, cancelled.\n6. Late onNext after cancellation: after any of the above, call `next(6)` on the publisher; `onDiscard(6)` is called synchronously and nothing throws.\n7. Existing one-arg overload: compiles unchanged and `FlowPublisherPekkoTest` still passes.\nRun `sbt \"core/testOnly ox.flow.reactive.*\"` and `sbt core/scalafmt`; all green."},{"title":"Exactly-once stress test: every element is emitted or discarded, never both or neither","description":"Add a test (same file as Task 1 or a sibling in core/src/test/scala/ox/flow/reactive/) that uses a demand-respecting publisher running on its own thread: it emits 1..N (N around 1000) only while `sent < requested` and not cancelled, calling `onNext` from that thread, then `onComplete`. Compose `Flow.fromPublisher(p, discarded.add)` with a consumer that stops at a random point (e.g. `take(k)` for random k, and separately a `map` that throws at a random element, and separately a `forkCancellable` cancelled after a random short delay). Collect emitted elements via `.tap(emitted.add)` into thread-safe collections. After the flow finishes and the publisher thread is joined, assert: `emitted` and `discarded` are disjoint, their union equals exactly the set of elements the publisher actually sent, and no element appears twice. Repeat each scenario in a loop (e.g. 20 iterations) with a small `given BufferCapacity = BufferCapacity(4)` to make buffer-full races likely. The publisher's late `onNext` calls (after cancellation) must land in `discarded`. If this exposes a race in Task 1's implementation, fix it in FlowCompanionReactiveOps.scala as part of this task. Run `sbt \"core/testOnly ox.flow.reactive.*\"`."},{"title":"Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","description":"In flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala add `def fromPublisher[T](p: org.reactivestreams.Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p), onDiscard)`, keep the one-arg overload, and update its scaladoc to point at the core overload for the callback semantics. Create flow-reactive-streams/src/test/scala/ox/flow/reactive/FlowReactiveStreamsFromPublisherTest.scala (AnyFunSuite with Matchers) with a small hand-written `org.reactivestreams.Publisher[Int]` (same shape as the core ManualPublisher, but implementing the reactive-streams interfaces; it cannot be shared because the module does not depend on core's test classpath) and two tests: elements flow through the one-arg variant; with `.take(1)` and the new overload, remaining pushed elements are passed to `onDiscard` and the subscription is cancelled. Run `sbt flowReactiveStreams/test` and `sbt flowReactiveStreams/scalafmt`."},{"title":"Document the onDiscard callback in doc/streaming/flows.md","description":"In doc/streaming/flows.md, section \"### Publisher -> Flow\" (around lines 211-219), add a short paragraph explaining: elements received from the publisher but not emitted downstream (buffered when the publisher fails, when the consumer stops early via exception, interruption or `take`, or arriving after cancellation) can be released by passing a second argument `onDiscard: T => Unit` to `Flow.fromPublisher` / `FlowReactiveStreams.fromPublisher`; it is called exactly once per such element, possibly from the publisher's thread, and should not throw; typical use is releasing reference-counted buffers. Also state that on publisher error, elements already received are emitted before the error is propagated. Keep the prose as short as the surrounding text. If you add a code example, use a ```` ```scala mdoc:compile-only ```` block and verify with `sbt compileDocumentation`; otherwise no build step is needed. Do not touch generated-doc/out."}],"brief":"## Modules and files\n\n- `core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala` — the only implementation file in core. Trait `FlowCompanionReactiveOps` (mixed into `Flow.type`) defines `def fromPublisher[T](p: java.util.concurrent.Flow.Publisher[T])(using BufferCapacity): Flow[T]`. Read the whole file (80 lines) before changing anything. Current structure: `usingEmitInline: emit => unsupervised { channel = BufferCapacity.newChannel[T]; forkUnsupervised { p.subscribe(Subscriber: onSubscribe -> subscriptionRef.set(s); s.request(capacity); onNext -> channel.send; onError -> channel.error; onComplete -> channel.done) }; repeatWhile { channel.receiveOrClosed() match Done -> false / Error -> throw / t -> emit(t); batched subscription.request } }.tapException(cancel subscription)`. The demand logic (`demandThreshold = ceil(capacity/2)`, `toDemand` counter, lazily cached `subscription` read from `subscriptionRef` after the first element) must be preserved.\n- `flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala` — `object FlowReactiveStreams { def fromPublisher[T](p: org.reactivestreams.Publisher[T])(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p)) }`. Module has no `src/test` dir yet; scalatest is already a test dependency (build.sbt `flowReactiveStreams`), it `dependsOn(core)` (compile only, not test->test).\n- `core/src/test/scala/ox/flow/reactive/` — existing reactive tests: `FlowPublisherPekkoTest.scala` (AnyFunSuite + Matchers, uses Pekko `Source...runWith(Sink.asPublisher)` and `FlowAdapters.toFlowPublisher`) and `FlowPublisherTckTest.scala`. Put new fromPublisher tests here. No test anywhere currently uses a hand-written Publisher; write one (see below).\n- `doc/streaming/flows.md` lines 211-219, section \"### Publisher -> Flow\" — the only user doc mentioning `fromPublisher`. `generated-doc/out/**` is mdoc output regenerated at release time by `updateDocs`; do NOT edit it. Code blocks in docs use ```` ```scala mdoc:compile-only ```` and are compiled by `sbt compileDocumentation` (slow; only needed if you add a code block). Doc dependency snippet convention: `// sbt dependency: \"com.softwaremill.ox\" %% \"flow-reactive-streams\" % \"@VERSION@\"`.\n\n## Key APIs (ox.channels, core/src/main/scala/ox/channels/Channel.scala)\n\n- `Sink.sendOrClosed(t): Unit | ChannelClosed` (non-throwing), `Sink.send(t)` throws `ChannelClosedException` when closed.\n- `Sink.doneOrClosed(): Unit | ChannelClosed` — \"buffered values will be delivered; send operations in progress complete normally when a receiver arrives\"; returns `ChannelClosed` instead of throwing if already closed. `Sink.errorOrClosed(reason)` — drops buffered values (that is the bug). Use the `*OrClosed` variants everywhere in the subscriber: a misbehaving publisher may call onError/onComplete twice.\n- `Source.receiveOrClosed(): T | ChannelClosed` (blocking); match `case ChannelClosed.Done => ...; case e: ChannelClosed.Error => throw e.toThrowable; case t: T @unchecked => ...`. `Source.tryReceiveOrClosed()` is non-blocking but may spuriously return None, so it is not suitable for a drain loop.\n- `Source.foreachOrError(f)` / `drainOrError()` from `SourceDrainOps.scala` loop until Done; usable for the drain, but write the drain inline so you can handle `InterruptedException` (see below).\n- jox semantics (com.softwaremill.jox channels 1.1.2, verified from source): after `done()`, buffered elements stay receivable; a receiver observing Done has already seen every element whose `send` started before `done()` (an in-progress sender is waited for); a `send` racing with `done()` either lands in the buffer or returns closed. Hence \"drain until Done, then `sendOrClosed` returns closed\" gives exactly-once discard with no extra locking.\n- `BufferCapacity`: `given default: BufferCapacity = 16`; `BufferCapacity.newChannel[T]`; `summon[BufferCapacity].toInt` (import `ox.channels.toInt`). Tests can override with `given BufferCapacity = BufferCapacity(4)`.\n- `ox.util`: `t.tapException(f)` runs `f(e)` for any Throwable and rethrows `e` (exceptions from `f` added as suppressed). `ox.pipe`, `ox.discard`, `ox.repeatWhile` already imported in the file.\n- `FlowOps.take(n)` stops upstream by throwing a private `ControlThrowable` (`abortTake`) out of `emit`, caught in `take` itself. So \"consumer stops early\" includes `take`, and the cleanup path must run for every Throwable, not just NonFatal. `tapException`/`catch case e: Throwable` already does this.\n- `unsupervised { ... }` + `forkUnsupervised { ... }` (import `ox.unsupervised`, `ox.forkUnsupervised`): scope end interrupts still-running forks once. `onNext`/`onError`/`onComplete` run on the publisher's threads (or on the `p.subscribe` fork for synchronous publishers), never on the consumer thread.\n- Test helpers: `ox.forkCancellable { ... }` inside `supervised` returns a `CancellableFork` with `cancel(): Either[Throwable, T]` (interrupts once, joins). Interruption inside the flow surfaces as `InterruptedException` thrown from a blocking call (e.g. `Semaphore.acquire`, `channel.receiveOrClosed`).\n\n## Design decisions (apply as-is)\n\n- API: add an overload `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg method delegate with `_ => ()`. Do not add a default parameter and do not change the existing signature: `core` runs MiMa (`enableMimaSettings` in build.sbt) and only additive changes are safe. Same shape in `FlowReactiveStreams`.\n- Subscriber: `onNext(t)` = `channel.sendOrClosed(t)` and if the result is `ChannelClosed` call `onDiscard(t)`. `onError(e)` = store `e` in an `AtomicReference[Throwable]`, then `channel.doneOrClosed().discard`. `onComplete()` = `channel.doneOrClosed().discard`. `onSubscribe(s)` = set `subscriptionRef`, then if a `cancelled: AtomicBoolean` is already set call `s.cancel()` else `s.request(capacity)` (both sides use volatile atomics, so at least one side sees the other; a double `cancel()` is allowed by the spec).\n- Consumer loop: on `ChannelClosed.Done`, read the stored error; if non-null throw it, else finish. Keep the `ChannelClosed.Error` branch for safety.\n- Cleanup, run on any Throwable escaping the loop (the stored publisher error included; cancel after onError is a spec-allowed no-op): set `cancelled`, cancel the subscription if `subscriptionRef` is non-null, `channel.doneOrClosed().discard`, then drain: loop `receiveOrClosed()` until `Done`, calling `onDiscard` for each element. The drain is bounded (channel is done, at most capacity elements plus in-flight sends) but blocking; if `InterruptedException` is thrown mid-drain, catch it, keep draining, and call `Thread.currentThread().interrupt()` once the drain finishes, so no element is lost on a second interrupt. Then rethrow the original exception. An element that was passed to `emit` and whose `emit` threw counts as emitted (downstream owns it): do not discard it.\n- Scaladoc on the new overload: state the exactly-once invariant, that `onDiscard` may be called from publisher threads, that it must not throw, and that on publisher error buffered elements are emitted before the error is rethrown (this is a deliberate behaviour change for the one-arg variant too; mention it in its doc).\n\n## Test publisher helper (write it in the core test, package `ox.flow.reactive`)\n\nA `ManualPublisher[T] extends java.util.concurrent.Flow.Publisher[T]` with: `@volatile var subscriber: Subscriber[? >: T]`, `requested: AtomicLong`, `cancelled: AtomicBoolean`, a `subscribed: CountDownLatch(1)`; `subscribe(s)` stores `s`, calls `s.onSubscribe(new Subscription { request(n) => requested.addAndGet(n); cancel() => cancelled.set(true) })`, counts down the latch. Helper methods `next(t)`, `error(e)`, `complete()` forward to the subscriber from the test thread. For the exactly-once stress test, a second variant that runs its own thread emitting `1..n` while `sent < requested` and not cancelled. Run the flow under test in `supervised { forkCancellable { ... } }` or `fork`, await `subscribed`, then drive the publisher. Collect emitted elements with `.tap(emitted += _)` or `runForeach`, discarded with `onDiscard = discarded += _` (use thread-safe collections such as `ConcurrentLinkedQueue` or `java.util.Collections.synchronizedList`, since `onDiscard` runs on other threads).\n\n## Conventions / build\n\n- Scala 3 indentation syntax, `end X` markers on long definitions, `.discard` on ignored results (compiler warns otherwise), scalafmt configured in `.scalafmt.conf` (140-column lines); run `sbt core/scalafmt flowReactiveStreams/scalafmt` after editing.\n- Comments: short, explain non-obvious reasoning only; never restate code. Scaladoc describes usage and invariants, not implementation.\n- Tests: scalatest 3.2.20; the reactive tests use `AnyFunSuite with Matchers` with `test(\"...\")` blocks and Scala 3 colon syntax. `core` runs tests forked (`Test / fork := true`).\n- Commands (sbt 2.0.10): `sbt \"core/testOnly ox.flow.reactive.*\"`, `sbt flowReactiveStreams/test`, `sbt core/mimaReportBinaryIssues` (only meaningful when a previous stable version resolves), `sbt compileDocumentation` for mdoc. Never commit; the user commits."}}],"stageStarts":[]} \ No newline at end of file +{"header":{"startingBranch":"zippy-astrodon-94aa2fb6","branch":"zippy-astrodon-94aa2fb6","branchMode":"Reused","userPrompt":"Implement GitHub issue softwaremill/ox#528: \"Flow.fromPublisher: allow releasing elements which are buffered but not emitted\".\n\nProblem: `Flow.fromPublisher` (core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala) buffers elements in a channel. Some of them never reach the consumer:\n- when the publisher fails, `channel.error` drops the buffered elements\n- when the consumer stops early (an exception, an interruption), the buffered elements are never received\n- elements can still arrive (onNext) after the subscription is cancelled\n\nIf elements hold resources (e.g. Netty's reference-counted ByteBufs), nobody releases them, so they leak.\n\nGoal: add an optional callback, e.g. `fromPublisher(p, onDiscard: T => Unit)`, called exactly once for each element received from the publisher but not emitted downstream. Existing call sites without the callback must keep compiling and behaving the same.\n\nSuggested implementation: on publisher error, store the error and call `channel.done()` (so buffered elements are kept and emitted before the error is rethrown). When the flow ends early, cancel the subscription, close the channel, and drain it, calling the callback for each drained element. In `onNext`, call the callback if the channel is already closed. Make sure there are no races where an element is neither emitted nor discarded.\n\nAlso update the variant in flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala (org.reactivestreams Publisher) to accept the same optional callback, and update any docs (doc/ directory) describing fromPublisher.\n\nAcceptance criteria:\n- tests covering: publisher error with buffered elements (elements still emitted, then error); consumer stopping early (exception / interruption) with buffered elements -> callback called for each; elements arriving after cancellation -> callback called; every element is either emitted or discarded, never both or neither.\n- `sbt compile test` passes for affected modules (core, flow-reactive-streams).\n- Code follows existing style in the repo; concise doc comments.\n\nContext: tapir currently tracks unconsumed elements itself using a lock: https://github.com/softwaremill/tapir/pull/5573","flow":{"type":"Catalog","name":"implement.sc"},"startingCommit":"664f01730cda34fd88e8c3db3deb600934eb1d1e"},"entries":[{"id":[{"name":"Plan","occurrence":0}],"resultJson":{"id":"ox-528-from-publisher-on-discard","description":"`Flow.fromPublisher` buffers elements received from a reactive `Publisher` in a channel. Today, elements that are buffered but never emitted are silently lost: `channel.error` drops the buffer on publisher error, a consumer that stops early (exception, interruption, `take`) never receives what is buffered, and `onNext` calls that arrive after the subscription is cancelled are dropped by `channel.send` throwing. When elements own resources (Netty reference-counted ByteBufs in tapir), this leaks them. Tapir currently works around it with its own lock-based tracking (softwaremill/tapir#5573).\n\nThis epic adds an optional `onDiscard: T => Unit` callback to `Flow.fromPublisher` (core, JDK `Flow.Publisher`) and `FlowReactiveStreams.fromPublisher` (flow-reactive-streams, `org.reactivestreams.Publisher`), with the invariant that every element received from the publisher is either emitted downstream or passed to `onDiscard`, exactly once, never both or neither. Publisher errors no longer drop buffered elements: the error is stored, the channel is completed with `done()`, buffered elements are emitted, and then the error is rethrown. When the flow ends early, the subscription is cancelled, the channel is closed and drained through `onDiscard`, and any late `onNext` goes to `onDiscard` as well. Existing single-argument call sites keep compiling (the new parameter is an overload, which also keeps MiMa happy for `core`). Docs in `doc/streaming/flows.md` describe the new callback.","tasks":[{"title":"Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","description":"In core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala add `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg `fromPublisher` delegate to it with `_ => ()`. Implement per the brief's design decisions: `onNext` uses `sendOrClosed` and discards on closed; `onError` stores the throwable in an AtomicReference and calls `doneOrClosed`; `onComplete` calls `doneOrClosed`; `onSubscribe` cancels immediately if a `cancelled` flag is already set, else requests `capacity`; the consumer loop rethrows the stored error when it sees `Done`; on any Throwable escaping the loop: set `cancelled`, cancel the subscription if present, `doneOrClosed` the channel, drain it calling `onDiscard` per element (keep draining across an InterruptedException and re-interrupt the thread afterwards), rethrow. Preserve the existing batched demand logic. Update the scaladoc of both overloads (exactly-once invariant, callback runs on publisher threads and must not throw, buffered elements are now emitted before a publisher error is rethrown).\n\nTests, in a new core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardTest.scala (AnyFunSuite with Matchers), using a hand-written `ManualPublisher[T]` helper as described in the brief (keep the helper in this file or a sibling file in the same package so Task 2 can reuse it):\n1. Publisher error with buffered elements: push 1,2,3 then `error(boom)` before/while the consumer runs; `runToList` throws boom and all of 1,2,3 were emitted; `onDiscard` never called.\n2. Publisher error with nothing buffered still throws.\n3. Consumer exception: `.map(x => if x == 2 then throw boom else x)`, push 1..5; `runToList` throws boom; discarded == 3,4,5 (order-independent compare), subscription cancelled.\n4. Consumer stops via `take(2)`: result List(1,2), discarded == 3,4,5, cancelled.\n5. Consumer interrupted: run `runForeach(x => { received.release(); gate.acquire() })` in `forkCancellable`, push 1..5, wait for `received`, `cancel()`; discarded == 2,3,4,5, cancelled.\n6. Late onNext after cancellation: after any of the above, call `next(6)` on the publisher; `onDiscard(6)` is called synchronously and nothing throws.\n7. Existing one-arg overload: compiles unchanged and `FlowPublisherPekkoTest` still passes.\nRun `sbt \"core/testOnly ox.flow.reactive.*\"` and `sbt core/scalafmt`; all green."},{"title":"Exactly-once stress test: every element is emitted or discarded, never both or neither","description":"Add a test (same file as Task 1 or a sibling in core/src/test/scala/ox/flow/reactive/) that uses a demand-respecting publisher running on its own thread: it emits 1..N (N around 1000) only while `sent < requested` and not cancelled, calling `onNext` from that thread, then `onComplete`. Compose `Flow.fromPublisher(p, discarded.add)` with a consumer that stops at a random point (e.g. `take(k)` for random k, and separately a `map` that throws at a random element, and separately a `forkCancellable` cancelled after a random short delay). Collect emitted elements via `.tap(emitted.add)` into thread-safe collections. After the flow finishes and the publisher thread is joined, assert: `emitted` and `discarded` are disjoint, their union equals exactly the set of elements the publisher actually sent, and no element appears twice. Repeat each scenario in a loop (e.g. 20 iterations) with a small `given BufferCapacity = BufferCapacity(4)` to make buffer-full races likely. The publisher's late `onNext` calls (after cancellation) must land in `discarded`. If this exposes a race in Task 1's implementation, fix it in FlowCompanionReactiveOps.scala as part of this task. Run `sbt \"core/testOnly ox.flow.reactive.*\"`."},{"title":"Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","description":"In flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala add `def fromPublisher[T](p: org.reactivestreams.Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p), onDiscard)`, keep the one-arg overload, and update its scaladoc to point at the core overload for the callback semantics. Create flow-reactive-streams/src/test/scala/ox/flow/reactive/FlowReactiveStreamsFromPublisherTest.scala (AnyFunSuite with Matchers) with a small hand-written `org.reactivestreams.Publisher[Int]` (same shape as the core ManualPublisher, but implementing the reactive-streams interfaces; it cannot be shared because the module does not depend on core's test classpath) and two tests: elements flow through the one-arg variant; with `.take(1)` and the new overload, remaining pushed elements are passed to `onDiscard` and the subscription is cancelled. Run `sbt flowReactiveStreams/test` and `sbt flowReactiveStreams/scalafmt`."},{"title":"Document the onDiscard callback in doc/streaming/flows.md","description":"In doc/streaming/flows.md, section \"### Publisher -> Flow\" (around lines 211-219), add a short paragraph explaining: elements received from the publisher but not emitted downstream (buffered when the publisher fails, when the consumer stops early via exception, interruption or `take`, or arriving after cancellation) can be released by passing a second argument `onDiscard: T => Unit` to `Flow.fromPublisher` / `FlowReactiveStreams.fromPublisher`; it is called exactly once per such element, possibly from the publisher's thread, and should not throw; typical use is releasing reference-counted buffers. Also state that on publisher error, elements already received are emitted before the error is propagated. Keep the prose as short as the surrounding text. If you add a code example, use a ```` ```scala mdoc:compile-only ```` block and verify with `sbt compileDocumentation`; otherwise no build step is needed. Do not touch generated-doc/out."}],"brief":"## Modules and files\n\n- `core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala` — the only implementation file in core. Trait `FlowCompanionReactiveOps` (mixed into `Flow.type`) defines `def fromPublisher[T](p: java.util.concurrent.Flow.Publisher[T])(using BufferCapacity): Flow[T]`. Read the whole file (80 lines) before changing anything. Current structure: `usingEmitInline: emit => unsupervised { channel = BufferCapacity.newChannel[T]; forkUnsupervised { p.subscribe(Subscriber: onSubscribe -> subscriptionRef.set(s); s.request(capacity); onNext -> channel.send; onError -> channel.error; onComplete -> channel.done) }; repeatWhile { channel.receiveOrClosed() match Done -> false / Error -> throw / t -> emit(t); batched subscription.request } }.tapException(cancel subscription)`. The demand logic (`demandThreshold = ceil(capacity/2)`, `toDemand` counter, lazily cached `subscription` read from `subscriptionRef` after the first element) must be preserved.\n- `flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala` — `object FlowReactiveStreams { def fromPublisher[T](p: org.reactivestreams.Publisher[T])(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p)) }`. Module has no `src/test` dir yet; scalatest is already a test dependency (build.sbt `flowReactiveStreams`), it `dependsOn(core)` (compile only, not test->test).\n- `core/src/test/scala/ox/flow/reactive/` — existing reactive tests: `FlowPublisherPekkoTest.scala` (AnyFunSuite + Matchers, uses Pekko `Source...runWith(Sink.asPublisher)` and `FlowAdapters.toFlowPublisher`) and `FlowPublisherTckTest.scala`. Put new fromPublisher tests here. No test anywhere currently uses a hand-written Publisher; write one (see below).\n- `doc/streaming/flows.md` lines 211-219, section \"### Publisher -> Flow\" — the only user doc mentioning `fromPublisher`. `generated-doc/out/**` is mdoc output regenerated at release time by `updateDocs`; do NOT edit it. Code blocks in docs use ```` ```scala mdoc:compile-only ```` and are compiled by `sbt compileDocumentation` (slow; only needed if you add a code block). Doc dependency snippet convention: `// sbt dependency: \"com.softwaremill.ox\" %% \"flow-reactive-streams\" % \"@VERSION@\"`.\n\n## Key APIs (ox.channels, core/src/main/scala/ox/channels/Channel.scala)\n\n- `Sink.sendOrClosed(t): Unit | ChannelClosed` (non-throwing), `Sink.send(t)` throws `ChannelClosedException` when closed.\n- `Sink.doneOrClosed(): Unit | ChannelClosed` — \"buffered values will be delivered; send operations in progress complete normally when a receiver arrives\"; returns `ChannelClosed` instead of throwing if already closed. `Sink.errorOrClosed(reason)` — drops buffered values (that is the bug). Use the `*OrClosed` variants everywhere in the subscriber: a misbehaving publisher may call onError/onComplete twice.\n- `Source.receiveOrClosed(): T | ChannelClosed` (blocking); match `case ChannelClosed.Done => ...; case e: ChannelClosed.Error => throw e.toThrowable; case t: T @unchecked => ...`. `Source.tryReceiveOrClosed()` is non-blocking but may spuriously return None, so it is not suitable for a drain loop.\n- `Source.foreachOrError(f)` / `drainOrError()` from `SourceDrainOps.scala` loop until Done; usable for the drain, but write the drain inline so you can handle `InterruptedException` (see below).\n- jox semantics (com.softwaremill.jox channels 1.1.2, verified from source): after `done()`, buffered elements stay receivable; a receiver observing Done has already seen every element whose `send` started before `done()` (an in-progress sender is waited for); a `send` racing with `done()` either lands in the buffer or returns closed. Hence \"drain until Done, then `sendOrClosed` returns closed\" gives exactly-once discard with no extra locking.\n- `BufferCapacity`: `given default: BufferCapacity = 16`; `BufferCapacity.newChannel[T]`; `summon[BufferCapacity].toInt` (import `ox.channels.toInt`). Tests can override with `given BufferCapacity = BufferCapacity(4)`.\n- `ox.util`: `t.tapException(f)` runs `f(e)` for any Throwable and rethrows `e` (exceptions from `f` added as suppressed). `ox.pipe`, `ox.discard`, `ox.repeatWhile` already imported in the file.\n- `FlowOps.take(n)` stops upstream by throwing a private `ControlThrowable` (`abortTake`) out of `emit`, caught in `take` itself. So \"consumer stops early\" includes `take`, and the cleanup path must run for every Throwable, not just NonFatal. `tapException`/`catch case e: Throwable` already does this.\n- `unsupervised { ... }` + `forkUnsupervised { ... }` (import `ox.unsupervised`, `ox.forkUnsupervised`): scope end interrupts still-running forks once. `onNext`/`onError`/`onComplete` run on the publisher's threads (or on the `p.subscribe` fork for synchronous publishers), never on the consumer thread.\n- Test helpers: `ox.forkCancellable { ... }` inside `supervised` returns a `CancellableFork` with `cancel(): Either[Throwable, T]` (interrupts once, joins). Interruption inside the flow surfaces as `InterruptedException` thrown from a blocking call (e.g. `Semaphore.acquire`, `channel.receiveOrClosed`).\n\n## Design decisions (apply as-is)\n\n- API: add an overload `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg method delegate with `_ => ()`. Do not add a default parameter and do not change the existing signature: `core` runs MiMa (`enableMimaSettings` in build.sbt) and only additive changes are safe. Same shape in `FlowReactiveStreams`.\n- Subscriber: `onNext(t)` = `channel.sendOrClosed(t)` and if the result is `ChannelClosed` call `onDiscard(t)`. `onError(e)` = store `e` in an `AtomicReference[Throwable]`, then `channel.doneOrClosed().discard`. `onComplete()` = `channel.doneOrClosed().discard`. `onSubscribe(s)` = set `subscriptionRef`, then if a `cancelled: AtomicBoolean` is already set call `s.cancel()` else `s.request(capacity)` (both sides use volatile atomics, so at least one side sees the other; a double `cancel()` is allowed by the spec).\n- Consumer loop: on `ChannelClosed.Done`, read the stored error; if non-null throw it, else finish. Keep the `ChannelClosed.Error` branch for safety.\n- Cleanup, run on any Throwable escaping the loop (the stored publisher error included; cancel after onError is a spec-allowed no-op): set `cancelled`, cancel the subscription if `subscriptionRef` is non-null, `channel.doneOrClosed().discard`, then drain: loop `receiveOrClosed()` until `Done`, calling `onDiscard` for each element. The drain is bounded (channel is done, at most capacity elements plus in-flight sends) but blocking; if `InterruptedException` is thrown mid-drain, catch it, keep draining, and call `Thread.currentThread().interrupt()` once the drain finishes, so no element is lost on a second interrupt. Then rethrow the original exception. An element that was passed to `emit` and whose `emit` threw counts as emitted (downstream owns it): do not discard it.\n- Scaladoc on the new overload: state the exactly-once invariant, that `onDiscard` may be called from publisher threads, that it must not throw, and that on publisher error buffered elements are emitted before the error is rethrown (this is a deliberate behaviour change for the one-arg variant too; mention it in its doc).\n\n## Test publisher helper (write it in the core test, package `ox.flow.reactive`)\n\nA `ManualPublisher[T] extends java.util.concurrent.Flow.Publisher[T]` with: `@volatile var subscriber: Subscriber[? >: T]`, `requested: AtomicLong`, `cancelled: AtomicBoolean`, a `subscribed: CountDownLatch(1)`; `subscribe(s)` stores `s`, calls `s.onSubscribe(new Subscription { request(n) => requested.addAndGet(n); cancel() => cancelled.set(true) })`, counts down the latch. Helper methods `next(t)`, `error(e)`, `complete()` forward to the subscriber from the test thread. For the exactly-once stress test, a second variant that runs its own thread emitting `1..n` while `sent < requested` and not cancelled. Run the flow under test in `supervised { forkCancellable { ... } }` or `fork`, await `subscribed`, then drive the publisher. Collect emitted elements with `.tap(emitted += _)` or `runForeach`, discarded with `onDiscard = discarded += _` (use thread-safe collections such as `ConcurrentLinkedQueue` or `java.util.Collections.synchronizedList`, since `onDiscard` runs on other threads).\n\n## Conventions / build\n\n- Scala 3 indentation syntax, `end X` markers on long definitions, `.discard` on ignored results (compiler warns otherwise), scalafmt configured in `.scalafmt.conf` (140-column lines); run `sbt core/scalafmt flowReactiveStreams/scalafmt` after editing.\n- Comments: short, explain non-obvious reasoning only; never restate code. Scaladoc describes usage and invariants, not implementation.\n- Tests: scalatest 3.2.20; the reactive tests use `AnyFunSuite with Matchers` with `test(\"...\")` blocks and Scala 3 colon syntax. `core` runs tests forked (`Test / fork := true`).\n- Commands (sbt 2.0.10): `sbt \"core/testOnly ox.flow.reactive.*\"`, `sbt flowReactiveStreams/test`, `sbt core/mimaReportBinaryIssues` (only meaningful when a previous stable version resolves), `sbt compileDocumentation` for mdoc. Never commit; the user commits."}},{"id":[{"name":"Task: Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","occurrence":0}],"resultJson":{"findings":[]}}],"stageStarts":[]} \ No newline at end of file diff --git a/core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala b/core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala index 370975fd..b1251416 100644 --- a/core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala +++ b/core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala @@ -1,11 +1,11 @@ package ox.flow import ox.channels.BufferCapacity +import ox.channels.Channel import ox.channels.ChannelClosed import ox.channels.toInt import ox.discard import ox.forkUnsupervised -import ox.pipe import ox.repeatWhile import ox.tapException import ox.unsupervised @@ -24,10 +24,34 @@ trait FlowCompanionReactiveOps: * The data is passed from a subscription to the flow using a [[ox.channel.Channel]], with a capacity given by the [[BufferCapacity]] in * scope. That's also how many elements will be at most requested from the publisher at a time. * + * If the publisher signals an error, elements received before the error are emitted first, and then the error is rethrown. + * + * Elements received from the publisher, but not emitted (e.g. because the flow failed, was interrupted or stopped early) are dropped. To + * release resources held by such elements, use the variant with an `onDiscard` callback. + * * The publisher parameter should implement the JDK 9+ `Flow.Publisher` API. To create a flow from a publisher implementing * `com.reactivestreams.Publisher`, use the `flow-reactive-streams` module. */ - def fromPublisher[T](p: Publisher[T])(using BufferCapacity): Flow[T] = usingEmitInline: emit => + def fromPublisher[T](p: Publisher[T])(using BufferCapacity): Flow[T] = fromPublisher(p, (_: T) => ()) + + /** Creates a [[Flow]] from a [[Publisher]], that is, which emits the elements received by subscribing to the publisher. A new + * subscription is created every time this flow is run. + * + * The data is passed from a subscription to the flow using a [[ox.channel.Channel]], with a capacity given by the [[BufferCapacity]] in + * scope. That's also how many elements will be at most requested from the publisher at a time. + * + * If the publisher signals an error, elements received before the error are emitted first, and then the error is rethrown. + * + * Every element received from the publisher is either emitted, or passed to `onDiscard` exactly once. Elements are discarded when the + * flow fails, is interrupted or stops early (e.g. due to `take`), including elements delivered by the publisher after the subscription + * is cancelled. Use this to release resources held by the elements (e.g. pooled buffers). + * + * `onDiscard` may be called from the publisher's threads, and must not throw. + * + * The publisher parameter should implement the JDK 9+ `Flow.Publisher` API. To create a flow from a publisher implementing + * `com.reactivestreams.Publisher`, use the `flow-reactive-streams` module. + */ + def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T] = usingEmitInline: emit => // using an unsafe scope for efficiency unsupervised { val channel = BufferCapacity.newChannel[T] @@ -35,8 +59,10 @@ trait FlowCompanionReactiveOps: val demandThreshold = math.ceil(capacity / 2.0).toInt // used to "extract" the subscription that is set in the subscription running in a fork - val subscriptionRef = new AtomicReference[Subscription]() + val state = new AtomicReference[SubscriptionState](SubscriptionState.NotSubscribed) var subscription: Subscription = null + // the error is stored, instead of erroring the channel, so that buffered elements are still received + val publisherError = new AtomicReference[Option[Throwable]](None) var toDemand = 0 @@ -45,24 +71,38 @@ trait FlowCompanionReactiveOps: forkUnsupervised { p.subscribe(new Subscriber[T]: def onSubscribe(s: Subscription): Unit = - subscriptionRef.set(s) - s.request(capacity) + val previous = state.getAndUpdate: + case SubscriptionState.Cancelled => SubscriptionState.Cancelled + case _ => SubscriptionState.Subscribed(s) + // the flow might have been cancelled before the subscription was set + if previous == SubscriptionState.Cancelled then s.cancel() else s.request(capacity) - def onNext(t: T): Unit = channel.send(t) - def onError(t: Throwable): Unit = channel.error(t) - def onComplete(): Unit = channel.done()) + def onNext(t: T): Unit = channel.sendOrClosed(t) match + case _: ChannelClosed => onDiscard(t) + case _ => () + + def onError(t: Throwable): Unit = + publisherError.set(Some(t)) + channel.doneOrClosed().discard + + def onComplete(): Unit = channel.doneOrClosed().discard) }.discard repeatWhile: val t = channel.receiveOrClosed() t match - case ChannelClosed.Done => false + case ChannelClosed.Done => + publisherError.get().foreach(e => throw e) + false case e: ChannelClosed.Error => throw e.toThrowable case t: T @unchecked => emit(t) // if we have an element, onSubscribe must have already happened; we can read the subscription and cache it for later - if subscription == null then subscription = subscriptionRef.get() + if subscription == null then + subscription = state.get() match + case SubscriptionState.Subscribed(s) => s + case other => throw new IllegalStateException(s"Unexpected subscription state: $other") // now that we'ver received an element from the channel, we can request more toDemand += 1 @@ -73,7 +113,34 @@ trait FlowCompanionReactiveOps: true end match - // exceptions might be propagated from the channel, but they might also originate from an interruption - }.tapException(_ => subscriptionRef.get().pipe(s => if s != null then s.cancel())) + // exceptions might be propagated from the publisher, from emit, but they might also originate from an interruption + }.tapException: _ => + state.getAndSet(SubscriptionState.Cancelled) match + case SubscriptionState.Subscribed(s) => s.cancel() + case _ => () + channel.doneOrClosed().discard + discardBuffered(channel, onDiscard) } + end fromPublisher end FlowCompanionReactiveOps + +private enum SubscriptionState: + case NotSubscribed + case Subscribed(s: Subscription) + case Cancelled + +/** Passes all elements remaining in a done `channel` to `onDiscard`. Interruptions are deferred until the channel is drained, so that no + * element is lost. + */ +private def discardBuffered[T](channel: Channel[T], onDiscard: T => Unit): Unit = + var interrupted = false + var draining = true + while draining do + try + channel.receiveOrClosed() match + case _: ChannelClosed => draining = false + case t: T @unchecked => onDiscard(t) + catch case _: InterruptedException => interrupted = true + end while + if interrupted then Thread.currentThread().interrupt() +end discardBuffered diff --git a/core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardTest.scala b/core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardTest.scala new file mode 100644 index 00000000..cdf4ff27 --- /dev/null +++ b/core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardTest.scala @@ -0,0 +1,106 @@ +package ox.flow.reactive + +import org.scalatest.funsuite.AnyFunSuite +import org.scalatest.matchers.should.Matchers +import ox.* +import ox.flow.Flow + +import java.util.concurrent.ConcurrentLinkedQueue +import java.util.concurrent.Semaphore +import scala.jdk.CollectionConverters.* +import scala.util.Failure +import scala.util.Try + +class FlowFromPublisherDiscardTest extends AnyFunSuite with Matchers: + private val boom = new RuntimeException("boom") + + private class Fixture: + val publisher = new ManualPublisher[Int] + val emitted = new ConcurrentLinkedQueue[Int]() + val discarded = new ConcurrentLinkedQueue[Int]() + val flow: Flow[Int] = Flow.fromPublisher(publisher, discarded.add(_).discard).tap(emitted.add(_).discard) + + def emittedList: List[Int] = emitted.asScala.toList + def discardedSorted: List[Int] = discarded.asScala.toList.sorted + end Fixture + + test("should emit buffered elements before rethrowing a publisher error"): + val f = Fixture() + val received = new Semaphore(0) + val gate = new Semaphore(0) + supervised: + val result = forkUnsupervised: + Try: + f.flow.runForeach: _ => + received.release() + gate.acquire() + f.publisher.subscribed.await() + f.publisher.next(1) + // the consumer is blocked, so 2 and 3 stay buffered when the error arrives + received.acquire() + f.publisher.next(2) + f.publisher.next(3) + f.publisher.error(boom) + gate.release(3) + result.join() shouldBe Failure(boom) + f.emittedList shouldBe List(1, 2, 3) + f.discarded shouldBe empty + + test("should discard undelivered elements when the consumer throws"): + val f = Fixture() + supervised: + val result = forkUnsupervised(Try(f.flow.map(x => if x == 2 then throw boom else x).runToList())) + f.publisher.subscribed.await() + (1 to 5).foreach(f.publisher.next) + result.join() shouldBe Failure(boom) + f.emittedList shouldBe List(1, 2) + f.discardedSorted shouldBe List(3, 4, 5) + f.publisher.cancelled.get() shouldBe true + + test("should discard undelivered elements when the consumer stops early"): + val f = Fixture() + supervised: + val result = fork(f.flow.take(2).runToList()) + f.publisher.subscribed.await() + (1 to 5).foreach(f.publisher.next) + result.join() shouldBe List(1, 2) + f.discardedSorted shouldBe List(3, 4, 5) + f.publisher.cancelled.get() shouldBe true + + test("should discard undelivered elements when the consumer is interrupted"): + val f = Fixture() + val received = new Semaphore(0) + val gate = new Semaphore(0) + supervised: + val result = forkCancellable: + f.flow.runForeach: _ => + received.release() + gate.acquire() + f.publisher.subscribed.await() + (1 to 5).foreach(f.publisher.next) + received.acquire() + result.cancel().discard + f.emittedList shouldBe List(1) + f.discardedSorted shouldBe List(2, 3, 4, 5) + f.publisher.cancelled.get() shouldBe true + + test("should discard elements delivered after cancellation"): + val f = Fixture() + supervised: + val result = fork(f.flow.take(1).runToList()) + f.publisher.subscribed.await() + f.publisher.next(1) + result.join() shouldBe List(1) + f.publisher.next(2) + f.discardedSorted shouldBe List(2) + + test("should cancel a subscription that arrives after the flow is cancelled"): + val publisher = new ManualPublisher[Int](deferOnSubscribe = true) + supervised: + val result = forkCancellable(Flow.fromPublisher(publisher).runDrain()) + publisher.subscribed.await() + result.cancel().discard + publisher.sendOnSubscribe() + publisher.cancelled.get() shouldBe true + publisher.requested.get() shouldBe 0 +end FlowFromPublisherDiscardTest diff --git a/core/src/test/scala/ox/flow/reactive/ManualPublisher.scala b/core/src/test/scala/ox/flow/reactive/ManualPublisher.scala new file mode 100644 index 00000000..351ae0e1 --- /dev/null +++ b/core/src/test/scala/ox/flow/reactive/ManualPublisher.scala @@ -0,0 +1,36 @@ +package ox.flow.reactive + +import ox.discard + +import java.util.concurrent.CountDownLatch +import java.util.concurrent.Flow.Publisher +import java.util.concurrent.Flow.Subscriber +import java.util.concurrent.Flow.Subscription +import java.util.concurrent.atomic.AtomicBoolean +import java.util.concurrent.atomic.AtomicLong + +/** A publisher driven from the test thread. Ignores demand; supports a single subscriber. With `deferOnSubscribe`, `onSubscribe` is only + * called by [[sendOnSubscribe]]. + */ +class ManualPublisher[T](deferOnSubscribe: Boolean = false) extends Publisher[T]: + @volatile var subscriber: Subscriber[? >: T] = null + val requested = new AtomicLong(0) + val cancelled = new AtomicBoolean(false) + val subscribed = new CountDownLatch(1) + + private val subscription = new Subscription: + override def request(n: Long): Unit = requested.addAndGet(n).discard + override def cancel(): Unit = cancelled.set(true) + + override def subscribe(s: Subscriber[? >: T]): Unit = + subscriber = s + if !deferOnSubscribe then sendOnSubscribe() + subscribed.countDown() + + def sendOnSubscribe(): Unit = subscriber.onSubscribe(subscription) + + def next(t: T): Unit = subscriber.onNext(t) + def error(e: Throwable): Unit = subscriber.onError(e) + def complete(): Unit = subscriber.onComplete() + +end ManualPublisher From f6ec23d83c3d6108a3e451dc1d0a6fe1aba06487 Mon Sep 17 00:00:00 2001 From: Adam Warski Date: Fri, 9 Oct 2026 15:22:12 +0000 Subject: [PATCH 04/11] Add stress tests for Flow.fromPublisher discard behavior --- .orca/runs/0a3279c8e3d7.progress.json | 2 +- .../FlowFromPublisherDiscardStressTest.scala | 95 +++++++++++++++++++ 2 files changed, 96 insertions(+), 1 deletion(-) create mode 100644 core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardStressTest.scala diff --git a/.orca/runs/0a3279c8e3d7.progress.json b/.orca/runs/0a3279c8e3d7.progress.json index 8e7a755e..9eea00c4 100644 --- a/.orca/runs/0a3279c8e3d7.progress.json +++ b/.orca/runs/0a3279c8e3d7.progress.json @@ -1 +1 @@ -{"header":{"startingBranch":"zippy-astrodon-94aa2fb6","branch":"zippy-astrodon-94aa2fb6","branchMode":"Reused","userPrompt":"Implement GitHub issue softwaremill/ox#528: \"Flow.fromPublisher: allow releasing elements which are buffered but not emitted\".\n\nProblem: `Flow.fromPublisher` (core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala) buffers elements in a channel. Some of them never reach the consumer:\n- when the publisher fails, `channel.error` drops the buffered elements\n- when the consumer stops early (an exception, an interruption), the buffered elements are never received\n- elements can still arrive (onNext) after the subscription is cancelled\n\nIf elements hold resources (e.g. Netty's reference-counted ByteBufs), nobody releases them, so they leak.\n\nGoal: add an optional callback, e.g. `fromPublisher(p, onDiscard: T => Unit)`, called exactly once for each element received from the publisher but not emitted downstream. Existing call sites without the callback must keep compiling and behaving the same.\n\nSuggested implementation: on publisher error, store the error and call `channel.done()` (so buffered elements are kept and emitted before the error is rethrown). When the flow ends early, cancel the subscription, close the channel, and drain it, calling the callback for each drained element. In `onNext`, call the callback if the channel is already closed. Make sure there are no races where an element is neither emitted nor discarded.\n\nAlso update the variant in flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala (org.reactivestreams Publisher) to accept the same optional callback, and update any docs (doc/ directory) describing fromPublisher.\n\nAcceptance criteria:\n- tests covering: publisher error with buffered elements (elements still emitted, then error); consumer stopping early (exception / interruption) with buffered elements -> callback called for each; elements arriving after cancellation -> callback called; every element is either emitted or discarded, never both or neither.\n- `sbt compile test` passes for affected modules (core, flow-reactive-streams).\n- Code follows existing style in the repo; concise doc comments.\n\nContext: tapir currently tracks unconsumed elements itself using a lock: https://github.com/softwaremill/tapir/pull/5573","flow":{"type":"Catalog","name":"implement.sc"},"startingCommit":"664f01730cda34fd88e8c3db3deb600934eb1d1e"},"entries":[{"id":[{"name":"Plan","occurrence":0}],"resultJson":{"id":"ox-528-from-publisher-on-discard","description":"`Flow.fromPublisher` buffers elements received from a reactive `Publisher` in a channel. Today, elements that are buffered but never emitted are silently lost: `channel.error` drops the buffer on publisher error, a consumer that stops early (exception, interruption, `take`) never receives what is buffered, and `onNext` calls that arrive after the subscription is cancelled are dropped by `channel.send` throwing. When elements own resources (Netty reference-counted ByteBufs in tapir), this leaks them. Tapir currently works around it with its own lock-based tracking (softwaremill/tapir#5573).\n\nThis epic adds an optional `onDiscard: T => Unit` callback to `Flow.fromPublisher` (core, JDK `Flow.Publisher`) and `FlowReactiveStreams.fromPublisher` (flow-reactive-streams, `org.reactivestreams.Publisher`), with the invariant that every element received from the publisher is either emitted downstream or passed to `onDiscard`, exactly once, never both or neither. Publisher errors no longer drop buffered elements: the error is stored, the channel is completed with `done()`, buffered elements are emitted, and then the error is rethrown. When the flow ends early, the subscription is cancelled, the channel is closed and drained through `onDiscard`, and any late `onNext` goes to `onDiscard` as well. Existing single-argument call sites keep compiling (the new parameter is an overload, which also keeps MiMa happy for `core`). Docs in `doc/streaming/flows.md` describe the new callback.","tasks":[{"title":"Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","description":"In core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala add `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg `fromPublisher` delegate to it with `_ => ()`. Implement per the brief's design decisions: `onNext` uses `sendOrClosed` and discards on closed; `onError` stores the throwable in an AtomicReference and calls `doneOrClosed`; `onComplete` calls `doneOrClosed`; `onSubscribe` cancels immediately if a `cancelled` flag is already set, else requests `capacity`; the consumer loop rethrows the stored error when it sees `Done`; on any Throwable escaping the loop: set `cancelled`, cancel the subscription if present, `doneOrClosed` the channel, drain it calling `onDiscard` per element (keep draining across an InterruptedException and re-interrupt the thread afterwards), rethrow. Preserve the existing batched demand logic. Update the scaladoc of both overloads (exactly-once invariant, callback runs on publisher threads and must not throw, buffered elements are now emitted before a publisher error is rethrown).\n\nTests, in a new core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardTest.scala (AnyFunSuite with Matchers), using a hand-written `ManualPublisher[T]` helper as described in the brief (keep the helper in this file or a sibling file in the same package so Task 2 can reuse it):\n1. Publisher error with buffered elements: push 1,2,3 then `error(boom)` before/while the consumer runs; `runToList` throws boom and all of 1,2,3 were emitted; `onDiscard` never called.\n2. Publisher error with nothing buffered still throws.\n3. Consumer exception: `.map(x => if x == 2 then throw boom else x)`, push 1..5; `runToList` throws boom; discarded == 3,4,5 (order-independent compare), subscription cancelled.\n4. Consumer stops via `take(2)`: result List(1,2), discarded == 3,4,5, cancelled.\n5. Consumer interrupted: run `runForeach(x => { received.release(); gate.acquire() })` in `forkCancellable`, push 1..5, wait for `received`, `cancel()`; discarded == 2,3,4,5, cancelled.\n6. Late onNext after cancellation: after any of the above, call `next(6)` on the publisher; `onDiscard(6)` is called synchronously and nothing throws.\n7. Existing one-arg overload: compiles unchanged and `FlowPublisherPekkoTest` still passes.\nRun `sbt \"core/testOnly ox.flow.reactive.*\"` and `sbt core/scalafmt`; all green."},{"title":"Exactly-once stress test: every element is emitted or discarded, never both or neither","description":"Add a test (same file as Task 1 or a sibling in core/src/test/scala/ox/flow/reactive/) that uses a demand-respecting publisher running on its own thread: it emits 1..N (N around 1000) only while `sent < requested` and not cancelled, calling `onNext` from that thread, then `onComplete`. Compose `Flow.fromPublisher(p, discarded.add)` with a consumer that stops at a random point (e.g. `take(k)` for random k, and separately a `map` that throws at a random element, and separately a `forkCancellable` cancelled after a random short delay). Collect emitted elements via `.tap(emitted.add)` into thread-safe collections. After the flow finishes and the publisher thread is joined, assert: `emitted` and `discarded` are disjoint, their union equals exactly the set of elements the publisher actually sent, and no element appears twice. Repeat each scenario in a loop (e.g. 20 iterations) with a small `given BufferCapacity = BufferCapacity(4)` to make buffer-full races likely. The publisher's late `onNext` calls (after cancellation) must land in `discarded`. If this exposes a race in Task 1's implementation, fix it in FlowCompanionReactiveOps.scala as part of this task. Run `sbt \"core/testOnly ox.flow.reactive.*\"`."},{"title":"Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","description":"In flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala add `def fromPublisher[T](p: org.reactivestreams.Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p), onDiscard)`, keep the one-arg overload, and update its scaladoc to point at the core overload for the callback semantics. Create flow-reactive-streams/src/test/scala/ox/flow/reactive/FlowReactiveStreamsFromPublisherTest.scala (AnyFunSuite with Matchers) with a small hand-written `org.reactivestreams.Publisher[Int]` (same shape as the core ManualPublisher, but implementing the reactive-streams interfaces; it cannot be shared because the module does not depend on core's test classpath) and two tests: elements flow through the one-arg variant; with `.take(1)` and the new overload, remaining pushed elements are passed to `onDiscard` and the subscription is cancelled. Run `sbt flowReactiveStreams/test` and `sbt flowReactiveStreams/scalafmt`."},{"title":"Document the onDiscard callback in doc/streaming/flows.md","description":"In doc/streaming/flows.md, section \"### Publisher -> Flow\" (around lines 211-219), add a short paragraph explaining: elements received from the publisher but not emitted downstream (buffered when the publisher fails, when the consumer stops early via exception, interruption or `take`, or arriving after cancellation) can be released by passing a second argument `onDiscard: T => Unit` to `Flow.fromPublisher` / `FlowReactiveStreams.fromPublisher`; it is called exactly once per such element, possibly from the publisher's thread, and should not throw; typical use is releasing reference-counted buffers. Also state that on publisher error, elements already received are emitted before the error is propagated. Keep the prose as short as the surrounding text. If you add a code example, use a ```` ```scala mdoc:compile-only ```` block and verify with `sbt compileDocumentation`; otherwise no build step is needed. Do not touch generated-doc/out."}],"brief":"## Modules and files\n\n- `core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala` — the only implementation file in core. Trait `FlowCompanionReactiveOps` (mixed into `Flow.type`) defines `def fromPublisher[T](p: java.util.concurrent.Flow.Publisher[T])(using BufferCapacity): Flow[T]`. Read the whole file (80 lines) before changing anything. Current structure: `usingEmitInline: emit => unsupervised { channel = BufferCapacity.newChannel[T]; forkUnsupervised { p.subscribe(Subscriber: onSubscribe -> subscriptionRef.set(s); s.request(capacity); onNext -> channel.send; onError -> channel.error; onComplete -> channel.done) }; repeatWhile { channel.receiveOrClosed() match Done -> false / Error -> throw / t -> emit(t); batched subscription.request } }.tapException(cancel subscription)`. The demand logic (`demandThreshold = ceil(capacity/2)`, `toDemand` counter, lazily cached `subscription` read from `subscriptionRef` after the first element) must be preserved.\n- `flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala` — `object FlowReactiveStreams { def fromPublisher[T](p: org.reactivestreams.Publisher[T])(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p)) }`. Module has no `src/test` dir yet; scalatest is already a test dependency (build.sbt `flowReactiveStreams`), it `dependsOn(core)` (compile only, not test->test).\n- `core/src/test/scala/ox/flow/reactive/` — existing reactive tests: `FlowPublisherPekkoTest.scala` (AnyFunSuite + Matchers, uses Pekko `Source...runWith(Sink.asPublisher)` and `FlowAdapters.toFlowPublisher`) and `FlowPublisherTckTest.scala`. Put new fromPublisher tests here. No test anywhere currently uses a hand-written Publisher; write one (see below).\n- `doc/streaming/flows.md` lines 211-219, section \"### Publisher -> Flow\" — the only user doc mentioning `fromPublisher`. `generated-doc/out/**` is mdoc output regenerated at release time by `updateDocs`; do NOT edit it. Code blocks in docs use ```` ```scala mdoc:compile-only ```` and are compiled by `sbt compileDocumentation` (slow; only needed if you add a code block). Doc dependency snippet convention: `// sbt dependency: \"com.softwaremill.ox\" %% \"flow-reactive-streams\" % \"@VERSION@\"`.\n\n## Key APIs (ox.channels, core/src/main/scala/ox/channels/Channel.scala)\n\n- `Sink.sendOrClosed(t): Unit | ChannelClosed` (non-throwing), `Sink.send(t)` throws `ChannelClosedException` when closed.\n- `Sink.doneOrClosed(): Unit | ChannelClosed` — \"buffered values will be delivered; send operations in progress complete normally when a receiver arrives\"; returns `ChannelClosed` instead of throwing if already closed. `Sink.errorOrClosed(reason)` — drops buffered values (that is the bug). Use the `*OrClosed` variants everywhere in the subscriber: a misbehaving publisher may call onError/onComplete twice.\n- `Source.receiveOrClosed(): T | ChannelClosed` (blocking); match `case ChannelClosed.Done => ...; case e: ChannelClosed.Error => throw e.toThrowable; case t: T @unchecked => ...`. `Source.tryReceiveOrClosed()` is non-blocking but may spuriously return None, so it is not suitable for a drain loop.\n- `Source.foreachOrError(f)` / `drainOrError()` from `SourceDrainOps.scala` loop until Done; usable for the drain, but write the drain inline so you can handle `InterruptedException` (see below).\n- jox semantics (com.softwaremill.jox channels 1.1.2, verified from source): after `done()`, buffered elements stay receivable; a receiver observing Done has already seen every element whose `send` started before `done()` (an in-progress sender is waited for); a `send` racing with `done()` either lands in the buffer or returns closed. Hence \"drain until Done, then `sendOrClosed` returns closed\" gives exactly-once discard with no extra locking.\n- `BufferCapacity`: `given default: BufferCapacity = 16`; `BufferCapacity.newChannel[T]`; `summon[BufferCapacity].toInt` (import `ox.channels.toInt`). Tests can override with `given BufferCapacity = BufferCapacity(4)`.\n- `ox.util`: `t.tapException(f)` runs `f(e)` for any Throwable and rethrows `e` (exceptions from `f` added as suppressed). `ox.pipe`, `ox.discard`, `ox.repeatWhile` already imported in the file.\n- `FlowOps.take(n)` stops upstream by throwing a private `ControlThrowable` (`abortTake`) out of `emit`, caught in `take` itself. So \"consumer stops early\" includes `take`, and the cleanup path must run for every Throwable, not just NonFatal. `tapException`/`catch case e: Throwable` already does this.\n- `unsupervised { ... }` + `forkUnsupervised { ... }` (import `ox.unsupervised`, `ox.forkUnsupervised`): scope end interrupts still-running forks once. `onNext`/`onError`/`onComplete` run on the publisher's threads (or on the `p.subscribe` fork for synchronous publishers), never on the consumer thread.\n- Test helpers: `ox.forkCancellable { ... }` inside `supervised` returns a `CancellableFork` with `cancel(): Either[Throwable, T]` (interrupts once, joins). Interruption inside the flow surfaces as `InterruptedException` thrown from a blocking call (e.g. `Semaphore.acquire`, `channel.receiveOrClosed`).\n\n## Design decisions (apply as-is)\n\n- API: add an overload `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg method delegate with `_ => ()`. Do not add a default parameter and do not change the existing signature: `core` runs MiMa (`enableMimaSettings` in build.sbt) and only additive changes are safe. Same shape in `FlowReactiveStreams`.\n- Subscriber: `onNext(t)` = `channel.sendOrClosed(t)` and if the result is `ChannelClosed` call `onDiscard(t)`. `onError(e)` = store `e` in an `AtomicReference[Throwable]`, then `channel.doneOrClosed().discard`. `onComplete()` = `channel.doneOrClosed().discard`. `onSubscribe(s)` = set `subscriptionRef`, then if a `cancelled: AtomicBoolean` is already set call `s.cancel()` else `s.request(capacity)` (both sides use volatile atomics, so at least one side sees the other; a double `cancel()` is allowed by the spec).\n- Consumer loop: on `ChannelClosed.Done`, read the stored error; if non-null throw it, else finish. Keep the `ChannelClosed.Error` branch for safety.\n- Cleanup, run on any Throwable escaping the loop (the stored publisher error included; cancel after onError is a spec-allowed no-op): set `cancelled`, cancel the subscription if `subscriptionRef` is non-null, `channel.doneOrClosed().discard`, then drain: loop `receiveOrClosed()` until `Done`, calling `onDiscard` for each element. The drain is bounded (channel is done, at most capacity elements plus in-flight sends) but blocking; if `InterruptedException` is thrown mid-drain, catch it, keep draining, and call `Thread.currentThread().interrupt()` once the drain finishes, so no element is lost on a second interrupt. Then rethrow the original exception. An element that was passed to `emit` and whose `emit` threw counts as emitted (downstream owns it): do not discard it.\n- Scaladoc on the new overload: state the exactly-once invariant, that `onDiscard` may be called from publisher threads, that it must not throw, and that on publisher error buffered elements are emitted before the error is rethrown (this is a deliberate behaviour change for the one-arg variant too; mention it in its doc).\n\n## Test publisher helper (write it in the core test, package `ox.flow.reactive`)\n\nA `ManualPublisher[T] extends java.util.concurrent.Flow.Publisher[T]` with: `@volatile var subscriber: Subscriber[? >: T]`, `requested: AtomicLong`, `cancelled: AtomicBoolean`, a `subscribed: CountDownLatch(1)`; `subscribe(s)` stores `s`, calls `s.onSubscribe(new Subscription { request(n) => requested.addAndGet(n); cancel() => cancelled.set(true) })`, counts down the latch. Helper methods `next(t)`, `error(e)`, `complete()` forward to the subscriber from the test thread. For the exactly-once stress test, a second variant that runs its own thread emitting `1..n` while `sent < requested` and not cancelled. Run the flow under test in `supervised { forkCancellable { ... } }` or `fork`, await `subscribed`, then drive the publisher. Collect emitted elements with `.tap(emitted += _)` or `runForeach`, discarded with `onDiscard = discarded += _` (use thread-safe collections such as `ConcurrentLinkedQueue` or `java.util.Collections.synchronizedList`, since `onDiscard` runs on other threads).\n\n## Conventions / build\n\n- Scala 3 indentation syntax, `end X` markers on long definitions, `.discard` on ignored results (compiler warns otherwise), scalafmt configured in `.scalafmt.conf` (140-column lines); run `sbt core/scalafmt flowReactiveStreams/scalafmt` after editing.\n- Comments: short, explain non-obvious reasoning only; never restate code. Scaladoc describes usage and invariants, not implementation.\n- Tests: scalatest 3.2.20; the reactive tests use `AnyFunSuite with Matchers` with `test(\"...\")` blocks and Scala 3 colon syntax. `core` runs tests forked (`Test / fork := true`).\n- Commands (sbt 2.0.10): `sbt \"core/testOnly ox.flow.reactive.*\"`, `sbt flowReactiveStreams/test`, `sbt core/mimaReportBinaryIssues` (only meaningful when a previous stable version resolves), `sbt compileDocumentation` for mdoc. Never commit; the user commits."}},{"id":[{"name":"Task: Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","occurrence":0}],"resultJson":{"findings":[]}}],"stageStarts":[]} \ No newline at end of file +{"header":{"startingBranch":"zippy-astrodon-94aa2fb6","branch":"zippy-astrodon-94aa2fb6","branchMode":"Reused","userPrompt":"Implement GitHub issue softwaremill/ox#528: \"Flow.fromPublisher: allow releasing elements which are buffered but not emitted\".\n\nProblem: `Flow.fromPublisher` (core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala) buffers elements in a channel. Some of them never reach the consumer:\n- when the publisher fails, `channel.error` drops the buffered elements\n- when the consumer stops early (an exception, an interruption), the buffered elements are never received\n- elements can still arrive (onNext) after the subscription is cancelled\n\nIf elements hold resources (e.g. Netty's reference-counted ByteBufs), nobody releases them, so they leak.\n\nGoal: add an optional callback, e.g. `fromPublisher(p, onDiscard: T => Unit)`, called exactly once for each element received from the publisher but not emitted downstream. Existing call sites without the callback must keep compiling and behaving the same.\n\nSuggested implementation: on publisher error, store the error and call `channel.done()` (so buffered elements are kept and emitted before the error is rethrown). When the flow ends early, cancel the subscription, close the channel, and drain it, calling the callback for each drained element. In `onNext`, call the callback if the channel is already closed. Make sure there are no races where an element is neither emitted nor discarded.\n\nAlso update the variant in flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala (org.reactivestreams Publisher) to accept the same optional callback, and update any docs (doc/ directory) describing fromPublisher.\n\nAcceptance criteria:\n- tests covering: publisher error with buffered elements (elements still emitted, then error); consumer stopping early (exception / interruption) with buffered elements -> callback called for each; elements arriving after cancellation -> callback called; every element is either emitted or discarded, never both or neither.\n- `sbt compile test` passes for affected modules (core, flow-reactive-streams).\n- Code follows existing style in the repo; concise doc comments.\n\nContext: tapir currently tracks unconsumed elements itself using a lock: https://github.com/softwaremill/tapir/pull/5573","flow":{"type":"Catalog","name":"implement.sc"},"startingCommit":"664f01730cda34fd88e8c3db3deb600934eb1d1e"},"entries":[{"id":[{"name":"Plan","occurrence":0}],"resultJson":{"id":"ox-528-from-publisher-on-discard","description":"`Flow.fromPublisher` buffers elements received from a reactive `Publisher` in a channel. Today, elements that are buffered but never emitted are silently lost: `channel.error` drops the buffer on publisher error, a consumer that stops early (exception, interruption, `take`) never receives what is buffered, and `onNext` calls that arrive after the subscription is cancelled are dropped by `channel.send` throwing. When elements own resources (Netty reference-counted ByteBufs in tapir), this leaks them. Tapir currently works around it with its own lock-based tracking (softwaremill/tapir#5573).\n\nThis epic adds an optional `onDiscard: T => Unit` callback to `Flow.fromPublisher` (core, JDK `Flow.Publisher`) and `FlowReactiveStreams.fromPublisher` (flow-reactive-streams, `org.reactivestreams.Publisher`), with the invariant that every element received from the publisher is either emitted downstream or passed to `onDiscard`, exactly once, never both or neither. Publisher errors no longer drop buffered elements: the error is stored, the channel is completed with `done()`, buffered elements are emitted, and then the error is rethrown. When the flow ends early, the subscription is cancelled, the channel is closed and drained through `onDiscard`, and any late `onNext` goes to `onDiscard` as well. Existing single-argument call sites keep compiling (the new parameter is an overload, which also keeps MiMa happy for `core`). Docs in `doc/streaming/flows.md` describe the new callback.","tasks":[{"title":"Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","description":"In core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala add `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg `fromPublisher` delegate to it with `_ => ()`. Implement per the brief's design decisions: `onNext` uses `sendOrClosed` and discards on closed; `onError` stores the throwable in an AtomicReference and calls `doneOrClosed`; `onComplete` calls `doneOrClosed`; `onSubscribe` cancels immediately if a `cancelled` flag is already set, else requests `capacity`; the consumer loop rethrows the stored error when it sees `Done`; on any Throwable escaping the loop: set `cancelled`, cancel the subscription if present, `doneOrClosed` the channel, drain it calling `onDiscard` per element (keep draining across an InterruptedException and re-interrupt the thread afterwards), rethrow. Preserve the existing batched demand logic. Update the scaladoc of both overloads (exactly-once invariant, callback runs on publisher threads and must not throw, buffered elements are now emitted before a publisher error is rethrown).\n\nTests, in a new core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardTest.scala (AnyFunSuite with Matchers), using a hand-written `ManualPublisher[T]` helper as described in the brief (keep the helper in this file or a sibling file in the same package so Task 2 can reuse it):\n1. Publisher error with buffered elements: push 1,2,3 then `error(boom)` before/while the consumer runs; `runToList` throws boom and all of 1,2,3 were emitted; `onDiscard` never called.\n2. Publisher error with nothing buffered still throws.\n3. Consumer exception: `.map(x => if x == 2 then throw boom else x)`, push 1..5; `runToList` throws boom; discarded == 3,4,5 (order-independent compare), subscription cancelled.\n4. Consumer stops via `take(2)`: result List(1,2), discarded == 3,4,5, cancelled.\n5. Consumer interrupted: run `runForeach(x => { received.release(); gate.acquire() })` in `forkCancellable`, push 1..5, wait for `received`, `cancel()`; discarded == 2,3,4,5, cancelled.\n6. Late onNext after cancellation: after any of the above, call `next(6)` on the publisher; `onDiscard(6)` is called synchronously and nothing throws.\n7. Existing one-arg overload: compiles unchanged and `FlowPublisherPekkoTest` still passes.\nRun `sbt \"core/testOnly ox.flow.reactive.*\"` and `sbt core/scalafmt`; all green."},{"title":"Exactly-once stress test: every element is emitted or discarded, never both or neither","description":"Add a test (same file as Task 1 or a sibling in core/src/test/scala/ox/flow/reactive/) that uses a demand-respecting publisher running on its own thread: it emits 1..N (N around 1000) only while `sent < requested` and not cancelled, calling `onNext` from that thread, then `onComplete`. Compose `Flow.fromPublisher(p, discarded.add)` with a consumer that stops at a random point (e.g. `take(k)` for random k, and separately a `map` that throws at a random element, and separately a `forkCancellable` cancelled after a random short delay). Collect emitted elements via `.tap(emitted.add)` into thread-safe collections. After the flow finishes and the publisher thread is joined, assert: `emitted` and `discarded` are disjoint, their union equals exactly the set of elements the publisher actually sent, and no element appears twice. Repeat each scenario in a loop (e.g. 20 iterations) with a small `given BufferCapacity = BufferCapacity(4)` to make buffer-full races likely. The publisher's late `onNext` calls (after cancellation) must land in `discarded`. If this exposes a race in Task 1's implementation, fix it in FlowCompanionReactiveOps.scala as part of this task. Run `sbt \"core/testOnly ox.flow.reactive.*\"`."},{"title":"Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","description":"In flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala add `def fromPublisher[T](p: org.reactivestreams.Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p), onDiscard)`, keep the one-arg overload, and update its scaladoc to point at the core overload for the callback semantics. Create flow-reactive-streams/src/test/scala/ox/flow/reactive/FlowReactiveStreamsFromPublisherTest.scala (AnyFunSuite with Matchers) with a small hand-written `org.reactivestreams.Publisher[Int]` (same shape as the core ManualPublisher, but implementing the reactive-streams interfaces; it cannot be shared because the module does not depend on core's test classpath) and two tests: elements flow through the one-arg variant; with `.take(1)` and the new overload, remaining pushed elements are passed to `onDiscard` and the subscription is cancelled. Run `sbt flowReactiveStreams/test` and `sbt flowReactiveStreams/scalafmt`."},{"title":"Document the onDiscard callback in doc/streaming/flows.md","description":"In doc/streaming/flows.md, section \"### Publisher -> Flow\" (around lines 211-219), add a short paragraph explaining: elements received from the publisher but not emitted downstream (buffered when the publisher fails, when the consumer stops early via exception, interruption or `take`, or arriving after cancellation) can be released by passing a second argument `onDiscard: T => Unit` to `Flow.fromPublisher` / `FlowReactiveStreams.fromPublisher`; it is called exactly once per such element, possibly from the publisher's thread, and should not throw; typical use is releasing reference-counted buffers. Also state that on publisher error, elements already received are emitted before the error is propagated. Keep the prose as short as the surrounding text. If you add a code example, use a ```` ```scala mdoc:compile-only ```` block and verify with `sbt compileDocumentation`; otherwise no build step is needed. Do not touch generated-doc/out."}],"brief":"## Modules and files\n\n- `core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala` — the only implementation file in core. Trait `FlowCompanionReactiveOps` (mixed into `Flow.type`) defines `def fromPublisher[T](p: java.util.concurrent.Flow.Publisher[T])(using BufferCapacity): Flow[T]`. Read the whole file (80 lines) before changing anything. Current structure: `usingEmitInline: emit => unsupervised { channel = BufferCapacity.newChannel[T]; forkUnsupervised { p.subscribe(Subscriber: onSubscribe -> subscriptionRef.set(s); s.request(capacity); onNext -> channel.send; onError -> channel.error; onComplete -> channel.done) }; repeatWhile { channel.receiveOrClosed() match Done -> false / Error -> throw / t -> emit(t); batched subscription.request } }.tapException(cancel subscription)`. The demand logic (`demandThreshold = ceil(capacity/2)`, `toDemand` counter, lazily cached `subscription` read from `subscriptionRef` after the first element) must be preserved.\n- `flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala` — `object FlowReactiveStreams { def fromPublisher[T](p: org.reactivestreams.Publisher[T])(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p)) }`. Module has no `src/test` dir yet; scalatest is already a test dependency (build.sbt `flowReactiveStreams`), it `dependsOn(core)` (compile only, not test->test).\n- `core/src/test/scala/ox/flow/reactive/` — existing reactive tests: `FlowPublisherPekkoTest.scala` (AnyFunSuite + Matchers, uses Pekko `Source...runWith(Sink.asPublisher)` and `FlowAdapters.toFlowPublisher`) and `FlowPublisherTckTest.scala`. Put new fromPublisher tests here. No test anywhere currently uses a hand-written Publisher; write one (see below).\n- `doc/streaming/flows.md` lines 211-219, section \"### Publisher -> Flow\" — the only user doc mentioning `fromPublisher`. `generated-doc/out/**` is mdoc output regenerated at release time by `updateDocs`; do NOT edit it. Code blocks in docs use ```` ```scala mdoc:compile-only ```` and are compiled by `sbt compileDocumentation` (slow; only needed if you add a code block). Doc dependency snippet convention: `// sbt dependency: \"com.softwaremill.ox\" %% \"flow-reactive-streams\" % \"@VERSION@\"`.\n\n## Key APIs (ox.channels, core/src/main/scala/ox/channels/Channel.scala)\n\n- `Sink.sendOrClosed(t): Unit | ChannelClosed` (non-throwing), `Sink.send(t)` throws `ChannelClosedException` when closed.\n- `Sink.doneOrClosed(): Unit | ChannelClosed` — \"buffered values will be delivered; send operations in progress complete normally when a receiver arrives\"; returns `ChannelClosed` instead of throwing if already closed. `Sink.errorOrClosed(reason)` — drops buffered values (that is the bug). Use the `*OrClosed` variants everywhere in the subscriber: a misbehaving publisher may call onError/onComplete twice.\n- `Source.receiveOrClosed(): T | ChannelClosed` (blocking); match `case ChannelClosed.Done => ...; case e: ChannelClosed.Error => throw e.toThrowable; case t: T @unchecked => ...`. `Source.tryReceiveOrClosed()` is non-blocking but may spuriously return None, so it is not suitable for a drain loop.\n- `Source.foreachOrError(f)` / `drainOrError()` from `SourceDrainOps.scala` loop until Done; usable for the drain, but write the drain inline so you can handle `InterruptedException` (see below).\n- jox semantics (com.softwaremill.jox channels 1.1.2, verified from source): after `done()`, buffered elements stay receivable; a receiver observing Done has already seen every element whose `send` started before `done()` (an in-progress sender is waited for); a `send` racing with `done()` either lands in the buffer or returns closed. Hence \"drain until Done, then `sendOrClosed` returns closed\" gives exactly-once discard with no extra locking.\n- `BufferCapacity`: `given default: BufferCapacity = 16`; `BufferCapacity.newChannel[T]`; `summon[BufferCapacity].toInt` (import `ox.channels.toInt`). Tests can override with `given BufferCapacity = BufferCapacity(4)`.\n- `ox.util`: `t.tapException(f)` runs `f(e)` for any Throwable and rethrows `e` (exceptions from `f` added as suppressed). `ox.pipe`, `ox.discard`, `ox.repeatWhile` already imported in the file.\n- `FlowOps.take(n)` stops upstream by throwing a private `ControlThrowable` (`abortTake`) out of `emit`, caught in `take` itself. So \"consumer stops early\" includes `take`, and the cleanup path must run for every Throwable, not just NonFatal. `tapException`/`catch case e: Throwable` already does this.\n- `unsupervised { ... }` + `forkUnsupervised { ... }` (import `ox.unsupervised`, `ox.forkUnsupervised`): scope end interrupts still-running forks once. `onNext`/`onError`/`onComplete` run on the publisher's threads (or on the `p.subscribe` fork for synchronous publishers), never on the consumer thread.\n- Test helpers: `ox.forkCancellable { ... }` inside `supervised` returns a `CancellableFork` with `cancel(): Either[Throwable, T]` (interrupts once, joins). Interruption inside the flow surfaces as `InterruptedException` thrown from a blocking call (e.g. `Semaphore.acquire`, `channel.receiveOrClosed`).\n\n## Design decisions (apply as-is)\n\n- API: add an overload `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg method delegate with `_ => ()`. Do not add a default parameter and do not change the existing signature: `core` runs MiMa (`enableMimaSettings` in build.sbt) and only additive changes are safe. Same shape in `FlowReactiveStreams`.\n- Subscriber: `onNext(t)` = `channel.sendOrClosed(t)` and if the result is `ChannelClosed` call `onDiscard(t)`. `onError(e)` = store `e` in an `AtomicReference[Throwable]`, then `channel.doneOrClosed().discard`. `onComplete()` = `channel.doneOrClosed().discard`. `onSubscribe(s)` = set `subscriptionRef`, then if a `cancelled: AtomicBoolean` is already set call `s.cancel()` else `s.request(capacity)` (both sides use volatile atomics, so at least one side sees the other; a double `cancel()` is allowed by the spec).\n- Consumer loop: on `ChannelClosed.Done`, read the stored error; if non-null throw it, else finish. Keep the `ChannelClosed.Error` branch for safety.\n- Cleanup, run on any Throwable escaping the loop (the stored publisher error included; cancel after onError is a spec-allowed no-op): set `cancelled`, cancel the subscription if `subscriptionRef` is non-null, `channel.doneOrClosed().discard`, then drain: loop `receiveOrClosed()` until `Done`, calling `onDiscard` for each element. The drain is bounded (channel is done, at most capacity elements plus in-flight sends) but blocking; if `InterruptedException` is thrown mid-drain, catch it, keep draining, and call `Thread.currentThread().interrupt()` once the drain finishes, so no element is lost on a second interrupt. Then rethrow the original exception. An element that was passed to `emit` and whose `emit` threw counts as emitted (downstream owns it): do not discard it.\n- Scaladoc on the new overload: state the exactly-once invariant, that `onDiscard` may be called from publisher threads, that it must not throw, and that on publisher error buffered elements are emitted before the error is rethrown (this is a deliberate behaviour change for the one-arg variant too; mention it in its doc).\n\n## Test publisher helper (write it in the core test, package `ox.flow.reactive`)\n\nA `ManualPublisher[T] extends java.util.concurrent.Flow.Publisher[T]` with: `@volatile var subscriber: Subscriber[? >: T]`, `requested: AtomicLong`, `cancelled: AtomicBoolean`, a `subscribed: CountDownLatch(1)`; `subscribe(s)` stores `s`, calls `s.onSubscribe(new Subscription { request(n) => requested.addAndGet(n); cancel() => cancelled.set(true) })`, counts down the latch. Helper methods `next(t)`, `error(e)`, `complete()` forward to the subscriber from the test thread. For the exactly-once stress test, a second variant that runs its own thread emitting `1..n` while `sent < requested` and not cancelled. Run the flow under test in `supervised { forkCancellable { ... } }` or `fork`, await `subscribed`, then drive the publisher. Collect emitted elements with `.tap(emitted += _)` or `runForeach`, discarded with `onDiscard = discarded += _` (use thread-safe collections such as `ConcurrentLinkedQueue` or `java.util.Collections.synchronizedList`, since `onDiscard` runs on other threads).\n\n## Conventions / build\n\n- Scala 3 indentation syntax, `end X` markers on long definitions, `.discard` on ignored results (compiler warns otherwise), scalafmt configured in `.scalafmt.conf` (140-column lines); run `sbt core/scalafmt flowReactiveStreams/scalafmt` after editing.\n- Comments: short, explain non-obvious reasoning only; never restate code. Scaladoc describes usage and invariants, not implementation.\n- Tests: scalatest 3.2.20; the reactive tests use `AnyFunSuite with Matchers` with `test(\"...\")` blocks and Scala 3 colon syntax. `core` runs tests forked (`Test / fork := true`).\n- Commands (sbt 2.0.10): `sbt \"core/testOnly ox.flow.reactive.*\"`, `sbt flowReactiveStreams/test`, `sbt core/mimaReportBinaryIssues` (only meaningful when a previous stable version resolves), `sbt compileDocumentation` for mdoc. Never commit; the user commits."}},{"id":[{"name":"Task: Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Exactly-once stress test: every element is emitted or discarded, never both or neither","occurrence":0}],"resultJson":{"findings":[]}}],"stageStarts":[]} \ No newline at end of file diff --git a/core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardStressTest.scala b/core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardStressTest.scala new file mode 100644 index 00000000..d2d86284 --- /dev/null +++ b/core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardStressTest.scala @@ -0,0 +1,95 @@ +package ox.flow.reactive + +import org.scalatest.funsuite.AnyFunSuite +import org.scalatest.matchers.should.Matchers +import ox.* +import ox.channels.BufferCapacity +import ox.flow.Flow + +import java.util.concurrent.ConcurrentLinkedQueue +import java.util.concurrent.Flow.Publisher +import java.util.concurrent.Flow.Subscriber +import java.util.concurrent.Flow.Subscription +import java.util.concurrent.atomic.AtomicBoolean +import java.util.concurrent.atomic.AtomicLong +import scala.collection.mutable.ArrayBuffer +import scala.concurrent.duration.* +import scala.jdk.CollectionConverters.* +import scala.util.Random +import scala.util.Try + +class FlowFromPublisherDiscardStressTest extends AnyFunSuite with Matchers: + private given BufferCapacity = BufferCapacity(4) + private val N = 1000 + private val Iterations = 20 + + private def checkExactlyOnce(consume: Flow[Int] => Unit): Unit = + val publisher = new ThreadedPublisher(N) + val emitted = new ConcurrentLinkedQueue[Int]() + val discarded = new ConcurrentLinkedQueue[Int]() + + consume(Flow.fromPublisher(publisher, discarded.add(_).discard).tap(emitted.add(_).discard)) + val sent = publisher.join() + + val handled = emitted.asScala.toList ++ discarded.asScala.toList + // `sent` has no duplicates, so this also catches elements emitted or discarded twice + withClue("duplicated:")(handled.diff(sent) shouldBe empty) + withClue("lost:")(sent.diff(handled) shouldBe empty) + end checkExactlyOnce + + test("should emit or discard each element exactly once when stopping early"): + for _ <- 1 to Iterations do + val k = Random.nextInt(N) + 1 + withClue(s"k = $k"): + checkExactlyOnce(_.take(k).runDrain()) + + test("should emit or discard each element exactly once when the consumer throws"): + for _ <- 1 to Iterations do + val failAt = Random.nextInt(N) + 1 + withClue(s"failAt = $failAt"): + checkExactlyOnce(f => Try(f.map(x => if x == failAt then throw new RuntimeException("boom") else x).runDrain()).discard) + + test("should emit or discard each element exactly once when the consumer is interrupted"): + for _ <- 1 to Iterations do + val delay = Random.nextInt(2000).micros + withClue(s"delay = $delay"): + checkExactlyOnce: f => + supervised: + val result = forkCancellable(f.runDrain()) + sleep(delay) + result.cancel().discard +end FlowFromPublisherDiscardStressTest + +/** Emits `1..n` from its own thread, respecting demand, then completes. After observing cancellation, sends a few more elements, as + * allowed by the spec. + */ +private class ThreadedPublisher(n: Int) extends Publisher[Int]: + private val requested = new AtomicLong(0) + private val cancelled = new AtomicBoolean(false) + // only accessed by the publisher thread; read after it's joined + private val sent = ArrayBuffer[Int]() + @volatile private var thread: Thread = null + + override def subscribe(s: Subscriber[? >: Int]): Unit = + thread = new Thread(() => + var next = 1 + def send(): Unit = + s.onNext(next) + sent += next + next += 1 + + while next <= n && !cancelled.get() do + if next <= requested.get() then send() else Thread.onSpinWait() + if cancelled.get() then (1 to 3).foreach(_ => send()) else s.onComplete() + ) + s.onSubscribe(new Subscription: + override def request(k: Long): Unit = requested.addAndGet(k).discard + override def cancel(): Unit = cancelled.set(true)) + thread.start() + end subscribe + + /** Waits for the publisher thread to finish, and returns the elements that were sent. Must be called after the flow completes. */ + def join(): List[Int] = + if thread != null then thread.join() + sent.toList +end ThreadedPublisher From 0347580cd1de54a77a583951f49f63a65a8cba11 Mon Sep 17 00:00:00 2001 From: Adam Warski Date: Fri, 9 Oct 2026 15:25:19 +0000 Subject: [PATCH 05/11] Add onDiscard overload to FlowReactiveStreams.fromPublisher --- .orca/runs/0a3279c8e3d7.progress.json | 2 +- .../flowReactiveStreamsExtension.scala | 17 +++++- ...FlowReactiveStreamsFromPublisherTest.scala | 53 +++++++++++++++++++ 3 files changed, 69 insertions(+), 3 deletions(-) create mode 100644 flow-reactive-streams/src/test/scala/ox/flow/reactive/FlowReactiveStreamsFromPublisherTest.scala diff --git a/.orca/runs/0a3279c8e3d7.progress.json b/.orca/runs/0a3279c8e3d7.progress.json index 9eea00c4..2681ba74 100644 --- a/.orca/runs/0a3279c8e3d7.progress.json +++ b/.orca/runs/0a3279c8e3d7.progress.json @@ -1 +1 @@ -{"header":{"startingBranch":"zippy-astrodon-94aa2fb6","branch":"zippy-astrodon-94aa2fb6","branchMode":"Reused","userPrompt":"Implement GitHub issue softwaremill/ox#528: \"Flow.fromPublisher: allow releasing elements which are buffered but not emitted\".\n\nProblem: `Flow.fromPublisher` (core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala) buffers elements in a channel. Some of them never reach the consumer:\n- when the publisher fails, `channel.error` drops the buffered elements\n- when the consumer stops early (an exception, an interruption), the buffered elements are never received\n- elements can still arrive (onNext) after the subscription is cancelled\n\nIf elements hold resources (e.g. Netty's reference-counted ByteBufs), nobody releases them, so they leak.\n\nGoal: add an optional callback, e.g. `fromPublisher(p, onDiscard: T => Unit)`, called exactly once for each element received from the publisher but not emitted downstream. Existing call sites without the callback must keep compiling and behaving the same.\n\nSuggested implementation: on publisher error, store the error and call `channel.done()` (so buffered elements are kept and emitted before the error is rethrown). When the flow ends early, cancel the subscription, close the channel, and drain it, calling the callback for each drained element. In `onNext`, call the callback if the channel is already closed. Make sure there are no races where an element is neither emitted nor discarded.\n\nAlso update the variant in flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala (org.reactivestreams Publisher) to accept the same optional callback, and update any docs (doc/ directory) describing fromPublisher.\n\nAcceptance criteria:\n- tests covering: publisher error with buffered elements (elements still emitted, then error); consumer stopping early (exception / interruption) with buffered elements -> callback called for each; elements arriving after cancellation -> callback called; every element is either emitted or discarded, never both or neither.\n- `sbt compile test` passes for affected modules (core, flow-reactive-streams).\n- Code follows existing style in the repo; concise doc comments.\n\nContext: tapir currently tracks unconsumed elements itself using a lock: https://github.com/softwaremill/tapir/pull/5573","flow":{"type":"Catalog","name":"implement.sc"},"startingCommit":"664f01730cda34fd88e8c3db3deb600934eb1d1e"},"entries":[{"id":[{"name":"Plan","occurrence":0}],"resultJson":{"id":"ox-528-from-publisher-on-discard","description":"`Flow.fromPublisher` buffers elements received from a reactive `Publisher` in a channel. Today, elements that are buffered but never emitted are silently lost: `channel.error` drops the buffer on publisher error, a consumer that stops early (exception, interruption, `take`) never receives what is buffered, and `onNext` calls that arrive after the subscription is cancelled are dropped by `channel.send` throwing. When elements own resources (Netty reference-counted ByteBufs in tapir), this leaks them. Tapir currently works around it with its own lock-based tracking (softwaremill/tapir#5573).\n\nThis epic adds an optional `onDiscard: T => Unit` callback to `Flow.fromPublisher` (core, JDK `Flow.Publisher`) and `FlowReactiveStreams.fromPublisher` (flow-reactive-streams, `org.reactivestreams.Publisher`), with the invariant that every element received from the publisher is either emitted downstream or passed to `onDiscard`, exactly once, never both or neither. Publisher errors no longer drop buffered elements: the error is stored, the channel is completed with `done()`, buffered elements are emitted, and then the error is rethrown. When the flow ends early, the subscription is cancelled, the channel is closed and drained through `onDiscard`, and any late `onNext` goes to `onDiscard` as well. Existing single-argument call sites keep compiling (the new parameter is an overload, which also keeps MiMa happy for `core`). Docs in `doc/streaming/flows.md` describe the new callback.","tasks":[{"title":"Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","description":"In core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala add `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg `fromPublisher` delegate to it with `_ => ()`. Implement per the brief's design decisions: `onNext` uses `sendOrClosed` and discards on closed; `onError` stores the throwable in an AtomicReference and calls `doneOrClosed`; `onComplete` calls `doneOrClosed`; `onSubscribe` cancels immediately if a `cancelled` flag is already set, else requests `capacity`; the consumer loop rethrows the stored error when it sees `Done`; on any Throwable escaping the loop: set `cancelled`, cancel the subscription if present, `doneOrClosed` the channel, drain it calling `onDiscard` per element (keep draining across an InterruptedException and re-interrupt the thread afterwards), rethrow. Preserve the existing batched demand logic. Update the scaladoc of both overloads (exactly-once invariant, callback runs on publisher threads and must not throw, buffered elements are now emitted before a publisher error is rethrown).\n\nTests, in a new core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardTest.scala (AnyFunSuite with Matchers), using a hand-written `ManualPublisher[T]` helper as described in the brief (keep the helper in this file or a sibling file in the same package so Task 2 can reuse it):\n1. Publisher error with buffered elements: push 1,2,3 then `error(boom)` before/while the consumer runs; `runToList` throws boom and all of 1,2,3 were emitted; `onDiscard` never called.\n2. Publisher error with nothing buffered still throws.\n3. Consumer exception: `.map(x => if x == 2 then throw boom else x)`, push 1..5; `runToList` throws boom; discarded == 3,4,5 (order-independent compare), subscription cancelled.\n4. Consumer stops via `take(2)`: result List(1,2), discarded == 3,4,5, cancelled.\n5. Consumer interrupted: run `runForeach(x => { received.release(); gate.acquire() })` in `forkCancellable`, push 1..5, wait for `received`, `cancel()`; discarded == 2,3,4,5, cancelled.\n6. Late onNext after cancellation: after any of the above, call `next(6)` on the publisher; `onDiscard(6)` is called synchronously and nothing throws.\n7. Existing one-arg overload: compiles unchanged and `FlowPublisherPekkoTest` still passes.\nRun `sbt \"core/testOnly ox.flow.reactive.*\"` and `sbt core/scalafmt`; all green."},{"title":"Exactly-once stress test: every element is emitted or discarded, never both or neither","description":"Add a test (same file as Task 1 or a sibling in core/src/test/scala/ox/flow/reactive/) that uses a demand-respecting publisher running on its own thread: it emits 1..N (N around 1000) only while `sent < requested` and not cancelled, calling `onNext` from that thread, then `onComplete`. Compose `Flow.fromPublisher(p, discarded.add)` with a consumer that stops at a random point (e.g. `take(k)` for random k, and separately a `map` that throws at a random element, and separately a `forkCancellable` cancelled after a random short delay). Collect emitted elements via `.tap(emitted.add)` into thread-safe collections. After the flow finishes and the publisher thread is joined, assert: `emitted` and `discarded` are disjoint, their union equals exactly the set of elements the publisher actually sent, and no element appears twice. Repeat each scenario in a loop (e.g. 20 iterations) with a small `given BufferCapacity = BufferCapacity(4)` to make buffer-full races likely. The publisher's late `onNext` calls (after cancellation) must land in `discarded`. If this exposes a race in Task 1's implementation, fix it in FlowCompanionReactiveOps.scala as part of this task. Run `sbt \"core/testOnly ox.flow.reactive.*\"`."},{"title":"Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","description":"In flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala add `def fromPublisher[T](p: org.reactivestreams.Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p), onDiscard)`, keep the one-arg overload, and update its scaladoc to point at the core overload for the callback semantics. Create flow-reactive-streams/src/test/scala/ox/flow/reactive/FlowReactiveStreamsFromPublisherTest.scala (AnyFunSuite with Matchers) with a small hand-written `org.reactivestreams.Publisher[Int]` (same shape as the core ManualPublisher, but implementing the reactive-streams interfaces; it cannot be shared because the module does not depend on core's test classpath) and two tests: elements flow through the one-arg variant; with `.take(1)` and the new overload, remaining pushed elements are passed to `onDiscard` and the subscription is cancelled. Run `sbt flowReactiveStreams/test` and `sbt flowReactiveStreams/scalafmt`."},{"title":"Document the onDiscard callback in doc/streaming/flows.md","description":"In doc/streaming/flows.md, section \"### Publisher -> Flow\" (around lines 211-219), add a short paragraph explaining: elements received from the publisher but not emitted downstream (buffered when the publisher fails, when the consumer stops early via exception, interruption or `take`, or arriving after cancellation) can be released by passing a second argument `onDiscard: T => Unit` to `Flow.fromPublisher` / `FlowReactiveStreams.fromPublisher`; it is called exactly once per such element, possibly from the publisher's thread, and should not throw; typical use is releasing reference-counted buffers. Also state that on publisher error, elements already received are emitted before the error is propagated. Keep the prose as short as the surrounding text. If you add a code example, use a ```` ```scala mdoc:compile-only ```` block and verify with `sbt compileDocumentation`; otherwise no build step is needed. Do not touch generated-doc/out."}],"brief":"## Modules and files\n\n- `core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala` — the only implementation file in core. Trait `FlowCompanionReactiveOps` (mixed into `Flow.type`) defines `def fromPublisher[T](p: java.util.concurrent.Flow.Publisher[T])(using BufferCapacity): Flow[T]`. Read the whole file (80 lines) before changing anything. Current structure: `usingEmitInline: emit => unsupervised { channel = BufferCapacity.newChannel[T]; forkUnsupervised { p.subscribe(Subscriber: onSubscribe -> subscriptionRef.set(s); s.request(capacity); onNext -> channel.send; onError -> channel.error; onComplete -> channel.done) }; repeatWhile { channel.receiveOrClosed() match Done -> false / Error -> throw / t -> emit(t); batched subscription.request } }.tapException(cancel subscription)`. The demand logic (`demandThreshold = ceil(capacity/2)`, `toDemand` counter, lazily cached `subscription` read from `subscriptionRef` after the first element) must be preserved.\n- `flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala` — `object FlowReactiveStreams { def fromPublisher[T](p: org.reactivestreams.Publisher[T])(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p)) }`. Module has no `src/test` dir yet; scalatest is already a test dependency (build.sbt `flowReactiveStreams`), it `dependsOn(core)` (compile only, not test->test).\n- `core/src/test/scala/ox/flow/reactive/` — existing reactive tests: `FlowPublisherPekkoTest.scala` (AnyFunSuite + Matchers, uses Pekko `Source...runWith(Sink.asPublisher)` and `FlowAdapters.toFlowPublisher`) and `FlowPublisherTckTest.scala`. Put new fromPublisher tests here. No test anywhere currently uses a hand-written Publisher; write one (see below).\n- `doc/streaming/flows.md` lines 211-219, section \"### Publisher -> Flow\" — the only user doc mentioning `fromPublisher`. `generated-doc/out/**` is mdoc output regenerated at release time by `updateDocs`; do NOT edit it. Code blocks in docs use ```` ```scala mdoc:compile-only ```` and are compiled by `sbt compileDocumentation` (slow; only needed if you add a code block). Doc dependency snippet convention: `// sbt dependency: \"com.softwaremill.ox\" %% \"flow-reactive-streams\" % \"@VERSION@\"`.\n\n## Key APIs (ox.channels, core/src/main/scala/ox/channels/Channel.scala)\n\n- `Sink.sendOrClosed(t): Unit | ChannelClosed` (non-throwing), `Sink.send(t)` throws `ChannelClosedException` when closed.\n- `Sink.doneOrClosed(): Unit | ChannelClosed` — \"buffered values will be delivered; send operations in progress complete normally when a receiver arrives\"; returns `ChannelClosed` instead of throwing if already closed. `Sink.errorOrClosed(reason)` — drops buffered values (that is the bug). Use the `*OrClosed` variants everywhere in the subscriber: a misbehaving publisher may call onError/onComplete twice.\n- `Source.receiveOrClosed(): T | ChannelClosed` (blocking); match `case ChannelClosed.Done => ...; case e: ChannelClosed.Error => throw e.toThrowable; case t: T @unchecked => ...`. `Source.tryReceiveOrClosed()` is non-blocking but may spuriously return None, so it is not suitable for a drain loop.\n- `Source.foreachOrError(f)` / `drainOrError()` from `SourceDrainOps.scala` loop until Done; usable for the drain, but write the drain inline so you can handle `InterruptedException` (see below).\n- jox semantics (com.softwaremill.jox channels 1.1.2, verified from source): after `done()`, buffered elements stay receivable; a receiver observing Done has already seen every element whose `send` started before `done()` (an in-progress sender is waited for); a `send` racing with `done()` either lands in the buffer or returns closed. Hence \"drain until Done, then `sendOrClosed` returns closed\" gives exactly-once discard with no extra locking.\n- `BufferCapacity`: `given default: BufferCapacity = 16`; `BufferCapacity.newChannel[T]`; `summon[BufferCapacity].toInt` (import `ox.channels.toInt`). Tests can override with `given BufferCapacity = BufferCapacity(4)`.\n- `ox.util`: `t.tapException(f)` runs `f(e)` for any Throwable and rethrows `e` (exceptions from `f` added as suppressed). `ox.pipe`, `ox.discard`, `ox.repeatWhile` already imported in the file.\n- `FlowOps.take(n)` stops upstream by throwing a private `ControlThrowable` (`abortTake`) out of `emit`, caught in `take` itself. So \"consumer stops early\" includes `take`, and the cleanup path must run for every Throwable, not just NonFatal. `tapException`/`catch case e: Throwable` already does this.\n- `unsupervised { ... }` + `forkUnsupervised { ... }` (import `ox.unsupervised`, `ox.forkUnsupervised`): scope end interrupts still-running forks once. `onNext`/`onError`/`onComplete` run on the publisher's threads (or on the `p.subscribe` fork for synchronous publishers), never on the consumer thread.\n- Test helpers: `ox.forkCancellable { ... }` inside `supervised` returns a `CancellableFork` with `cancel(): Either[Throwable, T]` (interrupts once, joins). Interruption inside the flow surfaces as `InterruptedException` thrown from a blocking call (e.g. `Semaphore.acquire`, `channel.receiveOrClosed`).\n\n## Design decisions (apply as-is)\n\n- API: add an overload `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg method delegate with `_ => ()`. Do not add a default parameter and do not change the existing signature: `core` runs MiMa (`enableMimaSettings` in build.sbt) and only additive changes are safe. Same shape in `FlowReactiveStreams`.\n- Subscriber: `onNext(t)` = `channel.sendOrClosed(t)` and if the result is `ChannelClosed` call `onDiscard(t)`. `onError(e)` = store `e` in an `AtomicReference[Throwable]`, then `channel.doneOrClosed().discard`. `onComplete()` = `channel.doneOrClosed().discard`. `onSubscribe(s)` = set `subscriptionRef`, then if a `cancelled: AtomicBoolean` is already set call `s.cancel()` else `s.request(capacity)` (both sides use volatile atomics, so at least one side sees the other; a double `cancel()` is allowed by the spec).\n- Consumer loop: on `ChannelClosed.Done`, read the stored error; if non-null throw it, else finish. Keep the `ChannelClosed.Error` branch for safety.\n- Cleanup, run on any Throwable escaping the loop (the stored publisher error included; cancel after onError is a spec-allowed no-op): set `cancelled`, cancel the subscription if `subscriptionRef` is non-null, `channel.doneOrClosed().discard`, then drain: loop `receiveOrClosed()` until `Done`, calling `onDiscard` for each element. The drain is bounded (channel is done, at most capacity elements plus in-flight sends) but blocking; if `InterruptedException` is thrown mid-drain, catch it, keep draining, and call `Thread.currentThread().interrupt()` once the drain finishes, so no element is lost on a second interrupt. Then rethrow the original exception. An element that was passed to `emit` and whose `emit` threw counts as emitted (downstream owns it): do not discard it.\n- Scaladoc on the new overload: state the exactly-once invariant, that `onDiscard` may be called from publisher threads, that it must not throw, and that on publisher error buffered elements are emitted before the error is rethrown (this is a deliberate behaviour change for the one-arg variant too; mention it in its doc).\n\n## Test publisher helper (write it in the core test, package `ox.flow.reactive`)\n\nA `ManualPublisher[T] extends java.util.concurrent.Flow.Publisher[T]` with: `@volatile var subscriber: Subscriber[? >: T]`, `requested: AtomicLong`, `cancelled: AtomicBoolean`, a `subscribed: CountDownLatch(1)`; `subscribe(s)` stores `s`, calls `s.onSubscribe(new Subscription { request(n) => requested.addAndGet(n); cancel() => cancelled.set(true) })`, counts down the latch. Helper methods `next(t)`, `error(e)`, `complete()` forward to the subscriber from the test thread. For the exactly-once stress test, a second variant that runs its own thread emitting `1..n` while `sent < requested` and not cancelled. Run the flow under test in `supervised { forkCancellable { ... } }` or `fork`, await `subscribed`, then drive the publisher. Collect emitted elements with `.tap(emitted += _)` or `runForeach`, discarded with `onDiscard = discarded += _` (use thread-safe collections such as `ConcurrentLinkedQueue` or `java.util.Collections.synchronizedList`, since `onDiscard` runs on other threads).\n\n## Conventions / build\n\n- Scala 3 indentation syntax, `end X` markers on long definitions, `.discard` on ignored results (compiler warns otherwise), scalafmt configured in `.scalafmt.conf` (140-column lines); run `sbt core/scalafmt flowReactiveStreams/scalafmt` after editing.\n- Comments: short, explain non-obvious reasoning only; never restate code. Scaladoc describes usage and invariants, not implementation.\n- Tests: scalatest 3.2.20; the reactive tests use `AnyFunSuite with Matchers` with `test(\"...\")` blocks and Scala 3 colon syntax. `core` runs tests forked (`Test / fork := true`).\n- Commands (sbt 2.0.10): `sbt \"core/testOnly ox.flow.reactive.*\"`, `sbt flowReactiveStreams/test`, `sbt core/mimaReportBinaryIssues` (only meaningful when a previous stable version resolves), `sbt compileDocumentation` for mdoc. Never commit; the user commits."}},{"id":[{"name":"Task: Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Exactly-once stress test: every element is emitted or discarded, never both or neither","occurrence":0}],"resultJson":{"findings":[]}}],"stageStarts":[]} \ No newline at end of file +{"header":{"startingBranch":"zippy-astrodon-94aa2fb6","branch":"zippy-astrodon-94aa2fb6","branchMode":"Reused","userPrompt":"Implement GitHub issue softwaremill/ox#528: \"Flow.fromPublisher: allow releasing elements which are buffered but not emitted\".\n\nProblem: `Flow.fromPublisher` (core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala) buffers elements in a channel. Some of them never reach the consumer:\n- when the publisher fails, `channel.error` drops the buffered elements\n- when the consumer stops early (an exception, an interruption), the buffered elements are never received\n- elements can still arrive (onNext) after the subscription is cancelled\n\nIf elements hold resources (e.g. Netty's reference-counted ByteBufs), nobody releases them, so they leak.\n\nGoal: add an optional callback, e.g. `fromPublisher(p, onDiscard: T => Unit)`, called exactly once for each element received from the publisher but not emitted downstream. Existing call sites without the callback must keep compiling and behaving the same.\n\nSuggested implementation: on publisher error, store the error and call `channel.done()` (so buffered elements are kept and emitted before the error is rethrown). When the flow ends early, cancel the subscription, close the channel, and drain it, calling the callback for each drained element. In `onNext`, call the callback if the channel is already closed. Make sure there are no races where an element is neither emitted nor discarded.\n\nAlso update the variant in flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala (org.reactivestreams Publisher) to accept the same optional callback, and update any docs (doc/ directory) describing fromPublisher.\n\nAcceptance criteria:\n- tests covering: publisher error with buffered elements (elements still emitted, then error); consumer stopping early (exception / interruption) with buffered elements -> callback called for each; elements arriving after cancellation -> callback called; every element is either emitted or discarded, never both or neither.\n- `sbt compile test` passes for affected modules (core, flow-reactive-streams).\n- Code follows existing style in the repo; concise doc comments.\n\nContext: tapir currently tracks unconsumed elements itself using a lock: https://github.com/softwaremill/tapir/pull/5573","flow":{"type":"Catalog","name":"implement.sc"},"startingCommit":"664f01730cda34fd88e8c3db3deb600934eb1d1e"},"entries":[{"id":[{"name":"Plan","occurrence":0}],"resultJson":{"id":"ox-528-from-publisher-on-discard","description":"`Flow.fromPublisher` buffers elements received from a reactive `Publisher` in a channel. Today, elements that are buffered but never emitted are silently lost: `channel.error` drops the buffer on publisher error, a consumer that stops early (exception, interruption, `take`) never receives what is buffered, and `onNext` calls that arrive after the subscription is cancelled are dropped by `channel.send` throwing. When elements own resources (Netty reference-counted ByteBufs in tapir), this leaks them. Tapir currently works around it with its own lock-based tracking (softwaremill/tapir#5573).\n\nThis epic adds an optional `onDiscard: T => Unit` callback to `Flow.fromPublisher` (core, JDK `Flow.Publisher`) and `FlowReactiveStreams.fromPublisher` (flow-reactive-streams, `org.reactivestreams.Publisher`), with the invariant that every element received from the publisher is either emitted downstream or passed to `onDiscard`, exactly once, never both or neither. Publisher errors no longer drop buffered elements: the error is stored, the channel is completed with `done()`, buffered elements are emitted, and then the error is rethrown. When the flow ends early, the subscription is cancelled, the channel is closed and drained through `onDiscard`, and any late `onNext` goes to `onDiscard` as well. Existing single-argument call sites keep compiling (the new parameter is an overload, which also keeps MiMa happy for `core`). Docs in `doc/streaming/flows.md` describe the new callback.","tasks":[{"title":"Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","description":"In core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala add `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg `fromPublisher` delegate to it with `_ => ()`. Implement per the brief's design decisions: `onNext` uses `sendOrClosed` and discards on closed; `onError` stores the throwable in an AtomicReference and calls `doneOrClosed`; `onComplete` calls `doneOrClosed`; `onSubscribe` cancels immediately if a `cancelled` flag is already set, else requests `capacity`; the consumer loop rethrows the stored error when it sees `Done`; on any Throwable escaping the loop: set `cancelled`, cancel the subscription if present, `doneOrClosed` the channel, drain it calling `onDiscard` per element (keep draining across an InterruptedException and re-interrupt the thread afterwards), rethrow. Preserve the existing batched demand logic. Update the scaladoc of both overloads (exactly-once invariant, callback runs on publisher threads and must not throw, buffered elements are now emitted before a publisher error is rethrown).\n\nTests, in a new core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardTest.scala (AnyFunSuite with Matchers), using a hand-written `ManualPublisher[T]` helper as described in the brief (keep the helper in this file or a sibling file in the same package so Task 2 can reuse it):\n1. Publisher error with buffered elements: push 1,2,3 then `error(boom)` before/while the consumer runs; `runToList` throws boom and all of 1,2,3 were emitted; `onDiscard` never called.\n2. Publisher error with nothing buffered still throws.\n3. Consumer exception: `.map(x => if x == 2 then throw boom else x)`, push 1..5; `runToList` throws boom; discarded == 3,4,5 (order-independent compare), subscription cancelled.\n4. Consumer stops via `take(2)`: result List(1,2), discarded == 3,4,5, cancelled.\n5. Consumer interrupted: run `runForeach(x => { received.release(); gate.acquire() })` in `forkCancellable`, push 1..5, wait for `received`, `cancel()`; discarded == 2,3,4,5, cancelled.\n6. Late onNext after cancellation: after any of the above, call `next(6)` on the publisher; `onDiscard(6)` is called synchronously and nothing throws.\n7. Existing one-arg overload: compiles unchanged and `FlowPublisherPekkoTest` still passes.\nRun `sbt \"core/testOnly ox.flow.reactive.*\"` and `sbt core/scalafmt`; all green."},{"title":"Exactly-once stress test: every element is emitted or discarded, never both or neither","description":"Add a test (same file as Task 1 or a sibling in core/src/test/scala/ox/flow/reactive/) that uses a demand-respecting publisher running on its own thread: it emits 1..N (N around 1000) only while `sent < requested` and not cancelled, calling `onNext` from that thread, then `onComplete`. Compose `Flow.fromPublisher(p, discarded.add)` with a consumer that stops at a random point (e.g. `take(k)` for random k, and separately a `map` that throws at a random element, and separately a `forkCancellable` cancelled after a random short delay). Collect emitted elements via `.tap(emitted.add)` into thread-safe collections. After the flow finishes and the publisher thread is joined, assert: `emitted` and `discarded` are disjoint, their union equals exactly the set of elements the publisher actually sent, and no element appears twice. Repeat each scenario in a loop (e.g. 20 iterations) with a small `given BufferCapacity = BufferCapacity(4)` to make buffer-full races likely. The publisher's late `onNext` calls (after cancellation) must land in `discarded`. If this exposes a race in Task 1's implementation, fix it in FlowCompanionReactiveOps.scala as part of this task. Run `sbt \"core/testOnly ox.flow.reactive.*\"`."},{"title":"Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","description":"In flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala add `def fromPublisher[T](p: org.reactivestreams.Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p), onDiscard)`, keep the one-arg overload, and update its scaladoc to point at the core overload for the callback semantics. Create flow-reactive-streams/src/test/scala/ox/flow/reactive/FlowReactiveStreamsFromPublisherTest.scala (AnyFunSuite with Matchers) with a small hand-written `org.reactivestreams.Publisher[Int]` (same shape as the core ManualPublisher, but implementing the reactive-streams interfaces; it cannot be shared because the module does not depend on core's test classpath) and two tests: elements flow through the one-arg variant; with `.take(1)` and the new overload, remaining pushed elements are passed to `onDiscard` and the subscription is cancelled. Run `sbt flowReactiveStreams/test` and `sbt flowReactiveStreams/scalafmt`."},{"title":"Document the onDiscard callback in doc/streaming/flows.md","description":"In doc/streaming/flows.md, section \"### Publisher -> Flow\" (around lines 211-219), add a short paragraph explaining: elements received from the publisher but not emitted downstream (buffered when the publisher fails, when the consumer stops early via exception, interruption or `take`, or arriving after cancellation) can be released by passing a second argument `onDiscard: T => Unit` to `Flow.fromPublisher` / `FlowReactiveStreams.fromPublisher`; it is called exactly once per such element, possibly from the publisher's thread, and should not throw; typical use is releasing reference-counted buffers. Also state that on publisher error, elements already received are emitted before the error is propagated. Keep the prose as short as the surrounding text. If you add a code example, use a ```` ```scala mdoc:compile-only ```` block and verify with `sbt compileDocumentation`; otherwise no build step is needed. Do not touch generated-doc/out."}],"brief":"## Modules and files\n\n- `core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala` — the only implementation file in core. Trait `FlowCompanionReactiveOps` (mixed into `Flow.type`) defines `def fromPublisher[T](p: java.util.concurrent.Flow.Publisher[T])(using BufferCapacity): Flow[T]`. Read the whole file (80 lines) before changing anything. Current structure: `usingEmitInline: emit => unsupervised { channel = BufferCapacity.newChannel[T]; forkUnsupervised { p.subscribe(Subscriber: onSubscribe -> subscriptionRef.set(s); s.request(capacity); onNext -> channel.send; onError -> channel.error; onComplete -> channel.done) }; repeatWhile { channel.receiveOrClosed() match Done -> false / Error -> throw / t -> emit(t); batched subscription.request } }.tapException(cancel subscription)`. The demand logic (`demandThreshold = ceil(capacity/2)`, `toDemand` counter, lazily cached `subscription` read from `subscriptionRef` after the first element) must be preserved.\n- `flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala` — `object FlowReactiveStreams { def fromPublisher[T](p: org.reactivestreams.Publisher[T])(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p)) }`. Module has no `src/test` dir yet; scalatest is already a test dependency (build.sbt `flowReactiveStreams`), it `dependsOn(core)` (compile only, not test->test).\n- `core/src/test/scala/ox/flow/reactive/` — existing reactive tests: `FlowPublisherPekkoTest.scala` (AnyFunSuite + Matchers, uses Pekko `Source...runWith(Sink.asPublisher)` and `FlowAdapters.toFlowPublisher`) and `FlowPublisherTckTest.scala`. Put new fromPublisher tests here. No test anywhere currently uses a hand-written Publisher; write one (see below).\n- `doc/streaming/flows.md` lines 211-219, section \"### Publisher -> Flow\" — the only user doc mentioning `fromPublisher`. `generated-doc/out/**` is mdoc output regenerated at release time by `updateDocs`; do NOT edit it. Code blocks in docs use ```` ```scala mdoc:compile-only ```` and are compiled by `sbt compileDocumentation` (slow; only needed if you add a code block). Doc dependency snippet convention: `// sbt dependency: \"com.softwaremill.ox\" %% \"flow-reactive-streams\" % \"@VERSION@\"`.\n\n## Key APIs (ox.channels, core/src/main/scala/ox/channels/Channel.scala)\n\n- `Sink.sendOrClosed(t): Unit | ChannelClosed` (non-throwing), `Sink.send(t)` throws `ChannelClosedException` when closed.\n- `Sink.doneOrClosed(): Unit | ChannelClosed` — \"buffered values will be delivered; send operations in progress complete normally when a receiver arrives\"; returns `ChannelClosed` instead of throwing if already closed. `Sink.errorOrClosed(reason)` — drops buffered values (that is the bug). Use the `*OrClosed` variants everywhere in the subscriber: a misbehaving publisher may call onError/onComplete twice.\n- `Source.receiveOrClosed(): T | ChannelClosed` (blocking); match `case ChannelClosed.Done => ...; case e: ChannelClosed.Error => throw e.toThrowable; case t: T @unchecked => ...`. `Source.tryReceiveOrClosed()` is non-blocking but may spuriously return None, so it is not suitable for a drain loop.\n- `Source.foreachOrError(f)` / `drainOrError()` from `SourceDrainOps.scala` loop until Done; usable for the drain, but write the drain inline so you can handle `InterruptedException` (see below).\n- jox semantics (com.softwaremill.jox channels 1.1.2, verified from source): after `done()`, buffered elements stay receivable; a receiver observing Done has already seen every element whose `send` started before `done()` (an in-progress sender is waited for); a `send` racing with `done()` either lands in the buffer or returns closed. Hence \"drain until Done, then `sendOrClosed` returns closed\" gives exactly-once discard with no extra locking.\n- `BufferCapacity`: `given default: BufferCapacity = 16`; `BufferCapacity.newChannel[T]`; `summon[BufferCapacity].toInt` (import `ox.channels.toInt`). Tests can override with `given BufferCapacity = BufferCapacity(4)`.\n- `ox.util`: `t.tapException(f)` runs `f(e)` for any Throwable and rethrows `e` (exceptions from `f` added as suppressed). `ox.pipe`, `ox.discard`, `ox.repeatWhile` already imported in the file.\n- `FlowOps.take(n)` stops upstream by throwing a private `ControlThrowable` (`abortTake`) out of `emit`, caught in `take` itself. So \"consumer stops early\" includes `take`, and the cleanup path must run for every Throwable, not just NonFatal. `tapException`/`catch case e: Throwable` already does this.\n- `unsupervised { ... }` + `forkUnsupervised { ... }` (import `ox.unsupervised`, `ox.forkUnsupervised`): scope end interrupts still-running forks once. `onNext`/`onError`/`onComplete` run on the publisher's threads (or on the `p.subscribe` fork for synchronous publishers), never on the consumer thread.\n- Test helpers: `ox.forkCancellable { ... }` inside `supervised` returns a `CancellableFork` with `cancel(): Either[Throwable, T]` (interrupts once, joins). Interruption inside the flow surfaces as `InterruptedException` thrown from a blocking call (e.g. `Semaphore.acquire`, `channel.receiveOrClosed`).\n\n## Design decisions (apply as-is)\n\n- API: add an overload `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg method delegate with `_ => ()`. Do not add a default parameter and do not change the existing signature: `core` runs MiMa (`enableMimaSettings` in build.sbt) and only additive changes are safe. Same shape in `FlowReactiveStreams`.\n- Subscriber: `onNext(t)` = `channel.sendOrClosed(t)` and if the result is `ChannelClosed` call `onDiscard(t)`. `onError(e)` = store `e` in an `AtomicReference[Throwable]`, then `channel.doneOrClosed().discard`. `onComplete()` = `channel.doneOrClosed().discard`. `onSubscribe(s)` = set `subscriptionRef`, then if a `cancelled: AtomicBoolean` is already set call `s.cancel()` else `s.request(capacity)` (both sides use volatile atomics, so at least one side sees the other; a double `cancel()` is allowed by the spec).\n- Consumer loop: on `ChannelClosed.Done`, read the stored error; if non-null throw it, else finish. Keep the `ChannelClosed.Error` branch for safety.\n- Cleanup, run on any Throwable escaping the loop (the stored publisher error included; cancel after onError is a spec-allowed no-op): set `cancelled`, cancel the subscription if `subscriptionRef` is non-null, `channel.doneOrClosed().discard`, then drain: loop `receiveOrClosed()` until `Done`, calling `onDiscard` for each element. The drain is bounded (channel is done, at most capacity elements plus in-flight sends) but blocking; if `InterruptedException` is thrown mid-drain, catch it, keep draining, and call `Thread.currentThread().interrupt()` once the drain finishes, so no element is lost on a second interrupt. Then rethrow the original exception. An element that was passed to `emit` and whose `emit` threw counts as emitted (downstream owns it): do not discard it.\n- Scaladoc on the new overload: state the exactly-once invariant, that `onDiscard` may be called from publisher threads, that it must not throw, and that on publisher error buffered elements are emitted before the error is rethrown (this is a deliberate behaviour change for the one-arg variant too; mention it in its doc).\n\n## Test publisher helper (write it in the core test, package `ox.flow.reactive`)\n\nA `ManualPublisher[T] extends java.util.concurrent.Flow.Publisher[T]` with: `@volatile var subscriber: Subscriber[? >: T]`, `requested: AtomicLong`, `cancelled: AtomicBoolean`, a `subscribed: CountDownLatch(1)`; `subscribe(s)` stores `s`, calls `s.onSubscribe(new Subscription { request(n) => requested.addAndGet(n); cancel() => cancelled.set(true) })`, counts down the latch. Helper methods `next(t)`, `error(e)`, `complete()` forward to the subscriber from the test thread. For the exactly-once stress test, a second variant that runs its own thread emitting `1..n` while `sent < requested` and not cancelled. Run the flow under test in `supervised { forkCancellable { ... } }` or `fork`, await `subscribed`, then drive the publisher. Collect emitted elements with `.tap(emitted += _)` or `runForeach`, discarded with `onDiscard = discarded += _` (use thread-safe collections such as `ConcurrentLinkedQueue` or `java.util.Collections.synchronizedList`, since `onDiscard` runs on other threads).\n\n## Conventions / build\n\n- Scala 3 indentation syntax, `end X` markers on long definitions, `.discard` on ignored results (compiler warns otherwise), scalafmt configured in `.scalafmt.conf` (140-column lines); run `sbt core/scalafmt flowReactiveStreams/scalafmt` after editing.\n- Comments: short, explain non-obvious reasoning only; never restate code. Scaladoc describes usage and invariants, not implementation.\n- Tests: scalatest 3.2.20; the reactive tests use `AnyFunSuite with Matchers` with `test(\"...\")` blocks and Scala 3 colon syntax. `core` runs tests forked (`Test / fork := true`).\n- Commands (sbt 2.0.10): `sbt \"core/testOnly ox.flow.reactive.*\"`, `sbt flowReactiveStreams/test`, `sbt core/mimaReportBinaryIssues` (only meaningful when a previous stable version resolves), `sbt compileDocumentation` for mdoc. Never commit; the user commits."}},{"id":[{"name":"Task: Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Exactly-once stress test: every element is emitted or discarded, never both or neither","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","occurrence":0}],"resultJson":{"findings":[{"id":"R1.I4.1","title":"sbt Test/compile fails with JGit NoWorkTreeException","reason":{"type":"Declined","text":"This is a problem with the environment, not the code. git sees the worktree as non-bare (`git rev-parse --is-bare-repository` prints false; the git dir is /home/adamw/ox/.git/worktrees/zippy-astrodon-94aa2fb6). The exception comes from the sbt-git plugin, which uses JGit by default and reads this linked worktree as bare while the build loads. Nothing in project/ or the build configures this; `zz-tmp-git.sbt` was my temporary workaround, not the cause. To check the code, I recreated it with `useReadableConsoleGit`, which makes sbt-git use command-line git: `sbt core/Test/compile flowReactiveStreams/Test/compile` succeeded (warnings only, in older test files, no errors) and flowReactiveStreams tests passed. I then deleted the file again. Changing the shared build's git plugin setup is outside the scope of this change."}}]}}],"stageStarts":[]} \ No newline at end of file diff --git a/flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala b/flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala index e0741f41..0b41b796 100644 --- a/flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala +++ b/flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala @@ -18,11 +18,24 @@ extension [A](flow: Flow[A]) end extension object FlowReactiveStreams: - /** This variant returns accepts an implementation of `org.reactivestreams.Publisher`, as opposed to `java.util.concurrent.Flow.Publisher` - * which is supported in the core module. + /** This variant accepts an implementation of `org.reactivestreams.Publisher`, as opposed to `java.util.concurrent.Flow.Publisher` which + * is supported in the core module. + * + * To release resources held by elements which are received but not emitted, use the overload with `onDiscard`. * * @see * [[Flow.fromPublisher]] */ def fromPublisher[T](p: Publisher[T])(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p)) + + /** This variant accepts an implementation of `org.reactivestreams.Publisher`, as opposed to `java.util.concurrent.Flow.Publisher` which + * is supported in the core module. + * + * See the core overload with `onDiscard` for the callback's semantics. + * + * @see + * [[Flow.fromPublisher]] + */ + def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T] = + Flow.fromPublisher(FlowAdapters.toFlowPublisher(p), onDiscard) end FlowReactiveStreams diff --git a/flow-reactive-streams/src/test/scala/ox/flow/reactive/FlowReactiveStreamsFromPublisherTest.scala b/flow-reactive-streams/src/test/scala/ox/flow/reactive/FlowReactiveStreamsFromPublisherTest.scala new file mode 100644 index 00000000..d57214d0 --- /dev/null +++ b/flow-reactive-streams/src/test/scala/ox/flow/reactive/FlowReactiveStreamsFromPublisherTest.scala @@ -0,0 +1,53 @@ +package ox.flow.reactive + +import org.reactivestreams.Publisher +import org.reactivestreams.Subscriber +import org.reactivestreams.Subscription +import org.scalatest.funsuite.AnyFunSuite +import org.scalatest.matchers.should.Matchers +import ox.* + +import java.util.concurrent.ConcurrentLinkedQueue +import java.util.concurrent.CountDownLatch +import java.util.concurrent.atomic.AtomicBoolean +import scala.jdk.CollectionConverters.* + +class FlowReactiveStreamsFromPublisherTest extends AnyFunSuite with Matchers: + + /** A publisher driven from the test thread. Ignores demand; supports a single subscriber. */ + private class ManualPublisher[T] extends Publisher[T]: + @volatile var subscriber: Subscriber[? >: T] = null + val cancelled = new AtomicBoolean(false) + val subscribed = new CountDownLatch(1) + + override def subscribe(s: Subscriber[? >: T]): Unit = + subscriber = s + s.onSubscribe(new Subscription: + override def request(n: Long): Unit = () + override def cancel(): Unit = cancelled.set(true)) + subscribed.countDown() + + def next(t: T): Unit = subscriber.onNext(t) + def complete(): Unit = subscriber.onComplete() + end ManualPublisher + + test("should emit elements from the publisher"): + val publisher = new ManualPublisher[Int] + supervised: + val result = fork(FlowReactiveStreams.fromPublisher(publisher).runToList()) + publisher.subscribed.await() + (1 to 3).foreach(publisher.next) + publisher.complete() + result.join() shouldBe List(1, 2, 3) + + test("should discard undelivered elements and cancel the subscription when the consumer stops early"): + val publisher = new ManualPublisher[Int] + val discarded = new ConcurrentLinkedQueue[Int]() + supervised: + val result = fork(FlowReactiveStreams.fromPublisher(publisher, discarded.add(_).discard).take(1).runToList()) + publisher.subscribed.await() + (1 to 5).foreach(publisher.next) + result.join() shouldBe List(1) + discarded.asScala.toList.sorted shouldBe List(2, 3, 4, 5) + publisher.cancelled.get() shouldBe true +end FlowReactiveStreamsFromPublisherTest From df4deeffadce4efb3573c797abacc20d39c618e3 Mon Sep 17 00:00:00 2001 From: Adam Warski Date: Fri, 9 Oct 2026 15:26:34 +0000 Subject: [PATCH 06/11] Document onDiscard callback and publisher failure behavior in flows docs --- .orca/runs/0a3279c8e3d7.progress.json | 2 +- doc/streaming/flows.md | 8 ++++++++ 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/.orca/runs/0a3279c8e3d7.progress.json b/.orca/runs/0a3279c8e3d7.progress.json index 2681ba74..538464eb 100644 --- a/.orca/runs/0a3279c8e3d7.progress.json +++ b/.orca/runs/0a3279c8e3d7.progress.json @@ -1 +1 @@ -{"header":{"startingBranch":"zippy-astrodon-94aa2fb6","branch":"zippy-astrodon-94aa2fb6","branchMode":"Reused","userPrompt":"Implement GitHub issue softwaremill/ox#528: \"Flow.fromPublisher: allow releasing elements which are buffered but not emitted\".\n\nProblem: `Flow.fromPublisher` (core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala) buffers elements in a channel. Some of them never reach the consumer:\n- when the publisher fails, `channel.error` drops the buffered elements\n- when the consumer stops early (an exception, an interruption), the buffered elements are never received\n- elements can still arrive (onNext) after the subscription is cancelled\n\nIf elements hold resources (e.g. Netty's reference-counted ByteBufs), nobody releases them, so they leak.\n\nGoal: add an optional callback, e.g. `fromPublisher(p, onDiscard: T => Unit)`, called exactly once for each element received from the publisher but not emitted downstream. Existing call sites without the callback must keep compiling and behaving the same.\n\nSuggested implementation: on publisher error, store the error and call `channel.done()` (so buffered elements are kept and emitted before the error is rethrown). When the flow ends early, cancel the subscription, close the channel, and drain it, calling the callback for each drained element. In `onNext`, call the callback if the channel is already closed. Make sure there are no races where an element is neither emitted nor discarded.\n\nAlso update the variant in flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala (org.reactivestreams Publisher) to accept the same optional callback, and update any docs (doc/ directory) describing fromPublisher.\n\nAcceptance criteria:\n- tests covering: publisher error with buffered elements (elements still emitted, then error); consumer stopping early (exception / interruption) with buffered elements -> callback called for each; elements arriving after cancellation -> callback called; every element is either emitted or discarded, never both or neither.\n- `sbt compile test` passes for affected modules (core, flow-reactive-streams).\n- Code follows existing style in the repo; concise doc comments.\n\nContext: tapir currently tracks unconsumed elements itself using a lock: https://github.com/softwaremill/tapir/pull/5573","flow":{"type":"Catalog","name":"implement.sc"},"startingCommit":"664f01730cda34fd88e8c3db3deb600934eb1d1e"},"entries":[{"id":[{"name":"Plan","occurrence":0}],"resultJson":{"id":"ox-528-from-publisher-on-discard","description":"`Flow.fromPublisher` buffers elements received from a reactive `Publisher` in a channel. Today, elements that are buffered but never emitted are silently lost: `channel.error` drops the buffer on publisher error, a consumer that stops early (exception, interruption, `take`) never receives what is buffered, and `onNext` calls that arrive after the subscription is cancelled are dropped by `channel.send` throwing. When elements own resources (Netty reference-counted ByteBufs in tapir), this leaks them. Tapir currently works around it with its own lock-based tracking (softwaremill/tapir#5573).\n\nThis epic adds an optional `onDiscard: T => Unit` callback to `Flow.fromPublisher` (core, JDK `Flow.Publisher`) and `FlowReactiveStreams.fromPublisher` (flow-reactive-streams, `org.reactivestreams.Publisher`), with the invariant that every element received from the publisher is either emitted downstream or passed to `onDiscard`, exactly once, never both or neither. Publisher errors no longer drop buffered elements: the error is stored, the channel is completed with `done()`, buffered elements are emitted, and then the error is rethrown. When the flow ends early, the subscription is cancelled, the channel is closed and drained through `onDiscard`, and any late `onNext` goes to `onDiscard` as well. Existing single-argument call sites keep compiling (the new parameter is an overload, which also keeps MiMa happy for `core`). Docs in `doc/streaming/flows.md` describe the new callback.","tasks":[{"title":"Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","description":"In core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala add `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg `fromPublisher` delegate to it with `_ => ()`. Implement per the brief's design decisions: `onNext` uses `sendOrClosed` and discards on closed; `onError` stores the throwable in an AtomicReference and calls `doneOrClosed`; `onComplete` calls `doneOrClosed`; `onSubscribe` cancels immediately if a `cancelled` flag is already set, else requests `capacity`; the consumer loop rethrows the stored error when it sees `Done`; on any Throwable escaping the loop: set `cancelled`, cancel the subscription if present, `doneOrClosed` the channel, drain it calling `onDiscard` per element (keep draining across an InterruptedException and re-interrupt the thread afterwards), rethrow. Preserve the existing batched demand logic. Update the scaladoc of both overloads (exactly-once invariant, callback runs on publisher threads and must not throw, buffered elements are now emitted before a publisher error is rethrown).\n\nTests, in a new core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardTest.scala (AnyFunSuite with Matchers), using a hand-written `ManualPublisher[T]` helper as described in the brief (keep the helper in this file or a sibling file in the same package so Task 2 can reuse it):\n1. Publisher error with buffered elements: push 1,2,3 then `error(boom)` before/while the consumer runs; `runToList` throws boom and all of 1,2,3 were emitted; `onDiscard` never called.\n2. Publisher error with nothing buffered still throws.\n3. Consumer exception: `.map(x => if x == 2 then throw boom else x)`, push 1..5; `runToList` throws boom; discarded == 3,4,5 (order-independent compare), subscription cancelled.\n4. Consumer stops via `take(2)`: result List(1,2), discarded == 3,4,5, cancelled.\n5. Consumer interrupted: run `runForeach(x => { received.release(); gate.acquire() })` in `forkCancellable`, push 1..5, wait for `received`, `cancel()`; discarded == 2,3,4,5, cancelled.\n6. Late onNext after cancellation: after any of the above, call `next(6)` on the publisher; `onDiscard(6)` is called synchronously and nothing throws.\n7. Existing one-arg overload: compiles unchanged and `FlowPublisherPekkoTest` still passes.\nRun `sbt \"core/testOnly ox.flow.reactive.*\"` and `sbt core/scalafmt`; all green."},{"title":"Exactly-once stress test: every element is emitted or discarded, never both or neither","description":"Add a test (same file as Task 1 or a sibling in core/src/test/scala/ox/flow/reactive/) that uses a demand-respecting publisher running on its own thread: it emits 1..N (N around 1000) only while `sent < requested` and not cancelled, calling `onNext` from that thread, then `onComplete`. Compose `Flow.fromPublisher(p, discarded.add)` with a consumer that stops at a random point (e.g. `take(k)` for random k, and separately a `map` that throws at a random element, and separately a `forkCancellable` cancelled after a random short delay). Collect emitted elements via `.tap(emitted.add)` into thread-safe collections. After the flow finishes and the publisher thread is joined, assert: `emitted` and `discarded` are disjoint, their union equals exactly the set of elements the publisher actually sent, and no element appears twice. Repeat each scenario in a loop (e.g. 20 iterations) with a small `given BufferCapacity = BufferCapacity(4)` to make buffer-full races likely. The publisher's late `onNext` calls (after cancellation) must land in `discarded`. If this exposes a race in Task 1's implementation, fix it in FlowCompanionReactiveOps.scala as part of this task. Run `sbt \"core/testOnly ox.flow.reactive.*\"`."},{"title":"Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","description":"In flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala add `def fromPublisher[T](p: org.reactivestreams.Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p), onDiscard)`, keep the one-arg overload, and update its scaladoc to point at the core overload for the callback semantics. Create flow-reactive-streams/src/test/scala/ox/flow/reactive/FlowReactiveStreamsFromPublisherTest.scala (AnyFunSuite with Matchers) with a small hand-written `org.reactivestreams.Publisher[Int]` (same shape as the core ManualPublisher, but implementing the reactive-streams interfaces; it cannot be shared because the module does not depend on core's test classpath) and two tests: elements flow through the one-arg variant; with `.take(1)` and the new overload, remaining pushed elements are passed to `onDiscard` and the subscription is cancelled. Run `sbt flowReactiveStreams/test` and `sbt flowReactiveStreams/scalafmt`."},{"title":"Document the onDiscard callback in doc/streaming/flows.md","description":"In doc/streaming/flows.md, section \"### Publisher -> Flow\" (around lines 211-219), add a short paragraph explaining: elements received from the publisher but not emitted downstream (buffered when the publisher fails, when the consumer stops early via exception, interruption or `take`, or arriving after cancellation) can be released by passing a second argument `onDiscard: T => Unit` to `Flow.fromPublisher` / `FlowReactiveStreams.fromPublisher`; it is called exactly once per such element, possibly from the publisher's thread, and should not throw; typical use is releasing reference-counted buffers. Also state that on publisher error, elements already received are emitted before the error is propagated. Keep the prose as short as the surrounding text. If you add a code example, use a ```` ```scala mdoc:compile-only ```` block and verify with `sbt compileDocumentation`; otherwise no build step is needed. Do not touch generated-doc/out."}],"brief":"## Modules and files\n\n- `core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala` — the only implementation file in core. Trait `FlowCompanionReactiveOps` (mixed into `Flow.type`) defines `def fromPublisher[T](p: java.util.concurrent.Flow.Publisher[T])(using BufferCapacity): Flow[T]`. Read the whole file (80 lines) before changing anything. Current structure: `usingEmitInline: emit => unsupervised { channel = BufferCapacity.newChannel[T]; forkUnsupervised { p.subscribe(Subscriber: onSubscribe -> subscriptionRef.set(s); s.request(capacity); onNext -> channel.send; onError -> channel.error; onComplete -> channel.done) }; repeatWhile { channel.receiveOrClosed() match Done -> false / Error -> throw / t -> emit(t); batched subscription.request } }.tapException(cancel subscription)`. The demand logic (`demandThreshold = ceil(capacity/2)`, `toDemand` counter, lazily cached `subscription` read from `subscriptionRef` after the first element) must be preserved.\n- `flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala` — `object FlowReactiveStreams { def fromPublisher[T](p: org.reactivestreams.Publisher[T])(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p)) }`. Module has no `src/test` dir yet; scalatest is already a test dependency (build.sbt `flowReactiveStreams`), it `dependsOn(core)` (compile only, not test->test).\n- `core/src/test/scala/ox/flow/reactive/` — existing reactive tests: `FlowPublisherPekkoTest.scala` (AnyFunSuite + Matchers, uses Pekko `Source...runWith(Sink.asPublisher)` and `FlowAdapters.toFlowPublisher`) and `FlowPublisherTckTest.scala`. Put new fromPublisher tests here. No test anywhere currently uses a hand-written Publisher; write one (see below).\n- `doc/streaming/flows.md` lines 211-219, section \"### Publisher -> Flow\" — the only user doc mentioning `fromPublisher`. `generated-doc/out/**` is mdoc output regenerated at release time by `updateDocs`; do NOT edit it. Code blocks in docs use ```` ```scala mdoc:compile-only ```` and are compiled by `sbt compileDocumentation` (slow; only needed if you add a code block). Doc dependency snippet convention: `// sbt dependency: \"com.softwaremill.ox\" %% \"flow-reactive-streams\" % \"@VERSION@\"`.\n\n## Key APIs (ox.channels, core/src/main/scala/ox/channels/Channel.scala)\n\n- `Sink.sendOrClosed(t): Unit | ChannelClosed` (non-throwing), `Sink.send(t)` throws `ChannelClosedException` when closed.\n- `Sink.doneOrClosed(): Unit | ChannelClosed` — \"buffered values will be delivered; send operations in progress complete normally when a receiver arrives\"; returns `ChannelClosed` instead of throwing if already closed. `Sink.errorOrClosed(reason)` — drops buffered values (that is the bug). Use the `*OrClosed` variants everywhere in the subscriber: a misbehaving publisher may call onError/onComplete twice.\n- `Source.receiveOrClosed(): T | ChannelClosed` (blocking); match `case ChannelClosed.Done => ...; case e: ChannelClosed.Error => throw e.toThrowable; case t: T @unchecked => ...`. `Source.tryReceiveOrClosed()` is non-blocking but may spuriously return None, so it is not suitable for a drain loop.\n- `Source.foreachOrError(f)` / `drainOrError()` from `SourceDrainOps.scala` loop until Done; usable for the drain, but write the drain inline so you can handle `InterruptedException` (see below).\n- jox semantics (com.softwaremill.jox channels 1.1.2, verified from source): after `done()`, buffered elements stay receivable; a receiver observing Done has already seen every element whose `send` started before `done()` (an in-progress sender is waited for); a `send` racing with `done()` either lands in the buffer or returns closed. Hence \"drain until Done, then `sendOrClosed` returns closed\" gives exactly-once discard with no extra locking.\n- `BufferCapacity`: `given default: BufferCapacity = 16`; `BufferCapacity.newChannel[T]`; `summon[BufferCapacity].toInt` (import `ox.channels.toInt`). Tests can override with `given BufferCapacity = BufferCapacity(4)`.\n- `ox.util`: `t.tapException(f)` runs `f(e)` for any Throwable and rethrows `e` (exceptions from `f` added as suppressed). `ox.pipe`, `ox.discard`, `ox.repeatWhile` already imported in the file.\n- `FlowOps.take(n)` stops upstream by throwing a private `ControlThrowable` (`abortTake`) out of `emit`, caught in `take` itself. So \"consumer stops early\" includes `take`, and the cleanup path must run for every Throwable, not just NonFatal. `tapException`/`catch case e: Throwable` already does this.\n- `unsupervised { ... }` + `forkUnsupervised { ... }` (import `ox.unsupervised`, `ox.forkUnsupervised`): scope end interrupts still-running forks once. `onNext`/`onError`/`onComplete` run on the publisher's threads (or on the `p.subscribe` fork for synchronous publishers), never on the consumer thread.\n- Test helpers: `ox.forkCancellable { ... }` inside `supervised` returns a `CancellableFork` with `cancel(): Either[Throwable, T]` (interrupts once, joins). Interruption inside the flow surfaces as `InterruptedException` thrown from a blocking call (e.g. `Semaphore.acquire`, `channel.receiveOrClosed`).\n\n## Design decisions (apply as-is)\n\n- API: add an overload `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg method delegate with `_ => ()`. Do not add a default parameter and do not change the existing signature: `core` runs MiMa (`enableMimaSettings` in build.sbt) and only additive changes are safe. Same shape in `FlowReactiveStreams`.\n- Subscriber: `onNext(t)` = `channel.sendOrClosed(t)` and if the result is `ChannelClosed` call `onDiscard(t)`. `onError(e)` = store `e` in an `AtomicReference[Throwable]`, then `channel.doneOrClosed().discard`. `onComplete()` = `channel.doneOrClosed().discard`. `onSubscribe(s)` = set `subscriptionRef`, then if a `cancelled: AtomicBoolean` is already set call `s.cancel()` else `s.request(capacity)` (both sides use volatile atomics, so at least one side sees the other; a double `cancel()` is allowed by the spec).\n- Consumer loop: on `ChannelClosed.Done`, read the stored error; if non-null throw it, else finish. Keep the `ChannelClosed.Error` branch for safety.\n- Cleanup, run on any Throwable escaping the loop (the stored publisher error included; cancel after onError is a spec-allowed no-op): set `cancelled`, cancel the subscription if `subscriptionRef` is non-null, `channel.doneOrClosed().discard`, then drain: loop `receiveOrClosed()` until `Done`, calling `onDiscard` for each element. The drain is bounded (channel is done, at most capacity elements plus in-flight sends) but blocking; if `InterruptedException` is thrown mid-drain, catch it, keep draining, and call `Thread.currentThread().interrupt()` once the drain finishes, so no element is lost on a second interrupt. Then rethrow the original exception. An element that was passed to `emit` and whose `emit` threw counts as emitted (downstream owns it): do not discard it.\n- Scaladoc on the new overload: state the exactly-once invariant, that `onDiscard` may be called from publisher threads, that it must not throw, and that on publisher error buffered elements are emitted before the error is rethrown (this is a deliberate behaviour change for the one-arg variant too; mention it in its doc).\n\n## Test publisher helper (write it in the core test, package `ox.flow.reactive`)\n\nA `ManualPublisher[T] extends java.util.concurrent.Flow.Publisher[T]` with: `@volatile var subscriber: Subscriber[? >: T]`, `requested: AtomicLong`, `cancelled: AtomicBoolean`, a `subscribed: CountDownLatch(1)`; `subscribe(s)` stores `s`, calls `s.onSubscribe(new Subscription { request(n) => requested.addAndGet(n); cancel() => cancelled.set(true) })`, counts down the latch. Helper methods `next(t)`, `error(e)`, `complete()` forward to the subscriber from the test thread. For the exactly-once stress test, a second variant that runs its own thread emitting `1..n` while `sent < requested` and not cancelled. Run the flow under test in `supervised { forkCancellable { ... } }` or `fork`, await `subscribed`, then drive the publisher. Collect emitted elements with `.tap(emitted += _)` or `runForeach`, discarded with `onDiscard = discarded += _` (use thread-safe collections such as `ConcurrentLinkedQueue` or `java.util.Collections.synchronizedList`, since `onDiscard` runs on other threads).\n\n## Conventions / build\n\n- Scala 3 indentation syntax, `end X` markers on long definitions, `.discard` on ignored results (compiler warns otherwise), scalafmt configured in `.scalafmt.conf` (140-column lines); run `sbt core/scalafmt flowReactiveStreams/scalafmt` after editing.\n- Comments: short, explain non-obvious reasoning only; never restate code. Scaladoc describes usage and invariants, not implementation.\n- Tests: scalatest 3.2.20; the reactive tests use `AnyFunSuite with Matchers` with `test(\"...\")` blocks and Scala 3 colon syntax. `core` runs tests forked (`Test / fork := true`).\n- Commands (sbt 2.0.10): `sbt \"core/testOnly ox.flow.reactive.*\"`, `sbt flowReactiveStreams/test`, `sbt core/mimaReportBinaryIssues` (only meaningful when a previous stable version resolves), `sbt compileDocumentation` for mdoc. Never commit; the user commits."}},{"id":[{"name":"Task: Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Exactly-once stress test: every element is emitted or discarded, never both or neither","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","occurrence":0}],"resultJson":{"findings":[{"id":"R1.I4.1","title":"sbt Test/compile fails with JGit NoWorkTreeException","reason":{"type":"Declined","text":"This is a problem with the environment, not the code. git sees the worktree as non-bare (`git rev-parse --is-bare-repository` prints false; the git dir is /home/adamw/ox/.git/worktrees/zippy-astrodon-94aa2fb6). The exception comes from the sbt-git plugin, which uses JGit by default and reads this linked worktree as bare while the build loads. Nothing in project/ or the build configures this; `zz-tmp-git.sbt` was my temporary workaround, not the cause. To check the code, I recreated it with `useReadableConsoleGit`, which makes sbt-git use command-line git: `sbt core/Test/compile flowReactiveStreams/Test/compile` succeeded (warnings only, in older test files, no errors) and flowReactiveStreams tests passed. I then deleted the file again. Changing the shared build's git plugin setup is outside the scope of this change."}}]}}],"stageStarts":[]} \ No newline at end of file +{"header":{"startingBranch":"zippy-astrodon-94aa2fb6","branch":"zippy-astrodon-94aa2fb6","branchMode":"Reused","userPrompt":"Implement GitHub issue softwaremill/ox#528: \"Flow.fromPublisher: allow releasing elements which are buffered but not emitted\".\n\nProblem: `Flow.fromPublisher` (core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala) buffers elements in a channel. Some of them never reach the consumer:\n- when the publisher fails, `channel.error` drops the buffered elements\n- when the consumer stops early (an exception, an interruption), the buffered elements are never received\n- elements can still arrive (onNext) after the subscription is cancelled\n\nIf elements hold resources (e.g. Netty's reference-counted ByteBufs), nobody releases them, so they leak.\n\nGoal: add an optional callback, e.g. `fromPublisher(p, onDiscard: T => Unit)`, called exactly once for each element received from the publisher but not emitted downstream. Existing call sites without the callback must keep compiling and behaving the same.\n\nSuggested implementation: on publisher error, store the error and call `channel.done()` (so buffered elements are kept and emitted before the error is rethrown). When the flow ends early, cancel the subscription, close the channel, and drain it, calling the callback for each drained element. In `onNext`, call the callback if the channel is already closed. Make sure there are no races where an element is neither emitted nor discarded.\n\nAlso update the variant in flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala (org.reactivestreams Publisher) to accept the same optional callback, and update any docs (doc/ directory) describing fromPublisher.\n\nAcceptance criteria:\n- tests covering: publisher error with buffered elements (elements still emitted, then error); consumer stopping early (exception / interruption) with buffered elements -> callback called for each; elements arriving after cancellation -> callback called; every element is either emitted or discarded, never both or neither.\n- `sbt compile test` passes for affected modules (core, flow-reactive-streams).\n- Code follows existing style in the repo; concise doc comments.\n\nContext: tapir currently tracks unconsumed elements itself using a lock: https://github.com/softwaremill/tapir/pull/5573","flow":{"type":"Catalog","name":"implement.sc"},"startingCommit":"664f01730cda34fd88e8c3db3deb600934eb1d1e"},"entries":[{"id":[{"name":"Plan","occurrence":0}],"resultJson":{"id":"ox-528-from-publisher-on-discard","description":"`Flow.fromPublisher` buffers elements received from a reactive `Publisher` in a channel. Today, elements that are buffered but never emitted are silently lost: `channel.error` drops the buffer on publisher error, a consumer that stops early (exception, interruption, `take`) never receives what is buffered, and `onNext` calls that arrive after the subscription is cancelled are dropped by `channel.send` throwing. When elements own resources (Netty reference-counted ByteBufs in tapir), this leaks them. Tapir currently works around it with its own lock-based tracking (softwaremill/tapir#5573).\n\nThis epic adds an optional `onDiscard: T => Unit` callback to `Flow.fromPublisher` (core, JDK `Flow.Publisher`) and `FlowReactiveStreams.fromPublisher` (flow-reactive-streams, `org.reactivestreams.Publisher`), with the invariant that every element received from the publisher is either emitted downstream or passed to `onDiscard`, exactly once, never both or neither. Publisher errors no longer drop buffered elements: the error is stored, the channel is completed with `done()`, buffered elements are emitted, and then the error is rethrown. When the flow ends early, the subscription is cancelled, the channel is closed and drained through `onDiscard`, and any late `onNext` goes to `onDiscard` as well. Existing single-argument call sites keep compiling (the new parameter is an overload, which also keeps MiMa happy for `core`). Docs in `doc/streaming/flows.md` describe the new callback.","tasks":[{"title":"Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","description":"In core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala add `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg `fromPublisher` delegate to it with `_ => ()`. Implement per the brief's design decisions: `onNext` uses `sendOrClosed` and discards on closed; `onError` stores the throwable in an AtomicReference and calls `doneOrClosed`; `onComplete` calls `doneOrClosed`; `onSubscribe` cancels immediately if a `cancelled` flag is already set, else requests `capacity`; the consumer loop rethrows the stored error when it sees `Done`; on any Throwable escaping the loop: set `cancelled`, cancel the subscription if present, `doneOrClosed` the channel, drain it calling `onDiscard` per element (keep draining across an InterruptedException and re-interrupt the thread afterwards), rethrow. Preserve the existing batched demand logic. Update the scaladoc of both overloads (exactly-once invariant, callback runs on publisher threads and must not throw, buffered elements are now emitted before a publisher error is rethrown).\n\nTests, in a new core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardTest.scala (AnyFunSuite with Matchers), using a hand-written `ManualPublisher[T]` helper as described in the brief (keep the helper in this file or a sibling file in the same package so Task 2 can reuse it):\n1. Publisher error with buffered elements: push 1,2,3 then `error(boom)` before/while the consumer runs; `runToList` throws boom and all of 1,2,3 were emitted; `onDiscard` never called.\n2. Publisher error with nothing buffered still throws.\n3. Consumer exception: `.map(x => if x == 2 then throw boom else x)`, push 1..5; `runToList` throws boom; discarded == 3,4,5 (order-independent compare), subscription cancelled.\n4. Consumer stops via `take(2)`: result List(1,2), discarded == 3,4,5, cancelled.\n5. Consumer interrupted: run `runForeach(x => { received.release(); gate.acquire() })` in `forkCancellable`, push 1..5, wait for `received`, `cancel()`; discarded == 2,3,4,5, cancelled.\n6. Late onNext after cancellation: after any of the above, call `next(6)` on the publisher; `onDiscard(6)` is called synchronously and nothing throws.\n7. Existing one-arg overload: compiles unchanged and `FlowPublisherPekkoTest` still passes.\nRun `sbt \"core/testOnly ox.flow.reactive.*\"` and `sbt core/scalafmt`; all green."},{"title":"Exactly-once stress test: every element is emitted or discarded, never both or neither","description":"Add a test (same file as Task 1 or a sibling in core/src/test/scala/ox/flow/reactive/) that uses a demand-respecting publisher running on its own thread: it emits 1..N (N around 1000) only while `sent < requested` and not cancelled, calling `onNext` from that thread, then `onComplete`. Compose `Flow.fromPublisher(p, discarded.add)` with a consumer that stops at a random point (e.g. `take(k)` for random k, and separately a `map` that throws at a random element, and separately a `forkCancellable` cancelled after a random short delay). Collect emitted elements via `.tap(emitted.add)` into thread-safe collections. After the flow finishes and the publisher thread is joined, assert: `emitted` and `discarded` are disjoint, their union equals exactly the set of elements the publisher actually sent, and no element appears twice. Repeat each scenario in a loop (e.g. 20 iterations) with a small `given BufferCapacity = BufferCapacity(4)` to make buffer-full races likely. The publisher's late `onNext` calls (after cancellation) must land in `discarded`. If this exposes a race in Task 1's implementation, fix it in FlowCompanionReactiveOps.scala as part of this task. Run `sbt \"core/testOnly ox.flow.reactive.*\"`."},{"title":"Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","description":"In flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala add `def fromPublisher[T](p: org.reactivestreams.Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p), onDiscard)`, keep the one-arg overload, and update its scaladoc to point at the core overload for the callback semantics. Create flow-reactive-streams/src/test/scala/ox/flow/reactive/FlowReactiveStreamsFromPublisherTest.scala (AnyFunSuite with Matchers) with a small hand-written `org.reactivestreams.Publisher[Int]` (same shape as the core ManualPublisher, but implementing the reactive-streams interfaces; it cannot be shared because the module does not depend on core's test classpath) and two tests: elements flow through the one-arg variant; with `.take(1)` and the new overload, remaining pushed elements are passed to `onDiscard` and the subscription is cancelled. Run `sbt flowReactiveStreams/test` and `sbt flowReactiveStreams/scalafmt`."},{"title":"Document the onDiscard callback in doc/streaming/flows.md","description":"In doc/streaming/flows.md, section \"### Publisher -> Flow\" (around lines 211-219), add a short paragraph explaining: elements received from the publisher but not emitted downstream (buffered when the publisher fails, when the consumer stops early via exception, interruption or `take`, or arriving after cancellation) can be released by passing a second argument `onDiscard: T => Unit` to `Flow.fromPublisher` / `FlowReactiveStreams.fromPublisher`; it is called exactly once per such element, possibly from the publisher's thread, and should not throw; typical use is releasing reference-counted buffers. Also state that on publisher error, elements already received are emitted before the error is propagated. Keep the prose as short as the surrounding text. If you add a code example, use a ```` ```scala mdoc:compile-only ```` block and verify with `sbt compileDocumentation`; otherwise no build step is needed. Do not touch generated-doc/out."}],"brief":"## Modules and files\n\n- `core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala` — the only implementation file in core. Trait `FlowCompanionReactiveOps` (mixed into `Flow.type`) defines `def fromPublisher[T](p: java.util.concurrent.Flow.Publisher[T])(using BufferCapacity): Flow[T]`. Read the whole file (80 lines) before changing anything. Current structure: `usingEmitInline: emit => unsupervised { channel = BufferCapacity.newChannel[T]; forkUnsupervised { p.subscribe(Subscriber: onSubscribe -> subscriptionRef.set(s); s.request(capacity); onNext -> channel.send; onError -> channel.error; onComplete -> channel.done) }; repeatWhile { channel.receiveOrClosed() match Done -> false / Error -> throw / t -> emit(t); batched subscription.request } }.tapException(cancel subscription)`. The demand logic (`demandThreshold = ceil(capacity/2)`, `toDemand` counter, lazily cached `subscription` read from `subscriptionRef` after the first element) must be preserved.\n- `flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala` — `object FlowReactiveStreams { def fromPublisher[T](p: org.reactivestreams.Publisher[T])(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p)) }`. Module has no `src/test` dir yet; scalatest is already a test dependency (build.sbt `flowReactiveStreams`), it `dependsOn(core)` (compile only, not test->test).\n- `core/src/test/scala/ox/flow/reactive/` — existing reactive tests: `FlowPublisherPekkoTest.scala` (AnyFunSuite + Matchers, uses Pekko `Source...runWith(Sink.asPublisher)` and `FlowAdapters.toFlowPublisher`) and `FlowPublisherTckTest.scala`. Put new fromPublisher tests here. No test anywhere currently uses a hand-written Publisher; write one (see below).\n- `doc/streaming/flows.md` lines 211-219, section \"### Publisher -> Flow\" — the only user doc mentioning `fromPublisher`. `generated-doc/out/**` is mdoc output regenerated at release time by `updateDocs`; do NOT edit it. Code blocks in docs use ```` ```scala mdoc:compile-only ```` and are compiled by `sbt compileDocumentation` (slow; only needed if you add a code block). Doc dependency snippet convention: `// sbt dependency: \"com.softwaremill.ox\" %% \"flow-reactive-streams\" % \"@VERSION@\"`.\n\n## Key APIs (ox.channels, core/src/main/scala/ox/channels/Channel.scala)\n\n- `Sink.sendOrClosed(t): Unit | ChannelClosed` (non-throwing), `Sink.send(t)` throws `ChannelClosedException` when closed.\n- `Sink.doneOrClosed(): Unit | ChannelClosed` — \"buffered values will be delivered; send operations in progress complete normally when a receiver arrives\"; returns `ChannelClosed` instead of throwing if already closed. `Sink.errorOrClosed(reason)` — drops buffered values (that is the bug). Use the `*OrClosed` variants everywhere in the subscriber: a misbehaving publisher may call onError/onComplete twice.\n- `Source.receiveOrClosed(): T | ChannelClosed` (blocking); match `case ChannelClosed.Done => ...; case e: ChannelClosed.Error => throw e.toThrowable; case t: T @unchecked => ...`. `Source.tryReceiveOrClosed()` is non-blocking but may spuriously return None, so it is not suitable for a drain loop.\n- `Source.foreachOrError(f)` / `drainOrError()` from `SourceDrainOps.scala` loop until Done; usable for the drain, but write the drain inline so you can handle `InterruptedException` (see below).\n- jox semantics (com.softwaremill.jox channels 1.1.2, verified from source): after `done()`, buffered elements stay receivable; a receiver observing Done has already seen every element whose `send` started before `done()` (an in-progress sender is waited for); a `send` racing with `done()` either lands in the buffer or returns closed. Hence \"drain until Done, then `sendOrClosed` returns closed\" gives exactly-once discard with no extra locking.\n- `BufferCapacity`: `given default: BufferCapacity = 16`; `BufferCapacity.newChannel[T]`; `summon[BufferCapacity].toInt` (import `ox.channels.toInt`). Tests can override with `given BufferCapacity = BufferCapacity(4)`.\n- `ox.util`: `t.tapException(f)` runs `f(e)` for any Throwable and rethrows `e` (exceptions from `f` added as suppressed). `ox.pipe`, `ox.discard`, `ox.repeatWhile` already imported in the file.\n- `FlowOps.take(n)` stops upstream by throwing a private `ControlThrowable` (`abortTake`) out of `emit`, caught in `take` itself. So \"consumer stops early\" includes `take`, and the cleanup path must run for every Throwable, not just NonFatal. `tapException`/`catch case e: Throwable` already does this.\n- `unsupervised { ... }` + `forkUnsupervised { ... }` (import `ox.unsupervised`, `ox.forkUnsupervised`): scope end interrupts still-running forks once. `onNext`/`onError`/`onComplete` run on the publisher's threads (or on the `p.subscribe` fork for synchronous publishers), never on the consumer thread.\n- Test helpers: `ox.forkCancellable { ... }` inside `supervised` returns a `CancellableFork` with `cancel(): Either[Throwable, T]` (interrupts once, joins). Interruption inside the flow surfaces as `InterruptedException` thrown from a blocking call (e.g. `Semaphore.acquire`, `channel.receiveOrClosed`).\n\n## Design decisions (apply as-is)\n\n- API: add an overload `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg method delegate with `_ => ()`. Do not add a default parameter and do not change the existing signature: `core` runs MiMa (`enableMimaSettings` in build.sbt) and only additive changes are safe. Same shape in `FlowReactiveStreams`.\n- Subscriber: `onNext(t)` = `channel.sendOrClosed(t)` and if the result is `ChannelClosed` call `onDiscard(t)`. `onError(e)` = store `e` in an `AtomicReference[Throwable]`, then `channel.doneOrClosed().discard`. `onComplete()` = `channel.doneOrClosed().discard`. `onSubscribe(s)` = set `subscriptionRef`, then if a `cancelled: AtomicBoolean` is already set call `s.cancel()` else `s.request(capacity)` (both sides use volatile atomics, so at least one side sees the other; a double `cancel()` is allowed by the spec).\n- Consumer loop: on `ChannelClosed.Done`, read the stored error; if non-null throw it, else finish. Keep the `ChannelClosed.Error` branch for safety.\n- Cleanup, run on any Throwable escaping the loop (the stored publisher error included; cancel after onError is a spec-allowed no-op): set `cancelled`, cancel the subscription if `subscriptionRef` is non-null, `channel.doneOrClosed().discard`, then drain: loop `receiveOrClosed()` until `Done`, calling `onDiscard` for each element. The drain is bounded (channel is done, at most capacity elements plus in-flight sends) but blocking; if `InterruptedException` is thrown mid-drain, catch it, keep draining, and call `Thread.currentThread().interrupt()` once the drain finishes, so no element is lost on a second interrupt. Then rethrow the original exception. An element that was passed to `emit` and whose `emit` threw counts as emitted (downstream owns it): do not discard it.\n- Scaladoc on the new overload: state the exactly-once invariant, that `onDiscard` may be called from publisher threads, that it must not throw, and that on publisher error buffered elements are emitted before the error is rethrown (this is a deliberate behaviour change for the one-arg variant too; mention it in its doc).\n\n## Test publisher helper (write it in the core test, package `ox.flow.reactive`)\n\nA `ManualPublisher[T] extends java.util.concurrent.Flow.Publisher[T]` with: `@volatile var subscriber: Subscriber[? >: T]`, `requested: AtomicLong`, `cancelled: AtomicBoolean`, a `subscribed: CountDownLatch(1)`; `subscribe(s)` stores `s`, calls `s.onSubscribe(new Subscription { request(n) => requested.addAndGet(n); cancel() => cancelled.set(true) })`, counts down the latch. Helper methods `next(t)`, `error(e)`, `complete()` forward to the subscriber from the test thread. For the exactly-once stress test, a second variant that runs its own thread emitting `1..n` while `sent < requested` and not cancelled. Run the flow under test in `supervised { forkCancellable { ... } }` or `fork`, await `subscribed`, then drive the publisher. Collect emitted elements with `.tap(emitted += _)` or `runForeach`, discarded with `onDiscard = discarded += _` (use thread-safe collections such as `ConcurrentLinkedQueue` or `java.util.Collections.synchronizedList`, since `onDiscard` runs on other threads).\n\n## Conventions / build\n\n- Scala 3 indentation syntax, `end X` markers on long definitions, `.discard` on ignored results (compiler warns otherwise), scalafmt configured in `.scalafmt.conf` (140-column lines); run `sbt core/scalafmt flowReactiveStreams/scalafmt` after editing.\n- Comments: short, explain non-obvious reasoning only; never restate code. Scaladoc describes usage and invariants, not implementation.\n- Tests: scalatest 3.2.20; the reactive tests use `AnyFunSuite with Matchers` with `test(\"...\")` blocks and Scala 3 colon syntax. `core` runs tests forked (`Test / fork := true`).\n- Commands (sbt 2.0.10): `sbt \"core/testOnly ox.flow.reactive.*\"`, `sbt flowReactiveStreams/test`, `sbt core/mimaReportBinaryIssues` (only meaningful when a previous stable version resolves), `sbt compileDocumentation` for mdoc. Never commit; the user commits."}},{"id":[{"name":"Task: Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Exactly-once stress test: every element is emitted or discarded, never both or neither","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","occurrence":0}],"resultJson":{"findings":[{"id":"R1.I4.1","title":"sbt Test/compile fails with JGit NoWorkTreeException","reason":{"type":"Declined","text":"This is a problem with the environment, not the code. git sees the worktree as non-bare (`git rev-parse --is-bare-repository` prints false; the git dir is /home/adamw/ox/.git/worktrees/zippy-astrodon-94aa2fb6). The exception comes from the sbt-git plugin, which uses JGit by default and reads this linked worktree as bare while the build loads. Nothing in project/ or the build configures this; `zz-tmp-git.sbt` was my temporary workaround, not the cause. To check the code, I recreated it with `useReadableConsoleGit`, which makes sbt-git use command-line git: `sbt core/Test/compile flowReactiveStreams/Test/compile` succeeded (warnings only, in older test files, no errors) and flowReactiveStreams tests passed. I then deleted the file again. Changing the shared build's git plugin setup is outside the scope of this change."}}]}},{"id":[{"name":"Task: Document the onDiscard callback in doc/streaming/flows.md","occurrence":0}],"resultJson":{"findings":[{"id":"R1.I4.1","title":"`sbt Test/compile` fails with no diagnostic","reason":{"type":"Declined","text":"Environmental. Full output shows sbt-git's JGit throws NoWorkTreeException (\"Bare Repository has neither a working tree, nor an index\") while loading the build, because this checkout is a git worktree; no source error is involved. With a temporary `ThisBuild / git.gitUncommittedChanges := false` override, `sbt Test/compile` succeeds across all modules (only pre-existing unrelated warnings). The temporary file was removed; no codebase change is needed."}}]}}],"stageStarts":[]} \ No newline at end of file diff --git a/doc/streaming/flows.md b/doc/streaming/flows.md index f6a3cacf..8af8a635 100644 --- a/doc/streaming/flows.md +++ b/doc/streaming/flows.md @@ -215,5 +215,13 @@ A `java.util.concurrent.Flow.Publisher` can be converted to a `Flow` using `Flow Internally, elements published to the subscription are buffered, using a buffer of capacity given by the `BufferCapacity` in scope. That's also how many elements will be at most requested from the publisher at a time. +If the publisher fails, elements already received are emitted before the error is propagated. + +Some elements might be received from the publisher, but never emitted downstream: e.g. when the consumer stops early +(through an exception, interruption or `take`), or when they arrive after cancellation. To release such elements +(e.g. reference-counted buffers), pass an `onDiscard: T => Unit` callback as the second argument to +`Flow.fromPublisher` / `FlowReactiveStreams.fromPublisher`. It is called exactly once for each such element, possibly +from the publisher's thread, and should not throw. + To convert a `org.reactivestreams.Publisher` instance, you'll need the same dependency as above and call the `FlowReactiveStreams.fromPublisher` method. \ No newline at end of file From b26896fabb67e9ec18ebf8f1e648eebeff6c2b3a Mon Sep 17 00:00:00 2001 From: Adam Warski Date: Fri, 9 Oct 2026 15:31:02 +0000 Subject: [PATCH 07/11] Remove the duplicated onDiscard docs from FlowCompanionReactiveOps and FlowReactiveStreams --- .orca/runs/0a3279c8e3d7.progress.json | 2 +- .../scala/ox/flow/FlowCompanionReactiveOps.scala | 15 +-------------- .../reactive/flowReactiveStreamsExtension.scala | 5 +---- 3 files changed, 3 insertions(+), 19 deletions(-) diff --git a/.orca/runs/0a3279c8e3d7.progress.json b/.orca/runs/0a3279c8e3d7.progress.json index 538464eb..e7f43fe3 100644 --- a/.orca/runs/0a3279c8e3d7.progress.json +++ b/.orca/runs/0a3279c8e3d7.progress.json @@ -1 +1 @@ -{"header":{"startingBranch":"zippy-astrodon-94aa2fb6","branch":"zippy-astrodon-94aa2fb6","branchMode":"Reused","userPrompt":"Implement GitHub issue softwaremill/ox#528: \"Flow.fromPublisher: allow releasing elements which are buffered but not emitted\".\n\nProblem: `Flow.fromPublisher` (core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala) buffers elements in a channel. Some of them never reach the consumer:\n- when the publisher fails, `channel.error` drops the buffered elements\n- when the consumer stops early (an exception, an interruption), the buffered elements are never received\n- elements can still arrive (onNext) after the subscription is cancelled\n\nIf elements hold resources (e.g. Netty's reference-counted ByteBufs), nobody releases them, so they leak.\n\nGoal: add an optional callback, e.g. `fromPublisher(p, onDiscard: T => Unit)`, called exactly once for each element received from the publisher but not emitted downstream. Existing call sites without the callback must keep compiling and behaving the same.\n\nSuggested implementation: on publisher error, store the error and call `channel.done()` (so buffered elements are kept and emitted before the error is rethrown). When the flow ends early, cancel the subscription, close the channel, and drain it, calling the callback for each drained element. In `onNext`, call the callback if the channel is already closed. Make sure there are no races where an element is neither emitted nor discarded.\n\nAlso update the variant in flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala (org.reactivestreams Publisher) to accept the same optional callback, and update any docs (doc/ directory) describing fromPublisher.\n\nAcceptance criteria:\n- tests covering: publisher error with buffered elements (elements still emitted, then error); consumer stopping early (exception / interruption) with buffered elements -> callback called for each; elements arriving after cancellation -> callback called; every element is either emitted or discarded, never both or neither.\n- `sbt compile test` passes for affected modules (core, flow-reactive-streams).\n- Code follows existing style in the repo; concise doc comments.\n\nContext: tapir currently tracks unconsumed elements itself using a lock: https://github.com/softwaremill/tapir/pull/5573","flow":{"type":"Catalog","name":"implement.sc"},"startingCommit":"664f01730cda34fd88e8c3db3deb600934eb1d1e"},"entries":[{"id":[{"name":"Plan","occurrence":0}],"resultJson":{"id":"ox-528-from-publisher-on-discard","description":"`Flow.fromPublisher` buffers elements received from a reactive `Publisher` in a channel. Today, elements that are buffered but never emitted are silently lost: `channel.error` drops the buffer on publisher error, a consumer that stops early (exception, interruption, `take`) never receives what is buffered, and `onNext` calls that arrive after the subscription is cancelled are dropped by `channel.send` throwing. When elements own resources (Netty reference-counted ByteBufs in tapir), this leaks them. Tapir currently works around it with its own lock-based tracking (softwaremill/tapir#5573).\n\nThis epic adds an optional `onDiscard: T => Unit` callback to `Flow.fromPublisher` (core, JDK `Flow.Publisher`) and `FlowReactiveStreams.fromPublisher` (flow-reactive-streams, `org.reactivestreams.Publisher`), with the invariant that every element received from the publisher is either emitted downstream or passed to `onDiscard`, exactly once, never both or neither. Publisher errors no longer drop buffered elements: the error is stored, the channel is completed with `done()`, buffered elements are emitted, and then the error is rethrown. When the flow ends early, the subscription is cancelled, the channel is closed and drained through `onDiscard`, and any late `onNext` goes to `onDiscard` as well. Existing single-argument call sites keep compiling (the new parameter is an overload, which also keeps MiMa happy for `core`). Docs in `doc/streaming/flows.md` describe the new callback.","tasks":[{"title":"Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","description":"In core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala add `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg `fromPublisher` delegate to it with `_ => ()`. Implement per the brief's design decisions: `onNext` uses `sendOrClosed` and discards on closed; `onError` stores the throwable in an AtomicReference and calls `doneOrClosed`; `onComplete` calls `doneOrClosed`; `onSubscribe` cancels immediately if a `cancelled` flag is already set, else requests `capacity`; the consumer loop rethrows the stored error when it sees `Done`; on any Throwable escaping the loop: set `cancelled`, cancel the subscription if present, `doneOrClosed` the channel, drain it calling `onDiscard` per element (keep draining across an InterruptedException and re-interrupt the thread afterwards), rethrow. Preserve the existing batched demand logic. Update the scaladoc of both overloads (exactly-once invariant, callback runs on publisher threads and must not throw, buffered elements are now emitted before a publisher error is rethrown).\n\nTests, in a new core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardTest.scala (AnyFunSuite with Matchers), using a hand-written `ManualPublisher[T]` helper as described in the brief (keep the helper in this file or a sibling file in the same package so Task 2 can reuse it):\n1. Publisher error with buffered elements: push 1,2,3 then `error(boom)` before/while the consumer runs; `runToList` throws boom and all of 1,2,3 were emitted; `onDiscard` never called.\n2. Publisher error with nothing buffered still throws.\n3. Consumer exception: `.map(x => if x == 2 then throw boom else x)`, push 1..5; `runToList` throws boom; discarded == 3,4,5 (order-independent compare), subscription cancelled.\n4. Consumer stops via `take(2)`: result List(1,2), discarded == 3,4,5, cancelled.\n5. Consumer interrupted: run `runForeach(x => { received.release(); gate.acquire() })` in `forkCancellable`, push 1..5, wait for `received`, `cancel()`; discarded == 2,3,4,5, cancelled.\n6. Late onNext after cancellation: after any of the above, call `next(6)` on the publisher; `onDiscard(6)` is called synchronously and nothing throws.\n7. Existing one-arg overload: compiles unchanged and `FlowPublisherPekkoTest` still passes.\nRun `sbt \"core/testOnly ox.flow.reactive.*\"` and `sbt core/scalafmt`; all green."},{"title":"Exactly-once stress test: every element is emitted or discarded, never both or neither","description":"Add a test (same file as Task 1 or a sibling in core/src/test/scala/ox/flow/reactive/) that uses a demand-respecting publisher running on its own thread: it emits 1..N (N around 1000) only while `sent < requested` and not cancelled, calling `onNext` from that thread, then `onComplete`. Compose `Flow.fromPublisher(p, discarded.add)` with a consumer that stops at a random point (e.g. `take(k)` for random k, and separately a `map` that throws at a random element, and separately a `forkCancellable` cancelled after a random short delay). Collect emitted elements via `.tap(emitted.add)` into thread-safe collections. After the flow finishes and the publisher thread is joined, assert: `emitted` and `discarded` are disjoint, their union equals exactly the set of elements the publisher actually sent, and no element appears twice. Repeat each scenario in a loop (e.g. 20 iterations) with a small `given BufferCapacity = BufferCapacity(4)` to make buffer-full races likely. The publisher's late `onNext` calls (after cancellation) must land in `discarded`. If this exposes a race in Task 1's implementation, fix it in FlowCompanionReactiveOps.scala as part of this task. Run `sbt \"core/testOnly ox.flow.reactive.*\"`."},{"title":"Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","description":"In flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala add `def fromPublisher[T](p: org.reactivestreams.Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p), onDiscard)`, keep the one-arg overload, and update its scaladoc to point at the core overload for the callback semantics. Create flow-reactive-streams/src/test/scala/ox/flow/reactive/FlowReactiveStreamsFromPublisherTest.scala (AnyFunSuite with Matchers) with a small hand-written `org.reactivestreams.Publisher[Int]` (same shape as the core ManualPublisher, but implementing the reactive-streams interfaces; it cannot be shared because the module does not depend on core's test classpath) and two tests: elements flow through the one-arg variant; with `.take(1)` and the new overload, remaining pushed elements are passed to `onDiscard` and the subscription is cancelled. Run `sbt flowReactiveStreams/test` and `sbt flowReactiveStreams/scalafmt`."},{"title":"Document the onDiscard callback in doc/streaming/flows.md","description":"In doc/streaming/flows.md, section \"### Publisher -> Flow\" (around lines 211-219), add a short paragraph explaining: elements received from the publisher but not emitted downstream (buffered when the publisher fails, when the consumer stops early via exception, interruption or `take`, or arriving after cancellation) can be released by passing a second argument `onDiscard: T => Unit` to `Flow.fromPublisher` / `FlowReactiveStreams.fromPublisher`; it is called exactly once per such element, possibly from the publisher's thread, and should not throw; typical use is releasing reference-counted buffers. Also state that on publisher error, elements already received are emitted before the error is propagated. Keep the prose as short as the surrounding text. If you add a code example, use a ```` ```scala mdoc:compile-only ```` block and verify with `sbt compileDocumentation`; otherwise no build step is needed. Do not touch generated-doc/out."}],"brief":"## Modules and files\n\n- `core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala` — the only implementation file in core. Trait `FlowCompanionReactiveOps` (mixed into `Flow.type`) defines `def fromPublisher[T](p: java.util.concurrent.Flow.Publisher[T])(using BufferCapacity): Flow[T]`. Read the whole file (80 lines) before changing anything. Current structure: `usingEmitInline: emit => unsupervised { channel = BufferCapacity.newChannel[T]; forkUnsupervised { p.subscribe(Subscriber: onSubscribe -> subscriptionRef.set(s); s.request(capacity); onNext -> channel.send; onError -> channel.error; onComplete -> channel.done) }; repeatWhile { channel.receiveOrClosed() match Done -> false / Error -> throw / t -> emit(t); batched subscription.request } }.tapException(cancel subscription)`. The demand logic (`demandThreshold = ceil(capacity/2)`, `toDemand` counter, lazily cached `subscription` read from `subscriptionRef` after the first element) must be preserved.\n- `flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala` — `object FlowReactiveStreams { def fromPublisher[T](p: org.reactivestreams.Publisher[T])(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p)) }`. Module has no `src/test` dir yet; scalatest is already a test dependency (build.sbt `flowReactiveStreams`), it `dependsOn(core)` (compile only, not test->test).\n- `core/src/test/scala/ox/flow/reactive/` — existing reactive tests: `FlowPublisherPekkoTest.scala` (AnyFunSuite + Matchers, uses Pekko `Source...runWith(Sink.asPublisher)` and `FlowAdapters.toFlowPublisher`) and `FlowPublisherTckTest.scala`. Put new fromPublisher tests here. No test anywhere currently uses a hand-written Publisher; write one (see below).\n- `doc/streaming/flows.md` lines 211-219, section \"### Publisher -> Flow\" — the only user doc mentioning `fromPublisher`. `generated-doc/out/**` is mdoc output regenerated at release time by `updateDocs`; do NOT edit it. Code blocks in docs use ```` ```scala mdoc:compile-only ```` and are compiled by `sbt compileDocumentation` (slow; only needed if you add a code block). Doc dependency snippet convention: `// sbt dependency: \"com.softwaremill.ox\" %% \"flow-reactive-streams\" % \"@VERSION@\"`.\n\n## Key APIs (ox.channels, core/src/main/scala/ox/channels/Channel.scala)\n\n- `Sink.sendOrClosed(t): Unit | ChannelClosed` (non-throwing), `Sink.send(t)` throws `ChannelClosedException` when closed.\n- `Sink.doneOrClosed(): Unit | ChannelClosed` — \"buffered values will be delivered; send operations in progress complete normally when a receiver arrives\"; returns `ChannelClosed` instead of throwing if already closed. `Sink.errorOrClosed(reason)` — drops buffered values (that is the bug). Use the `*OrClosed` variants everywhere in the subscriber: a misbehaving publisher may call onError/onComplete twice.\n- `Source.receiveOrClosed(): T | ChannelClosed` (blocking); match `case ChannelClosed.Done => ...; case e: ChannelClosed.Error => throw e.toThrowable; case t: T @unchecked => ...`. `Source.tryReceiveOrClosed()` is non-blocking but may spuriously return None, so it is not suitable for a drain loop.\n- `Source.foreachOrError(f)` / `drainOrError()` from `SourceDrainOps.scala` loop until Done; usable for the drain, but write the drain inline so you can handle `InterruptedException` (see below).\n- jox semantics (com.softwaremill.jox channels 1.1.2, verified from source): after `done()`, buffered elements stay receivable; a receiver observing Done has already seen every element whose `send` started before `done()` (an in-progress sender is waited for); a `send` racing with `done()` either lands in the buffer or returns closed. Hence \"drain until Done, then `sendOrClosed` returns closed\" gives exactly-once discard with no extra locking.\n- `BufferCapacity`: `given default: BufferCapacity = 16`; `BufferCapacity.newChannel[T]`; `summon[BufferCapacity].toInt` (import `ox.channels.toInt`). Tests can override with `given BufferCapacity = BufferCapacity(4)`.\n- `ox.util`: `t.tapException(f)` runs `f(e)` for any Throwable and rethrows `e` (exceptions from `f` added as suppressed). `ox.pipe`, `ox.discard`, `ox.repeatWhile` already imported in the file.\n- `FlowOps.take(n)` stops upstream by throwing a private `ControlThrowable` (`abortTake`) out of `emit`, caught in `take` itself. So \"consumer stops early\" includes `take`, and the cleanup path must run for every Throwable, not just NonFatal. `tapException`/`catch case e: Throwable` already does this.\n- `unsupervised { ... }` + `forkUnsupervised { ... }` (import `ox.unsupervised`, `ox.forkUnsupervised`): scope end interrupts still-running forks once. `onNext`/`onError`/`onComplete` run on the publisher's threads (or on the `p.subscribe` fork for synchronous publishers), never on the consumer thread.\n- Test helpers: `ox.forkCancellable { ... }` inside `supervised` returns a `CancellableFork` with `cancel(): Either[Throwable, T]` (interrupts once, joins). Interruption inside the flow surfaces as `InterruptedException` thrown from a blocking call (e.g. `Semaphore.acquire`, `channel.receiveOrClosed`).\n\n## Design decisions (apply as-is)\n\n- API: add an overload `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg method delegate with `_ => ()`. Do not add a default parameter and do not change the existing signature: `core` runs MiMa (`enableMimaSettings` in build.sbt) and only additive changes are safe. Same shape in `FlowReactiveStreams`.\n- Subscriber: `onNext(t)` = `channel.sendOrClosed(t)` and if the result is `ChannelClosed` call `onDiscard(t)`. `onError(e)` = store `e` in an `AtomicReference[Throwable]`, then `channel.doneOrClosed().discard`. `onComplete()` = `channel.doneOrClosed().discard`. `onSubscribe(s)` = set `subscriptionRef`, then if a `cancelled: AtomicBoolean` is already set call `s.cancel()` else `s.request(capacity)` (both sides use volatile atomics, so at least one side sees the other; a double `cancel()` is allowed by the spec).\n- Consumer loop: on `ChannelClosed.Done`, read the stored error; if non-null throw it, else finish. Keep the `ChannelClosed.Error` branch for safety.\n- Cleanup, run on any Throwable escaping the loop (the stored publisher error included; cancel after onError is a spec-allowed no-op): set `cancelled`, cancel the subscription if `subscriptionRef` is non-null, `channel.doneOrClosed().discard`, then drain: loop `receiveOrClosed()` until `Done`, calling `onDiscard` for each element. The drain is bounded (channel is done, at most capacity elements plus in-flight sends) but blocking; if `InterruptedException` is thrown mid-drain, catch it, keep draining, and call `Thread.currentThread().interrupt()` once the drain finishes, so no element is lost on a second interrupt. Then rethrow the original exception. An element that was passed to `emit` and whose `emit` threw counts as emitted (downstream owns it): do not discard it.\n- Scaladoc on the new overload: state the exactly-once invariant, that `onDiscard` may be called from publisher threads, that it must not throw, and that on publisher error buffered elements are emitted before the error is rethrown (this is a deliberate behaviour change for the one-arg variant too; mention it in its doc).\n\n## Test publisher helper (write it in the core test, package `ox.flow.reactive`)\n\nA `ManualPublisher[T] extends java.util.concurrent.Flow.Publisher[T]` with: `@volatile var subscriber: Subscriber[? >: T]`, `requested: AtomicLong`, `cancelled: AtomicBoolean`, a `subscribed: CountDownLatch(1)`; `subscribe(s)` stores `s`, calls `s.onSubscribe(new Subscription { request(n) => requested.addAndGet(n); cancel() => cancelled.set(true) })`, counts down the latch. Helper methods `next(t)`, `error(e)`, `complete()` forward to the subscriber from the test thread. For the exactly-once stress test, a second variant that runs its own thread emitting `1..n` while `sent < requested` and not cancelled. Run the flow under test in `supervised { forkCancellable { ... } }` or `fork`, await `subscribed`, then drive the publisher. Collect emitted elements with `.tap(emitted += _)` or `runForeach`, discarded with `onDiscard = discarded += _` (use thread-safe collections such as `ConcurrentLinkedQueue` or `java.util.Collections.synchronizedList`, since `onDiscard` runs on other threads).\n\n## Conventions / build\n\n- Scala 3 indentation syntax, `end X` markers on long definitions, `.discard` on ignored results (compiler warns otherwise), scalafmt configured in `.scalafmt.conf` (140-column lines); run `sbt core/scalafmt flowReactiveStreams/scalafmt` after editing.\n- Comments: short, explain non-obvious reasoning only; never restate code. Scaladoc describes usage and invariants, not implementation.\n- Tests: scalatest 3.2.20; the reactive tests use `AnyFunSuite with Matchers` with `test(\"...\")` blocks and Scala 3 colon syntax. `core` runs tests forked (`Test / fork := true`).\n- Commands (sbt 2.0.10): `sbt \"core/testOnly ox.flow.reactive.*\"`, `sbt flowReactiveStreams/test`, `sbt core/mimaReportBinaryIssues` (only meaningful when a previous stable version resolves), `sbt compileDocumentation` for mdoc. Never commit; the user commits."}},{"id":[{"name":"Task: Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Exactly-once stress test: every element is emitted or discarded, never both or neither","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","occurrence":0}],"resultJson":{"findings":[{"id":"R1.I4.1","title":"sbt Test/compile fails with JGit NoWorkTreeException","reason":{"type":"Declined","text":"This is a problem with the environment, not the code. git sees the worktree as non-bare (`git rev-parse --is-bare-repository` prints false; the git dir is /home/adamw/ox/.git/worktrees/zippy-astrodon-94aa2fb6). The exception comes from the sbt-git plugin, which uses JGit by default and reads this linked worktree as bare while the build loads. Nothing in project/ or the build configures this; `zz-tmp-git.sbt` was my temporary workaround, not the cause. To check the code, I recreated it with `useReadableConsoleGit`, which makes sbt-git use command-line git: `sbt core/Test/compile flowReactiveStreams/Test/compile` succeeded (warnings only, in older test files, no errors) and flowReactiveStreams tests passed. I then deleted the file again. Changing the shared build's git plugin setup is outside the scope of this change."}}]}},{"id":[{"name":"Task: Document the onDiscard callback in doc/streaming/flows.md","occurrence":0}],"resultJson":{"findings":[{"id":"R1.I4.1","title":"`sbt Test/compile` fails with no diagnostic","reason":{"type":"Declined","text":"Environmental. Full output shows sbt-git's JGit throws NoWorkTreeException (\"Bare Repository has neither a working tree, nor an index\") while loading the build, because this checkout is a git worktree; no source error is involved. With a temporary `ThisBuild / git.gitUncommittedChanges := false` override, `sbt Test/compile` succeeds across all modules (only pre-existing unrelated warnings). The temporary file was removed; no codebase change is needed."}}]}}],"stageStarts":[]} \ No newline at end of file +{"header":{"startingBranch":"zippy-astrodon-94aa2fb6","branch":"zippy-astrodon-94aa2fb6","branchMode":"Reused","userPrompt":"Implement GitHub issue softwaremill/ox#528: \"Flow.fromPublisher: allow releasing elements which are buffered but not emitted\".\n\nProblem: `Flow.fromPublisher` (core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala) buffers elements in a channel. Some of them never reach the consumer:\n- when the publisher fails, `channel.error` drops the buffered elements\n- when the consumer stops early (an exception, an interruption), the buffered elements are never received\n- elements can still arrive (onNext) after the subscription is cancelled\n\nIf elements hold resources (e.g. Netty's reference-counted ByteBufs), nobody releases them, so they leak.\n\nGoal: add an optional callback, e.g. `fromPublisher(p, onDiscard: T => Unit)`, called exactly once for each element received from the publisher but not emitted downstream. Existing call sites without the callback must keep compiling and behaving the same.\n\nSuggested implementation: on publisher error, store the error and call `channel.done()` (so buffered elements are kept and emitted before the error is rethrown). When the flow ends early, cancel the subscription, close the channel, and drain it, calling the callback for each drained element. In `onNext`, call the callback if the channel is already closed. Make sure there are no races where an element is neither emitted nor discarded.\n\nAlso update the variant in flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala (org.reactivestreams Publisher) to accept the same optional callback, and update any docs (doc/ directory) describing fromPublisher.\n\nAcceptance criteria:\n- tests covering: publisher error with buffered elements (elements still emitted, then error); consumer stopping early (exception / interruption) with buffered elements -> callback called for each; elements arriving after cancellation -> callback called; every element is either emitted or discarded, never both or neither.\n- `sbt compile test` passes for affected modules (core, flow-reactive-streams).\n- Code follows existing style in the repo; concise doc comments.\n\nContext: tapir currently tracks unconsumed elements itself using a lock: https://github.com/softwaremill/tapir/pull/5573","flow":{"type":"Catalog","name":"implement.sc"},"startingCommit":"664f01730cda34fd88e8c3db3deb600934eb1d1e"},"entries":[{"id":[{"name":"Plan","occurrence":0}],"resultJson":{"id":"ox-528-from-publisher-on-discard","description":"`Flow.fromPublisher` buffers elements received from a reactive `Publisher` in a channel. Today, elements that are buffered but never emitted are silently lost: `channel.error` drops the buffer on publisher error, a consumer that stops early (exception, interruption, `take`) never receives what is buffered, and `onNext` calls that arrive after the subscription is cancelled are dropped by `channel.send` throwing. When elements own resources (Netty reference-counted ByteBufs in tapir), this leaks them. Tapir currently works around it with its own lock-based tracking (softwaremill/tapir#5573).\n\nThis epic adds an optional `onDiscard: T => Unit` callback to `Flow.fromPublisher` (core, JDK `Flow.Publisher`) and `FlowReactiveStreams.fromPublisher` (flow-reactive-streams, `org.reactivestreams.Publisher`), with the invariant that every element received from the publisher is either emitted downstream or passed to `onDiscard`, exactly once, never both or neither. Publisher errors no longer drop buffered elements: the error is stored, the channel is completed with `done()`, buffered elements are emitted, and then the error is rethrown. When the flow ends early, the subscription is cancelled, the channel is closed and drained through `onDiscard`, and any late `onNext` goes to `onDiscard` as well. Existing single-argument call sites keep compiling (the new parameter is an overload, which also keeps MiMa happy for `core`). Docs in `doc/streaming/flows.md` describe the new callback.","tasks":[{"title":"Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","description":"In core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala add `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg `fromPublisher` delegate to it with `_ => ()`. Implement per the brief's design decisions: `onNext` uses `sendOrClosed` and discards on closed; `onError` stores the throwable in an AtomicReference and calls `doneOrClosed`; `onComplete` calls `doneOrClosed`; `onSubscribe` cancels immediately if a `cancelled` flag is already set, else requests `capacity`; the consumer loop rethrows the stored error when it sees `Done`; on any Throwable escaping the loop: set `cancelled`, cancel the subscription if present, `doneOrClosed` the channel, drain it calling `onDiscard` per element (keep draining across an InterruptedException and re-interrupt the thread afterwards), rethrow. Preserve the existing batched demand logic. Update the scaladoc of both overloads (exactly-once invariant, callback runs on publisher threads and must not throw, buffered elements are now emitted before a publisher error is rethrown).\n\nTests, in a new core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardTest.scala (AnyFunSuite with Matchers), using a hand-written `ManualPublisher[T]` helper as described in the brief (keep the helper in this file or a sibling file in the same package so Task 2 can reuse it):\n1. Publisher error with buffered elements: push 1,2,3 then `error(boom)` before/while the consumer runs; `runToList` throws boom and all of 1,2,3 were emitted; `onDiscard` never called.\n2. Publisher error with nothing buffered still throws.\n3. Consumer exception: `.map(x => if x == 2 then throw boom else x)`, push 1..5; `runToList` throws boom; discarded == 3,4,5 (order-independent compare), subscription cancelled.\n4. Consumer stops via `take(2)`: result List(1,2), discarded == 3,4,5, cancelled.\n5. Consumer interrupted: run `runForeach(x => { received.release(); gate.acquire() })` in `forkCancellable`, push 1..5, wait for `received`, `cancel()`; discarded == 2,3,4,5, cancelled.\n6. Late onNext after cancellation: after any of the above, call `next(6)` on the publisher; `onDiscard(6)` is called synchronously and nothing throws.\n7. Existing one-arg overload: compiles unchanged and `FlowPublisherPekkoTest` still passes.\nRun `sbt \"core/testOnly ox.flow.reactive.*\"` and `sbt core/scalafmt`; all green."},{"title":"Exactly-once stress test: every element is emitted or discarded, never both or neither","description":"Add a test (same file as Task 1 or a sibling in core/src/test/scala/ox/flow/reactive/) that uses a demand-respecting publisher running on its own thread: it emits 1..N (N around 1000) only while `sent < requested` and not cancelled, calling `onNext` from that thread, then `onComplete`. Compose `Flow.fromPublisher(p, discarded.add)` with a consumer that stops at a random point (e.g. `take(k)` for random k, and separately a `map` that throws at a random element, and separately a `forkCancellable` cancelled after a random short delay). Collect emitted elements via `.tap(emitted.add)` into thread-safe collections. After the flow finishes and the publisher thread is joined, assert: `emitted` and `discarded` are disjoint, their union equals exactly the set of elements the publisher actually sent, and no element appears twice. Repeat each scenario in a loop (e.g. 20 iterations) with a small `given BufferCapacity = BufferCapacity(4)` to make buffer-full races likely. The publisher's late `onNext` calls (after cancellation) must land in `discarded`. If this exposes a race in Task 1's implementation, fix it in FlowCompanionReactiveOps.scala as part of this task. Run `sbt \"core/testOnly ox.flow.reactive.*\"`."},{"title":"Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","description":"In flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala add `def fromPublisher[T](p: org.reactivestreams.Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p), onDiscard)`, keep the one-arg overload, and update its scaladoc to point at the core overload for the callback semantics. Create flow-reactive-streams/src/test/scala/ox/flow/reactive/FlowReactiveStreamsFromPublisherTest.scala (AnyFunSuite with Matchers) with a small hand-written `org.reactivestreams.Publisher[Int]` (same shape as the core ManualPublisher, but implementing the reactive-streams interfaces; it cannot be shared because the module does not depend on core's test classpath) and two tests: elements flow through the one-arg variant; with `.take(1)` and the new overload, remaining pushed elements are passed to `onDiscard` and the subscription is cancelled. Run `sbt flowReactiveStreams/test` and `sbt flowReactiveStreams/scalafmt`."},{"title":"Document the onDiscard callback in doc/streaming/flows.md","description":"In doc/streaming/flows.md, section \"### Publisher -> Flow\" (around lines 211-219), add a short paragraph explaining: elements received from the publisher but not emitted downstream (buffered when the publisher fails, when the consumer stops early via exception, interruption or `take`, or arriving after cancellation) can be released by passing a second argument `onDiscard: T => Unit` to `Flow.fromPublisher` / `FlowReactiveStreams.fromPublisher`; it is called exactly once per such element, possibly from the publisher's thread, and should not throw; typical use is releasing reference-counted buffers. Also state that on publisher error, elements already received are emitted before the error is propagated. Keep the prose as short as the surrounding text. If you add a code example, use a ```` ```scala mdoc:compile-only ```` block and verify with `sbt compileDocumentation`; otherwise no build step is needed. Do not touch generated-doc/out."}],"brief":"## Modules and files\n\n- `core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala` — the only implementation file in core. Trait `FlowCompanionReactiveOps` (mixed into `Flow.type`) defines `def fromPublisher[T](p: java.util.concurrent.Flow.Publisher[T])(using BufferCapacity): Flow[T]`. Read the whole file (80 lines) before changing anything. Current structure: `usingEmitInline: emit => unsupervised { channel = BufferCapacity.newChannel[T]; forkUnsupervised { p.subscribe(Subscriber: onSubscribe -> subscriptionRef.set(s); s.request(capacity); onNext -> channel.send; onError -> channel.error; onComplete -> channel.done) }; repeatWhile { channel.receiveOrClosed() match Done -> false / Error -> throw / t -> emit(t); batched subscription.request } }.tapException(cancel subscription)`. The demand logic (`demandThreshold = ceil(capacity/2)`, `toDemand` counter, lazily cached `subscription` read from `subscriptionRef` after the first element) must be preserved.\n- `flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala` — `object FlowReactiveStreams { def fromPublisher[T](p: org.reactivestreams.Publisher[T])(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p)) }`. Module has no `src/test` dir yet; scalatest is already a test dependency (build.sbt `flowReactiveStreams`), it `dependsOn(core)` (compile only, not test->test).\n- `core/src/test/scala/ox/flow/reactive/` — existing reactive tests: `FlowPublisherPekkoTest.scala` (AnyFunSuite + Matchers, uses Pekko `Source...runWith(Sink.asPublisher)` and `FlowAdapters.toFlowPublisher`) and `FlowPublisherTckTest.scala`. Put new fromPublisher tests here. No test anywhere currently uses a hand-written Publisher; write one (see below).\n- `doc/streaming/flows.md` lines 211-219, section \"### Publisher -> Flow\" — the only user doc mentioning `fromPublisher`. `generated-doc/out/**` is mdoc output regenerated at release time by `updateDocs`; do NOT edit it. Code blocks in docs use ```` ```scala mdoc:compile-only ```` and are compiled by `sbt compileDocumentation` (slow; only needed if you add a code block). Doc dependency snippet convention: `// sbt dependency: \"com.softwaremill.ox\" %% \"flow-reactive-streams\" % \"@VERSION@\"`.\n\n## Key APIs (ox.channels, core/src/main/scala/ox/channels/Channel.scala)\n\n- `Sink.sendOrClosed(t): Unit | ChannelClosed` (non-throwing), `Sink.send(t)` throws `ChannelClosedException` when closed.\n- `Sink.doneOrClosed(): Unit | ChannelClosed` — \"buffered values will be delivered; send operations in progress complete normally when a receiver arrives\"; returns `ChannelClosed` instead of throwing if already closed. `Sink.errorOrClosed(reason)` — drops buffered values (that is the bug). Use the `*OrClosed` variants everywhere in the subscriber: a misbehaving publisher may call onError/onComplete twice.\n- `Source.receiveOrClosed(): T | ChannelClosed` (blocking); match `case ChannelClosed.Done => ...; case e: ChannelClosed.Error => throw e.toThrowable; case t: T @unchecked => ...`. `Source.tryReceiveOrClosed()` is non-blocking but may spuriously return None, so it is not suitable for a drain loop.\n- `Source.foreachOrError(f)` / `drainOrError()` from `SourceDrainOps.scala` loop until Done; usable for the drain, but write the drain inline so you can handle `InterruptedException` (see below).\n- jox semantics (com.softwaremill.jox channels 1.1.2, verified from source): after `done()`, buffered elements stay receivable; a receiver observing Done has already seen every element whose `send` started before `done()` (an in-progress sender is waited for); a `send` racing with `done()` either lands in the buffer or returns closed. Hence \"drain until Done, then `sendOrClosed` returns closed\" gives exactly-once discard with no extra locking.\n- `BufferCapacity`: `given default: BufferCapacity = 16`; `BufferCapacity.newChannel[T]`; `summon[BufferCapacity].toInt` (import `ox.channels.toInt`). Tests can override with `given BufferCapacity = BufferCapacity(4)`.\n- `ox.util`: `t.tapException(f)` runs `f(e)` for any Throwable and rethrows `e` (exceptions from `f` added as suppressed). `ox.pipe`, `ox.discard`, `ox.repeatWhile` already imported in the file.\n- `FlowOps.take(n)` stops upstream by throwing a private `ControlThrowable` (`abortTake`) out of `emit`, caught in `take` itself. So \"consumer stops early\" includes `take`, and the cleanup path must run for every Throwable, not just NonFatal. `tapException`/`catch case e: Throwable` already does this.\n- `unsupervised { ... }` + `forkUnsupervised { ... }` (import `ox.unsupervised`, `ox.forkUnsupervised`): scope end interrupts still-running forks once. `onNext`/`onError`/`onComplete` run on the publisher's threads (or on the `p.subscribe` fork for synchronous publishers), never on the consumer thread.\n- Test helpers: `ox.forkCancellable { ... }` inside `supervised` returns a `CancellableFork` with `cancel(): Either[Throwable, T]` (interrupts once, joins). Interruption inside the flow surfaces as `InterruptedException` thrown from a blocking call (e.g. `Semaphore.acquire`, `channel.receiveOrClosed`).\n\n## Design decisions (apply as-is)\n\n- API: add an overload `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg method delegate with `_ => ()`. Do not add a default parameter and do not change the existing signature: `core` runs MiMa (`enableMimaSettings` in build.sbt) and only additive changes are safe. Same shape in `FlowReactiveStreams`.\n- Subscriber: `onNext(t)` = `channel.sendOrClosed(t)` and if the result is `ChannelClosed` call `onDiscard(t)`. `onError(e)` = store `e` in an `AtomicReference[Throwable]`, then `channel.doneOrClosed().discard`. `onComplete()` = `channel.doneOrClosed().discard`. `onSubscribe(s)` = set `subscriptionRef`, then if a `cancelled: AtomicBoolean` is already set call `s.cancel()` else `s.request(capacity)` (both sides use volatile atomics, so at least one side sees the other; a double `cancel()` is allowed by the spec).\n- Consumer loop: on `ChannelClosed.Done`, read the stored error; if non-null throw it, else finish. Keep the `ChannelClosed.Error` branch for safety.\n- Cleanup, run on any Throwable escaping the loop (the stored publisher error included; cancel after onError is a spec-allowed no-op): set `cancelled`, cancel the subscription if `subscriptionRef` is non-null, `channel.doneOrClosed().discard`, then drain: loop `receiveOrClosed()` until `Done`, calling `onDiscard` for each element. The drain is bounded (channel is done, at most capacity elements plus in-flight sends) but blocking; if `InterruptedException` is thrown mid-drain, catch it, keep draining, and call `Thread.currentThread().interrupt()` once the drain finishes, so no element is lost on a second interrupt. Then rethrow the original exception. An element that was passed to `emit` and whose `emit` threw counts as emitted (downstream owns it): do not discard it.\n- Scaladoc on the new overload: state the exactly-once invariant, that `onDiscard` may be called from publisher threads, that it must not throw, and that on publisher error buffered elements are emitted before the error is rethrown (this is a deliberate behaviour change for the one-arg variant too; mention it in its doc).\n\n## Test publisher helper (write it in the core test, package `ox.flow.reactive`)\n\nA `ManualPublisher[T] extends java.util.concurrent.Flow.Publisher[T]` with: `@volatile var subscriber: Subscriber[? >: T]`, `requested: AtomicLong`, `cancelled: AtomicBoolean`, a `subscribed: CountDownLatch(1)`; `subscribe(s)` stores `s`, calls `s.onSubscribe(new Subscription { request(n) => requested.addAndGet(n); cancel() => cancelled.set(true) })`, counts down the latch. Helper methods `next(t)`, `error(e)`, `complete()` forward to the subscriber from the test thread. For the exactly-once stress test, a second variant that runs its own thread emitting `1..n` while `sent < requested` and not cancelled. Run the flow under test in `supervised { forkCancellable { ... } }` or `fork`, await `subscribed`, then drive the publisher. Collect emitted elements with `.tap(emitted += _)` or `runForeach`, discarded with `onDiscard = discarded += _` (use thread-safe collections such as `ConcurrentLinkedQueue` or `java.util.Collections.synchronizedList`, since `onDiscard` runs on other threads).\n\n## Conventions / build\n\n- Scala 3 indentation syntax, `end X` markers on long definitions, `.discard` on ignored results (compiler warns otherwise), scalafmt configured in `.scalafmt.conf` (140-column lines); run `sbt core/scalafmt flowReactiveStreams/scalafmt` after editing.\n- Comments: short, explain non-obvious reasoning only; never restate code. Scaladoc describes usage and invariants, not implementation.\n- Tests: scalatest 3.2.20; the reactive tests use `AnyFunSuite with Matchers` with `test(\"...\")` blocks and Scala 3 colon syntax. `core` runs tests forked (`Test / fork := true`).\n- Commands (sbt 2.0.10): `sbt \"core/testOnly ox.flow.reactive.*\"`, `sbt flowReactiveStreams/test`, `sbt core/mimaReportBinaryIssues` (only meaningful when a previous stable version resolves), `sbt compileDocumentation` for mdoc. Never commit; the user commits."}},{"id":[{"name":"Task: Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Exactly-once stress test: every element is emitted or discarded, never both or neither","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","occurrence":0}],"resultJson":{"findings":[{"id":"R1.I4.1","title":"sbt Test/compile fails with JGit NoWorkTreeException","reason":{"type":"Declined","text":"This is a problem with the environment, not the code. git sees the worktree as non-bare (`git rev-parse --is-bare-repository` prints false; the git dir is /home/adamw/ox/.git/worktrees/zippy-astrodon-94aa2fb6). The exception comes from the sbt-git plugin, which uses JGit by default and reads this linked worktree as bare while the build loads. Nothing in project/ or the build configures this; `zz-tmp-git.sbt` was my temporary workaround, not the cause. To check the code, I recreated it with `useReadableConsoleGit`, which makes sbt-git use command-line git: `sbt core/Test/compile flowReactiveStreams/Test/compile` succeeded (warnings only, in older test files, no errors) and flowReactiveStreams tests passed. I then deleted the file again. Changing the shared build's git plugin setup is outside the scope of this change."}}]}},{"id":[{"name":"Task: Document the onDiscard callback in doc/streaming/flows.md","occurrence":0}],"resultJson":{"findings":[{"id":"R1.I4.1","title":"`sbt Test/compile` fails with no diagnostic","reason":{"type":"Declined","text":"Environmental. Full output shows sbt-git's JGit throws NoWorkTreeException (\"Bare Repository has neither a working tree, nor an index\") while loading the build, because this checkout is a git worktree; no source error is involved. With a temporary `ThisBuild / git.gitUncommittedChanges := false` override, `sbt Test/compile` succeeds across all modules (only pre-existing unrelated warnings). The temporary file was removed; no codebase change is needed."}}]}},{"id":[{"name":"Final review","occurrence":0}],"resultJson":{"findings":[{"id":"S1","title":"sbt Test/compile fails with JGit NoWorkTreeException","reason":{"type":"Declined","text":"This is a problem with the environment, not the code. git sees the worktree as non-bare (`git rev-parse --is-bare-repository` prints false; the git dir is /home/adamw/ox/.git/worktrees/zippy-astrodon-94aa2fb6). The exception comes from the sbt-git plugin, which uses JGit by default and reads this linked worktree as bare while the build loads. Nothing in project/ or the build configures this; `zz-tmp-git.sbt` was my temporary workaround, not the cause. To check the code, I recreated it with `useReadableConsoleGit`, which makes sbt-git use command-line git: `sbt core/Test/compile flowReactiveStreams/Test/compile` succeeded (warnings only, in older test files, no errors) and flowReactiveStreams tests passed. I then deleted the file again. Changing the shared build's git plugin setup is outside the scope of this change."}},{"id":"S2","title":"`sbt Test/compile` fails with no diagnostic","reason":{"type":"Declined","text":"Environmental. Full output shows sbt-git's JGit throws NoWorkTreeException (\"Bare Repository has neither a working tree, nor an index\") while loading the build, because this checkout is a git worktree; no source error is involved. With a temporary `ThisBuild / git.gitUncommittedChanges := false` override, `sbt Test/compile` succeeds across all modules (only pre-existing unrelated warnings). The temporary file was removed; no codebase change is needed."}},{"id":"R1.I2.1","title":"No test covers the drain's deferred-interrupt path in discardBuffered","reason":{"type":"Declined","text":"A deterministic test can't be written. In jox 1.1.2, `receiveOrClosed` only throws `InterruptedException` when it parks (`Continuation.await`). After `doneOrClosed()` has finished on the draining thread, a receive never parks: buffered or in-flight sends are returned or retried, and later cells are already CLOSED. The catch is reached only in a narrow race, when a concurrent `onComplete`/`onError` is still closing cells. I tried the suggested test (interrupt plus throw in the consumer, then assert the discards and the interrupt flag). It passed both with the catch removed and with the re-interrupt removed, so it did not cover the branch, and I reverted it. The suggested capacity-1 variant also doesn't help, because a blocked sender's Continuation is resumed without parking."},"location":{"file":"core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala","line":141}},{"id":"R1.I5.1","title":"sbt Test/compile exited 1 with no compile error in the output","reason":{"type":"Declined","text":"Environmental. sbt-git's JGit fails in this git worktree with `NoWorkTreeException: Bare Repository has neither a working tree, nor an index` while evaluating `gitUncommittedChanges`, so the build fails before compiling. It is not a code problem. With a temporary `zz-tmp-git.sbt` that sets `ThisBuild / git.gitUncommittedChanges := false`, Test/compile and the core reactive and flowReactiveStreams tests all pass. I deleted the temporary file afterwards."}},{"id":"R2.I3.1","title":"`sbt Test/compile` exits 1 with no compiler diagnostics","reason":{"type":"Declined","text":"Environmental, not a code problem. In this git worktree, sbt-git's JGit throws `NoWorkTreeException: Bare Repository has neither a working tree, nor an index` while evaluating `gitUncommittedChanges` during build load. The build fails before any compilation, and the full output shows that stack trace above the timing line. With a temporary `zz-tmp-git.sbt` setting `ThisBuild / git.gitUncommittedChanges := false`, `Test/compile` succeeds. With the same file, `core/testOnly ox.flow.reactive.*` (44 passed, 7 ignored) and `flowReactiveStreams/test` (2 passed) also pass. I deleted the file afterwards, so it is not left in the tree."}}]}}],"stageStarts":[]} \ No newline at end of file diff --git a/core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala b/core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala index b1251416..9623e69b 100644 --- a/core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala +++ b/core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala @@ -18,20 +18,7 @@ import java.util.concurrent.atomic.AtomicReference trait FlowCompanionReactiveOps: this: Flow.type => - /** Creates a [[Flow]] from a [[Publisher]], that is, which emits the elements received by subscribing to the publisher. A new - * subscription is created every time this flow is run. - * - * The data is passed from a subscription to the flow using a [[ox.channel.Channel]], with a capacity given by the [[BufferCapacity]] in - * scope. That's also how many elements will be at most requested from the publisher at a time. - * - * If the publisher signals an error, elements received before the error are emitted first, and then the error is rethrown. - * - * Elements received from the publisher, but not emitted (e.g. because the flow failed, was interrupted or stopped early) are dropped. To - * release resources held by such elements, use the variant with an `onDiscard` callback. - * - * The publisher parameter should implement the JDK 9+ `Flow.Publisher` API. To create a flow from a publisher implementing - * `com.reactivestreams.Publisher`, use the `flow-reactive-streams` module. - */ + /** Same as the overload with `onDiscard`, except that elements received from the publisher, but not emitted, are dropped. */ def fromPublisher[T](p: Publisher[T])(using BufferCapacity): Flow[T] = fromPublisher(p, (_: T) => ()) /** Creates a [[Flow]] from a [[Publisher]], that is, which emits the elements received by subscribing to the publisher. A new diff --git a/flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala b/flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala index 0b41b796..df92d607 100644 --- a/flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala +++ b/flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala @@ -28,10 +28,7 @@ object FlowReactiveStreams: */ def fromPublisher[T](p: Publisher[T])(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p)) - /** This variant accepts an implementation of `org.reactivestreams.Publisher`, as opposed to `java.util.concurrent.Flow.Publisher` which - * is supported in the core module. - * - * See the core overload with `onDiscard` for the callback's semantics. + /** Same as the overload without `onDiscard`, but passes elements which are received but not emitted to `onDiscard`. * * @see * [[Flow.fromPublisher]] From 821325424e75d7150d6cc201c84ea0157416d014 Mon Sep 17 00:00:00 2001 From: Adam Warski Date: Fri, 9 Oct 2026 15:31:05 +0000 Subject: [PATCH 08/11] stage: Push branch --- .orca/runs/0a3279c8e3d7.progress.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.orca/runs/0a3279c8e3d7.progress.json b/.orca/runs/0a3279c8e3d7.progress.json index e7f43fe3..b02af71e 100644 --- a/.orca/runs/0a3279c8e3d7.progress.json +++ b/.orca/runs/0a3279c8e3d7.progress.json @@ -1 +1 @@ -{"header":{"startingBranch":"zippy-astrodon-94aa2fb6","branch":"zippy-astrodon-94aa2fb6","branchMode":"Reused","userPrompt":"Implement GitHub issue softwaremill/ox#528: \"Flow.fromPublisher: allow releasing elements which are buffered but not emitted\".\n\nProblem: `Flow.fromPublisher` (core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala) buffers elements in a channel. Some of them never reach the consumer:\n- when the publisher fails, `channel.error` drops the buffered elements\n- when the consumer stops early (an exception, an interruption), the buffered elements are never received\n- elements can still arrive (onNext) after the subscription is cancelled\n\nIf elements hold resources (e.g. Netty's reference-counted ByteBufs), nobody releases them, so they leak.\n\nGoal: add an optional callback, e.g. `fromPublisher(p, onDiscard: T => Unit)`, called exactly once for each element received from the publisher but not emitted downstream. Existing call sites without the callback must keep compiling and behaving the same.\n\nSuggested implementation: on publisher error, store the error and call `channel.done()` (so buffered elements are kept and emitted before the error is rethrown). When the flow ends early, cancel the subscription, close the channel, and drain it, calling the callback for each drained element. In `onNext`, call the callback if the channel is already closed. Make sure there are no races where an element is neither emitted nor discarded.\n\nAlso update the variant in flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala (org.reactivestreams Publisher) to accept the same optional callback, and update any docs (doc/ directory) describing fromPublisher.\n\nAcceptance criteria:\n- tests covering: publisher error with buffered elements (elements still emitted, then error); consumer stopping early (exception / interruption) with buffered elements -> callback called for each; elements arriving after cancellation -> callback called; every element is either emitted or discarded, never both or neither.\n- `sbt compile test` passes for affected modules (core, flow-reactive-streams).\n- Code follows existing style in the repo; concise doc comments.\n\nContext: tapir currently tracks unconsumed elements itself using a lock: https://github.com/softwaremill/tapir/pull/5573","flow":{"type":"Catalog","name":"implement.sc"},"startingCommit":"664f01730cda34fd88e8c3db3deb600934eb1d1e"},"entries":[{"id":[{"name":"Plan","occurrence":0}],"resultJson":{"id":"ox-528-from-publisher-on-discard","description":"`Flow.fromPublisher` buffers elements received from a reactive `Publisher` in a channel. Today, elements that are buffered but never emitted are silently lost: `channel.error` drops the buffer on publisher error, a consumer that stops early (exception, interruption, `take`) never receives what is buffered, and `onNext` calls that arrive after the subscription is cancelled are dropped by `channel.send` throwing. When elements own resources (Netty reference-counted ByteBufs in tapir), this leaks them. Tapir currently works around it with its own lock-based tracking (softwaremill/tapir#5573).\n\nThis epic adds an optional `onDiscard: T => Unit` callback to `Flow.fromPublisher` (core, JDK `Flow.Publisher`) and `FlowReactiveStreams.fromPublisher` (flow-reactive-streams, `org.reactivestreams.Publisher`), with the invariant that every element received from the publisher is either emitted downstream or passed to `onDiscard`, exactly once, never both or neither. Publisher errors no longer drop buffered elements: the error is stored, the channel is completed with `done()`, buffered elements are emitted, and then the error is rethrown. When the flow ends early, the subscription is cancelled, the channel is closed and drained through `onDiscard`, and any late `onNext` goes to `onDiscard` as well. Existing single-argument call sites keep compiling (the new parameter is an overload, which also keeps MiMa happy for `core`). Docs in `doc/streaming/flows.md` describe the new callback.","tasks":[{"title":"Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","description":"In core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala add `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg `fromPublisher` delegate to it with `_ => ()`. Implement per the brief's design decisions: `onNext` uses `sendOrClosed` and discards on closed; `onError` stores the throwable in an AtomicReference and calls `doneOrClosed`; `onComplete` calls `doneOrClosed`; `onSubscribe` cancels immediately if a `cancelled` flag is already set, else requests `capacity`; the consumer loop rethrows the stored error when it sees `Done`; on any Throwable escaping the loop: set `cancelled`, cancel the subscription if present, `doneOrClosed` the channel, drain it calling `onDiscard` per element (keep draining across an InterruptedException and re-interrupt the thread afterwards), rethrow. Preserve the existing batched demand logic. Update the scaladoc of both overloads (exactly-once invariant, callback runs on publisher threads and must not throw, buffered elements are now emitted before a publisher error is rethrown).\n\nTests, in a new core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardTest.scala (AnyFunSuite with Matchers), using a hand-written `ManualPublisher[T]` helper as described in the brief (keep the helper in this file or a sibling file in the same package so Task 2 can reuse it):\n1. Publisher error with buffered elements: push 1,2,3 then `error(boom)` before/while the consumer runs; `runToList` throws boom and all of 1,2,3 were emitted; `onDiscard` never called.\n2. Publisher error with nothing buffered still throws.\n3. Consumer exception: `.map(x => if x == 2 then throw boom else x)`, push 1..5; `runToList` throws boom; discarded == 3,4,5 (order-independent compare), subscription cancelled.\n4. Consumer stops via `take(2)`: result List(1,2), discarded == 3,4,5, cancelled.\n5. Consumer interrupted: run `runForeach(x => { received.release(); gate.acquire() })` in `forkCancellable`, push 1..5, wait for `received`, `cancel()`; discarded == 2,3,4,5, cancelled.\n6. Late onNext after cancellation: after any of the above, call `next(6)` on the publisher; `onDiscard(6)` is called synchronously and nothing throws.\n7. Existing one-arg overload: compiles unchanged and `FlowPublisherPekkoTest` still passes.\nRun `sbt \"core/testOnly ox.flow.reactive.*\"` and `sbt core/scalafmt`; all green."},{"title":"Exactly-once stress test: every element is emitted or discarded, never both or neither","description":"Add a test (same file as Task 1 or a sibling in core/src/test/scala/ox/flow/reactive/) that uses a demand-respecting publisher running on its own thread: it emits 1..N (N around 1000) only while `sent < requested` and not cancelled, calling `onNext` from that thread, then `onComplete`. Compose `Flow.fromPublisher(p, discarded.add)` with a consumer that stops at a random point (e.g. `take(k)` for random k, and separately a `map` that throws at a random element, and separately a `forkCancellable` cancelled after a random short delay). Collect emitted elements via `.tap(emitted.add)` into thread-safe collections. After the flow finishes and the publisher thread is joined, assert: `emitted` and `discarded` are disjoint, their union equals exactly the set of elements the publisher actually sent, and no element appears twice. Repeat each scenario in a loop (e.g. 20 iterations) with a small `given BufferCapacity = BufferCapacity(4)` to make buffer-full races likely. The publisher's late `onNext` calls (after cancellation) must land in `discarded`. If this exposes a race in Task 1's implementation, fix it in FlowCompanionReactiveOps.scala as part of this task. Run `sbt \"core/testOnly ox.flow.reactive.*\"`."},{"title":"Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","description":"In flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala add `def fromPublisher[T](p: org.reactivestreams.Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p), onDiscard)`, keep the one-arg overload, and update its scaladoc to point at the core overload for the callback semantics. Create flow-reactive-streams/src/test/scala/ox/flow/reactive/FlowReactiveStreamsFromPublisherTest.scala (AnyFunSuite with Matchers) with a small hand-written `org.reactivestreams.Publisher[Int]` (same shape as the core ManualPublisher, but implementing the reactive-streams interfaces; it cannot be shared because the module does not depend on core's test classpath) and two tests: elements flow through the one-arg variant; with `.take(1)` and the new overload, remaining pushed elements are passed to `onDiscard` and the subscription is cancelled. Run `sbt flowReactiveStreams/test` and `sbt flowReactiveStreams/scalafmt`."},{"title":"Document the onDiscard callback in doc/streaming/flows.md","description":"In doc/streaming/flows.md, section \"### Publisher -> Flow\" (around lines 211-219), add a short paragraph explaining: elements received from the publisher but not emitted downstream (buffered when the publisher fails, when the consumer stops early via exception, interruption or `take`, or arriving after cancellation) can be released by passing a second argument `onDiscard: T => Unit` to `Flow.fromPublisher` / `FlowReactiveStreams.fromPublisher`; it is called exactly once per such element, possibly from the publisher's thread, and should not throw; typical use is releasing reference-counted buffers. Also state that on publisher error, elements already received are emitted before the error is propagated. Keep the prose as short as the surrounding text. If you add a code example, use a ```` ```scala mdoc:compile-only ```` block and verify with `sbt compileDocumentation`; otherwise no build step is needed. Do not touch generated-doc/out."}],"brief":"## Modules and files\n\n- `core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala` — the only implementation file in core. Trait `FlowCompanionReactiveOps` (mixed into `Flow.type`) defines `def fromPublisher[T](p: java.util.concurrent.Flow.Publisher[T])(using BufferCapacity): Flow[T]`. Read the whole file (80 lines) before changing anything. Current structure: `usingEmitInline: emit => unsupervised { channel = BufferCapacity.newChannel[T]; forkUnsupervised { p.subscribe(Subscriber: onSubscribe -> subscriptionRef.set(s); s.request(capacity); onNext -> channel.send; onError -> channel.error; onComplete -> channel.done) }; repeatWhile { channel.receiveOrClosed() match Done -> false / Error -> throw / t -> emit(t); batched subscription.request } }.tapException(cancel subscription)`. The demand logic (`demandThreshold = ceil(capacity/2)`, `toDemand` counter, lazily cached `subscription` read from `subscriptionRef` after the first element) must be preserved.\n- `flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala` — `object FlowReactiveStreams { def fromPublisher[T](p: org.reactivestreams.Publisher[T])(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p)) }`. Module has no `src/test` dir yet; scalatest is already a test dependency (build.sbt `flowReactiveStreams`), it `dependsOn(core)` (compile only, not test->test).\n- `core/src/test/scala/ox/flow/reactive/` — existing reactive tests: `FlowPublisherPekkoTest.scala` (AnyFunSuite + Matchers, uses Pekko `Source...runWith(Sink.asPublisher)` and `FlowAdapters.toFlowPublisher`) and `FlowPublisherTckTest.scala`. Put new fromPublisher tests here. No test anywhere currently uses a hand-written Publisher; write one (see below).\n- `doc/streaming/flows.md` lines 211-219, section \"### Publisher -> Flow\" — the only user doc mentioning `fromPublisher`. `generated-doc/out/**` is mdoc output regenerated at release time by `updateDocs`; do NOT edit it. Code blocks in docs use ```` ```scala mdoc:compile-only ```` and are compiled by `sbt compileDocumentation` (slow; only needed if you add a code block). Doc dependency snippet convention: `// sbt dependency: \"com.softwaremill.ox\" %% \"flow-reactive-streams\" % \"@VERSION@\"`.\n\n## Key APIs (ox.channels, core/src/main/scala/ox/channels/Channel.scala)\n\n- `Sink.sendOrClosed(t): Unit | ChannelClosed` (non-throwing), `Sink.send(t)` throws `ChannelClosedException` when closed.\n- `Sink.doneOrClosed(): Unit | ChannelClosed` — \"buffered values will be delivered; send operations in progress complete normally when a receiver arrives\"; returns `ChannelClosed` instead of throwing if already closed. `Sink.errorOrClosed(reason)` — drops buffered values (that is the bug). Use the `*OrClosed` variants everywhere in the subscriber: a misbehaving publisher may call onError/onComplete twice.\n- `Source.receiveOrClosed(): T | ChannelClosed` (blocking); match `case ChannelClosed.Done => ...; case e: ChannelClosed.Error => throw e.toThrowable; case t: T @unchecked => ...`. `Source.tryReceiveOrClosed()` is non-blocking but may spuriously return None, so it is not suitable for a drain loop.\n- `Source.foreachOrError(f)` / `drainOrError()` from `SourceDrainOps.scala` loop until Done; usable for the drain, but write the drain inline so you can handle `InterruptedException` (see below).\n- jox semantics (com.softwaremill.jox channels 1.1.2, verified from source): after `done()`, buffered elements stay receivable; a receiver observing Done has already seen every element whose `send` started before `done()` (an in-progress sender is waited for); a `send` racing with `done()` either lands in the buffer or returns closed. Hence \"drain until Done, then `sendOrClosed` returns closed\" gives exactly-once discard with no extra locking.\n- `BufferCapacity`: `given default: BufferCapacity = 16`; `BufferCapacity.newChannel[T]`; `summon[BufferCapacity].toInt` (import `ox.channels.toInt`). Tests can override with `given BufferCapacity = BufferCapacity(4)`.\n- `ox.util`: `t.tapException(f)` runs `f(e)` for any Throwable and rethrows `e` (exceptions from `f` added as suppressed). `ox.pipe`, `ox.discard`, `ox.repeatWhile` already imported in the file.\n- `FlowOps.take(n)` stops upstream by throwing a private `ControlThrowable` (`abortTake`) out of `emit`, caught in `take` itself. So \"consumer stops early\" includes `take`, and the cleanup path must run for every Throwable, not just NonFatal. `tapException`/`catch case e: Throwable` already does this.\n- `unsupervised { ... }` + `forkUnsupervised { ... }` (import `ox.unsupervised`, `ox.forkUnsupervised`): scope end interrupts still-running forks once. `onNext`/`onError`/`onComplete` run on the publisher's threads (or on the `p.subscribe` fork for synchronous publishers), never on the consumer thread.\n- Test helpers: `ox.forkCancellable { ... }` inside `supervised` returns a `CancellableFork` with `cancel(): Either[Throwable, T]` (interrupts once, joins). Interruption inside the flow surfaces as `InterruptedException` thrown from a blocking call (e.g. `Semaphore.acquire`, `channel.receiveOrClosed`).\n\n## Design decisions (apply as-is)\n\n- API: add an overload `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg method delegate with `_ => ()`. Do not add a default parameter and do not change the existing signature: `core` runs MiMa (`enableMimaSettings` in build.sbt) and only additive changes are safe. Same shape in `FlowReactiveStreams`.\n- Subscriber: `onNext(t)` = `channel.sendOrClosed(t)` and if the result is `ChannelClosed` call `onDiscard(t)`. `onError(e)` = store `e` in an `AtomicReference[Throwable]`, then `channel.doneOrClosed().discard`. `onComplete()` = `channel.doneOrClosed().discard`. `onSubscribe(s)` = set `subscriptionRef`, then if a `cancelled: AtomicBoolean` is already set call `s.cancel()` else `s.request(capacity)` (both sides use volatile atomics, so at least one side sees the other; a double `cancel()` is allowed by the spec).\n- Consumer loop: on `ChannelClosed.Done`, read the stored error; if non-null throw it, else finish. Keep the `ChannelClosed.Error` branch for safety.\n- Cleanup, run on any Throwable escaping the loop (the stored publisher error included; cancel after onError is a spec-allowed no-op): set `cancelled`, cancel the subscription if `subscriptionRef` is non-null, `channel.doneOrClosed().discard`, then drain: loop `receiveOrClosed()` until `Done`, calling `onDiscard` for each element. The drain is bounded (channel is done, at most capacity elements plus in-flight sends) but blocking; if `InterruptedException` is thrown mid-drain, catch it, keep draining, and call `Thread.currentThread().interrupt()` once the drain finishes, so no element is lost on a second interrupt. Then rethrow the original exception. An element that was passed to `emit` and whose `emit` threw counts as emitted (downstream owns it): do not discard it.\n- Scaladoc on the new overload: state the exactly-once invariant, that `onDiscard` may be called from publisher threads, that it must not throw, and that on publisher error buffered elements are emitted before the error is rethrown (this is a deliberate behaviour change for the one-arg variant too; mention it in its doc).\n\n## Test publisher helper (write it in the core test, package `ox.flow.reactive`)\n\nA `ManualPublisher[T] extends java.util.concurrent.Flow.Publisher[T]` with: `@volatile var subscriber: Subscriber[? >: T]`, `requested: AtomicLong`, `cancelled: AtomicBoolean`, a `subscribed: CountDownLatch(1)`; `subscribe(s)` stores `s`, calls `s.onSubscribe(new Subscription { request(n) => requested.addAndGet(n); cancel() => cancelled.set(true) })`, counts down the latch. Helper methods `next(t)`, `error(e)`, `complete()` forward to the subscriber from the test thread. For the exactly-once stress test, a second variant that runs its own thread emitting `1..n` while `sent < requested` and not cancelled. Run the flow under test in `supervised { forkCancellable { ... } }` or `fork`, await `subscribed`, then drive the publisher. Collect emitted elements with `.tap(emitted += _)` or `runForeach`, discarded with `onDiscard = discarded += _` (use thread-safe collections such as `ConcurrentLinkedQueue` or `java.util.Collections.synchronizedList`, since `onDiscard` runs on other threads).\n\n## Conventions / build\n\n- Scala 3 indentation syntax, `end X` markers on long definitions, `.discard` on ignored results (compiler warns otherwise), scalafmt configured in `.scalafmt.conf` (140-column lines); run `sbt core/scalafmt flowReactiveStreams/scalafmt` after editing.\n- Comments: short, explain non-obvious reasoning only; never restate code. Scaladoc describes usage and invariants, not implementation.\n- Tests: scalatest 3.2.20; the reactive tests use `AnyFunSuite with Matchers` with `test(\"...\")` blocks and Scala 3 colon syntax. `core` runs tests forked (`Test / fork := true`).\n- Commands (sbt 2.0.10): `sbt \"core/testOnly ox.flow.reactive.*\"`, `sbt flowReactiveStreams/test`, `sbt core/mimaReportBinaryIssues` (only meaningful when a previous stable version resolves), `sbt compileDocumentation` for mdoc. Never commit; the user commits."}},{"id":[{"name":"Task: Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Exactly-once stress test: every element is emitted or discarded, never both or neither","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","occurrence":0}],"resultJson":{"findings":[{"id":"R1.I4.1","title":"sbt Test/compile fails with JGit NoWorkTreeException","reason":{"type":"Declined","text":"This is a problem with the environment, not the code. git sees the worktree as non-bare (`git rev-parse --is-bare-repository` prints false; the git dir is /home/adamw/ox/.git/worktrees/zippy-astrodon-94aa2fb6). The exception comes from the sbt-git plugin, which uses JGit by default and reads this linked worktree as bare while the build loads. Nothing in project/ or the build configures this; `zz-tmp-git.sbt` was my temporary workaround, not the cause. To check the code, I recreated it with `useReadableConsoleGit`, which makes sbt-git use command-line git: `sbt core/Test/compile flowReactiveStreams/Test/compile` succeeded (warnings only, in older test files, no errors) and flowReactiveStreams tests passed. I then deleted the file again. Changing the shared build's git plugin setup is outside the scope of this change."}}]}},{"id":[{"name":"Task: Document the onDiscard callback in doc/streaming/flows.md","occurrence":0}],"resultJson":{"findings":[{"id":"R1.I4.1","title":"`sbt Test/compile` fails with no diagnostic","reason":{"type":"Declined","text":"Environmental. Full output shows sbt-git's JGit throws NoWorkTreeException (\"Bare Repository has neither a working tree, nor an index\") while loading the build, because this checkout is a git worktree; no source error is involved. With a temporary `ThisBuild / git.gitUncommittedChanges := false` override, `sbt Test/compile` succeeds across all modules (only pre-existing unrelated warnings). The temporary file was removed; no codebase change is needed."}}]}},{"id":[{"name":"Final review","occurrence":0}],"resultJson":{"findings":[{"id":"S1","title":"sbt Test/compile fails with JGit NoWorkTreeException","reason":{"type":"Declined","text":"This is a problem with the environment, not the code. git sees the worktree as non-bare (`git rev-parse --is-bare-repository` prints false; the git dir is /home/adamw/ox/.git/worktrees/zippy-astrodon-94aa2fb6). The exception comes from the sbt-git plugin, which uses JGit by default and reads this linked worktree as bare while the build loads. Nothing in project/ or the build configures this; `zz-tmp-git.sbt` was my temporary workaround, not the cause. To check the code, I recreated it with `useReadableConsoleGit`, which makes sbt-git use command-line git: `sbt core/Test/compile flowReactiveStreams/Test/compile` succeeded (warnings only, in older test files, no errors) and flowReactiveStreams tests passed. I then deleted the file again. Changing the shared build's git plugin setup is outside the scope of this change."}},{"id":"S2","title":"`sbt Test/compile` fails with no diagnostic","reason":{"type":"Declined","text":"Environmental. Full output shows sbt-git's JGit throws NoWorkTreeException (\"Bare Repository has neither a working tree, nor an index\") while loading the build, because this checkout is a git worktree; no source error is involved. With a temporary `ThisBuild / git.gitUncommittedChanges := false` override, `sbt Test/compile` succeeds across all modules (only pre-existing unrelated warnings). The temporary file was removed; no codebase change is needed."}},{"id":"R1.I2.1","title":"No test covers the drain's deferred-interrupt path in discardBuffered","reason":{"type":"Declined","text":"A deterministic test can't be written. In jox 1.1.2, `receiveOrClosed` only throws `InterruptedException` when it parks (`Continuation.await`). After `doneOrClosed()` has finished on the draining thread, a receive never parks: buffered or in-flight sends are returned or retried, and later cells are already CLOSED. The catch is reached only in a narrow race, when a concurrent `onComplete`/`onError` is still closing cells. I tried the suggested test (interrupt plus throw in the consumer, then assert the discards and the interrupt flag). It passed both with the catch removed and with the re-interrupt removed, so it did not cover the branch, and I reverted it. The suggested capacity-1 variant also doesn't help, because a blocked sender's Continuation is resumed without parking."},"location":{"file":"core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala","line":141}},{"id":"R1.I5.1","title":"sbt Test/compile exited 1 with no compile error in the output","reason":{"type":"Declined","text":"Environmental. sbt-git's JGit fails in this git worktree with `NoWorkTreeException: Bare Repository has neither a working tree, nor an index` while evaluating `gitUncommittedChanges`, so the build fails before compiling. It is not a code problem. With a temporary `zz-tmp-git.sbt` that sets `ThisBuild / git.gitUncommittedChanges := false`, Test/compile and the core reactive and flowReactiveStreams tests all pass. I deleted the temporary file afterwards."}},{"id":"R2.I3.1","title":"`sbt Test/compile` exits 1 with no compiler diagnostics","reason":{"type":"Declined","text":"Environmental, not a code problem. In this git worktree, sbt-git's JGit throws `NoWorkTreeException: Bare Repository has neither a working tree, nor an index` while evaluating `gitUncommittedChanges` during build load. The build fails before any compilation, and the full output shows that stack trace above the timing line. With a temporary `zz-tmp-git.sbt` setting `ThisBuild / git.gitUncommittedChanges := false`, `Test/compile` succeeds. With the same file, `core/testOnly ox.flow.reactive.*` (44 passed, 7 ignored) and `flowReactiveStreams/test` (2 passed) also pass. I deleted the file afterwards, so it is not left in the tree."}}]}}],"stageStarts":[]} \ No newline at end of file +{"header":{"startingBranch":"zippy-astrodon-94aa2fb6","branch":"zippy-astrodon-94aa2fb6","branchMode":"Reused","userPrompt":"Implement GitHub issue softwaremill/ox#528: \"Flow.fromPublisher: allow releasing elements which are buffered but not emitted\".\n\nProblem: `Flow.fromPublisher` (core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala) buffers elements in a channel. Some of them never reach the consumer:\n- when the publisher fails, `channel.error` drops the buffered elements\n- when the consumer stops early (an exception, an interruption), the buffered elements are never received\n- elements can still arrive (onNext) after the subscription is cancelled\n\nIf elements hold resources (e.g. Netty's reference-counted ByteBufs), nobody releases them, so they leak.\n\nGoal: add an optional callback, e.g. `fromPublisher(p, onDiscard: T => Unit)`, called exactly once for each element received from the publisher but not emitted downstream. Existing call sites without the callback must keep compiling and behaving the same.\n\nSuggested implementation: on publisher error, store the error and call `channel.done()` (so buffered elements are kept and emitted before the error is rethrown). When the flow ends early, cancel the subscription, close the channel, and drain it, calling the callback for each drained element. In `onNext`, call the callback if the channel is already closed. Make sure there are no races where an element is neither emitted nor discarded.\n\nAlso update the variant in flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala (org.reactivestreams Publisher) to accept the same optional callback, and update any docs (doc/ directory) describing fromPublisher.\n\nAcceptance criteria:\n- tests covering: publisher error with buffered elements (elements still emitted, then error); consumer stopping early (exception / interruption) with buffered elements -> callback called for each; elements arriving after cancellation -> callback called; every element is either emitted or discarded, never both or neither.\n- `sbt compile test` passes for affected modules (core, flow-reactive-streams).\n- Code follows existing style in the repo; concise doc comments.\n\nContext: tapir currently tracks unconsumed elements itself using a lock: https://github.com/softwaremill/tapir/pull/5573","flow":{"type":"Catalog","name":"implement.sc"},"startingCommit":"664f01730cda34fd88e8c3db3deb600934eb1d1e"},"entries":[{"id":[{"name":"Plan","occurrence":0}],"resultJson":{"id":"ox-528-from-publisher-on-discard","description":"`Flow.fromPublisher` buffers elements received from a reactive `Publisher` in a channel. Today, elements that are buffered but never emitted are silently lost: `channel.error` drops the buffer on publisher error, a consumer that stops early (exception, interruption, `take`) never receives what is buffered, and `onNext` calls that arrive after the subscription is cancelled are dropped by `channel.send` throwing. When elements own resources (Netty reference-counted ByteBufs in tapir), this leaks them. Tapir currently works around it with its own lock-based tracking (softwaremill/tapir#5573).\n\nThis epic adds an optional `onDiscard: T => Unit` callback to `Flow.fromPublisher` (core, JDK `Flow.Publisher`) and `FlowReactiveStreams.fromPublisher` (flow-reactive-streams, `org.reactivestreams.Publisher`), with the invariant that every element received from the publisher is either emitted downstream or passed to `onDiscard`, exactly once, never both or neither. Publisher errors no longer drop buffered elements: the error is stored, the channel is completed with `done()`, buffered elements are emitted, and then the error is rethrown. When the flow ends early, the subscription is cancelled, the channel is closed and drained through `onDiscard`, and any late `onNext` goes to `onDiscard` as well. Existing single-argument call sites keep compiling (the new parameter is an overload, which also keeps MiMa happy for `core`). Docs in `doc/streaming/flows.md` describe the new callback.","tasks":[{"title":"Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","description":"In core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala add `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg `fromPublisher` delegate to it with `_ => ()`. Implement per the brief's design decisions: `onNext` uses `sendOrClosed` and discards on closed; `onError` stores the throwable in an AtomicReference and calls `doneOrClosed`; `onComplete` calls `doneOrClosed`; `onSubscribe` cancels immediately if a `cancelled` flag is already set, else requests `capacity`; the consumer loop rethrows the stored error when it sees `Done`; on any Throwable escaping the loop: set `cancelled`, cancel the subscription if present, `doneOrClosed` the channel, drain it calling `onDiscard` per element (keep draining across an InterruptedException and re-interrupt the thread afterwards), rethrow. Preserve the existing batched demand logic. Update the scaladoc of both overloads (exactly-once invariant, callback runs on publisher threads and must not throw, buffered elements are now emitted before a publisher error is rethrown).\n\nTests, in a new core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardTest.scala (AnyFunSuite with Matchers), using a hand-written `ManualPublisher[T]` helper as described in the brief (keep the helper in this file or a sibling file in the same package so Task 2 can reuse it):\n1. Publisher error with buffered elements: push 1,2,3 then `error(boom)` before/while the consumer runs; `runToList` throws boom and all of 1,2,3 were emitted; `onDiscard` never called.\n2. Publisher error with nothing buffered still throws.\n3. Consumer exception: `.map(x => if x == 2 then throw boom else x)`, push 1..5; `runToList` throws boom; discarded == 3,4,5 (order-independent compare), subscription cancelled.\n4. Consumer stops via `take(2)`: result List(1,2), discarded == 3,4,5, cancelled.\n5. Consumer interrupted: run `runForeach(x => { received.release(); gate.acquire() })` in `forkCancellable`, push 1..5, wait for `received`, `cancel()`; discarded == 2,3,4,5, cancelled.\n6. Late onNext after cancellation: after any of the above, call `next(6)` on the publisher; `onDiscard(6)` is called synchronously and nothing throws.\n7. Existing one-arg overload: compiles unchanged and `FlowPublisherPekkoTest` still passes.\nRun `sbt \"core/testOnly ox.flow.reactive.*\"` and `sbt core/scalafmt`; all green."},{"title":"Exactly-once stress test: every element is emitted or discarded, never both or neither","description":"Add a test (same file as Task 1 or a sibling in core/src/test/scala/ox/flow/reactive/) that uses a demand-respecting publisher running on its own thread: it emits 1..N (N around 1000) only while `sent < requested` and not cancelled, calling `onNext` from that thread, then `onComplete`. Compose `Flow.fromPublisher(p, discarded.add)` with a consumer that stops at a random point (e.g. `take(k)` for random k, and separately a `map` that throws at a random element, and separately a `forkCancellable` cancelled after a random short delay). Collect emitted elements via `.tap(emitted.add)` into thread-safe collections. After the flow finishes and the publisher thread is joined, assert: `emitted` and `discarded` are disjoint, their union equals exactly the set of elements the publisher actually sent, and no element appears twice. Repeat each scenario in a loop (e.g. 20 iterations) with a small `given BufferCapacity = BufferCapacity(4)` to make buffer-full races likely. The publisher's late `onNext` calls (after cancellation) must land in `discarded`. If this exposes a race in Task 1's implementation, fix it in FlowCompanionReactiveOps.scala as part of this task. Run `sbt \"core/testOnly ox.flow.reactive.*\"`."},{"title":"Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","description":"In flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala add `def fromPublisher[T](p: org.reactivestreams.Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p), onDiscard)`, keep the one-arg overload, and update its scaladoc to point at the core overload for the callback semantics. Create flow-reactive-streams/src/test/scala/ox/flow/reactive/FlowReactiveStreamsFromPublisherTest.scala (AnyFunSuite with Matchers) with a small hand-written `org.reactivestreams.Publisher[Int]` (same shape as the core ManualPublisher, but implementing the reactive-streams interfaces; it cannot be shared because the module does not depend on core's test classpath) and two tests: elements flow through the one-arg variant; with `.take(1)` and the new overload, remaining pushed elements are passed to `onDiscard` and the subscription is cancelled. Run `sbt flowReactiveStreams/test` and `sbt flowReactiveStreams/scalafmt`."},{"title":"Document the onDiscard callback in doc/streaming/flows.md","description":"In doc/streaming/flows.md, section \"### Publisher -> Flow\" (around lines 211-219), add a short paragraph explaining: elements received from the publisher but not emitted downstream (buffered when the publisher fails, when the consumer stops early via exception, interruption or `take`, or arriving after cancellation) can be released by passing a second argument `onDiscard: T => Unit` to `Flow.fromPublisher` / `FlowReactiveStreams.fromPublisher`; it is called exactly once per such element, possibly from the publisher's thread, and should not throw; typical use is releasing reference-counted buffers. Also state that on publisher error, elements already received are emitted before the error is propagated. Keep the prose as short as the surrounding text. If you add a code example, use a ```` ```scala mdoc:compile-only ```` block and verify with `sbt compileDocumentation`; otherwise no build step is needed. Do not touch generated-doc/out."}],"brief":"## Modules and files\n\n- `core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala` — the only implementation file in core. Trait `FlowCompanionReactiveOps` (mixed into `Flow.type`) defines `def fromPublisher[T](p: java.util.concurrent.Flow.Publisher[T])(using BufferCapacity): Flow[T]`. Read the whole file (80 lines) before changing anything. Current structure: `usingEmitInline: emit => unsupervised { channel = BufferCapacity.newChannel[T]; forkUnsupervised { p.subscribe(Subscriber: onSubscribe -> subscriptionRef.set(s); s.request(capacity); onNext -> channel.send; onError -> channel.error; onComplete -> channel.done) }; repeatWhile { channel.receiveOrClosed() match Done -> false / Error -> throw / t -> emit(t); batched subscription.request } }.tapException(cancel subscription)`. The demand logic (`demandThreshold = ceil(capacity/2)`, `toDemand` counter, lazily cached `subscription` read from `subscriptionRef` after the first element) must be preserved.\n- `flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala` — `object FlowReactiveStreams { def fromPublisher[T](p: org.reactivestreams.Publisher[T])(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p)) }`. Module has no `src/test` dir yet; scalatest is already a test dependency (build.sbt `flowReactiveStreams`), it `dependsOn(core)` (compile only, not test->test).\n- `core/src/test/scala/ox/flow/reactive/` — existing reactive tests: `FlowPublisherPekkoTest.scala` (AnyFunSuite + Matchers, uses Pekko `Source...runWith(Sink.asPublisher)` and `FlowAdapters.toFlowPublisher`) and `FlowPublisherTckTest.scala`. Put new fromPublisher tests here. No test anywhere currently uses a hand-written Publisher; write one (see below).\n- `doc/streaming/flows.md` lines 211-219, section \"### Publisher -> Flow\" — the only user doc mentioning `fromPublisher`. `generated-doc/out/**` is mdoc output regenerated at release time by `updateDocs`; do NOT edit it. Code blocks in docs use ```` ```scala mdoc:compile-only ```` and are compiled by `sbt compileDocumentation` (slow; only needed if you add a code block). Doc dependency snippet convention: `// sbt dependency: \"com.softwaremill.ox\" %% \"flow-reactive-streams\" % \"@VERSION@\"`.\n\n## Key APIs (ox.channels, core/src/main/scala/ox/channels/Channel.scala)\n\n- `Sink.sendOrClosed(t): Unit | ChannelClosed` (non-throwing), `Sink.send(t)` throws `ChannelClosedException` when closed.\n- `Sink.doneOrClosed(): Unit | ChannelClosed` — \"buffered values will be delivered; send operations in progress complete normally when a receiver arrives\"; returns `ChannelClosed` instead of throwing if already closed. `Sink.errorOrClosed(reason)` — drops buffered values (that is the bug). Use the `*OrClosed` variants everywhere in the subscriber: a misbehaving publisher may call onError/onComplete twice.\n- `Source.receiveOrClosed(): T | ChannelClosed` (blocking); match `case ChannelClosed.Done => ...; case e: ChannelClosed.Error => throw e.toThrowable; case t: T @unchecked => ...`. `Source.tryReceiveOrClosed()` is non-blocking but may spuriously return None, so it is not suitable for a drain loop.\n- `Source.foreachOrError(f)` / `drainOrError()` from `SourceDrainOps.scala` loop until Done; usable for the drain, but write the drain inline so you can handle `InterruptedException` (see below).\n- jox semantics (com.softwaremill.jox channels 1.1.2, verified from source): after `done()`, buffered elements stay receivable; a receiver observing Done has already seen every element whose `send` started before `done()` (an in-progress sender is waited for); a `send` racing with `done()` either lands in the buffer or returns closed. Hence \"drain until Done, then `sendOrClosed` returns closed\" gives exactly-once discard with no extra locking.\n- `BufferCapacity`: `given default: BufferCapacity = 16`; `BufferCapacity.newChannel[T]`; `summon[BufferCapacity].toInt` (import `ox.channels.toInt`). Tests can override with `given BufferCapacity = BufferCapacity(4)`.\n- `ox.util`: `t.tapException(f)` runs `f(e)` for any Throwable and rethrows `e` (exceptions from `f` added as suppressed). `ox.pipe`, `ox.discard`, `ox.repeatWhile` already imported in the file.\n- `FlowOps.take(n)` stops upstream by throwing a private `ControlThrowable` (`abortTake`) out of `emit`, caught in `take` itself. So \"consumer stops early\" includes `take`, and the cleanup path must run for every Throwable, not just NonFatal. `tapException`/`catch case e: Throwable` already does this.\n- `unsupervised { ... }` + `forkUnsupervised { ... }` (import `ox.unsupervised`, `ox.forkUnsupervised`): scope end interrupts still-running forks once. `onNext`/`onError`/`onComplete` run on the publisher's threads (or on the `p.subscribe` fork for synchronous publishers), never on the consumer thread.\n- Test helpers: `ox.forkCancellable { ... }` inside `supervised` returns a `CancellableFork` with `cancel(): Either[Throwable, T]` (interrupts once, joins). Interruption inside the flow surfaces as `InterruptedException` thrown from a blocking call (e.g. `Semaphore.acquire`, `channel.receiveOrClosed`).\n\n## Design decisions (apply as-is)\n\n- API: add an overload `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg method delegate with `_ => ()`. Do not add a default parameter and do not change the existing signature: `core` runs MiMa (`enableMimaSettings` in build.sbt) and only additive changes are safe. Same shape in `FlowReactiveStreams`.\n- Subscriber: `onNext(t)` = `channel.sendOrClosed(t)` and if the result is `ChannelClosed` call `onDiscard(t)`. `onError(e)` = store `e` in an `AtomicReference[Throwable]`, then `channel.doneOrClosed().discard`. `onComplete()` = `channel.doneOrClosed().discard`. `onSubscribe(s)` = set `subscriptionRef`, then if a `cancelled: AtomicBoolean` is already set call `s.cancel()` else `s.request(capacity)` (both sides use volatile atomics, so at least one side sees the other; a double `cancel()` is allowed by the spec).\n- Consumer loop: on `ChannelClosed.Done`, read the stored error; if non-null throw it, else finish. Keep the `ChannelClosed.Error` branch for safety.\n- Cleanup, run on any Throwable escaping the loop (the stored publisher error included; cancel after onError is a spec-allowed no-op): set `cancelled`, cancel the subscription if `subscriptionRef` is non-null, `channel.doneOrClosed().discard`, then drain: loop `receiveOrClosed()` until `Done`, calling `onDiscard` for each element. The drain is bounded (channel is done, at most capacity elements plus in-flight sends) but blocking; if `InterruptedException` is thrown mid-drain, catch it, keep draining, and call `Thread.currentThread().interrupt()` once the drain finishes, so no element is lost on a second interrupt. Then rethrow the original exception. An element that was passed to `emit` and whose `emit` threw counts as emitted (downstream owns it): do not discard it.\n- Scaladoc on the new overload: state the exactly-once invariant, that `onDiscard` may be called from publisher threads, that it must not throw, and that on publisher error buffered elements are emitted before the error is rethrown (this is a deliberate behaviour change for the one-arg variant too; mention it in its doc).\n\n## Test publisher helper (write it in the core test, package `ox.flow.reactive`)\n\nA `ManualPublisher[T] extends java.util.concurrent.Flow.Publisher[T]` with: `@volatile var subscriber: Subscriber[? >: T]`, `requested: AtomicLong`, `cancelled: AtomicBoolean`, a `subscribed: CountDownLatch(1)`; `subscribe(s)` stores `s`, calls `s.onSubscribe(new Subscription { request(n) => requested.addAndGet(n); cancel() => cancelled.set(true) })`, counts down the latch. Helper methods `next(t)`, `error(e)`, `complete()` forward to the subscriber from the test thread. For the exactly-once stress test, a second variant that runs its own thread emitting `1..n` while `sent < requested` and not cancelled. Run the flow under test in `supervised { forkCancellable { ... } }` or `fork`, await `subscribed`, then drive the publisher. Collect emitted elements with `.tap(emitted += _)` or `runForeach`, discarded with `onDiscard = discarded += _` (use thread-safe collections such as `ConcurrentLinkedQueue` or `java.util.Collections.synchronizedList`, since `onDiscard` runs on other threads).\n\n## Conventions / build\n\n- Scala 3 indentation syntax, `end X` markers on long definitions, `.discard` on ignored results (compiler warns otherwise), scalafmt configured in `.scalafmt.conf` (140-column lines); run `sbt core/scalafmt flowReactiveStreams/scalafmt` after editing.\n- Comments: short, explain non-obvious reasoning only; never restate code. Scaladoc describes usage and invariants, not implementation.\n- Tests: scalatest 3.2.20; the reactive tests use `AnyFunSuite with Matchers` with `test(\"...\")` blocks and Scala 3 colon syntax. `core` runs tests forked (`Test / fork := true`).\n- Commands (sbt 2.0.10): `sbt \"core/testOnly ox.flow.reactive.*\"`, `sbt flowReactiveStreams/test`, `sbt core/mimaReportBinaryIssues` (only meaningful when a previous stable version resolves), `sbt compileDocumentation` for mdoc. Never commit; the user commits."}},{"id":[{"name":"Task: Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Exactly-once stress test: every element is emitted or discarded, never both or neither","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","occurrence":0}],"resultJson":{"findings":[{"id":"R1.I4.1","title":"sbt Test/compile fails with JGit NoWorkTreeException","reason":{"type":"Declined","text":"This is a problem with the environment, not the code. git sees the worktree as non-bare (`git rev-parse --is-bare-repository` prints false; the git dir is /home/adamw/ox/.git/worktrees/zippy-astrodon-94aa2fb6). The exception comes from the sbt-git plugin, which uses JGit by default and reads this linked worktree as bare while the build loads. Nothing in project/ or the build configures this; `zz-tmp-git.sbt` was my temporary workaround, not the cause. To check the code, I recreated it with `useReadableConsoleGit`, which makes sbt-git use command-line git: `sbt core/Test/compile flowReactiveStreams/Test/compile` succeeded (warnings only, in older test files, no errors) and flowReactiveStreams tests passed. I then deleted the file again. Changing the shared build's git plugin setup is outside the scope of this change."}}]}},{"id":[{"name":"Task: Document the onDiscard callback in doc/streaming/flows.md","occurrence":0}],"resultJson":{"findings":[{"id":"R1.I4.1","title":"`sbt Test/compile` fails with no diagnostic","reason":{"type":"Declined","text":"Environmental. Full output shows sbt-git's JGit throws NoWorkTreeException (\"Bare Repository has neither a working tree, nor an index\") while loading the build, because this checkout is a git worktree; no source error is involved. With a temporary `ThisBuild / git.gitUncommittedChanges := false` override, `sbt Test/compile` succeeds across all modules (only pre-existing unrelated warnings). The temporary file was removed; no codebase change is needed."}}]}},{"id":[{"name":"Final review","occurrence":0}],"resultJson":{"findings":[{"id":"S1","title":"sbt Test/compile fails with JGit NoWorkTreeException","reason":{"type":"Declined","text":"This is a problem with the environment, not the code. git sees the worktree as non-bare (`git rev-parse --is-bare-repository` prints false; the git dir is /home/adamw/ox/.git/worktrees/zippy-astrodon-94aa2fb6). The exception comes from the sbt-git plugin, which uses JGit by default and reads this linked worktree as bare while the build loads. Nothing in project/ or the build configures this; `zz-tmp-git.sbt` was my temporary workaround, not the cause. To check the code, I recreated it with `useReadableConsoleGit`, which makes sbt-git use command-line git: `sbt core/Test/compile flowReactiveStreams/Test/compile` succeeded (warnings only, in older test files, no errors) and flowReactiveStreams tests passed. I then deleted the file again. Changing the shared build's git plugin setup is outside the scope of this change."}},{"id":"S2","title":"`sbt Test/compile` fails with no diagnostic","reason":{"type":"Declined","text":"Environmental. Full output shows sbt-git's JGit throws NoWorkTreeException (\"Bare Repository has neither a working tree, nor an index\") while loading the build, because this checkout is a git worktree; no source error is involved. With a temporary `ThisBuild / git.gitUncommittedChanges := false` override, `sbt Test/compile` succeeds across all modules (only pre-existing unrelated warnings). The temporary file was removed; no codebase change is needed."}},{"id":"R1.I2.1","title":"No test covers the drain's deferred-interrupt path in discardBuffered","reason":{"type":"Declined","text":"A deterministic test can't be written. In jox 1.1.2, `receiveOrClosed` only throws `InterruptedException` when it parks (`Continuation.await`). After `doneOrClosed()` has finished on the draining thread, a receive never parks: buffered or in-flight sends are returned or retried, and later cells are already CLOSED. The catch is reached only in a narrow race, when a concurrent `onComplete`/`onError` is still closing cells. I tried the suggested test (interrupt plus throw in the consumer, then assert the discards and the interrupt flag). It passed both with the catch removed and with the re-interrupt removed, so it did not cover the branch, and I reverted it. The suggested capacity-1 variant also doesn't help, because a blocked sender's Continuation is resumed without parking."},"location":{"file":"core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala","line":141}},{"id":"R1.I5.1","title":"sbt Test/compile exited 1 with no compile error in the output","reason":{"type":"Declined","text":"Environmental. sbt-git's JGit fails in this git worktree with `NoWorkTreeException: Bare Repository has neither a working tree, nor an index` while evaluating `gitUncommittedChanges`, so the build fails before compiling. It is not a code problem. With a temporary `zz-tmp-git.sbt` that sets `ThisBuild / git.gitUncommittedChanges := false`, Test/compile and the core reactive and flowReactiveStreams tests all pass. I deleted the temporary file afterwards."}},{"id":"R2.I3.1","title":"`sbt Test/compile` exits 1 with no compiler diagnostics","reason":{"type":"Declined","text":"Environmental, not a code problem. In this git worktree, sbt-git's JGit throws `NoWorkTreeException: Bare Repository has neither a working tree, nor an index` while evaluating `gitUncommittedChanges` during build load. The build fails before any compilation, and the full output shows that stack trace above the timing line. With a temporary `zz-tmp-git.sbt` setting `ThisBuild / git.gitUncommittedChanges := false`, `Test/compile` succeeds. With the same file, `core/testOnly ox.flow.reactive.*` (44 passed, 7 ignored) and `flowReactiveStreams/test` (2 passed) also pass. I deleted the file afterwards, so it is not left in the tree."}}]}},{"id":[{"name":"Push branch","occurrence":0}],"resultJson":{"type":"Pushed"}}],"stageStarts":[]} \ No newline at end of file From 2003c449ab147e6cef6bfcfab5090d40e5c0b872 Mon Sep 17 00:00:00 2001 From: Adam Warski Date: Fri, 9 Oct 2026 15:31:13 +0000 Subject: [PATCH 09/11] stage: Generate PR title and description --- .orca/runs/0a3279c8e3d7.progress.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.orca/runs/0a3279c8e3d7.progress.json b/.orca/runs/0a3279c8e3d7.progress.json index b02af71e..1072e185 100644 --- a/.orca/runs/0a3279c8e3d7.progress.json +++ b/.orca/runs/0a3279c8e3d7.progress.json @@ -1 +1 @@ -{"header":{"startingBranch":"zippy-astrodon-94aa2fb6","branch":"zippy-astrodon-94aa2fb6","branchMode":"Reused","userPrompt":"Implement GitHub issue softwaremill/ox#528: \"Flow.fromPublisher: allow releasing elements which are buffered but not emitted\".\n\nProblem: `Flow.fromPublisher` (core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala) buffers elements in a channel. Some of them never reach the consumer:\n- when the publisher fails, `channel.error` drops the buffered elements\n- when the consumer stops early (an exception, an interruption), the buffered elements are never received\n- elements can still arrive (onNext) after the subscription is cancelled\n\nIf elements hold resources (e.g. Netty's reference-counted ByteBufs), nobody releases them, so they leak.\n\nGoal: add an optional callback, e.g. `fromPublisher(p, onDiscard: T => Unit)`, called exactly once for each element received from the publisher but not emitted downstream. Existing call sites without the callback must keep compiling and behaving the same.\n\nSuggested implementation: on publisher error, store the error and call `channel.done()` (so buffered elements are kept and emitted before the error is rethrown). When the flow ends early, cancel the subscription, close the channel, and drain it, calling the callback for each drained element. In `onNext`, call the callback if the channel is already closed. Make sure there are no races where an element is neither emitted nor discarded.\n\nAlso update the variant in flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala (org.reactivestreams Publisher) to accept the same optional callback, and update any docs (doc/ directory) describing fromPublisher.\n\nAcceptance criteria:\n- tests covering: publisher error with buffered elements (elements still emitted, then error); consumer stopping early (exception / interruption) with buffered elements -> callback called for each; elements arriving after cancellation -> callback called; every element is either emitted or discarded, never both or neither.\n- `sbt compile test` passes for affected modules (core, flow-reactive-streams).\n- Code follows existing style in the repo; concise doc comments.\n\nContext: tapir currently tracks unconsumed elements itself using a lock: https://github.com/softwaremill/tapir/pull/5573","flow":{"type":"Catalog","name":"implement.sc"},"startingCommit":"664f01730cda34fd88e8c3db3deb600934eb1d1e"},"entries":[{"id":[{"name":"Plan","occurrence":0}],"resultJson":{"id":"ox-528-from-publisher-on-discard","description":"`Flow.fromPublisher` buffers elements received from a reactive `Publisher` in a channel. Today, elements that are buffered but never emitted are silently lost: `channel.error` drops the buffer on publisher error, a consumer that stops early (exception, interruption, `take`) never receives what is buffered, and `onNext` calls that arrive after the subscription is cancelled are dropped by `channel.send` throwing. When elements own resources (Netty reference-counted ByteBufs in tapir), this leaks them. Tapir currently works around it with its own lock-based tracking (softwaremill/tapir#5573).\n\nThis epic adds an optional `onDiscard: T => Unit` callback to `Flow.fromPublisher` (core, JDK `Flow.Publisher`) and `FlowReactiveStreams.fromPublisher` (flow-reactive-streams, `org.reactivestreams.Publisher`), with the invariant that every element received from the publisher is either emitted downstream or passed to `onDiscard`, exactly once, never both or neither. Publisher errors no longer drop buffered elements: the error is stored, the channel is completed with `done()`, buffered elements are emitted, and then the error is rethrown. When the flow ends early, the subscription is cancelled, the channel is closed and drained through `onDiscard`, and any late `onNext` goes to `onDiscard` as well. Existing single-argument call sites keep compiling (the new parameter is an overload, which also keeps MiMa happy for `core`). Docs in `doc/streaming/flows.md` describe the new callback.","tasks":[{"title":"Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","description":"In core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala add `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg `fromPublisher` delegate to it with `_ => ()`. Implement per the brief's design decisions: `onNext` uses `sendOrClosed` and discards on closed; `onError` stores the throwable in an AtomicReference and calls `doneOrClosed`; `onComplete` calls `doneOrClosed`; `onSubscribe` cancels immediately if a `cancelled` flag is already set, else requests `capacity`; the consumer loop rethrows the stored error when it sees `Done`; on any Throwable escaping the loop: set `cancelled`, cancel the subscription if present, `doneOrClosed` the channel, drain it calling `onDiscard` per element (keep draining across an InterruptedException and re-interrupt the thread afterwards), rethrow. Preserve the existing batched demand logic. Update the scaladoc of both overloads (exactly-once invariant, callback runs on publisher threads and must not throw, buffered elements are now emitted before a publisher error is rethrown).\n\nTests, in a new core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardTest.scala (AnyFunSuite with Matchers), using a hand-written `ManualPublisher[T]` helper as described in the brief (keep the helper in this file or a sibling file in the same package so Task 2 can reuse it):\n1. Publisher error with buffered elements: push 1,2,3 then `error(boom)` before/while the consumer runs; `runToList` throws boom and all of 1,2,3 were emitted; `onDiscard` never called.\n2. Publisher error with nothing buffered still throws.\n3. Consumer exception: `.map(x => if x == 2 then throw boom else x)`, push 1..5; `runToList` throws boom; discarded == 3,4,5 (order-independent compare), subscription cancelled.\n4. Consumer stops via `take(2)`: result List(1,2), discarded == 3,4,5, cancelled.\n5. Consumer interrupted: run `runForeach(x => { received.release(); gate.acquire() })` in `forkCancellable`, push 1..5, wait for `received`, `cancel()`; discarded == 2,3,4,5, cancelled.\n6. Late onNext after cancellation: after any of the above, call `next(6)` on the publisher; `onDiscard(6)` is called synchronously and nothing throws.\n7. Existing one-arg overload: compiles unchanged and `FlowPublisherPekkoTest` still passes.\nRun `sbt \"core/testOnly ox.flow.reactive.*\"` and `sbt core/scalafmt`; all green."},{"title":"Exactly-once stress test: every element is emitted or discarded, never both or neither","description":"Add a test (same file as Task 1 or a sibling in core/src/test/scala/ox/flow/reactive/) that uses a demand-respecting publisher running on its own thread: it emits 1..N (N around 1000) only while `sent < requested` and not cancelled, calling `onNext` from that thread, then `onComplete`. Compose `Flow.fromPublisher(p, discarded.add)` with a consumer that stops at a random point (e.g. `take(k)` for random k, and separately a `map` that throws at a random element, and separately a `forkCancellable` cancelled after a random short delay). Collect emitted elements via `.tap(emitted.add)` into thread-safe collections. After the flow finishes and the publisher thread is joined, assert: `emitted` and `discarded` are disjoint, their union equals exactly the set of elements the publisher actually sent, and no element appears twice. Repeat each scenario in a loop (e.g. 20 iterations) with a small `given BufferCapacity = BufferCapacity(4)` to make buffer-full races likely. The publisher's late `onNext` calls (after cancellation) must land in `discarded`. If this exposes a race in Task 1's implementation, fix it in FlowCompanionReactiveOps.scala as part of this task. Run `sbt \"core/testOnly ox.flow.reactive.*\"`."},{"title":"Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","description":"In flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala add `def fromPublisher[T](p: org.reactivestreams.Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p), onDiscard)`, keep the one-arg overload, and update its scaladoc to point at the core overload for the callback semantics. Create flow-reactive-streams/src/test/scala/ox/flow/reactive/FlowReactiveStreamsFromPublisherTest.scala (AnyFunSuite with Matchers) with a small hand-written `org.reactivestreams.Publisher[Int]` (same shape as the core ManualPublisher, but implementing the reactive-streams interfaces; it cannot be shared because the module does not depend on core's test classpath) and two tests: elements flow through the one-arg variant; with `.take(1)` and the new overload, remaining pushed elements are passed to `onDiscard` and the subscription is cancelled. Run `sbt flowReactiveStreams/test` and `sbt flowReactiveStreams/scalafmt`."},{"title":"Document the onDiscard callback in doc/streaming/flows.md","description":"In doc/streaming/flows.md, section \"### Publisher -> Flow\" (around lines 211-219), add a short paragraph explaining: elements received from the publisher but not emitted downstream (buffered when the publisher fails, when the consumer stops early via exception, interruption or `take`, or arriving after cancellation) can be released by passing a second argument `onDiscard: T => Unit` to `Flow.fromPublisher` / `FlowReactiveStreams.fromPublisher`; it is called exactly once per such element, possibly from the publisher's thread, and should not throw; typical use is releasing reference-counted buffers. Also state that on publisher error, elements already received are emitted before the error is propagated. Keep the prose as short as the surrounding text. If you add a code example, use a ```` ```scala mdoc:compile-only ```` block and verify with `sbt compileDocumentation`; otherwise no build step is needed. Do not touch generated-doc/out."}],"brief":"## Modules and files\n\n- `core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala` — the only implementation file in core. Trait `FlowCompanionReactiveOps` (mixed into `Flow.type`) defines `def fromPublisher[T](p: java.util.concurrent.Flow.Publisher[T])(using BufferCapacity): Flow[T]`. Read the whole file (80 lines) before changing anything. Current structure: `usingEmitInline: emit => unsupervised { channel = BufferCapacity.newChannel[T]; forkUnsupervised { p.subscribe(Subscriber: onSubscribe -> subscriptionRef.set(s); s.request(capacity); onNext -> channel.send; onError -> channel.error; onComplete -> channel.done) }; repeatWhile { channel.receiveOrClosed() match Done -> false / Error -> throw / t -> emit(t); batched subscription.request } }.tapException(cancel subscription)`. The demand logic (`demandThreshold = ceil(capacity/2)`, `toDemand` counter, lazily cached `subscription` read from `subscriptionRef` after the first element) must be preserved.\n- `flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala` — `object FlowReactiveStreams { def fromPublisher[T](p: org.reactivestreams.Publisher[T])(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p)) }`. Module has no `src/test` dir yet; scalatest is already a test dependency (build.sbt `flowReactiveStreams`), it `dependsOn(core)` (compile only, not test->test).\n- `core/src/test/scala/ox/flow/reactive/` — existing reactive tests: `FlowPublisherPekkoTest.scala` (AnyFunSuite + Matchers, uses Pekko `Source...runWith(Sink.asPublisher)` and `FlowAdapters.toFlowPublisher`) and `FlowPublisherTckTest.scala`. Put new fromPublisher tests here. No test anywhere currently uses a hand-written Publisher; write one (see below).\n- `doc/streaming/flows.md` lines 211-219, section \"### Publisher -> Flow\" — the only user doc mentioning `fromPublisher`. `generated-doc/out/**` is mdoc output regenerated at release time by `updateDocs`; do NOT edit it. Code blocks in docs use ```` ```scala mdoc:compile-only ```` and are compiled by `sbt compileDocumentation` (slow; only needed if you add a code block). Doc dependency snippet convention: `// sbt dependency: \"com.softwaremill.ox\" %% \"flow-reactive-streams\" % \"@VERSION@\"`.\n\n## Key APIs (ox.channels, core/src/main/scala/ox/channels/Channel.scala)\n\n- `Sink.sendOrClosed(t): Unit | ChannelClosed` (non-throwing), `Sink.send(t)` throws `ChannelClosedException` when closed.\n- `Sink.doneOrClosed(): Unit | ChannelClosed` — \"buffered values will be delivered; send operations in progress complete normally when a receiver arrives\"; returns `ChannelClosed` instead of throwing if already closed. `Sink.errorOrClosed(reason)` — drops buffered values (that is the bug). Use the `*OrClosed` variants everywhere in the subscriber: a misbehaving publisher may call onError/onComplete twice.\n- `Source.receiveOrClosed(): T | ChannelClosed` (blocking); match `case ChannelClosed.Done => ...; case e: ChannelClosed.Error => throw e.toThrowable; case t: T @unchecked => ...`. `Source.tryReceiveOrClosed()` is non-blocking but may spuriously return None, so it is not suitable for a drain loop.\n- `Source.foreachOrError(f)` / `drainOrError()` from `SourceDrainOps.scala` loop until Done; usable for the drain, but write the drain inline so you can handle `InterruptedException` (see below).\n- jox semantics (com.softwaremill.jox channels 1.1.2, verified from source): after `done()`, buffered elements stay receivable; a receiver observing Done has already seen every element whose `send` started before `done()` (an in-progress sender is waited for); a `send` racing with `done()` either lands in the buffer or returns closed. Hence \"drain until Done, then `sendOrClosed` returns closed\" gives exactly-once discard with no extra locking.\n- `BufferCapacity`: `given default: BufferCapacity = 16`; `BufferCapacity.newChannel[T]`; `summon[BufferCapacity].toInt` (import `ox.channels.toInt`). Tests can override with `given BufferCapacity = BufferCapacity(4)`.\n- `ox.util`: `t.tapException(f)` runs `f(e)` for any Throwable and rethrows `e` (exceptions from `f` added as suppressed). `ox.pipe`, `ox.discard`, `ox.repeatWhile` already imported in the file.\n- `FlowOps.take(n)` stops upstream by throwing a private `ControlThrowable` (`abortTake`) out of `emit`, caught in `take` itself. So \"consumer stops early\" includes `take`, and the cleanup path must run for every Throwable, not just NonFatal. `tapException`/`catch case e: Throwable` already does this.\n- `unsupervised { ... }` + `forkUnsupervised { ... }` (import `ox.unsupervised`, `ox.forkUnsupervised`): scope end interrupts still-running forks once. `onNext`/`onError`/`onComplete` run on the publisher's threads (or on the `p.subscribe` fork for synchronous publishers), never on the consumer thread.\n- Test helpers: `ox.forkCancellable { ... }` inside `supervised` returns a `CancellableFork` with `cancel(): Either[Throwable, T]` (interrupts once, joins). Interruption inside the flow surfaces as `InterruptedException` thrown from a blocking call (e.g. `Semaphore.acquire`, `channel.receiveOrClosed`).\n\n## Design decisions (apply as-is)\n\n- API: add an overload `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg method delegate with `_ => ()`. Do not add a default parameter and do not change the existing signature: `core` runs MiMa (`enableMimaSettings` in build.sbt) and only additive changes are safe. Same shape in `FlowReactiveStreams`.\n- Subscriber: `onNext(t)` = `channel.sendOrClosed(t)` and if the result is `ChannelClosed` call `onDiscard(t)`. `onError(e)` = store `e` in an `AtomicReference[Throwable]`, then `channel.doneOrClosed().discard`. `onComplete()` = `channel.doneOrClosed().discard`. `onSubscribe(s)` = set `subscriptionRef`, then if a `cancelled: AtomicBoolean` is already set call `s.cancel()` else `s.request(capacity)` (both sides use volatile atomics, so at least one side sees the other; a double `cancel()` is allowed by the spec).\n- Consumer loop: on `ChannelClosed.Done`, read the stored error; if non-null throw it, else finish. Keep the `ChannelClosed.Error` branch for safety.\n- Cleanup, run on any Throwable escaping the loop (the stored publisher error included; cancel after onError is a spec-allowed no-op): set `cancelled`, cancel the subscription if `subscriptionRef` is non-null, `channel.doneOrClosed().discard`, then drain: loop `receiveOrClosed()` until `Done`, calling `onDiscard` for each element. The drain is bounded (channel is done, at most capacity elements plus in-flight sends) but blocking; if `InterruptedException` is thrown mid-drain, catch it, keep draining, and call `Thread.currentThread().interrupt()` once the drain finishes, so no element is lost on a second interrupt. Then rethrow the original exception. An element that was passed to `emit` and whose `emit` threw counts as emitted (downstream owns it): do not discard it.\n- Scaladoc on the new overload: state the exactly-once invariant, that `onDiscard` may be called from publisher threads, that it must not throw, and that on publisher error buffered elements are emitted before the error is rethrown (this is a deliberate behaviour change for the one-arg variant too; mention it in its doc).\n\n## Test publisher helper (write it in the core test, package `ox.flow.reactive`)\n\nA `ManualPublisher[T] extends java.util.concurrent.Flow.Publisher[T]` with: `@volatile var subscriber: Subscriber[? >: T]`, `requested: AtomicLong`, `cancelled: AtomicBoolean`, a `subscribed: CountDownLatch(1)`; `subscribe(s)` stores `s`, calls `s.onSubscribe(new Subscription { request(n) => requested.addAndGet(n); cancel() => cancelled.set(true) })`, counts down the latch. Helper methods `next(t)`, `error(e)`, `complete()` forward to the subscriber from the test thread. For the exactly-once stress test, a second variant that runs its own thread emitting `1..n` while `sent < requested` and not cancelled. Run the flow under test in `supervised { forkCancellable { ... } }` or `fork`, await `subscribed`, then drive the publisher. Collect emitted elements with `.tap(emitted += _)` or `runForeach`, discarded with `onDiscard = discarded += _` (use thread-safe collections such as `ConcurrentLinkedQueue` or `java.util.Collections.synchronizedList`, since `onDiscard` runs on other threads).\n\n## Conventions / build\n\n- Scala 3 indentation syntax, `end X` markers on long definitions, `.discard` on ignored results (compiler warns otherwise), scalafmt configured in `.scalafmt.conf` (140-column lines); run `sbt core/scalafmt flowReactiveStreams/scalafmt` after editing.\n- Comments: short, explain non-obvious reasoning only; never restate code. Scaladoc describes usage and invariants, not implementation.\n- Tests: scalatest 3.2.20; the reactive tests use `AnyFunSuite with Matchers` with `test(\"...\")` blocks and Scala 3 colon syntax. `core` runs tests forked (`Test / fork := true`).\n- Commands (sbt 2.0.10): `sbt \"core/testOnly ox.flow.reactive.*\"`, `sbt flowReactiveStreams/test`, `sbt core/mimaReportBinaryIssues` (only meaningful when a previous stable version resolves), `sbt compileDocumentation` for mdoc. Never commit; the user commits."}},{"id":[{"name":"Task: Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Exactly-once stress test: every element is emitted or discarded, never both or neither","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","occurrence":0}],"resultJson":{"findings":[{"id":"R1.I4.1","title":"sbt Test/compile fails with JGit NoWorkTreeException","reason":{"type":"Declined","text":"This is a problem with the environment, not the code. git sees the worktree as non-bare (`git rev-parse --is-bare-repository` prints false; the git dir is /home/adamw/ox/.git/worktrees/zippy-astrodon-94aa2fb6). The exception comes from the sbt-git plugin, which uses JGit by default and reads this linked worktree as bare while the build loads. Nothing in project/ or the build configures this; `zz-tmp-git.sbt` was my temporary workaround, not the cause. To check the code, I recreated it with `useReadableConsoleGit`, which makes sbt-git use command-line git: `sbt core/Test/compile flowReactiveStreams/Test/compile` succeeded (warnings only, in older test files, no errors) and flowReactiveStreams tests passed. I then deleted the file again. Changing the shared build's git plugin setup is outside the scope of this change."}}]}},{"id":[{"name":"Task: Document the onDiscard callback in doc/streaming/flows.md","occurrence":0}],"resultJson":{"findings":[{"id":"R1.I4.1","title":"`sbt Test/compile` fails with no diagnostic","reason":{"type":"Declined","text":"Environmental. Full output shows sbt-git's JGit throws NoWorkTreeException (\"Bare Repository has neither a working tree, nor an index\") while loading the build, because this checkout is a git worktree; no source error is involved. With a temporary `ThisBuild / git.gitUncommittedChanges := false` override, `sbt Test/compile` succeeds across all modules (only pre-existing unrelated warnings). The temporary file was removed; no codebase change is needed."}}]}},{"id":[{"name":"Final review","occurrence":0}],"resultJson":{"findings":[{"id":"S1","title":"sbt Test/compile fails with JGit NoWorkTreeException","reason":{"type":"Declined","text":"This is a problem with the environment, not the code. git sees the worktree as non-bare (`git rev-parse --is-bare-repository` prints false; the git dir is /home/adamw/ox/.git/worktrees/zippy-astrodon-94aa2fb6). The exception comes from the sbt-git plugin, which uses JGit by default and reads this linked worktree as bare while the build loads. Nothing in project/ or the build configures this; `zz-tmp-git.sbt` was my temporary workaround, not the cause. To check the code, I recreated it with `useReadableConsoleGit`, which makes sbt-git use command-line git: `sbt core/Test/compile flowReactiveStreams/Test/compile` succeeded (warnings only, in older test files, no errors) and flowReactiveStreams tests passed. I then deleted the file again. Changing the shared build's git plugin setup is outside the scope of this change."}},{"id":"S2","title":"`sbt Test/compile` fails with no diagnostic","reason":{"type":"Declined","text":"Environmental. Full output shows sbt-git's JGit throws NoWorkTreeException (\"Bare Repository has neither a working tree, nor an index\") while loading the build, because this checkout is a git worktree; no source error is involved. With a temporary `ThisBuild / git.gitUncommittedChanges := false` override, `sbt Test/compile` succeeds across all modules (only pre-existing unrelated warnings). The temporary file was removed; no codebase change is needed."}},{"id":"R1.I2.1","title":"No test covers the drain's deferred-interrupt path in discardBuffered","reason":{"type":"Declined","text":"A deterministic test can't be written. In jox 1.1.2, `receiveOrClosed` only throws `InterruptedException` when it parks (`Continuation.await`). After `doneOrClosed()` has finished on the draining thread, a receive never parks: buffered or in-flight sends are returned or retried, and later cells are already CLOSED. The catch is reached only in a narrow race, when a concurrent `onComplete`/`onError` is still closing cells. I tried the suggested test (interrupt plus throw in the consumer, then assert the discards and the interrupt flag). It passed both with the catch removed and with the re-interrupt removed, so it did not cover the branch, and I reverted it. The suggested capacity-1 variant also doesn't help, because a blocked sender's Continuation is resumed without parking."},"location":{"file":"core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala","line":141}},{"id":"R1.I5.1","title":"sbt Test/compile exited 1 with no compile error in the output","reason":{"type":"Declined","text":"Environmental. sbt-git's JGit fails in this git worktree with `NoWorkTreeException: Bare Repository has neither a working tree, nor an index` while evaluating `gitUncommittedChanges`, so the build fails before compiling. It is not a code problem. With a temporary `zz-tmp-git.sbt` that sets `ThisBuild / git.gitUncommittedChanges := false`, Test/compile and the core reactive and flowReactiveStreams tests all pass. I deleted the temporary file afterwards."}},{"id":"R2.I3.1","title":"`sbt Test/compile` exits 1 with no compiler diagnostics","reason":{"type":"Declined","text":"Environmental, not a code problem. In this git worktree, sbt-git's JGit throws `NoWorkTreeException: Bare Repository has neither a working tree, nor an index` while evaluating `gitUncommittedChanges` during build load. The build fails before any compilation, and the full output shows that stack trace above the timing line. With a temporary `zz-tmp-git.sbt` setting `ThisBuild / git.gitUncommittedChanges := false`, `Test/compile` succeeds. With the same file, `core/testOnly ox.flow.reactive.*` (44 passed, 7 ignored) and `flowReactiveStreams/test` (2 passed) also pass. I deleted the file afterwards, so it is not left in the tree."}}]}},{"id":[{"name":"Push branch","occurrence":0}],"resultJson":{"type":"Pushed"}}],"stageStarts":[]} \ No newline at end of file +{"header":{"startingBranch":"zippy-astrodon-94aa2fb6","branch":"zippy-astrodon-94aa2fb6","branchMode":"Reused","userPrompt":"Implement GitHub issue softwaremill/ox#528: \"Flow.fromPublisher: allow releasing elements which are buffered but not emitted\".\n\nProblem: `Flow.fromPublisher` (core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala) buffers elements in a channel. Some of them never reach the consumer:\n- when the publisher fails, `channel.error` drops the buffered elements\n- when the consumer stops early (an exception, an interruption), the buffered elements are never received\n- elements can still arrive (onNext) after the subscription is cancelled\n\nIf elements hold resources (e.g. Netty's reference-counted ByteBufs), nobody releases them, so they leak.\n\nGoal: add an optional callback, e.g. `fromPublisher(p, onDiscard: T => Unit)`, called exactly once for each element received from the publisher but not emitted downstream. Existing call sites without the callback must keep compiling and behaving the same.\n\nSuggested implementation: on publisher error, store the error and call `channel.done()` (so buffered elements are kept and emitted before the error is rethrown). When the flow ends early, cancel the subscription, close the channel, and drain it, calling the callback for each drained element. In `onNext`, call the callback if the channel is already closed. Make sure there are no races where an element is neither emitted nor discarded.\n\nAlso update the variant in flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala (org.reactivestreams Publisher) to accept the same optional callback, and update any docs (doc/ directory) describing fromPublisher.\n\nAcceptance criteria:\n- tests covering: publisher error with buffered elements (elements still emitted, then error); consumer stopping early (exception / interruption) with buffered elements -> callback called for each; elements arriving after cancellation -> callback called; every element is either emitted or discarded, never both or neither.\n- `sbt compile test` passes for affected modules (core, flow-reactive-streams).\n- Code follows existing style in the repo; concise doc comments.\n\nContext: tapir currently tracks unconsumed elements itself using a lock: https://github.com/softwaremill/tapir/pull/5573","flow":{"type":"Catalog","name":"implement.sc"},"startingCommit":"664f01730cda34fd88e8c3db3deb600934eb1d1e"},"entries":[{"id":[{"name":"Plan","occurrence":0}],"resultJson":{"id":"ox-528-from-publisher-on-discard","description":"`Flow.fromPublisher` buffers elements received from a reactive `Publisher` in a channel. Today, elements that are buffered but never emitted are silently lost: `channel.error` drops the buffer on publisher error, a consumer that stops early (exception, interruption, `take`) never receives what is buffered, and `onNext` calls that arrive after the subscription is cancelled are dropped by `channel.send` throwing. When elements own resources (Netty reference-counted ByteBufs in tapir), this leaks them. Tapir currently works around it with its own lock-based tracking (softwaremill/tapir#5573).\n\nThis epic adds an optional `onDiscard: T => Unit` callback to `Flow.fromPublisher` (core, JDK `Flow.Publisher`) and `FlowReactiveStreams.fromPublisher` (flow-reactive-streams, `org.reactivestreams.Publisher`), with the invariant that every element received from the publisher is either emitted downstream or passed to `onDiscard`, exactly once, never both or neither. Publisher errors no longer drop buffered elements: the error is stored, the channel is completed with `done()`, buffered elements are emitted, and then the error is rethrown. When the flow ends early, the subscription is cancelled, the channel is closed and drained through `onDiscard`, and any late `onNext` goes to `onDiscard` as well. Existing single-argument call sites keep compiling (the new parameter is an overload, which also keeps MiMa happy for `core`). Docs in `doc/streaming/flows.md` describe the new callback.","tasks":[{"title":"Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","description":"In core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala add `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg `fromPublisher` delegate to it with `_ => ()`. Implement per the brief's design decisions: `onNext` uses `sendOrClosed` and discards on closed; `onError` stores the throwable in an AtomicReference and calls `doneOrClosed`; `onComplete` calls `doneOrClosed`; `onSubscribe` cancels immediately if a `cancelled` flag is already set, else requests `capacity`; the consumer loop rethrows the stored error when it sees `Done`; on any Throwable escaping the loop: set `cancelled`, cancel the subscription if present, `doneOrClosed` the channel, drain it calling `onDiscard` per element (keep draining across an InterruptedException and re-interrupt the thread afterwards), rethrow. Preserve the existing batched demand logic. Update the scaladoc of both overloads (exactly-once invariant, callback runs on publisher threads and must not throw, buffered elements are now emitted before a publisher error is rethrown).\n\nTests, in a new core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardTest.scala (AnyFunSuite with Matchers), using a hand-written `ManualPublisher[T]` helper as described in the brief (keep the helper in this file or a sibling file in the same package so Task 2 can reuse it):\n1. Publisher error with buffered elements: push 1,2,3 then `error(boom)` before/while the consumer runs; `runToList` throws boom and all of 1,2,3 were emitted; `onDiscard` never called.\n2. Publisher error with nothing buffered still throws.\n3. Consumer exception: `.map(x => if x == 2 then throw boom else x)`, push 1..5; `runToList` throws boom; discarded == 3,4,5 (order-independent compare), subscription cancelled.\n4. Consumer stops via `take(2)`: result List(1,2), discarded == 3,4,5, cancelled.\n5. Consumer interrupted: run `runForeach(x => { received.release(); gate.acquire() })` in `forkCancellable`, push 1..5, wait for `received`, `cancel()`; discarded == 2,3,4,5, cancelled.\n6. Late onNext after cancellation: after any of the above, call `next(6)` on the publisher; `onDiscard(6)` is called synchronously and nothing throws.\n7. Existing one-arg overload: compiles unchanged and `FlowPublisherPekkoTest` still passes.\nRun `sbt \"core/testOnly ox.flow.reactive.*\"` and `sbt core/scalafmt`; all green."},{"title":"Exactly-once stress test: every element is emitted or discarded, never both or neither","description":"Add a test (same file as Task 1 or a sibling in core/src/test/scala/ox/flow/reactive/) that uses a demand-respecting publisher running on its own thread: it emits 1..N (N around 1000) only while `sent < requested` and not cancelled, calling `onNext` from that thread, then `onComplete`. Compose `Flow.fromPublisher(p, discarded.add)` with a consumer that stops at a random point (e.g. `take(k)` for random k, and separately a `map` that throws at a random element, and separately a `forkCancellable` cancelled after a random short delay). Collect emitted elements via `.tap(emitted.add)` into thread-safe collections. After the flow finishes and the publisher thread is joined, assert: `emitted` and `discarded` are disjoint, their union equals exactly the set of elements the publisher actually sent, and no element appears twice. Repeat each scenario in a loop (e.g. 20 iterations) with a small `given BufferCapacity = BufferCapacity(4)` to make buffer-full races likely. The publisher's late `onNext` calls (after cancellation) must land in `discarded`. If this exposes a race in Task 1's implementation, fix it in FlowCompanionReactiveOps.scala as part of this task. Run `sbt \"core/testOnly ox.flow.reactive.*\"`."},{"title":"Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","description":"In flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala add `def fromPublisher[T](p: org.reactivestreams.Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p), onDiscard)`, keep the one-arg overload, and update its scaladoc to point at the core overload for the callback semantics. Create flow-reactive-streams/src/test/scala/ox/flow/reactive/FlowReactiveStreamsFromPublisherTest.scala (AnyFunSuite with Matchers) with a small hand-written `org.reactivestreams.Publisher[Int]` (same shape as the core ManualPublisher, but implementing the reactive-streams interfaces; it cannot be shared because the module does not depend on core's test classpath) and two tests: elements flow through the one-arg variant; with `.take(1)` and the new overload, remaining pushed elements are passed to `onDiscard` and the subscription is cancelled. Run `sbt flowReactiveStreams/test` and `sbt flowReactiveStreams/scalafmt`."},{"title":"Document the onDiscard callback in doc/streaming/flows.md","description":"In doc/streaming/flows.md, section \"### Publisher -> Flow\" (around lines 211-219), add a short paragraph explaining: elements received from the publisher but not emitted downstream (buffered when the publisher fails, when the consumer stops early via exception, interruption or `take`, or arriving after cancellation) can be released by passing a second argument `onDiscard: T => Unit` to `Flow.fromPublisher` / `FlowReactiveStreams.fromPublisher`; it is called exactly once per such element, possibly from the publisher's thread, and should not throw; typical use is releasing reference-counted buffers. Also state that on publisher error, elements already received are emitted before the error is propagated. Keep the prose as short as the surrounding text. If you add a code example, use a ```` ```scala mdoc:compile-only ```` block and verify with `sbt compileDocumentation`; otherwise no build step is needed. Do not touch generated-doc/out."}],"brief":"## Modules and files\n\n- `core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala` — the only implementation file in core. Trait `FlowCompanionReactiveOps` (mixed into `Flow.type`) defines `def fromPublisher[T](p: java.util.concurrent.Flow.Publisher[T])(using BufferCapacity): Flow[T]`. Read the whole file (80 lines) before changing anything. Current structure: `usingEmitInline: emit => unsupervised { channel = BufferCapacity.newChannel[T]; forkUnsupervised { p.subscribe(Subscriber: onSubscribe -> subscriptionRef.set(s); s.request(capacity); onNext -> channel.send; onError -> channel.error; onComplete -> channel.done) }; repeatWhile { channel.receiveOrClosed() match Done -> false / Error -> throw / t -> emit(t); batched subscription.request } }.tapException(cancel subscription)`. The demand logic (`demandThreshold = ceil(capacity/2)`, `toDemand` counter, lazily cached `subscription` read from `subscriptionRef` after the first element) must be preserved.\n- `flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala` — `object FlowReactiveStreams { def fromPublisher[T](p: org.reactivestreams.Publisher[T])(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p)) }`. Module has no `src/test` dir yet; scalatest is already a test dependency (build.sbt `flowReactiveStreams`), it `dependsOn(core)` (compile only, not test->test).\n- `core/src/test/scala/ox/flow/reactive/` — existing reactive tests: `FlowPublisherPekkoTest.scala` (AnyFunSuite + Matchers, uses Pekko `Source...runWith(Sink.asPublisher)` and `FlowAdapters.toFlowPublisher`) and `FlowPublisherTckTest.scala`. Put new fromPublisher tests here. No test anywhere currently uses a hand-written Publisher; write one (see below).\n- `doc/streaming/flows.md` lines 211-219, section \"### Publisher -> Flow\" — the only user doc mentioning `fromPublisher`. `generated-doc/out/**` is mdoc output regenerated at release time by `updateDocs`; do NOT edit it. Code blocks in docs use ```` ```scala mdoc:compile-only ```` and are compiled by `sbt compileDocumentation` (slow; only needed if you add a code block). Doc dependency snippet convention: `// sbt dependency: \"com.softwaremill.ox\" %% \"flow-reactive-streams\" % \"@VERSION@\"`.\n\n## Key APIs (ox.channels, core/src/main/scala/ox/channels/Channel.scala)\n\n- `Sink.sendOrClosed(t): Unit | ChannelClosed` (non-throwing), `Sink.send(t)` throws `ChannelClosedException` when closed.\n- `Sink.doneOrClosed(): Unit | ChannelClosed` — \"buffered values will be delivered; send operations in progress complete normally when a receiver arrives\"; returns `ChannelClosed` instead of throwing if already closed. `Sink.errorOrClosed(reason)` — drops buffered values (that is the bug). Use the `*OrClosed` variants everywhere in the subscriber: a misbehaving publisher may call onError/onComplete twice.\n- `Source.receiveOrClosed(): T | ChannelClosed` (blocking); match `case ChannelClosed.Done => ...; case e: ChannelClosed.Error => throw e.toThrowable; case t: T @unchecked => ...`. `Source.tryReceiveOrClosed()` is non-blocking but may spuriously return None, so it is not suitable for a drain loop.\n- `Source.foreachOrError(f)` / `drainOrError()` from `SourceDrainOps.scala` loop until Done; usable for the drain, but write the drain inline so you can handle `InterruptedException` (see below).\n- jox semantics (com.softwaremill.jox channels 1.1.2, verified from source): after `done()`, buffered elements stay receivable; a receiver observing Done has already seen every element whose `send` started before `done()` (an in-progress sender is waited for); a `send` racing with `done()` either lands in the buffer or returns closed. Hence \"drain until Done, then `sendOrClosed` returns closed\" gives exactly-once discard with no extra locking.\n- `BufferCapacity`: `given default: BufferCapacity = 16`; `BufferCapacity.newChannel[T]`; `summon[BufferCapacity].toInt` (import `ox.channels.toInt`). Tests can override with `given BufferCapacity = BufferCapacity(4)`.\n- `ox.util`: `t.tapException(f)` runs `f(e)` for any Throwable and rethrows `e` (exceptions from `f` added as suppressed). `ox.pipe`, `ox.discard`, `ox.repeatWhile` already imported in the file.\n- `FlowOps.take(n)` stops upstream by throwing a private `ControlThrowable` (`abortTake`) out of `emit`, caught in `take` itself. So \"consumer stops early\" includes `take`, and the cleanup path must run for every Throwable, not just NonFatal. `tapException`/`catch case e: Throwable` already does this.\n- `unsupervised { ... }` + `forkUnsupervised { ... }` (import `ox.unsupervised`, `ox.forkUnsupervised`): scope end interrupts still-running forks once. `onNext`/`onError`/`onComplete` run on the publisher's threads (or on the `p.subscribe` fork for synchronous publishers), never on the consumer thread.\n- Test helpers: `ox.forkCancellable { ... }` inside `supervised` returns a `CancellableFork` with `cancel(): Either[Throwable, T]` (interrupts once, joins). Interruption inside the flow surfaces as `InterruptedException` thrown from a blocking call (e.g. `Semaphore.acquire`, `channel.receiveOrClosed`).\n\n## Design decisions (apply as-is)\n\n- API: add an overload `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg method delegate with `_ => ()`. Do not add a default parameter and do not change the existing signature: `core` runs MiMa (`enableMimaSettings` in build.sbt) and only additive changes are safe. Same shape in `FlowReactiveStreams`.\n- Subscriber: `onNext(t)` = `channel.sendOrClosed(t)` and if the result is `ChannelClosed` call `onDiscard(t)`. `onError(e)` = store `e` in an `AtomicReference[Throwable]`, then `channel.doneOrClosed().discard`. `onComplete()` = `channel.doneOrClosed().discard`. `onSubscribe(s)` = set `subscriptionRef`, then if a `cancelled: AtomicBoolean` is already set call `s.cancel()` else `s.request(capacity)` (both sides use volatile atomics, so at least one side sees the other; a double `cancel()` is allowed by the spec).\n- Consumer loop: on `ChannelClosed.Done`, read the stored error; if non-null throw it, else finish. Keep the `ChannelClosed.Error` branch for safety.\n- Cleanup, run on any Throwable escaping the loop (the stored publisher error included; cancel after onError is a spec-allowed no-op): set `cancelled`, cancel the subscription if `subscriptionRef` is non-null, `channel.doneOrClosed().discard`, then drain: loop `receiveOrClosed()` until `Done`, calling `onDiscard` for each element. The drain is bounded (channel is done, at most capacity elements plus in-flight sends) but blocking; if `InterruptedException` is thrown mid-drain, catch it, keep draining, and call `Thread.currentThread().interrupt()` once the drain finishes, so no element is lost on a second interrupt. Then rethrow the original exception. An element that was passed to `emit` and whose `emit` threw counts as emitted (downstream owns it): do not discard it.\n- Scaladoc on the new overload: state the exactly-once invariant, that `onDiscard` may be called from publisher threads, that it must not throw, and that on publisher error buffered elements are emitted before the error is rethrown (this is a deliberate behaviour change for the one-arg variant too; mention it in its doc).\n\n## Test publisher helper (write it in the core test, package `ox.flow.reactive`)\n\nA `ManualPublisher[T] extends java.util.concurrent.Flow.Publisher[T]` with: `@volatile var subscriber: Subscriber[? >: T]`, `requested: AtomicLong`, `cancelled: AtomicBoolean`, a `subscribed: CountDownLatch(1)`; `subscribe(s)` stores `s`, calls `s.onSubscribe(new Subscription { request(n) => requested.addAndGet(n); cancel() => cancelled.set(true) })`, counts down the latch. Helper methods `next(t)`, `error(e)`, `complete()` forward to the subscriber from the test thread. For the exactly-once stress test, a second variant that runs its own thread emitting `1..n` while `sent < requested` and not cancelled. Run the flow under test in `supervised { forkCancellable { ... } }` or `fork`, await `subscribed`, then drive the publisher. Collect emitted elements with `.tap(emitted += _)` or `runForeach`, discarded with `onDiscard = discarded += _` (use thread-safe collections such as `ConcurrentLinkedQueue` or `java.util.Collections.synchronizedList`, since `onDiscard` runs on other threads).\n\n## Conventions / build\n\n- Scala 3 indentation syntax, `end X` markers on long definitions, `.discard` on ignored results (compiler warns otherwise), scalafmt configured in `.scalafmt.conf` (140-column lines); run `sbt core/scalafmt flowReactiveStreams/scalafmt` after editing.\n- Comments: short, explain non-obvious reasoning only; never restate code. Scaladoc describes usage and invariants, not implementation.\n- Tests: scalatest 3.2.20; the reactive tests use `AnyFunSuite with Matchers` with `test(\"...\")` blocks and Scala 3 colon syntax. `core` runs tests forked (`Test / fork := true`).\n- Commands (sbt 2.0.10): `sbt \"core/testOnly ox.flow.reactive.*\"`, `sbt flowReactiveStreams/test`, `sbt core/mimaReportBinaryIssues` (only meaningful when a previous stable version resolves), `sbt compileDocumentation` for mdoc. Never commit; the user commits."}},{"id":[{"name":"Task: Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Exactly-once stress test: every element is emitted or discarded, never both or neither","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","occurrence":0}],"resultJson":{"findings":[{"id":"R1.I4.1","title":"sbt Test/compile fails with JGit NoWorkTreeException","reason":{"type":"Declined","text":"This is a problem with the environment, not the code. git sees the worktree as non-bare (`git rev-parse --is-bare-repository` prints false; the git dir is /home/adamw/ox/.git/worktrees/zippy-astrodon-94aa2fb6). The exception comes from the sbt-git plugin, which uses JGit by default and reads this linked worktree as bare while the build loads. Nothing in project/ or the build configures this; `zz-tmp-git.sbt` was my temporary workaround, not the cause. To check the code, I recreated it with `useReadableConsoleGit`, which makes sbt-git use command-line git: `sbt core/Test/compile flowReactiveStreams/Test/compile` succeeded (warnings only, in older test files, no errors) and flowReactiveStreams tests passed. I then deleted the file again. Changing the shared build's git plugin setup is outside the scope of this change."}}]}},{"id":[{"name":"Task: Document the onDiscard callback in doc/streaming/flows.md","occurrence":0}],"resultJson":{"findings":[{"id":"R1.I4.1","title":"`sbt Test/compile` fails with no diagnostic","reason":{"type":"Declined","text":"Environmental. Full output shows sbt-git's JGit throws NoWorkTreeException (\"Bare Repository has neither a working tree, nor an index\") while loading the build, because this checkout is a git worktree; no source error is involved. With a temporary `ThisBuild / git.gitUncommittedChanges := false` override, `sbt Test/compile` succeeds across all modules (only pre-existing unrelated warnings). The temporary file was removed; no codebase change is needed."}}]}},{"id":[{"name":"Final review","occurrence":0}],"resultJson":{"findings":[{"id":"S1","title":"sbt Test/compile fails with JGit NoWorkTreeException","reason":{"type":"Declined","text":"This is a problem with the environment, not the code. git sees the worktree as non-bare (`git rev-parse --is-bare-repository` prints false; the git dir is /home/adamw/ox/.git/worktrees/zippy-astrodon-94aa2fb6). The exception comes from the sbt-git plugin, which uses JGit by default and reads this linked worktree as bare while the build loads. Nothing in project/ or the build configures this; `zz-tmp-git.sbt` was my temporary workaround, not the cause. To check the code, I recreated it with `useReadableConsoleGit`, which makes sbt-git use command-line git: `sbt core/Test/compile flowReactiveStreams/Test/compile` succeeded (warnings only, in older test files, no errors) and flowReactiveStreams tests passed. I then deleted the file again. Changing the shared build's git plugin setup is outside the scope of this change."}},{"id":"S2","title":"`sbt Test/compile` fails with no diagnostic","reason":{"type":"Declined","text":"Environmental. Full output shows sbt-git's JGit throws NoWorkTreeException (\"Bare Repository has neither a working tree, nor an index\") while loading the build, because this checkout is a git worktree; no source error is involved. With a temporary `ThisBuild / git.gitUncommittedChanges := false` override, `sbt Test/compile` succeeds across all modules (only pre-existing unrelated warnings). The temporary file was removed; no codebase change is needed."}},{"id":"R1.I2.1","title":"No test covers the drain's deferred-interrupt path in discardBuffered","reason":{"type":"Declined","text":"A deterministic test can't be written. In jox 1.1.2, `receiveOrClosed` only throws `InterruptedException` when it parks (`Continuation.await`). After `doneOrClosed()` has finished on the draining thread, a receive never parks: buffered or in-flight sends are returned or retried, and later cells are already CLOSED. The catch is reached only in a narrow race, when a concurrent `onComplete`/`onError` is still closing cells. I tried the suggested test (interrupt plus throw in the consumer, then assert the discards and the interrupt flag). It passed both with the catch removed and with the re-interrupt removed, so it did not cover the branch, and I reverted it. The suggested capacity-1 variant also doesn't help, because a blocked sender's Continuation is resumed without parking."},"location":{"file":"core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala","line":141}},{"id":"R1.I5.1","title":"sbt Test/compile exited 1 with no compile error in the output","reason":{"type":"Declined","text":"Environmental. sbt-git's JGit fails in this git worktree with `NoWorkTreeException: Bare Repository has neither a working tree, nor an index` while evaluating `gitUncommittedChanges`, so the build fails before compiling. It is not a code problem. With a temporary `zz-tmp-git.sbt` that sets `ThisBuild / git.gitUncommittedChanges := false`, Test/compile and the core reactive and flowReactiveStreams tests all pass. I deleted the temporary file afterwards."}},{"id":"R2.I3.1","title":"`sbt Test/compile` exits 1 with no compiler diagnostics","reason":{"type":"Declined","text":"Environmental, not a code problem. In this git worktree, sbt-git's JGit throws `NoWorkTreeException: Bare Repository has neither a working tree, nor an index` while evaluating `gitUncommittedChanges` during build load. The build fails before any compilation, and the full output shows that stack trace above the timing line. With a temporary `zz-tmp-git.sbt` setting `ThisBuild / git.gitUncommittedChanges := false`, `Test/compile` succeeds. With the same file, `core/testOnly ox.flow.reactive.*` (44 passed, 7 ignored) and `flowReactiveStreams/test` (2 passed) also pass. I deleted the file afterwards, so it is not left in the tree."}}]}},{"id":[{"name":"Push branch","occurrence":0}],"resultJson":{"type":"Pushed"}},{"id":[{"name":"Generate PR title and description","occurrence":0}],"resultJson":{"title":"Add onDiscard callback to Flow.fromPublisher","body":"`Flow.fromPublisher` buffers elements in a channel. Some of them could be received from the publisher but never emitted, and nothing released them. That leaks elements which hold resources, such as Netty's ref-counted ByteBufs. Tapir currently tracks these elements itself (softwaremill/tapir#5573).\n\n- New overload `fromPublisher(p, onDiscard: T => Unit)`, in core and in `FlowReactiveStreams`. Each element received from the publisher is either emitted, or passed to `onDiscard` exactly once. Elements are discarded when the flow fails, is interrupted, or stops early (e.g. `take`), and also when they arrive after the subscription is cancelled. `onDiscard` may run on publisher threads and must not throw.\n- Behaviour change, also for the one-argument variant: when the publisher fails, elements already buffered are now emitted first, and then the error is rethrown. Before, they were dropped.\n- The existing one-argument overloads are unchanged and pass a no-op callback. The new method is an overload, not a default parameter, so binary compatibility is kept.\n- Added tests for publisher error, consumer exception, interruption, `take`, and elements arriving after cancellation. Also added a stress test that checks each element is either emitted or discarded, never both or neither. Updated `doc/streaming/flows.md`.\n\nCloses #528"}}],"stageStarts":[]} \ No newline at end of file From c21b4817e72ccea75080b8ed30a7842ee48bb45c Mon Sep 17 00:00:00 2001 From: Adam Warski Date: Fri, 9 Oct 2026 15:31:16 +0000 Subject: [PATCH 10/11] stage: Open PR --- .orca/runs/0a3279c8e3d7.progress.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.orca/runs/0a3279c8e3d7.progress.json b/.orca/runs/0a3279c8e3d7.progress.json index 1072e185..2f3a8d53 100644 --- a/.orca/runs/0a3279c8e3d7.progress.json +++ b/.orca/runs/0a3279c8e3d7.progress.json @@ -1 +1 @@ -{"header":{"startingBranch":"zippy-astrodon-94aa2fb6","branch":"zippy-astrodon-94aa2fb6","branchMode":"Reused","userPrompt":"Implement GitHub issue softwaremill/ox#528: \"Flow.fromPublisher: allow releasing elements which are buffered but not emitted\".\n\nProblem: `Flow.fromPublisher` (core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala) buffers elements in a channel. Some of them never reach the consumer:\n- when the publisher fails, `channel.error` drops the buffered elements\n- when the consumer stops early (an exception, an interruption), the buffered elements are never received\n- elements can still arrive (onNext) after the subscription is cancelled\n\nIf elements hold resources (e.g. Netty's reference-counted ByteBufs), nobody releases them, so they leak.\n\nGoal: add an optional callback, e.g. `fromPublisher(p, onDiscard: T => Unit)`, called exactly once for each element received from the publisher but not emitted downstream. Existing call sites without the callback must keep compiling and behaving the same.\n\nSuggested implementation: on publisher error, store the error and call `channel.done()` (so buffered elements are kept and emitted before the error is rethrown). When the flow ends early, cancel the subscription, close the channel, and drain it, calling the callback for each drained element. In `onNext`, call the callback if the channel is already closed. Make sure there are no races where an element is neither emitted nor discarded.\n\nAlso update the variant in flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala (org.reactivestreams Publisher) to accept the same optional callback, and update any docs (doc/ directory) describing fromPublisher.\n\nAcceptance criteria:\n- tests covering: publisher error with buffered elements (elements still emitted, then error); consumer stopping early (exception / interruption) with buffered elements -> callback called for each; elements arriving after cancellation -> callback called; every element is either emitted or discarded, never both or neither.\n- `sbt compile test` passes for affected modules (core, flow-reactive-streams).\n- Code follows existing style in the repo; concise doc comments.\n\nContext: tapir currently tracks unconsumed elements itself using a lock: https://github.com/softwaremill/tapir/pull/5573","flow":{"type":"Catalog","name":"implement.sc"},"startingCommit":"664f01730cda34fd88e8c3db3deb600934eb1d1e"},"entries":[{"id":[{"name":"Plan","occurrence":0}],"resultJson":{"id":"ox-528-from-publisher-on-discard","description":"`Flow.fromPublisher` buffers elements received from a reactive `Publisher` in a channel. Today, elements that are buffered but never emitted are silently lost: `channel.error` drops the buffer on publisher error, a consumer that stops early (exception, interruption, `take`) never receives what is buffered, and `onNext` calls that arrive after the subscription is cancelled are dropped by `channel.send` throwing. When elements own resources (Netty reference-counted ByteBufs in tapir), this leaks them. Tapir currently works around it with its own lock-based tracking (softwaremill/tapir#5573).\n\nThis epic adds an optional `onDiscard: T => Unit` callback to `Flow.fromPublisher` (core, JDK `Flow.Publisher`) and `FlowReactiveStreams.fromPublisher` (flow-reactive-streams, `org.reactivestreams.Publisher`), with the invariant that every element received from the publisher is either emitted downstream or passed to `onDiscard`, exactly once, never both or neither. Publisher errors no longer drop buffered elements: the error is stored, the channel is completed with `done()`, buffered elements are emitted, and then the error is rethrown. When the flow ends early, the subscription is cancelled, the channel is closed and drained through `onDiscard`, and any late `onNext` goes to `onDiscard` as well. Existing single-argument call sites keep compiling (the new parameter is an overload, which also keeps MiMa happy for `core`). Docs in `doc/streaming/flows.md` describe the new callback.","tasks":[{"title":"Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","description":"In core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala add `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg `fromPublisher` delegate to it with `_ => ()`. Implement per the brief's design decisions: `onNext` uses `sendOrClosed` and discards on closed; `onError` stores the throwable in an AtomicReference and calls `doneOrClosed`; `onComplete` calls `doneOrClosed`; `onSubscribe` cancels immediately if a `cancelled` flag is already set, else requests `capacity`; the consumer loop rethrows the stored error when it sees `Done`; on any Throwable escaping the loop: set `cancelled`, cancel the subscription if present, `doneOrClosed` the channel, drain it calling `onDiscard` per element (keep draining across an InterruptedException and re-interrupt the thread afterwards), rethrow. Preserve the existing batched demand logic. Update the scaladoc of both overloads (exactly-once invariant, callback runs on publisher threads and must not throw, buffered elements are now emitted before a publisher error is rethrown).\n\nTests, in a new core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardTest.scala (AnyFunSuite with Matchers), using a hand-written `ManualPublisher[T]` helper as described in the brief (keep the helper in this file or a sibling file in the same package so Task 2 can reuse it):\n1. Publisher error with buffered elements: push 1,2,3 then `error(boom)` before/while the consumer runs; `runToList` throws boom and all of 1,2,3 were emitted; `onDiscard` never called.\n2. Publisher error with nothing buffered still throws.\n3. Consumer exception: `.map(x => if x == 2 then throw boom else x)`, push 1..5; `runToList` throws boom; discarded == 3,4,5 (order-independent compare), subscription cancelled.\n4. Consumer stops via `take(2)`: result List(1,2), discarded == 3,4,5, cancelled.\n5. Consumer interrupted: run `runForeach(x => { received.release(); gate.acquire() })` in `forkCancellable`, push 1..5, wait for `received`, `cancel()`; discarded == 2,3,4,5, cancelled.\n6. Late onNext after cancellation: after any of the above, call `next(6)` on the publisher; `onDiscard(6)` is called synchronously and nothing throws.\n7. Existing one-arg overload: compiles unchanged and `FlowPublisherPekkoTest` still passes.\nRun `sbt \"core/testOnly ox.flow.reactive.*\"` and `sbt core/scalafmt`; all green."},{"title":"Exactly-once stress test: every element is emitted or discarded, never both or neither","description":"Add a test (same file as Task 1 or a sibling in core/src/test/scala/ox/flow/reactive/) that uses a demand-respecting publisher running on its own thread: it emits 1..N (N around 1000) only while `sent < requested` and not cancelled, calling `onNext` from that thread, then `onComplete`. Compose `Flow.fromPublisher(p, discarded.add)` with a consumer that stops at a random point (e.g. `take(k)` for random k, and separately a `map` that throws at a random element, and separately a `forkCancellable` cancelled after a random short delay). Collect emitted elements via `.tap(emitted.add)` into thread-safe collections. After the flow finishes and the publisher thread is joined, assert: `emitted` and `discarded` are disjoint, their union equals exactly the set of elements the publisher actually sent, and no element appears twice. Repeat each scenario in a loop (e.g. 20 iterations) with a small `given BufferCapacity = BufferCapacity(4)` to make buffer-full races likely. The publisher's late `onNext` calls (after cancellation) must land in `discarded`. If this exposes a race in Task 1's implementation, fix it in FlowCompanionReactiveOps.scala as part of this task. Run `sbt \"core/testOnly ox.flow.reactive.*\"`."},{"title":"Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","description":"In flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala add `def fromPublisher[T](p: org.reactivestreams.Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p), onDiscard)`, keep the one-arg overload, and update its scaladoc to point at the core overload for the callback semantics. Create flow-reactive-streams/src/test/scala/ox/flow/reactive/FlowReactiveStreamsFromPublisherTest.scala (AnyFunSuite with Matchers) with a small hand-written `org.reactivestreams.Publisher[Int]` (same shape as the core ManualPublisher, but implementing the reactive-streams interfaces; it cannot be shared because the module does not depend on core's test classpath) and two tests: elements flow through the one-arg variant; with `.take(1)` and the new overload, remaining pushed elements are passed to `onDiscard` and the subscription is cancelled. Run `sbt flowReactiveStreams/test` and `sbt flowReactiveStreams/scalafmt`."},{"title":"Document the onDiscard callback in doc/streaming/flows.md","description":"In doc/streaming/flows.md, section \"### Publisher -> Flow\" (around lines 211-219), add a short paragraph explaining: elements received from the publisher but not emitted downstream (buffered when the publisher fails, when the consumer stops early via exception, interruption or `take`, or arriving after cancellation) can be released by passing a second argument `onDiscard: T => Unit` to `Flow.fromPublisher` / `FlowReactiveStreams.fromPublisher`; it is called exactly once per such element, possibly from the publisher's thread, and should not throw; typical use is releasing reference-counted buffers. Also state that on publisher error, elements already received are emitted before the error is propagated. Keep the prose as short as the surrounding text. If you add a code example, use a ```` ```scala mdoc:compile-only ```` block and verify with `sbt compileDocumentation`; otherwise no build step is needed. Do not touch generated-doc/out."}],"brief":"## Modules and files\n\n- `core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala` — the only implementation file in core. Trait `FlowCompanionReactiveOps` (mixed into `Flow.type`) defines `def fromPublisher[T](p: java.util.concurrent.Flow.Publisher[T])(using BufferCapacity): Flow[T]`. Read the whole file (80 lines) before changing anything. Current structure: `usingEmitInline: emit => unsupervised { channel = BufferCapacity.newChannel[T]; forkUnsupervised { p.subscribe(Subscriber: onSubscribe -> subscriptionRef.set(s); s.request(capacity); onNext -> channel.send; onError -> channel.error; onComplete -> channel.done) }; repeatWhile { channel.receiveOrClosed() match Done -> false / Error -> throw / t -> emit(t); batched subscription.request } }.tapException(cancel subscription)`. The demand logic (`demandThreshold = ceil(capacity/2)`, `toDemand` counter, lazily cached `subscription` read from `subscriptionRef` after the first element) must be preserved.\n- `flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala` — `object FlowReactiveStreams { def fromPublisher[T](p: org.reactivestreams.Publisher[T])(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p)) }`. Module has no `src/test` dir yet; scalatest is already a test dependency (build.sbt `flowReactiveStreams`), it `dependsOn(core)` (compile only, not test->test).\n- `core/src/test/scala/ox/flow/reactive/` — existing reactive tests: `FlowPublisherPekkoTest.scala` (AnyFunSuite + Matchers, uses Pekko `Source...runWith(Sink.asPublisher)` and `FlowAdapters.toFlowPublisher`) and `FlowPublisherTckTest.scala`. Put new fromPublisher tests here. No test anywhere currently uses a hand-written Publisher; write one (see below).\n- `doc/streaming/flows.md` lines 211-219, section \"### Publisher -> Flow\" — the only user doc mentioning `fromPublisher`. `generated-doc/out/**` is mdoc output regenerated at release time by `updateDocs`; do NOT edit it. Code blocks in docs use ```` ```scala mdoc:compile-only ```` and are compiled by `sbt compileDocumentation` (slow; only needed if you add a code block). Doc dependency snippet convention: `// sbt dependency: \"com.softwaremill.ox\" %% \"flow-reactive-streams\" % \"@VERSION@\"`.\n\n## Key APIs (ox.channels, core/src/main/scala/ox/channels/Channel.scala)\n\n- `Sink.sendOrClosed(t): Unit | ChannelClosed` (non-throwing), `Sink.send(t)` throws `ChannelClosedException` when closed.\n- `Sink.doneOrClosed(): Unit | ChannelClosed` — \"buffered values will be delivered; send operations in progress complete normally when a receiver arrives\"; returns `ChannelClosed` instead of throwing if already closed. `Sink.errorOrClosed(reason)` — drops buffered values (that is the bug). Use the `*OrClosed` variants everywhere in the subscriber: a misbehaving publisher may call onError/onComplete twice.\n- `Source.receiveOrClosed(): T | ChannelClosed` (blocking); match `case ChannelClosed.Done => ...; case e: ChannelClosed.Error => throw e.toThrowable; case t: T @unchecked => ...`. `Source.tryReceiveOrClosed()` is non-blocking but may spuriously return None, so it is not suitable for a drain loop.\n- `Source.foreachOrError(f)` / `drainOrError()` from `SourceDrainOps.scala` loop until Done; usable for the drain, but write the drain inline so you can handle `InterruptedException` (see below).\n- jox semantics (com.softwaremill.jox channels 1.1.2, verified from source): after `done()`, buffered elements stay receivable; a receiver observing Done has already seen every element whose `send` started before `done()` (an in-progress sender is waited for); a `send` racing with `done()` either lands in the buffer or returns closed. Hence \"drain until Done, then `sendOrClosed` returns closed\" gives exactly-once discard with no extra locking.\n- `BufferCapacity`: `given default: BufferCapacity = 16`; `BufferCapacity.newChannel[T]`; `summon[BufferCapacity].toInt` (import `ox.channels.toInt`). Tests can override with `given BufferCapacity = BufferCapacity(4)`.\n- `ox.util`: `t.tapException(f)` runs `f(e)` for any Throwable and rethrows `e` (exceptions from `f` added as suppressed). `ox.pipe`, `ox.discard`, `ox.repeatWhile` already imported in the file.\n- `FlowOps.take(n)` stops upstream by throwing a private `ControlThrowable` (`abortTake`) out of `emit`, caught in `take` itself. So \"consumer stops early\" includes `take`, and the cleanup path must run for every Throwable, not just NonFatal. `tapException`/`catch case e: Throwable` already does this.\n- `unsupervised { ... }` + `forkUnsupervised { ... }` (import `ox.unsupervised`, `ox.forkUnsupervised`): scope end interrupts still-running forks once. `onNext`/`onError`/`onComplete` run on the publisher's threads (or on the `p.subscribe` fork for synchronous publishers), never on the consumer thread.\n- Test helpers: `ox.forkCancellable { ... }` inside `supervised` returns a `CancellableFork` with `cancel(): Either[Throwable, T]` (interrupts once, joins). Interruption inside the flow surfaces as `InterruptedException` thrown from a blocking call (e.g. `Semaphore.acquire`, `channel.receiveOrClosed`).\n\n## Design decisions (apply as-is)\n\n- API: add an overload `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg method delegate with `_ => ()`. Do not add a default parameter and do not change the existing signature: `core` runs MiMa (`enableMimaSettings` in build.sbt) and only additive changes are safe. Same shape in `FlowReactiveStreams`.\n- Subscriber: `onNext(t)` = `channel.sendOrClosed(t)` and if the result is `ChannelClosed` call `onDiscard(t)`. `onError(e)` = store `e` in an `AtomicReference[Throwable]`, then `channel.doneOrClosed().discard`. `onComplete()` = `channel.doneOrClosed().discard`. `onSubscribe(s)` = set `subscriptionRef`, then if a `cancelled: AtomicBoolean` is already set call `s.cancel()` else `s.request(capacity)` (both sides use volatile atomics, so at least one side sees the other; a double `cancel()` is allowed by the spec).\n- Consumer loop: on `ChannelClosed.Done`, read the stored error; if non-null throw it, else finish. Keep the `ChannelClosed.Error` branch for safety.\n- Cleanup, run on any Throwable escaping the loop (the stored publisher error included; cancel after onError is a spec-allowed no-op): set `cancelled`, cancel the subscription if `subscriptionRef` is non-null, `channel.doneOrClosed().discard`, then drain: loop `receiveOrClosed()` until `Done`, calling `onDiscard` for each element. The drain is bounded (channel is done, at most capacity elements plus in-flight sends) but blocking; if `InterruptedException` is thrown mid-drain, catch it, keep draining, and call `Thread.currentThread().interrupt()` once the drain finishes, so no element is lost on a second interrupt. Then rethrow the original exception. An element that was passed to `emit` and whose `emit` threw counts as emitted (downstream owns it): do not discard it.\n- Scaladoc on the new overload: state the exactly-once invariant, that `onDiscard` may be called from publisher threads, that it must not throw, and that on publisher error buffered elements are emitted before the error is rethrown (this is a deliberate behaviour change for the one-arg variant too; mention it in its doc).\n\n## Test publisher helper (write it in the core test, package `ox.flow.reactive`)\n\nA `ManualPublisher[T] extends java.util.concurrent.Flow.Publisher[T]` with: `@volatile var subscriber: Subscriber[? >: T]`, `requested: AtomicLong`, `cancelled: AtomicBoolean`, a `subscribed: CountDownLatch(1)`; `subscribe(s)` stores `s`, calls `s.onSubscribe(new Subscription { request(n) => requested.addAndGet(n); cancel() => cancelled.set(true) })`, counts down the latch. Helper methods `next(t)`, `error(e)`, `complete()` forward to the subscriber from the test thread. For the exactly-once stress test, a second variant that runs its own thread emitting `1..n` while `sent < requested` and not cancelled. Run the flow under test in `supervised { forkCancellable { ... } }` or `fork`, await `subscribed`, then drive the publisher. Collect emitted elements with `.tap(emitted += _)` or `runForeach`, discarded with `onDiscard = discarded += _` (use thread-safe collections such as `ConcurrentLinkedQueue` or `java.util.Collections.synchronizedList`, since `onDiscard` runs on other threads).\n\n## Conventions / build\n\n- Scala 3 indentation syntax, `end X` markers on long definitions, `.discard` on ignored results (compiler warns otherwise), scalafmt configured in `.scalafmt.conf` (140-column lines); run `sbt core/scalafmt flowReactiveStreams/scalafmt` after editing.\n- Comments: short, explain non-obvious reasoning only; never restate code. Scaladoc describes usage and invariants, not implementation.\n- Tests: scalatest 3.2.20; the reactive tests use `AnyFunSuite with Matchers` with `test(\"...\")` blocks and Scala 3 colon syntax. `core` runs tests forked (`Test / fork := true`).\n- Commands (sbt 2.0.10): `sbt \"core/testOnly ox.flow.reactive.*\"`, `sbt flowReactiveStreams/test`, `sbt core/mimaReportBinaryIssues` (only meaningful when a previous stable version resolves), `sbt compileDocumentation` for mdoc. Never commit; the user commits."}},{"id":[{"name":"Task: Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Exactly-once stress test: every element is emitted or discarded, never both or neither","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","occurrence":0}],"resultJson":{"findings":[{"id":"R1.I4.1","title":"sbt Test/compile fails with JGit NoWorkTreeException","reason":{"type":"Declined","text":"This is a problem with the environment, not the code. git sees the worktree as non-bare (`git rev-parse --is-bare-repository` prints false; the git dir is /home/adamw/ox/.git/worktrees/zippy-astrodon-94aa2fb6). The exception comes from the sbt-git plugin, which uses JGit by default and reads this linked worktree as bare while the build loads. Nothing in project/ or the build configures this; `zz-tmp-git.sbt` was my temporary workaround, not the cause. To check the code, I recreated it with `useReadableConsoleGit`, which makes sbt-git use command-line git: `sbt core/Test/compile flowReactiveStreams/Test/compile` succeeded (warnings only, in older test files, no errors) and flowReactiveStreams tests passed. I then deleted the file again. Changing the shared build's git plugin setup is outside the scope of this change."}}]}},{"id":[{"name":"Task: Document the onDiscard callback in doc/streaming/flows.md","occurrence":0}],"resultJson":{"findings":[{"id":"R1.I4.1","title":"`sbt Test/compile` fails with no diagnostic","reason":{"type":"Declined","text":"Environmental. Full output shows sbt-git's JGit throws NoWorkTreeException (\"Bare Repository has neither a working tree, nor an index\") while loading the build, because this checkout is a git worktree; no source error is involved. With a temporary `ThisBuild / git.gitUncommittedChanges := false` override, `sbt Test/compile` succeeds across all modules (only pre-existing unrelated warnings). The temporary file was removed; no codebase change is needed."}}]}},{"id":[{"name":"Final review","occurrence":0}],"resultJson":{"findings":[{"id":"S1","title":"sbt Test/compile fails with JGit NoWorkTreeException","reason":{"type":"Declined","text":"This is a problem with the environment, not the code. git sees the worktree as non-bare (`git rev-parse --is-bare-repository` prints false; the git dir is /home/adamw/ox/.git/worktrees/zippy-astrodon-94aa2fb6). The exception comes from the sbt-git plugin, which uses JGit by default and reads this linked worktree as bare while the build loads. Nothing in project/ or the build configures this; `zz-tmp-git.sbt` was my temporary workaround, not the cause. To check the code, I recreated it with `useReadableConsoleGit`, which makes sbt-git use command-line git: `sbt core/Test/compile flowReactiveStreams/Test/compile` succeeded (warnings only, in older test files, no errors) and flowReactiveStreams tests passed. I then deleted the file again. Changing the shared build's git plugin setup is outside the scope of this change."}},{"id":"S2","title":"`sbt Test/compile` fails with no diagnostic","reason":{"type":"Declined","text":"Environmental. Full output shows sbt-git's JGit throws NoWorkTreeException (\"Bare Repository has neither a working tree, nor an index\") while loading the build, because this checkout is a git worktree; no source error is involved. With a temporary `ThisBuild / git.gitUncommittedChanges := false` override, `sbt Test/compile` succeeds across all modules (only pre-existing unrelated warnings). The temporary file was removed; no codebase change is needed."}},{"id":"R1.I2.1","title":"No test covers the drain's deferred-interrupt path in discardBuffered","reason":{"type":"Declined","text":"A deterministic test can't be written. In jox 1.1.2, `receiveOrClosed` only throws `InterruptedException` when it parks (`Continuation.await`). After `doneOrClosed()` has finished on the draining thread, a receive never parks: buffered or in-flight sends are returned or retried, and later cells are already CLOSED. The catch is reached only in a narrow race, when a concurrent `onComplete`/`onError` is still closing cells. I tried the suggested test (interrupt plus throw in the consumer, then assert the discards and the interrupt flag). It passed both with the catch removed and with the re-interrupt removed, so it did not cover the branch, and I reverted it. The suggested capacity-1 variant also doesn't help, because a blocked sender's Continuation is resumed without parking."},"location":{"file":"core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala","line":141}},{"id":"R1.I5.1","title":"sbt Test/compile exited 1 with no compile error in the output","reason":{"type":"Declined","text":"Environmental. sbt-git's JGit fails in this git worktree with `NoWorkTreeException: Bare Repository has neither a working tree, nor an index` while evaluating `gitUncommittedChanges`, so the build fails before compiling. It is not a code problem. With a temporary `zz-tmp-git.sbt` that sets `ThisBuild / git.gitUncommittedChanges := false`, Test/compile and the core reactive and flowReactiveStreams tests all pass. I deleted the temporary file afterwards."}},{"id":"R2.I3.1","title":"`sbt Test/compile` exits 1 with no compiler diagnostics","reason":{"type":"Declined","text":"Environmental, not a code problem. In this git worktree, sbt-git's JGit throws `NoWorkTreeException: Bare Repository has neither a working tree, nor an index` while evaluating `gitUncommittedChanges` during build load. The build fails before any compilation, and the full output shows that stack trace above the timing line. With a temporary `zz-tmp-git.sbt` setting `ThisBuild / git.gitUncommittedChanges := false`, `Test/compile` succeeds. With the same file, `core/testOnly ox.flow.reactive.*` (44 passed, 7 ignored) and `flowReactiveStreams/test` (2 passed) also pass. I deleted the file afterwards, so it is not left in the tree."}}]}},{"id":[{"name":"Push branch","occurrence":0}],"resultJson":{"type":"Pushed"}},{"id":[{"name":"Generate PR title and description","occurrence":0}],"resultJson":{"title":"Add onDiscard callback to Flow.fromPublisher","body":"`Flow.fromPublisher` buffers elements in a channel. Some of them could be received from the publisher but never emitted, and nothing released them. That leaks elements which hold resources, such as Netty's ref-counted ByteBufs. Tapir currently tracks these elements itself (softwaremill/tapir#5573).\n\n- New overload `fromPublisher(p, onDiscard: T => Unit)`, in core and in `FlowReactiveStreams`. Each element received from the publisher is either emitted, or passed to `onDiscard` exactly once. Elements are discarded when the flow fails, is interrupted, or stops early (e.g. `take`), and also when they arrive after the subscription is cancelled. `onDiscard` may run on publisher threads and must not throw.\n- Behaviour change, also for the one-argument variant: when the publisher fails, elements already buffered are now emitted first, and then the error is rethrown. Before, they were dropped.\n- The existing one-argument overloads are unchanged and pass a no-op callback. The new method is an overload, not a default parameter, so binary compatibility is kept.\n- Added tests for publisher error, consumer exception, interruption, `take`, and elements arriving after cancellation. Also added a stress test that checks each element is either emitted or discarded, never both or neither. Updated `doc/streaming/flows.md`.\n\nCloses #528"}}],"stageStarts":[]} \ No newline at end of file +{"header":{"startingBranch":"zippy-astrodon-94aa2fb6","branch":"zippy-astrodon-94aa2fb6","branchMode":"Reused","userPrompt":"Implement GitHub issue softwaremill/ox#528: \"Flow.fromPublisher: allow releasing elements which are buffered but not emitted\".\n\nProblem: `Flow.fromPublisher` (core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala) buffers elements in a channel. Some of them never reach the consumer:\n- when the publisher fails, `channel.error` drops the buffered elements\n- when the consumer stops early (an exception, an interruption), the buffered elements are never received\n- elements can still arrive (onNext) after the subscription is cancelled\n\nIf elements hold resources (e.g. Netty's reference-counted ByteBufs), nobody releases them, so they leak.\n\nGoal: add an optional callback, e.g. `fromPublisher(p, onDiscard: T => Unit)`, called exactly once for each element received from the publisher but not emitted downstream. Existing call sites without the callback must keep compiling and behaving the same.\n\nSuggested implementation: on publisher error, store the error and call `channel.done()` (so buffered elements are kept and emitted before the error is rethrown). When the flow ends early, cancel the subscription, close the channel, and drain it, calling the callback for each drained element. In `onNext`, call the callback if the channel is already closed. Make sure there are no races where an element is neither emitted nor discarded.\n\nAlso update the variant in flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala (org.reactivestreams Publisher) to accept the same optional callback, and update any docs (doc/ directory) describing fromPublisher.\n\nAcceptance criteria:\n- tests covering: publisher error with buffered elements (elements still emitted, then error); consumer stopping early (exception / interruption) with buffered elements -> callback called for each; elements arriving after cancellation -> callback called; every element is either emitted or discarded, never both or neither.\n- `sbt compile test` passes for affected modules (core, flow-reactive-streams).\n- Code follows existing style in the repo; concise doc comments.\n\nContext: tapir currently tracks unconsumed elements itself using a lock: https://github.com/softwaremill/tapir/pull/5573","flow":{"type":"Catalog","name":"implement.sc"},"startingCommit":"664f01730cda34fd88e8c3db3deb600934eb1d1e"},"entries":[{"id":[{"name":"Plan","occurrence":0}],"resultJson":{"id":"ox-528-from-publisher-on-discard","description":"`Flow.fromPublisher` buffers elements received from a reactive `Publisher` in a channel. Today, elements that are buffered but never emitted are silently lost: `channel.error` drops the buffer on publisher error, a consumer that stops early (exception, interruption, `take`) never receives what is buffered, and `onNext` calls that arrive after the subscription is cancelled are dropped by `channel.send` throwing. When elements own resources (Netty reference-counted ByteBufs in tapir), this leaks them. Tapir currently works around it with its own lock-based tracking (softwaremill/tapir#5573).\n\nThis epic adds an optional `onDiscard: T => Unit` callback to `Flow.fromPublisher` (core, JDK `Flow.Publisher`) and `FlowReactiveStreams.fromPublisher` (flow-reactive-streams, `org.reactivestreams.Publisher`), with the invariant that every element received from the publisher is either emitted downstream or passed to `onDiscard`, exactly once, never both or neither. Publisher errors no longer drop buffered elements: the error is stored, the channel is completed with `done()`, buffered elements are emitted, and then the error is rethrown. When the flow ends early, the subscription is cancelled, the channel is closed and drained through `onDiscard`, and any late `onNext` goes to `onDiscard` as well. Existing single-argument call sites keep compiling (the new parameter is an overload, which also keeps MiMa happy for `core`). Docs in `doc/streaming/flows.md` describe the new callback.","tasks":[{"title":"Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","description":"In core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala add `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg `fromPublisher` delegate to it with `_ => ()`. Implement per the brief's design decisions: `onNext` uses `sendOrClosed` and discards on closed; `onError` stores the throwable in an AtomicReference and calls `doneOrClosed`; `onComplete` calls `doneOrClosed`; `onSubscribe` cancels immediately if a `cancelled` flag is already set, else requests `capacity`; the consumer loop rethrows the stored error when it sees `Done`; on any Throwable escaping the loop: set `cancelled`, cancel the subscription if present, `doneOrClosed` the channel, drain it calling `onDiscard` per element (keep draining across an InterruptedException and re-interrupt the thread afterwards), rethrow. Preserve the existing batched demand logic. Update the scaladoc of both overloads (exactly-once invariant, callback runs on publisher threads and must not throw, buffered elements are now emitted before a publisher error is rethrown).\n\nTests, in a new core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardTest.scala (AnyFunSuite with Matchers), using a hand-written `ManualPublisher[T]` helper as described in the brief (keep the helper in this file or a sibling file in the same package so Task 2 can reuse it):\n1. Publisher error with buffered elements: push 1,2,3 then `error(boom)` before/while the consumer runs; `runToList` throws boom and all of 1,2,3 were emitted; `onDiscard` never called.\n2. Publisher error with nothing buffered still throws.\n3. Consumer exception: `.map(x => if x == 2 then throw boom else x)`, push 1..5; `runToList` throws boom; discarded == 3,4,5 (order-independent compare), subscription cancelled.\n4. Consumer stops via `take(2)`: result List(1,2), discarded == 3,4,5, cancelled.\n5. Consumer interrupted: run `runForeach(x => { received.release(); gate.acquire() })` in `forkCancellable`, push 1..5, wait for `received`, `cancel()`; discarded == 2,3,4,5, cancelled.\n6. Late onNext after cancellation: after any of the above, call `next(6)` on the publisher; `onDiscard(6)` is called synchronously and nothing throws.\n7. Existing one-arg overload: compiles unchanged and `FlowPublisherPekkoTest` still passes.\nRun `sbt \"core/testOnly ox.flow.reactive.*\"` and `sbt core/scalafmt`; all green."},{"title":"Exactly-once stress test: every element is emitted or discarded, never both or neither","description":"Add a test (same file as Task 1 or a sibling in core/src/test/scala/ox/flow/reactive/) that uses a demand-respecting publisher running on its own thread: it emits 1..N (N around 1000) only while `sent < requested` and not cancelled, calling `onNext` from that thread, then `onComplete`. Compose `Flow.fromPublisher(p, discarded.add)` with a consumer that stops at a random point (e.g. `take(k)` for random k, and separately a `map` that throws at a random element, and separately a `forkCancellable` cancelled after a random short delay). Collect emitted elements via `.tap(emitted.add)` into thread-safe collections. After the flow finishes and the publisher thread is joined, assert: `emitted` and `discarded` are disjoint, their union equals exactly the set of elements the publisher actually sent, and no element appears twice. Repeat each scenario in a loop (e.g. 20 iterations) with a small `given BufferCapacity = BufferCapacity(4)` to make buffer-full races likely. The publisher's late `onNext` calls (after cancellation) must land in `discarded`. If this exposes a race in Task 1's implementation, fix it in FlowCompanionReactiveOps.scala as part of this task. Run `sbt \"core/testOnly ox.flow.reactive.*\"`."},{"title":"Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","description":"In flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala add `def fromPublisher[T](p: org.reactivestreams.Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p), onDiscard)`, keep the one-arg overload, and update its scaladoc to point at the core overload for the callback semantics. Create flow-reactive-streams/src/test/scala/ox/flow/reactive/FlowReactiveStreamsFromPublisherTest.scala (AnyFunSuite with Matchers) with a small hand-written `org.reactivestreams.Publisher[Int]` (same shape as the core ManualPublisher, but implementing the reactive-streams interfaces; it cannot be shared because the module does not depend on core's test classpath) and two tests: elements flow through the one-arg variant; with `.take(1)` and the new overload, remaining pushed elements are passed to `onDiscard` and the subscription is cancelled. Run `sbt flowReactiveStreams/test` and `sbt flowReactiveStreams/scalafmt`."},{"title":"Document the onDiscard callback in doc/streaming/flows.md","description":"In doc/streaming/flows.md, section \"### Publisher -> Flow\" (around lines 211-219), add a short paragraph explaining: elements received from the publisher but not emitted downstream (buffered when the publisher fails, when the consumer stops early via exception, interruption or `take`, or arriving after cancellation) can be released by passing a second argument `onDiscard: T => Unit` to `Flow.fromPublisher` / `FlowReactiveStreams.fromPublisher`; it is called exactly once per such element, possibly from the publisher's thread, and should not throw; typical use is releasing reference-counted buffers. Also state that on publisher error, elements already received are emitted before the error is propagated. Keep the prose as short as the surrounding text. If you add a code example, use a ```` ```scala mdoc:compile-only ```` block and verify with `sbt compileDocumentation`; otherwise no build step is needed. Do not touch generated-doc/out."}],"brief":"## Modules and files\n\n- `core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala` — the only implementation file in core. Trait `FlowCompanionReactiveOps` (mixed into `Flow.type`) defines `def fromPublisher[T](p: java.util.concurrent.Flow.Publisher[T])(using BufferCapacity): Flow[T]`. Read the whole file (80 lines) before changing anything. Current structure: `usingEmitInline: emit => unsupervised { channel = BufferCapacity.newChannel[T]; forkUnsupervised { p.subscribe(Subscriber: onSubscribe -> subscriptionRef.set(s); s.request(capacity); onNext -> channel.send; onError -> channel.error; onComplete -> channel.done) }; repeatWhile { channel.receiveOrClosed() match Done -> false / Error -> throw / t -> emit(t); batched subscription.request } }.tapException(cancel subscription)`. The demand logic (`demandThreshold = ceil(capacity/2)`, `toDemand` counter, lazily cached `subscription` read from `subscriptionRef` after the first element) must be preserved.\n- `flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala` — `object FlowReactiveStreams { def fromPublisher[T](p: org.reactivestreams.Publisher[T])(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p)) }`. Module has no `src/test` dir yet; scalatest is already a test dependency (build.sbt `flowReactiveStreams`), it `dependsOn(core)` (compile only, not test->test).\n- `core/src/test/scala/ox/flow/reactive/` — existing reactive tests: `FlowPublisherPekkoTest.scala` (AnyFunSuite + Matchers, uses Pekko `Source...runWith(Sink.asPublisher)` and `FlowAdapters.toFlowPublisher`) and `FlowPublisherTckTest.scala`. Put new fromPublisher tests here. No test anywhere currently uses a hand-written Publisher; write one (see below).\n- `doc/streaming/flows.md` lines 211-219, section \"### Publisher -> Flow\" — the only user doc mentioning `fromPublisher`. `generated-doc/out/**` is mdoc output regenerated at release time by `updateDocs`; do NOT edit it. Code blocks in docs use ```` ```scala mdoc:compile-only ```` and are compiled by `sbt compileDocumentation` (slow; only needed if you add a code block). Doc dependency snippet convention: `// sbt dependency: \"com.softwaremill.ox\" %% \"flow-reactive-streams\" % \"@VERSION@\"`.\n\n## Key APIs (ox.channels, core/src/main/scala/ox/channels/Channel.scala)\n\n- `Sink.sendOrClosed(t): Unit | ChannelClosed` (non-throwing), `Sink.send(t)` throws `ChannelClosedException` when closed.\n- `Sink.doneOrClosed(): Unit | ChannelClosed` — \"buffered values will be delivered; send operations in progress complete normally when a receiver arrives\"; returns `ChannelClosed` instead of throwing if already closed. `Sink.errorOrClosed(reason)` — drops buffered values (that is the bug). Use the `*OrClosed` variants everywhere in the subscriber: a misbehaving publisher may call onError/onComplete twice.\n- `Source.receiveOrClosed(): T | ChannelClosed` (blocking); match `case ChannelClosed.Done => ...; case e: ChannelClosed.Error => throw e.toThrowable; case t: T @unchecked => ...`. `Source.tryReceiveOrClosed()` is non-blocking but may spuriously return None, so it is not suitable for a drain loop.\n- `Source.foreachOrError(f)` / `drainOrError()` from `SourceDrainOps.scala` loop until Done; usable for the drain, but write the drain inline so you can handle `InterruptedException` (see below).\n- jox semantics (com.softwaremill.jox channels 1.1.2, verified from source): after `done()`, buffered elements stay receivable; a receiver observing Done has already seen every element whose `send` started before `done()` (an in-progress sender is waited for); a `send` racing with `done()` either lands in the buffer or returns closed. Hence \"drain until Done, then `sendOrClosed` returns closed\" gives exactly-once discard with no extra locking.\n- `BufferCapacity`: `given default: BufferCapacity = 16`; `BufferCapacity.newChannel[T]`; `summon[BufferCapacity].toInt` (import `ox.channels.toInt`). Tests can override with `given BufferCapacity = BufferCapacity(4)`.\n- `ox.util`: `t.tapException(f)` runs `f(e)` for any Throwable and rethrows `e` (exceptions from `f` added as suppressed). `ox.pipe`, `ox.discard`, `ox.repeatWhile` already imported in the file.\n- `FlowOps.take(n)` stops upstream by throwing a private `ControlThrowable` (`abortTake`) out of `emit`, caught in `take` itself. So \"consumer stops early\" includes `take`, and the cleanup path must run for every Throwable, not just NonFatal. `tapException`/`catch case e: Throwable` already does this.\n- `unsupervised { ... }` + `forkUnsupervised { ... }` (import `ox.unsupervised`, `ox.forkUnsupervised`): scope end interrupts still-running forks once. `onNext`/`onError`/`onComplete` run on the publisher's threads (or on the `p.subscribe` fork for synchronous publishers), never on the consumer thread.\n- Test helpers: `ox.forkCancellable { ... }` inside `supervised` returns a `CancellableFork` with `cancel(): Either[Throwable, T]` (interrupts once, joins). Interruption inside the flow surfaces as `InterruptedException` thrown from a blocking call (e.g. `Semaphore.acquire`, `channel.receiveOrClosed`).\n\n## Design decisions (apply as-is)\n\n- API: add an overload `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg method delegate with `_ => ()`. Do not add a default parameter and do not change the existing signature: `core` runs MiMa (`enableMimaSettings` in build.sbt) and only additive changes are safe. Same shape in `FlowReactiveStreams`.\n- Subscriber: `onNext(t)` = `channel.sendOrClosed(t)` and if the result is `ChannelClosed` call `onDiscard(t)`. `onError(e)` = store `e` in an `AtomicReference[Throwable]`, then `channel.doneOrClosed().discard`. `onComplete()` = `channel.doneOrClosed().discard`. `onSubscribe(s)` = set `subscriptionRef`, then if a `cancelled: AtomicBoolean` is already set call `s.cancel()` else `s.request(capacity)` (both sides use volatile atomics, so at least one side sees the other; a double `cancel()` is allowed by the spec).\n- Consumer loop: on `ChannelClosed.Done`, read the stored error; if non-null throw it, else finish. Keep the `ChannelClosed.Error` branch for safety.\n- Cleanup, run on any Throwable escaping the loop (the stored publisher error included; cancel after onError is a spec-allowed no-op): set `cancelled`, cancel the subscription if `subscriptionRef` is non-null, `channel.doneOrClosed().discard`, then drain: loop `receiveOrClosed()` until `Done`, calling `onDiscard` for each element. The drain is bounded (channel is done, at most capacity elements plus in-flight sends) but blocking; if `InterruptedException` is thrown mid-drain, catch it, keep draining, and call `Thread.currentThread().interrupt()` once the drain finishes, so no element is lost on a second interrupt. Then rethrow the original exception. An element that was passed to `emit` and whose `emit` threw counts as emitted (downstream owns it): do not discard it.\n- Scaladoc on the new overload: state the exactly-once invariant, that `onDiscard` may be called from publisher threads, that it must not throw, and that on publisher error buffered elements are emitted before the error is rethrown (this is a deliberate behaviour change for the one-arg variant too; mention it in its doc).\n\n## Test publisher helper (write it in the core test, package `ox.flow.reactive`)\n\nA `ManualPublisher[T] extends java.util.concurrent.Flow.Publisher[T]` with: `@volatile var subscriber: Subscriber[? >: T]`, `requested: AtomicLong`, `cancelled: AtomicBoolean`, a `subscribed: CountDownLatch(1)`; `subscribe(s)` stores `s`, calls `s.onSubscribe(new Subscription { request(n) => requested.addAndGet(n); cancel() => cancelled.set(true) })`, counts down the latch. Helper methods `next(t)`, `error(e)`, `complete()` forward to the subscriber from the test thread. For the exactly-once stress test, a second variant that runs its own thread emitting `1..n` while `sent < requested` and not cancelled. Run the flow under test in `supervised { forkCancellable { ... } }` or `fork`, await `subscribed`, then drive the publisher. Collect emitted elements with `.tap(emitted += _)` or `runForeach`, discarded with `onDiscard = discarded += _` (use thread-safe collections such as `ConcurrentLinkedQueue` or `java.util.Collections.synchronizedList`, since `onDiscard` runs on other threads).\n\n## Conventions / build\n\n- Scala 3 indentation syntax, `end X` markers on long definitions, `.discard` on ignored results (compiler warns otherwise), scalafmt configured in `.scalafmt.conf` (140-column lines); run `sbt core/scalafmt flowReactiveStreams/scalafmt` after editing.\n- Comments: short, explain non-obvious reasoning only; never restate code. Scaladoc describes usage and invariants, not implementation.\n- Tests: scalatest 3.2.20; the reactive tests use `AnyFunSuite with Matchers` with `test(\"...\")` blocks and Scala 3 colon syntax. `core` runs tests forked (`Test / fork := true`).\n- Commands (sbt 2.0.10): `sbt \"core/testOnly ox.flow.reactive.*\"`, `sbt flowReactiveStreams/test`, `sbt core/mimaReportBinaryIssues` (only meaningful when a previous stable version resolves), `sbt compileDocumentation` for mdoc. Never commit; the user commits."}},{"id":[{"name":"Task: Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Exactly-once stress test: every element is emitted or discarded, never both or neither","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","occurrence":0}],"resultJson":{"findings":[{"id":"R1.I4.1","title":"sbt Test/compile fails with JGit NoWorkTreeException","reason":{"type":"Declined","text":"This is a problem with the environment, not the code. git sees the worktree as non-bare (`git rev-parse --is-bare-repository` prints false; the git dir is /home/adamw/ox/.git/worktrees/zippy-astrodon-94aa2fb6). The exception comes from the sbt-git plugin, which uses JGit by default and reads this linked worktree as bare while the build loads. Nothing in project/ or the build configures this; `zz-tmp-git.sbt` was my temporary workaround, not the cause. To check the code, I recreated it with `useReadableConsoleGit`, which makes sbt-git use command-line git: `sbt core/Test/compile flowReactiveStreams/Test/compile` succeeded (warnings only, in older test files, no errors) and flowReactiveStreams tests passed. I then deleted the file again. Changing the shared build's git plugin setup is outside the scope of this change."}}]}},{"id":[{"name":"Task: Document the onDiscard callback in doc/streaming/flows.md","occurrence":0}],"resultJson":{"findings":[{"id":"R1.I4.1","title":"`sbt Test/compile` fails with no diagnostic","reason":{"type":"Declined","text":"Environmental. Full output shows sbt-git's JGit throws NoWorkTreeException (\"Bare Repository has neither a working tree, nor an index\") while loading the build, because this checkout is a git worktree; no source error is involved. With a temporary `ThisBuild / git.gitUncommittedChanges := false` override, `sbt Test/compile` succeeds across all modules (only pre-existing unrelated warnings). The temporary file was removed; no codebase change is needed."}}]}},{"id":[{"name":"Final review","occurrence":0}],"resultJson":{"findings":[{"id":"S1","title":"sbt Test/compile fails with JGit NoWorkTreeException","reason":{"type":"Declined","text":"This is a problem with the environment, not the code. git sees the worktree as non-bare (`git rev-parse --is-bare-repository` prints false; the git dir is /home/adamw/ox/.git/worktrees/zippy-astrodon-94aa2fb6). The exception comes from the sbt-git plugin, which uses JGit by default and reads this linked worktree as bare while the build loads. Nothing in project/ or the build configures this; `zz-tmp-git.sbt` was my temporary workaround, not the cause. To check the code, I recreated it with `useReadableConsoleGit`, which makes sbt-git use command-line git: `sbt core/Test/compile flowReactiveStreams/Test/compile` succeeded (warnings only, in older test files, no errors) and flowReactiveStreams tests passed. I then deleted the file again. Changing the shared build's git plugin setup is outside the scope of this change."}},{"id":"S2","title":"`sbt Test/compile` fails with no diagnostic","reason":{"type":"Declined","text":"Environmental. Full output shows sbt-git's JGit throws NoWorkTreeException (\"Bare Repository has neither a working tree, nor an index\") while loading the build, because this checkout is a git worktree; no source error is involved. With a temporary `ThisBuild / git.gitUncommittedChanges := false` override, `sbt Test/compile` succeeds across all modules (only pre-existing unrelated warnings). The temporary file was removed; no codebase change is needed."}},{"id":"R1.I2.1","title":"No test covers the drain's deferred-interrupt path in discardBuffered","reason":{"type":"Declined","text":"A deterministic test can't be written. In jox 1.1.2, `receiveOrClosed` only throws `InterruptedException` when it parks (`Continuation.await`). After `doneOrClosed()` has finished on the draining thread, a receive never parks: buffered or in-flight sends are returned or retried, and later cells are already CLOSED. The catch is reached only in a narrow race, when a concurrent `onComplete`/`onError` is still closing cells. I tried the suggested test (interrupt plus throw in the consumer, then assert the discards and the interrupt flag). It passed both with the catch removed and with the re-interrupt removed, so it did not cover the branch, and I reverted it. The suggested capacity-1 variant also doesn't help, because a blocked sender's Continuation is resumed without parking."},"location":{"file":"core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala","line":141}},{"id":"R1.I5.1","title":"sbt Test/compile exited 1 with no compile error in the output","reason":{"type":"Declined","text":"Environmental. sbt-git's JGit fails in this git worktree with `NoWorkTreeException: Bare Repository has neither a working tree, nor an index` while evaluating `gitUncommittedChanges`, so the build fails before compiling. It is not a code problem. With a temporary `zz-tmp-git.sbt` that sets `ThisBuild / git.gitUncommittedChanges := false`, Test/compile and the core reactive and flowReactiveStreams tests all pass. I deleted the temporary file afterwards."}},{"id":"R2.I3.1","title":"`sbt Test/compile` exits 1 with no compiler diagnostics","reason":{"type":"Declined","text":"Environmental, not a code problem. In this git worktree, sbt-git's JGit throws `NoWorkTreeException: Bare Repository has neither a working tree, nor an index` while evaluating `gitUncommittedChanges` during build load. The build fails before any compilation, and the full output shows that stack trace above the timing line. With a temporary `zz-tmp-git.sbt` setting `ThisBuild / git.gitUncommittedChanges := false`, `Test/compile` succeeds. With the same file, `core/testOnly ox.flow.reactive.*` (44 passed, 7 ignored) and `flowReactiveStreams/test` (2 passed) also pass. I deleted the file afterwards, so it is not left in the tree."}}]}},{"id":[{"name":"Push branch","occurrence":0}],"resultJson":{"type":"Pushed"}},{"id":[{"name":"Generate PR title and description","occurrence":0}],"resultJson":{"title":"Add onDiscard callback to Flow.fromPublisher","body":"`Flow.fromPublisher` buffers elements in a channel. Some of them could be received from the publisher but never emitted, and nothing released them. That leaks elements which hold resources, such as Netty's ref-counted ByteBufs. Tapir currently tracks these elements itself (softwaremill/tapir#5573).\n\n- New overload `fromPublisher(p, onDiscard: T => Unit)`, in core and in `FlowReactiveStreams`. Each element received from the publisher is either emitted, or passed to `onDiscard` exactly once. Elements are discarded when the flow fails, is interrupted, or stops early (e.g. `take`), and also when they arrive after the subscription is cancelled. `onDiscard` may run on publisher threads and must not throw.\n- Behaviour change, also for the one-argument variant: when the publisher fails, elements already buffered are now emitted first, and then the error is rethrown. Before, they were dropped.\n- The existing one-argument overloads are unchanged and pass a no-op callback. The new method is an overload, not a default parameter, so binary compatibility is kept.\n- Added tests for publisher error, consumer exception, interruption, `take`, and elements arriving after cancellation. Also added a stress test that checks each element is either emitted or discarded, never both or neither. Updated `doc/streaming/flows.md`.\n\nCloses #528"}},{"id":[{"name":"Open PR","occurrence":0}],"resultJson":{"type":"Opened","pr":"https://github.com/softwaremill/ox/pull/529"}}],"published":{"reference":"https://github.com/softwaremill/ox/pull/529"},"stageStarts":[]} \ No newline at end of file From a3467c6a14a56aefd5c72142abd33113158e6708 Mon Sep 17 00:00:00 2001 From: Adam Warski Date: Fri, 9 Oct 2026 15:31:16 +0000 Subject: [PATCH 11/11] orca: remove progress log --- .orca/runs/0a3279c8e3d7.progress.json | 1 - 1 file changed, 1 deletion(-) delete mode 100644 .orca/runs/0a3279c8e3d7.progress.json diff --git a/.orca/runs/0a3279c8e3d7.progress.json b/.orca/runs/0a3279c8e3d7.progress.json deleted file mode 100644 index 2f3a8d53..00000000 --- a/.orca/runs/0a3279c8e3d7.progress.json +++ /dev/null @@ -1 +0,0 @@ -{"header":{"startingBranch":"zippy-astrodon-94aa2fb6","branch":"zippy-astrodon-94aa2fb6","branchMode":"Reused","userPrompt":"Implement GitHub issue softwaremill/ox#528: \"Flow.fromPublisher: allow releasing elements which are buffered but not emitted\".\n\nProblem: `Flow.fromPublisher` (core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala) buffers elements in a channel. Some of them never reach the consumer:\n- when the publisher fails, `channel.error` drops the buffered elements\n- when the consumer stops early (an exception, an interruption), the buffered elements are never received\n- elements can still arrive (onNext) after the subscription is cancelled\n\nIf elements hold resources (e.g. Netty's reference-counted ByteBufs), nobody releases them, so they leak.\n\nGoal: add an optional callback, e.g. `fromPublisher(p, onDiscard: T => Unit)`, called exactly once for each element received from the publisher but not emitted downstream. Existing call sites without the callback must keep compiling and behaving the same.\n\nSuggested implementation: on publisher error, store the error and call `channel.done()` (so buffered elements are kept and emitted before the error is rethrown). When the flow ends early, cancel the subscription, close the channel, and drain it, calling the callback for each drained element. In `onNext`, call the callback if the channel is already closed. Make sure there are no races where an element is neither emitted nor discarded.\n\nAlso update the variant in flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala (org.reactivestreams Publisher) to accept the same optional callback, and update any docs (doc/ directory) describing fromPublisher.\n\nAcceptance criteria:\n- tests covering: publisher error with buffered elements (elements still emitted, then error); consumer stopping early (exception / interruption) with buffered elements -> callback called for each; elements arriving after cancellation -> callback called; every element is either emitted or discarded, never both or neither.\n- `sbt compile test` passes for affected modules (core, flow-reactive-streams).\n- Code follows existing style in the repo; concise doc comments.\n\nContext: tapir currently tracks unconsumed elements itself using a lock: https://github.com/softwaremill/tapir/pull/5573","flow":{"type":"Catalog","name":"implement.sc"},"startingCommit":"664f01730cda34fd88e8c3db3deb600934eb1d1e"},"entries":[{"id":[{"name":"Plan","occurrence":0}],"resultJson":{"id":"ox-528-from-publisher-on-discard","description":"`Flow.fromPublisher` buffers elements received from a reactive `Publisher` in a channel. Today, elements that are buffered but never emitted are silently lost: `channel.error` drops the buffer on publisher error, a consumer that stops early (exception, interruption, `take`) never receives what is buffered, and `onNext` calls that arrive after the subscription is cancelled are dropped by `channel.send` throwing. When elements own resources (Netty reference-counted ByteBufs in tapir), this leaks them. Tapir currently works around it with its own lock-based tracking (softwaremill/tapir#5573).\n\nThis epic adds an optional `onDiscard: T => Unit` callback to `Flow.fromPublisher` (core, JDK `Flow.Publisher`) and `FlowReactiveStreams.fromPublisher` (flow-reactive-streams, `org.reactivestreams.Publisher`), with the invariant that every element received from the publisher is either emitted downstream or passed to `onDiscard`, exactly once, never both or neither. Publisher errors no longer drop buffered elements: the error is stored, the channel is completed with `done()`, buffered elements are emitted, and then the error is rethrown. When the flow ends early, the subscription is cancelled, the channel is closed and drained through `onDiscard`, and any late `onNext` goes to `onDiscard` as well. Existing single-argument call sites keep compiling (the new parameter is an overload, which also keeps MiMa happy for `core`). Docs in `doc/streaming/flows.md` describe the new callback.","tasks":[{"title":"Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","description":"In core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala add `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg `fromPublisher` delegate to it with `_ => ()`. Implement per the brief's design decisions: `onNext` uses `sendOrClosed` and discards on closed; `onError` stores the throwable in an AtomicReference and calls `doneOrClosed`; `onComplete` calls `doneOrClosed`; `onSubscribe` cancels immediately if a `cancelled` flag is already set, else requests `capacity`; the consumer loop rethrows the stored error when it sees `Done`; on any Throwable escaping the loop: set `cancelled`, cancel the subscription if present, `doneOrClosed` the channel, drain it calling `onDiscard` per element (keep draining across an InterruptedException and re-interrupt the thread afterwards), rethrow. Preserve the existing batched demand logic. Update the scaladoc of both overloads (exactly-once invariant, callback runs on publisher threads and must not throw, buffered elements are now emitted before a publisher error is rethrown).\n\nTests, in a new core/src/test/scala/ox/flow/reactive/FlowFromPublisherDiscardTest.scala (AnyFunSuite with Matchers), using a hand-written `ManualPublisher[T]` helper as described in the brief (keep the helper in this file or a sibling file in the same package so Task 2 can reuse it):\n1. Publisher error with buffered elements: push 1,2,3 then `error(boom)` before/while the consumer runs; `runToList` throws boom and all of 1,2,3 were emitted; `onDiscard` never called.\n2. Publisher error with nothing buffered still throws.\n3. Consumer exception: `.map(x => if x == 2 then throw boom else x)`, push 1..5; `runToList` throws boom; discarded == 3,4,5 (order-independent compare), subscription cancelled.\n4. Consumer stops via `take(2)`: result List(1,2), discarded == 3,4,5, cancelled.\n5. Consumer interrupted: run `runForeach(x => { received.release(); gate.acquire() })` in `forkCancellable`, push 1..5, wait for `received`, `cancel()`; discarded == 2,3,4,5, cancelled.\n6. Late onNext after cancellation: after any of the above, call `next(6)` on the publisher; `onDiscard(6)` is called synchronously and nothing throws.\n7. Existing one-arg overload: compiles unchanged and `FlowPublisherPekkoTest` still passes.\nRun `sbt \"core/testOnly ox.flow.reactive.*\"` and `sbt core/scalafmt`; all green."},{"title":"Exactly-once stress test: every element is emitted or discarded, never both or neither","description":"Add a test (same file as Task 1 or a sibling in core/src/test/scala/ox/flow/reactive/) that uses a demand-respecting publisher running on its own thread: it emits 1..N (N around 1000) only while `sent < requested` and not cancelled, calling `onNext` from that thread, then `onComplete`. Compose `Flow.fromPublisher(p, discarded.add)` with a consumer that stops at a random point (e.g. `take(k)` for random k, and separately a `map` that throws at a random element, and separately a `forkCancellable` cancelled after a random short delay). Collect emitted elements via `.tap(emitted.add)` into thread-safe collections. After the flow finishes and the publisher thread is joined, assert: `emitted` and `discarded` are disjoint, their union equals exactly the set of elements the publisher actually sent, and no element appears twice. Repeat each scenario in a loop (e.g. 20 iterations) with a small `given BufferCapacity = BufferCapacity(4)` to make buffer-full races likely. The publisher's late `onNext` calls (after cancellation) must land in `discarded`. If this exposes a race in Task 1's implementation, fix it in FlowCompanionReactiveOps.scala as part of this task. Run `sbt \"core/testOnly ox.flow.reactive.*\"`."},{"title":"Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","description":"In flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala add `def fromPublisher[T](p: org.reactivestreams.Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p), onDiscard)`, keep the one-arg overload, and update its scaladoc to point at the core overload for the callback semantics. Create flow-reactive-streams/src/test/scala/ox/flow/reactive/FlowReactiveStreamsFromPublisherTest.scala (AnyFunSuite with Matchers) with a small hand-written `org.reactivestreams.Publisher[Int]` (same shape as the core ManualPublisher, but implementing the reactive-streams interfaces; it cannot be shared because the module does not depend on core's test classpath) and two tests: elements flow through the one-arg variant; with `.take(1)` and the new overload, remaining pushed elements are passed to `onDiscard` and the subscription is cancelled. Run `sbt flowReactiveStreams/test` and `sbt flowReactiveStreams/scalafmt`."},{"title":"Document the onDiscard callback in doc/streaming/flows.md","description":"In doc/streaming/flows.md, section \"### Publisher -> Flow\" (around lines 211-219), add a short paragraph explaining: elements received from the publisher but not emitted downstream (buffered when the publisher fails, when the consumer stops early via exception, interruption or `take`, or arriving after cancellation) can be released by passing a second argument `onDiscard: T => Unit` to `Flow.fromPublisher` / `FlowReactiveStreams.fromPublisher`; it is called exactly once per such element, possibly from the publisher's thread, and should not throw; typical use is releasing reference-counted buffers. Also state that on publisher error, elements already received are emitted before the error is propagated. Keep the prose as short as the surrounding text. If you add a code example, use a ```` ```scala mdoc:compile-only ```` block and verify with `sbt compileDocumentation`; otherwise no build step is needed. Do not touch generated-doc/out."}],"brief":"## Modules and files\n\n- `core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala` — the only implementation file in core. Trait `FlowCompanionReactiveOps` (mixed into `Flow.type`) defines `def fromPublisher[T](p: java.util.concurrent.Flow.Publisher[T])(using BufferCapacity): Flow[T]`. Read the whole file (80 lines) before changing anything. Current structure: `usingEmitInline: emit => unsupervised { channel = BufferCapacity.newChannel[T]; forkUnsupervised { p.subscribe(Subscriber: onSubscribe -> subscriptionRef.set(s); s.request(capacity); onNext -> channel.send; onError -> channel.error; onComplete -> channel.done) }; repeatWhile { channel.receiveOrClosed() match Done -> false / Error -> throw / t -> emit(t); batched subscription.request } }.tapException(cancel subscription)`. The demand logic (`demandThreshold = ceil(capacity/2)`, `toDemand` counter, lazily cached `subscription` read from `subscriptionRef` after the first element) must be preserved.\n- `flow-reactive-streams/src/main/scala/ox/flow/reactive/flowReactiveStreamsExtension.scala` — `object FlowReactiveStreams { def fromPublisher[T](p: org.reactivestreams.Publisher[T])(using BufferCapacity): Flow[T] = Flow.fromPublisher(FlowAdapters.toFlowPublisher(p)) }`. Module has no `src/test` dir yet; scalatest is already a test dependency (build.sbt `flowReactiveStreams`), it `dependsOn(core)` (compile only, not test->test).\n- `core/src/test/scala/ox/flow/reactive/` — existing reactive tests: `FlowPublisherPekkoTest.scala` (AnyFunSuite + Matchers, uses Pekko `Source...runWith(Sink.asPublisher)` and `FlowAdapters.toFlowPublisher`) and `FlowPublisherTckTest.scala`. Put new fromPublisher tests here. No test anywhere currently uses a hand-written Publisher; write one (see below).\n- `doc/streaming/flows.md` lines 211-219, section \"### Publisher -> Flow\" — the only user doc mentioning `fromPublisher`. `generated-doc/out/**` is mdoc output regenerated at release time by `updateDocs`; do NOT edit it. Code blocks in docs use ```` ```scala mdoc:compile-only ```` and are compiled by `sbt compileDocumentation` (slow; only needed if you add a code block). Doc dependency snippet convention: `// sbt dependency: \"com.softwaremill.ox\" %% \"flow-reactive-streams\" % \"@VERSION@\"`.\n\n## Key APIs (ox.channels, core/src/main/scala/ox/channels/Channel.scala)\n\n- `Sink.sendOrClosed(t): Unit | ChannelClosed` (non-throwing), `Sink.send(t)` throws `ChannelClosedException` when closed.\n- `Sink.doneOrClosed(): Unit | ChannelClosed` — \"buffered values will be delivered; send operations in progress complete normally when a receiver arrives\"; returns `ChannelClosed` instead of throwing if already closed. `Sink.errorOrClosed(reason)` — drops buffered values (that is the bug). Use the `*OrClosed` variants everywhere in the subscriber: a misbehaving publisher may call onError/onComplete twice.\n- `Source.receiveOrClosed(): T | ChannelClosed` (blocking); match `case ChannelClosed.Done => ...; case e: ChannelClosed.Error => throw e.toThrowable; case t: T @unchecked => ...`. `Source.tryReceiveOrClosed()` is non-blocking but may spuriously return None, so it is not suitable for a drain loop.\n- `Source.foreachOrError(f)` / `drainOrError()` from `SourceDrainOps.scala` loop until Done; usable for the drain, but write the drain inline so you can handle `InterruptedException` (see below).\n- jox semantics (com.softwaremill.jox channels 1.1.2, verified from source): after `done()`, buffered elements stay receivable; a receiver observing Done has already seen every element whose `send` started before `done()` (an in-progress sender is waited for); a `send` racing with `done()` either lands in the buffer or returns closed. Hence \"drain until Done, then `sendOrClosed` returns closed\" gives exactly-once discard with no extra locking.\n- `BufferCapacity`: `given default: BufferCapacity = 16`; `BufferCapacity.newChannel[T]`; `summon[BufferCapacity].toInt` (import `ox.channels.toInt`). Tests can override with `given BufferCapacity = BufferCapacity(4)`.\n- `ox.util`: `t.tapException(f)` runs `f(e)` for any Throwable and rethrows `e` (exceptions from `f` added as suppressed). `ox.pipe`, `ox.discard`, `ox.repeatWhile` already imported in the file.\n- `FlowOps.take(n)` stops upstream by throwing a private `ControlThrowable` (`abortTake`) out of `emit`, caught in `take` itself. So \"consumer stops early\" includes `take`, and the cleanup path must run for every Throwable, not just NonFatal. `tapException`/`catch case e: Throwable` already does this.\n- `unsupervised { ... }` + `forkUnsupervised { ... }` (import `ox.unsupervised`, `ox.forkUnsupervised`): scope end interrupts still-running forks once. `onNext`/`onError`/`onComplete` run on the publisher's threads (or on the `p.subscribe` fork for synchronous publishers), never on the consumer thread.\n- Test helpers: `ox.forkCancellable { ... }` inside `supervised` returns a `CancellableFork` with `cancel(): Either[Throwable, T]` (interrupts once, joins). Interruption inside the flow surfaces as `InterruptedException` thrown from a blocking call (e.g. `Semaphore.acquire`, `channel.receiveOrClosed`).\n\n## Design decisions (apply as-is)\n\n- API: add an overload `def fromPublisher[T](p: Publisher[T], onDiscard: T => Unit)(using BufferCapacity): Flow[T]` and make the existing one-arg method delegate with `_ => ()`. Do not add a default parameter and do not change the existing signature: `core` runs MiMa (`enableMimaSettings` in build.sbt) and only additive changes are safe. Same shape in `FlowReactiveStreams`.\n- Subscriber: `onNext(t)` = `channel.sendOrClosed(t)` and if the result is `ChannelClosed` call `onDiscard(t)`. `onError(e)` = store `e` in an `AtomicReference[Throwable]`, then `channel.doneOrClosed().discard`. `onComplete()` = `channel.doneOrClosed().discard`. `onSubscribe(s)` = set `subscriptionRef`, then if a `cancelled: AtomicBoolean` is already set call `s.cancel()` else `s.request(capacity)` (both sides use volatile atomics, so at least one side sees the other; a double `cancel()` is allowed by the spec).\n- Consumer loop: on `ChannelClosed.Done`, read the stored error; if non-null throw it, else finish. Keep the `ChannelClosed.Error` branch for safety.\n- Cleanup, run on any Throwable escaping the loop (the stored publisher error included; cancel after onError is a spec-allowed no-op): set `cancelled`, cancel the subscription if `subscriptionRef` is non-null, `channel.doneOrClosed().discard`, then drain: loop `receiveOrClosed()` until `Done`, calling `onDiscard` for each element. The drain is bounded (channel is done, at most capacity elements plus in-flight sends) but blocking; if `InterruptedException` is thrown mid-drain, catch it, keep draining, and call `Thread.currentThread().interrupt()` once the drain finishes, so no element is lost on a second interrupt. Then rethrow the original exception. An element that was passed to `emit` and whose `emit` threw counts as emitted (downstream owns it): do not discard it.\n- Scaladoc on the new overload: state the exactly-once invariant, that `onDiscard` may be called from publisher threads, that it must not throw, and that on publisher error buffered elements are emitted before the error is rethrown (this is a deliberate behaviour change for the one-arg variant too; mention it in its doc).\n\n## Test publisher helper (write it in the core test, package `ox.flow.reactive`)\n\nA `ManualPublisher[T] extends java.util.concurrent.Flow.Publisher[T]` with: `@volatile var subscriber: Subscriber[? >: T]`, `requested: AtomicLong`, `cancelled: AtomicBoolean`, a `subscribed: CountDownLatch(1)`; `subscribe(s)` stores `s`, calls `s.onSubscribe(new Subscription { request(n) => requested.addAndGet(n); cancel() => cancelled.set(true) })`, counts down the latch. Helper methods `next(t)`, `error(e)`, `complete()` forward to the subscriber from the test thread. For the exactly-once stress test, a second variant that runs its own thread emitting `1..n` while `sent < requested` and not cancelled. Run the flow under test in `supervised { forkCancellable { ... } }` or `fork`, await `subscribed`, then drive the publisher. Collect emitted elements with `.tap(emitted += _)` or `runForeach`, discarded with `onDiscard = discarded += _` (use thread-safe collections such as `ConcurrentLinkedQueue` or `java.util.Collections.synchronizedList`, since `onDiscard` runs on other threads).\n\n## Conventions / build\n\n- Scala 3 indentation syntax, `end X` markers on long definitions, `.discard` on ignored results (compiler warns otherwise), scalafmt configured in `.scalafmt.conf` (140-column lines); run `sbt core/scalafmt flowReactiveStreams/scalafmt` after editing.\n- Comments: short, explain non-obvious reasoning only; never restate code. Scaladoc describes usage and invariants, not implementation.\n- Tests: scalatest 3.2.20; the reactive tests use `AnyFunSuite with Matchers` with `test(\"...\")` blocks and Scala 3 colon syntax. `core` runs tests forked (`Test / fork := true`).\n- Commands (sbt 2.0.10): `sbt \"core/testOnly ox.flow.reactive.*\"`, `sbt flowReactiveStreams/test`, `sbt core/mimaReportBinaryIssues` (only meaningful when a previous stable version resolves), `sbt compileDocumentation` for mdoc. Never commit; the user commits."}},{"id":[{"name":"Task: Add onDiscard overload to Flow.fromPublisher in core, with tests for error, early stop and late onNext","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Exactly-once stress test: every element is emitted or discarded, never both or neither","occurrence":0}],"resultJson":{"findings":[]}},{"id":[{"name":"Task: Add onDiscard overload to FlowReactiveStreams.fromPublisher with a module test","occurrence":0}],"resultJson":{"findings":[{"id":"R1.I4.1","title":"sbt Test/compile fails with JGit NoWorkTreeException","reason":{"type":"Declined","text":"This is a problem with the environment, not the code. git sees the worktree as non-bare (`git rev-parse --is-bare-repository` prints false; the git dir is /home/adamw/ox/.git/worktrees/zippy-astrodon-94aa2fb6). The exception comes from the sbt-git plugin, which uses JGit by default and reads this linked worktree as bare while the build loads. Nothing in project/ or the build configures this; `zz-tmp-git.sbt` was my temporary workaround, not the cause. To check the code, I recreated it with `useReadableConsoleGit`, which makes sbt-git use command-line git: `sbt core/Test/compile flowReactiveStreams/Test/compile` succeeded (warnings only, in older test files, no errors) and flowReactiveStreams tests passed. I then deleted the file again. Changing the shared build's git plugin setup is outside the scope of this change."}}]}},{"id":[{"name":"Task: Document the onDiscard callback in doc/streaming/flows.md","occurrence":0}],"resultJson":{"findings":[{"id":"R1.I4.1","title":"`sbt Test/compile` fails with no diagnostic","reason":{"type":"Declined","text":"Environmental. Full output shows sbt-git's JGit throws NoWorkTreeException (\"Bare Repository has neither a working tree, nor an index\") while loading the build, because this checkout is a git worktree; no source error is involved. With a temporary `ThisBuild / git.gitUncommittedChanges := false` override, `sbt Test/compile` succeeds across all modules (only pre-existing unrelated warnings). The temporary file was removed; no codebase change is needed."}}]}},{"id":[{"name":"Final review","occurrence":0}],"resultJson":{"findings":[{"id":"S1","title":"sbt Test/compile fails with JGit NoWorkTreeException","reason":{"type":"Declined","text":"This is a problem with the environment, not the code. git sees the worktree as non-bare (`git rev-parse --is-bare-repository` prints false; the git dir is /home/adamw/ox/.git/worktrees/zippy-astrodon-94aa2fb6). The exception comes from the sbt-git plugin, which uses JGit by default and reads this linked worktree as bare while the build loads. Nothing in project/ or the build configures this; `zz-tmp-git.sbt` was my temporary workaround, not the cause. To check the code, I recreated it with `useReadableConsoleGit`, which makes sbt-git use command-line git: `sbt core/Test/compile flowReactiveStreams/Test/compile` succeeded (warnings only, in older test files, no errors) and flowReactiveStreams tests passed. I then deleted the file again. Changing the shared build's git plugin setup is outside the scope of this change."}},{"id":"S2","title":"`sbt Test/compile` fails with no diagnostic","reason":{"type":"Declined","text":"Environmental. Full output shows sbt-git's JGit throws NoWorkTreeException (\"Bare Repository has neither a working tree, nor an index\") while loading the build, because this checkout is a git worktree; no source error is involved. With a temporary `ThisBuild / git.gitUncommittedChanges := false` override, `sbt Test/compile` succeeds across all modules (only pre-existing unrelated warnings). The temporary file was removed; no codebase change is needed."}},{"id":"R1.I2.1","title":"No test covers the drain's deferred-interrupt path in discardBuffered","reason":{"type":"Declined","text":"A deterministic test can't be written. In jox 1.1.2, `receiveOrClosed` only throws `InterruptedException` when it parks (`Continuation.await`). After `doneOrClosed()` has finished on the draining thread, a receive never parks: buffered or in-flight sends are returned or retried, and later cells are already CLOSED. The catch is reached only in a narrow race, when a concurrent `onComplete`/`onError` is still closing cells. I tried the suggested test (interrupt plus throw in the consumer, then assert the discards and the interrupt flag). It passed both with the catch removed and with the re-interrupt removed, so it did not cover the branch, and I reverted it. The suggested capacity-1 variant also doesn't help, because a blocked sender's Continuation is resumed without parking."},"location":{"file":"core/src/main/scala/ox/flow/FlowCompanionReactiveOps.scala","line":141}},{"id":"R1.I5.1","title":"sbt Test/compile exited 1 with no compile error in the output","reason":{"type":"Declined","text":"Environmental. sbt-git's JGit fails in this git worktree with `NoWorkTreeException: Bare Repository has neither a working tree, nor an index` while evaluating `gitUncommittedChanges`, so the build fails before compiling. It is not a code problem. With a temporary `zz-tmp-git.sbt` that sets `ThisBuild / git.gitUncommittedChanges := false`, Test/compile and the core reactive and flowReactiveStreams tests all pass. I deleted the temporary file afterwards."}},{"id":"R2.I3.1","title":"`sbt Test/compile` exits 1 with no compiler diagnostics","reason":{"type":"Declined","text":"Environmental, not a code problem. In this git worktree, sbt-git's JGit throws `NoWorkTreeException: Bare Repository has neither a working tree, nor an index` while evaluating `gitUncommittedChanges` during build load. The build fails before any compilation, and the full output shows that stack trace above the timing line. With a temporary `zz-tmp-git.sbt` setting `ThisBuild / git.gitUncommittedChanges := false`, `Test/compile` succeeds. With the same file, `core/testOnly ox.flow.reactive.*` (44 passed, 7 ignored) and `flowReactiveStreams/test` (2 passed) also pass. I deleted the file afterwards, so it is not left in the tree."}}]}},{"id":[{"name":"Push branch","occurrence":0}],"resultJson":{"type":"Pushed"}},{"id":[{"name":"Generate PR title and description","occurrence":0}],"resultJson":{"title":"Add onDiscard callback to Flow.fromPublisher","body":"`Flow.fromPublisher` buffers elements in a channel. Some of them could be received from the publisher but never emitted, and nothing released them. That leaks elements which hold resources, such as Netty's ref-counted ByteBufs. Tapir currently tracks these elements itself (softwaremill/tapir#5573).\n\n- New overload `fromPublisher(p, onDiscard: T => Unit)`, in core and in `FlowReactiveStreams`. Each element received from the publisher is either emitted, or passed to `onDiscard` exactly once. Elements are discarded when the flow fails, is interrupted, or stops early (e.g. `take`), and also when they arrive after the subscription is cancelled. `onDiscard` may run on publisher threads and must not throw.\n- Behaviour change, also for the one-argument variant: when the publisher fails, elements already buffered are now emitted first, and then the error is rethrown. Before, they were dropped.\n- The existing one-argument overloads are unchanged and pass a no-op callback. The new method is an overload, not a default parameter, so binary compatibility is kept.\n- Added tests for publisher error, consumer exception, interruption, `take`, and elements arriving after cancellation. Also added a stress test that checks each element is either emitted or discarded, never both or neither. Updated `doc/streaming/flows.md`.\n\nCloses #528"}},{"id":[{"name":"Open PR","occurrence":0}],"resultJson":{"type":"Opened","pr":"https://github.com/softwaremill/ox/pull/529"}}],"published":{"reference":"https://github.com/softwaremill/ox/pull/529"},"stageStarts":[]} \ No newline at end of file