Skip to content

client: report stream lifecycle RPC failure as error codes - #112

Open
davidonuoha wants to merge 1 commit into
pothosware:masterfrom
davidonuoha:fix/stream-lifecycle-rpc-error-codes
Open

davidonuoha wants to merge 1 commit into
pothosware:masterfrom
davidonuoha:fix/stream-lifecycle-rpc-error-codes

Conversation

@davidonuoha

Copy link
Copy Markdown

When the TCP control connection to the server is gone, the client throws
std::runtime_error from deactivateStream(), activateStream() and
closeStream(). SoapySDR documents these as error-code / void calls
(SoapySDR/Device.hpp), and their real call sites are the constructor and
destructor of GNU Radio's gr::block_executor (block_executor.cc:254), which
cannot handle an exception.

deactivateStream() throwing from that destructor aborts the host application
through std::terminate. Reported as gqrx-sdr/gqrx#1445: gqrx aborts about one
time in four when the modulation is changed or audio recording is started with an
SDRplay RSPdx behind SoapyRemote, because each of those actions tears down and
rebuilds the flow graph and therefore deactivates/reactivates the remote stream:

terminate reached from thread id: ... Got std::runtime_error
SoapyRPCUnpacker::recv(header) FAIL: recv() [104: Connection reset by peer]
  SoapyRemoteDevice::deactivateStream(...) [clone .cold]+0x2b
  gr::block_executor::~block_executor()+0x2b2

activateStream() is called from the same class's constructor, where an exception
leaves the scheduler's start barrier unsatisfied and the next flow-graph restart
deadlocks instead of failing. closeStream() is called from the client device's
destructor and returns void, so it has no way to report the failure at all.

This change logs the transport error via SoapySDR's logger and returns
SOAPY_SDR_STREAM_ERROR from deactivateStream()/activateStream(); the logging
keeps the original reason visible (SoapyRemoteDevice::deactivateStream() FAIL: ...), so this reports a dead link rather than hiding it. closeStream() logs and
still releases the local stream data, which it previously leaked when the call
failed. setupStream() keeps throwing, as documented. No protocol, API or ABI
change.

Tested without SDR hardware: a dummy SoapySDR device served by a real
SoapySDRServer, the control link closed with FIN, with RST, and with RST after
the request was forwarded, teardown exercised from a noexcept destructor on a
worker thread. Before: 4 of 9 cases abort with SIGABRT and the messages above.
After: no aborts, every failure logged and returned as -2; healthy
activate/deactivate cycles and SoapySDRUtil --check=remote unchanged. Still needs
one confirmation run on real hardware.

The SoapySDR stream API reports failures through the return value:
deactivateStream() is documented as "0 for success or error code on
failure" and closeStream() returns void. These calls are also made from
places that cannot handle an exception:

- deactivateStream() is called from the destructor of gr::block_executor
  (GNU Radio), which is implicitly noexcept. The client throws
  std::runtime_error when the control connection is gone ("Connection
  reset by peer", or EOF when the server went away), so losing the remote
  link aborts the whole host process instead of reporting a dead device.
  This is gqrx-sdr/gqrx#1445:

    terminate reached from thread id: ... Got std::runtime_error
    SoapyRPCUnpacker::recv(header) FAIL: recv() [104: Connection reset by peer]
    ... SoapyRemoteDevice::deactivateStream(...) [clone .cold]+0x2b
    ... gr::block_executor::~block_executor()+0x2b2

- activateStream() is called from the block_executor constructor on the
  block threads. An exception there leaves the scheduler's start barrier
  unsatisfied, so the host process deadlocks in top_block::unlock() on the
  next flow graph reconfiguration instead of just failing.

- closeStream() is called from the destructor of the client device, and
  returns void, so it has no way to report such a failure at all. Also
  release the local stream data unconditionally, so a failed remote call
  no longer leaks the endpoint.

Log the transport error through SoapySDR's logger and report
SOAPY_SDR_STREAM_ERROR, so that a lost link is visible as a device error.
setupStream() keeps throwing, as documented by SoapySDR.

Fixes: gqrx-sdr/gqrx#1445
@zuckschwerdt

Copy link
Copy Markdown
Member

Thanks! Seems sensible, errors in teardown (deactivateStream, closeStream) should not be fatal. And for activateStream there should already be an error path for SOAPY_SDR_STREAM_ERROR returns.

Let's keep this open for discussion for a few weeks, good to merge otherwise.

@transmogrify-cell

Copy link
Copy Markdown

I ran an independent test of this branch (69f5b89), without SDR hardware.

Setup

  • A real SoapySDRServer (master baa9279) serving a fake RX device.
  • The client is driven through the stream lifecycle the way GNU Radio 3.10 drives gr-osmosdr: activateStream() in the block_executor constructor before the start barrier, deactivateStream() in its implicitly noexcept destructor, and closeStream() plus unmake() from a destructor.
  • The link was killed four ways: kill -9 of the server, FIN and RST through a proxy, and an abort() inside the server-side driver's activateStream().
Scenario master this PR
teardown after the link dies (kill -9 / FIN / RST) SIGABRT ×3 clean ×3
flowgraph start on a dead link (kill -9 / FIN / RST) start barrier never released, hang ×3 clean ×3
server crashes inside activateStream() hang clean
link dies, setFrequency(), then stop SIGABRT at stop clean; setFrequency() still throws, as intended
5× server restart + device reopen SIGABRT on the first cycle clean, all 5 recover
100 healthy activate/deactivate cycles ok identical

A UBSan build of this branch was clean on the failure scenarios. Platform: macOS arm64, SoapySDR 0.8.1.

Separately, #113 fixes a hang in the RPC payload receive that this PR doesn't touch.

Disclosure: done with AI assistance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants