Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
649074e
add connection state and disconnect_on decorator
tomkane-dls Nov 12, 2026
0ff9d0f
update builtin connections with a @disconnect_on decorator
tomkane-dls Nov 12, 2026
2727833
remove old controller connection methods, and add new connected attri…
tomkane-dls Nov 12, 2026
6013ff9
add a gate to scans that is set on disconnect, group scans by controller
tomkane-dls Nov 12, 2026
a54c67c
check connection holders, and implement reconnect logic
tomkane-dls Nov 12, 2026
a97a078
remove unhelpful comments
tomkane-dls Nov 12, 2026
1c826e3
add reconnect attempts and period, class and instane overridable
tomkane-dls Nov 12, 2026
09ee695
add fatal error handling and reconnect introspection comparison
tomkane-dls Nov 12, 2026
ddb5bff
update docs with stale connection descriptions
tomkane-dls Nov 12, 2026
31a04bd
add Connected attribute to epics tests
tomkane-dls Nov 12, 2026
699c2d2
tests for depends_on ordering
tomkane-dls Nov 12, 2026
54d97b0
update disconnect_on errors, now that standard connection error handl…
tomkane-dls Nov 13, 2026
d087058
tango test fails because of Connected fix
tomkane-dls Nov 13, 2026
ea169c5
move structure comparison to the connection to avoid mutation of intr…
tomkane-dls Nov 13, 2026
11bfddd
wrap serve coroutines in tasks so we can cleanup on FatalErrors
tomkane-dls Nov 13, 2026
03f7882
tidy up disconnect errors
tomkane-dls Nov 13, 2026
fa6e63f
add context so that connect can use other methods with disconnect_on …
tomkane-dls Nov 13, 2026
79425c8
fix scan explanation
tomkane-dls Nov 16, 2026
c73ec1a
improve fatal error handling
tomkane-dls Nov 16, 2026
5c6d27c
set default reconnect settings to None, and improve implementation of…
tomkane-dls Nov 16, 2026
d3fb8be
check reconnect_period is not None when reconnect_attempts is set, at…
tomkane-dls Nov 16, 2026
0db84f6
update how disconnect_on works, explicitly needing anything beyond Di…
tomkane-dls Nov 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,7 @@
nitpick_ignore_regex = [
("py:class", r"fastcs.*.DType_T"),
("py:class", r"fastcs.*.Numeric_T"),
("py:class", r"fastcs.*._MethodT"),
("py:obj", r"fastcs.*.DType_T"),
(r"py:.*", r"fastcs\.demo.*"),
(r"py:.*", r"tickit.*"),
Expand Down
105 changes: 102 additions & 3 deletions docs/explanations/connections.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,7 @@ reply = await connection.query("R?") # sends "R?\r\n", returns the reply withou
For binary protocols, `write_bytes` and `query_bytes` send raw bytes without a
terminator. If an exchange fails or takes longer than `timeout`, the link is closed and
later calls raise `DisconnectedError`. If the link fails, such as the device closing the
socket or a serial adapter being unplugged, the exchange raises `ConnectionError`, which
`DisconnectedError` subclasses, so
`except (ConnectionError, TimeoutError)` catches any link failure.
socket or a serial adapter being unplugged, the exchange raises `ConnectionError`.

`HTTPConnection` talks to a device with a JSON API. `get`, `put`, `post` and `delete`
take a path relative to its `url`, send JSON, and return the parsed reply. An error
Expand All @@ -45,3 +43,104 @@ Like the other connections, it raises `TimeoutError` when the device doesn't ans
time, but leaves the link open, as one slow request doesn't mean the server has gone. It
raises `ConnectionError` when the server can't be reached, and closes the link so later
calls raise `DisconnectedError`.

## When the link goes down

A connection marks the IO methods whose failure means the link has gone with
`@disconnect_on`:

```python
from fastcs.connections import Connection, disconnect_on


class EigerConnection(Connection):
disconnect_after_timeouts = 3
...

@disconnect_on(ConnectionError, TimeoutError)
async def get(self, key: str) -> Any:
return await self.client.get(f"/detector/{key}")
```

If the method raises a `DisconnectedError` or one of the listed exceptions, the
connection is marked down and the exception carries on to the caller. Bare
`@disconnect_on` only lists `DisconnectedError`, so list `ConnectionError` too if it
means the link has gone.

If `TimeoutError` is listed, directly or through a base class like `OSError`, timeouts
are counted: the connection is marked down after `disconnect_after_timeouts` in a row,
1 by default, and any call that works starts the count again.

While it's down, decorated methods raise `DisconnectedError` without touching the
device, except when called from the connection's own `connect()` or `introspect()` while
FastCS reconnects it. Other exceptions, like an error reply from the device, leave the
link alone. The built-in connections already decorate their IO methods.

When a connection is marked down, FastCS:

1. Sets `connected` to `False` on every controller holding it, and pauses their scans.
2. Waits for the connections it `depends_on` to be up, then calls `close()`, `connect()`
and `introspect()`. The first try is straight away, and later ones are
`reconnect_period` seconds apart.
3. Once that works and the introspection's structure is unchanged, marks the connection
up, sets `connected` back to `True`, re-runs the controllers' `ONCE` scans and
resumes the others.
4. After `reconnect_attempts` failed tries, stops with a `ReconnectFailedError`.

If the new introspection has different fields or keys from the one the controllers were
built from, FastCS stops with an `IntrospectionMismatchError`. Changed values are fine.

Both are a `FastCSError`: FastCS shuts down, so that it can be restarted and build its
controllers again.

### Reconnect settings

Each connection's block in `fastcs.yaml` can set how it reconnects, and how many
timeouts mean it has gone:

```yaml
connections:
connection:
ip: 10.0.0.5
port: 4000
reconnect_attempts: 5
reconnect_period: 2.0
disconnect_after_timeouts: 1
```

`reconnect_attempts: 0` or `null` means stop as soon as the connection goes down. That
is the default for a connection class that doesn't set its own, which it does with:

```python
class EigerConnection(Connection):
reconnect_attempts = 20
reconnect_period = 3.0
```

A class that sets `reconnect_attempts` must set `reconnect_period` too, or defining it
raises a `TypeError`.

| Connection | `reconnect_attempts` | `reconnect_period` |
|---|---|---|
| `Connection` | `None` | `None` |
| `IPConnection` | 30 | 2.0 |
| `HTTPConnection` | 60 | 5.0 |
| `SerialConnection` | 10 | 1.0 |

`disconnect_after_timeouts` is 1 unless a class sets it, and must be at least 1. It only
applies to methods whose `@disconnect_on` lists `TimeoutError`, so `HTTPConnection`'s
timeouts never mark it down. `IPConnection` and `SerialConnection` close the link on a
timeout, so a late reply isn't read as the reply to the next exchange, and a higher
value only delays marking them down until the next call.

### Which controllers a connection belongs to

FastCS finds the controllers to mark down by looking for the connection on them, so:

- The controller that uses a connection stores it, e.g. `self.connection = connection`,
and passes it to any sub controllers that need it, which store it too.
- Each controller stores at most one connection.
- Every connection a controller stores was opened by FastCS, from the top-level
controller's arguments.

FastCS raises a `ValueError` at startup if any of these is broken.
33 changes: 14 additions & 19 deletions docs/explanations/controllers.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,28 +12,31 @@ role, FastCS opens the `Connection`s it is given, constructs it, calls its lifec
hooks and runs the scan tasks it creates on the event loop.

**Sub controller:** attached to a parent controller via `add_sub_controller()` or by
assigning it as an attribute. In this role, the sub controller's `reconnect` hook is
not called automatically by FastCS. The parent controller is responsible for calling it
as part of its own lifecycle, if required.
assigning it as an attribute. Its attributes are exposed under a prefixed path.

### Lifecycle hooks

| Method | Purpose |
|---|---|
| `setup` | Called by the framework after all attributes are added, before serving the application. Called on every controller in the tree |
| `reconnect` | Re-open connection after scan error |

Connections to the device are not opened by the controller. They are `Connection`s
passed to its constructor, which FastCS opens before constructing it and closes on
shutdown.
passed to its constructor, which FastCS opens before constructing it, reconnects if the
link goes down, and closes on shutdown. A controller stores the connection it uses, and
passes it on to any sub controllers that need it. See
[Connections](connections.md).

### Scan task behaviour

When used as the root controller, FastCS collects all `@scan` methods and readable
attributes with `update_period` set, across the whole controller hierarchy to be run as
background tasks by FastCS. Scan tasks are gated on the `_connected` flag: if a scan
raises an exception, `_connected` is set to `False` and tasks pause until `reconnect`
sets it back to `True`.
FastCS collects the `@scan` methods and readable attributes with `update_period` set
from every controller in the tree, and runs them as background tasks. A scan that
raises an exception is logged, and runs again a second later.

Every controller has a read-only `connected` attribute. When the connection a controller
holds goes down, FastCS sets `connected` to `False` and pauses that controller's scans.
Once the connection is back, it sets `connected` to `True`, re-runs the controller's
`ONCE` scans and resumes the others. Controllers that hold no connection, or a different
one, carry on.

```python
from fastcs.controllers import Controller
Expand All @@ -50,14 +53,6 @@ class TemperatureController(Controller):
super().__init__()
self._connection = connection

async def reconnect(self):
try:
await self._connection.close()
await self._connection.connect()
self._connected = True
except Exception:
logger.error("Failed to reconnect")

@scan(period=1.0)
async def update_temperature(self):
value = await self._connection.get_temperature()
Expand Down
6 changes: 3 additions & 3 deletions docs/explanations/what-is-fastcs.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ using `ControllerVector`. See [Controllers](controllers.md) for details.

To create a driver, subclass `Controller`, declare attributes and commands as typed
class members, take the `Connection`s it needs as constructor arguments, and implement
lifecycle hooks (`setup`, `reconnect`) and periodic scan tasks. The [tutorials](../tutorials.md) walk through a complete example.
the `setup` lifecycle hook and periodic scan tasks. The [tutorials](../tutorials.md) walk through a complete example.

## Deploying as an application

Expand All @@ -77,8 +77,8 @@ device logic does not require a running IOC or device server.
**Multiple transports from a single driver** — the same controller can serve EPICS CA,
EPICS PVA, Tango, REST, and GraphQL simultaneously or in any combination.

**Automatic reconnection** — if a scan task raises an exception, FastCS pauses all
tasks and calls `reconnect()` automatically, resuming once the connection is restored.
**Automatic reconnection** — when a connection's link goes down, FastCS pauses the scans
of the controllers using it and reconnects it, resuming them once the link is back.

**Auto-generated OPI screens** — EPICS transports generate CSS-Phoebus screen files
from the controller's attribute metadata, with no additional effort from the driver
Expand Down
4 changes: 2 additions & 2 deletions docs/how-to/update-attributes-from-device.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,8 +175,8 @@ class MultiChannelController(Controller):
await channel.voltage.update(float(voltage))
```

The scan period (here `0.1` seconds) sets how often the batched query runs. Scans that
raise an exception will pause and wait for `reconnect()` to be called before resuming.
The scan period (here `0.1` seconds) sets how often the batched query runs. If the
connection goes down, the scan pauses until FastCS has reconnected it.

### Scan as a cache for `AttributeIO.update`

Expand Down
2 changes: 2 additions & 0 deletions src/fastcs/connections/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,9 @@
from fastcs.util import optional_dependency

from .connection import Connection as Connection
from .connection import ConnectionState as ConnectionState
from .connection import DisconnectedError as DisconnectedError
from .connection import disconnect_on as disconnect_on
from .ip_connection import IPConnection as IPConnection

with optional_dependency("httpx"):
Expand Down
Loading
Loading