Skip to content

Connections - #459

Draft
tomkane-dls wants to merge 48 commits into
mainfrom
connections
Draft

tomkane-dls wants to merge 48 commits into
mainfrom
connections

Conversation

@tomkane-dls

@tomkane-dls tomkane-dls commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Connections - Closes #458

example ioc - > fastcs-thorlabs-mff


Questions

Design: open questions

  1. Per-instance depends_on (not implemented). depends_on is currently declared at class level, by connection type. A dependent therefore waits for every connection of that type, e.g. each meta writer waits for both Odins. A per-instance depends_on("conn") in fastcs.yaml, naming the connection, would override the class-level types. It would also fix the case of two parents of the same type.
  2. Simulated connections. fastcs.yaml has no way to swap a real connection for a simulated one. Options considered:
    • a union type hint (RealConnection | SimConnection)
    • accepting any subclass of the hinted connection type
    • a dedicated sim setting
  3. Unions of connection types, e.g. IPConnection | SerialConnection. _connection_hint doesn't recognise a union as a connection, so launch treats the argument as the controller's options argument. This is unchanged from current behaviour, but it isn't defined.
  4. Reconnects that cycle behind a terminal server (e.g. Moxa). A reconnect only reopens TCP to the terminal server. That succeeds even when the device behind it is off, unless connect() or introspect() actually queries the device. FastCS then cycles down → up → down and never uses up reconnect_attempts. Should the built-in connections, or the docs, require a check that the device itself answers?
  5. What counts as "structure" in the introspection comparison. After a reconnect, the new introspect() result is compared with the original to detect whether the device changed. Leaf types aren't compared, so a parameter that changes type isn't caught. Devices that describe their types as strings would need those strings compared.
  6. Push-based connections (streams, subscriptions). No caller sits on the failure path, so @disconnect_on never sees their errors and they never mark the connection down. Not designed yet.

Design: known gaps (accepted for now)

  1. Setpoints after a device reboot. The device may come back with its default setpoints while clients still show the old values.
  2. Undecorated methods aren't watched for failures, and aren't blocked while the connection is down. Only @disconnect_on methods are.

Design: decisions

  1. Ownership. Each controller holds at most one connection, and every opened connection must be held by a controller. Both rules are enforced when the controller tree is sealed.
  2. Calls on unheld connections. A ContextVar records which controller is currently acting. A call into a connection from a controller that doesn't hold it (e.g. a sibling's or parent's connection) can then be detected.
  3. Calls during reconnect. While a connection is reconnecting, its connect() and introspect() may call its own @disconnect_on methods. Only the reconnecting task may do this; other callers stay blocked. This is also enforced with a ContextVar.
  4. Subconnection naming and passing (PR 4):
    • The runner always names them <parent>_<name>, e.g. odin_meta.
    • The controller receives each one as an argument with that name, which must be hinted X | None.
    • A yaml block with the same name overrides the discovered subconnection.
  5. Specs (PR 4). ControllerSpec keeps typed arguments and takes connection instances. Launch and discovery build connections through ConnectionSpec. We chose this over full symmetry, where everything would go through specs.
  6. Positional connections in a ControllerSpec (PR 4) get their names by binding them to the controller's signature. Anything passed through *args falls back to its class name.
  7. Dependencies on discovered connections (PR 4). At startup, an explicitly given connection may not depend on a connection that can only be discovered. It is rejected rather than held back until discovery finishes.

Reconnect defaults

  1. Reconnect settings per class:

    Class reconnect_attempts reconnect_period disconnect_after_timeouts
    Connection None None 1
    IPConnection 30 2.0 s 1
    HTTPConnection 60 5.0 s 3
    SerialConnection 10 1.0 s 1
  2. disconnect_after_timeouts is the number of consecutive timeouts before the connection is marked down. It defaults to 1 on the base class and must be at least 1.

  3. The first reconnect attempt is immediate. reconnect_period only applies between failed attempts.

  4. While a scan method is failing, it retries after 1 s, whatever its configured period.

Connection defaults

  1. IPConnection: terminator="\r\n", encoding="utf-8", timeout=5.0, limit=2**16 (the limit doesn't count the terminator).
  2. SerialConnection: baud=115200, 8 data bits, no parity, 1 stop bit, no flow control, terminator="\r\n", encoding="utf-8", timeout=5.0, limit=2**16. The port is opened exclusively.
  3. HTTPConnection: timeout=5.0, verify=True, follow_redirects=True. Its connect() sends GET "" and treats any response as "reachable", even an error status.

Eiger example

eiger/
  connections.py
  controllers.py
  __main__.py
fastcs.yaml

connections.py

import asyncio
import json
from dataclasses import dataclass
from typing import Any

from fastcs.connections import (
    Connection,
    ConnectionSpec,
    HTTPConnection,
    Introspection,
    disconnect_on,
)


@dataclass
class EigerIntrospection:
    # {"count_time": {"value": 0.5, "value_type": "float", "access_mode": "rw"}, ...}
    parameters: dict[str, dict[str, Any]]


class EigerConnection(HTTPConnection[EigerIntrospection]):
    # HTTPConnection already gives connect(), close(), and get()/put() with
    # @disconnect_on, and reconnects 60 times, 5 seconds apart

    async def introspect(self) -> Introspection[EigerIntrospection]:
        return Introspection(EigerIntrospection(await self.get("/config")))


@dataclass
class OdinIntrospection:
    parameters: dict[str, dict[str, Any]]
    meta_writer_ip: str
    meta_writer_port: int


class OdinConnection(HTTPConnection[OdinIntrospection]):
    # Opened after the detector, and waits for it to be up before reconnecting
    depends_on = EigerConnection

    async def introspect(self) -> Introspection[OdinIntrospection]:
        tree = await self.get("/config")
        meta_writer = tree.pop("meta_writer")
        info = OdinIntrospection(tree, meta_writer["ip"], meta_writer["port"])

        # Renamed "odin_meta" by the runner, after this connection's name
        meta = ConnectionSpec(
            "meta", MetaWriterConnection, ip=info.meta_writer_ip, port=info.meta_writer_port
        )
        return Introspection(info, subconnections=[meta])


class MetaWriterConnection(Connection):
    """A line-based JSON protocol over TCP"""

    reconnect_attempts = 20
    reconnect_period = 1.0

    def __init__(self, ip: str, port: int) -> None:
        self.ip = ip
        self.port = port

    async def connect(self) -> None:
        self._reader, self._writer = await asyncio.open_connection(self.ip, self.port)

    async def close(self) -> None:
        self._writer.close()
        await self._writer.wait_closed()

    @disconnect_on(TimeoutError)
    async def request(self, command: str, **params: Any) -> dict[str, Any]:
        self._writer.write(json.dumps({"command": command, **params}).encode() + b"\n")
        await self._writer.drain()
        reply = await asyncio.wait_for(self._reader.readline(), timeout=2.0)
        if not reply:
            raise ConnectionResetError("The meta writer closed the link")

        return json.loads(reply)

controllers.py

from typing import Any

from fastcs.attributes import Attribute, AttrR, AttrRW, Polled
from fastcs.connections import HTTPConnection
from fastcs.controllers import Controller

from eiger.connections import EigerConnection, MetaWriterConnection, OdinConnection

_DATATYPES: dict[str, type] = {"float": float, "int": int, "string": str, "bool": bool}


def _attribute(connection: HTTPConnection, path: str, parameter: dict[str, Any]) -> Attribute:
    """An attribute for one parameter of a device that describes itself"""
    datatype = _DATATYPES[parameter["value_type"]]
    # Metadata is passed as keyword arguments, e.g. units="s" for a number
    meta = {"units": parameter["units"]} if "units" in parameter else {}

    async def get() -> Any:
        return (await connection.get(path))["value"]

    if parameter["access_mode"] == "rw":

        async def put(value: Any) -> None:
            await connection.put(path, json={"value": value})

        # Read once at startup and after each reconnect, then on every set
        return AttrRW(datatype, getter=get, setter=put, **meta)

    # Status the device changes by itself
    return AttrR(datatype, getter=Polled(get, period=0.2), **meta)


class EigerController(Controller):
    """The top-level controller, whose arguments are the connections"""

    def __init__(
        self,
        eiger: EigerConnection,
        odin: OdinConnection,
        odin_meta: MetaWriterConnection | None = None,
    ) -> None:
        super().__init__()
        self.DET = EigerDetController(eiger)
        self.ODIN = OdinController(odin, odin_meta)


class EigerDetController(Controller):
    def __init__(self, connection: EigerConnection) -> None:
        super().__init__()
        self.connection = connection
        for name, parameter in connection.introspection.parameters.items():
            self.add_attribute(name, _attribute(connection, f"/config/{name}", parameter))


class OdinController(Controller):
    def __init__(
        self, connection: OdinConnection, meta: MetaWriterConnection | None
    ) -> None:
        super().__init__()
        self.connection = connection
        for name, parameter in connection.introspection.parameters.items():
            self.add_attribute(name, _attribute(connection, f"/config/{name}", parameter))

        # Not stored here: each controller holds at most one connection
        if meta is not None:
            self.META = MetaWriterController(meta)


class MetaWriterController(Controller):
    def __init__(self, connection: MetaWriterConnection) -> None:
        super().__init__()
        self.connection = connection
        self.writing = AttrR(bool, getter=Polled(self._get_writing, period=0.5))
        self.frames_written = AttrR(
            int,
            getter=Polled(self._get_frames_written, period=0.5),
            description="Frames written to the current file",
        )
        self.file_path = AttrRW(
            str, getter=self._get_file_path, setter=self._set_file_path, length=256
        )

    async def _get_writing(self) -> bool:
        return (await self.connection.request("status"))["writing"]

    async def _get_frames_written(self) -> int:
        return (await self.connection.request("status"))["frames_written"]

    async def _get_file_path(self) -> str:
        return (await self.connection.request("status"))["file_path"]

    async def _set_file_path(self, value: str) -> None:
        await self.connection.request("configure", file_path=value)

__main__.py

from fastcs.launch import launch

from eiger.controllers import EigerController

launch(EigerController)

fastcs.yaml

controllers:
  - id: EIGER
    type: eiger.EigerController
    connections:
      eiger:
        url: http://10.0.0.1/detector/api/1.8.0
      odin:
        url: http://10.0.0.2:8888/api/0.1
        reconnect_attempts: 10
        reconnect_period: 2.0
      # odin_meta is discovered from odin. Give it here to set it by hand instead:
      # odin_meta:
      #   ip: 10.0.0.3
      #   port: 5659
transport:
  - epicsca: {}

Without fastcs.yaml

The same thing built in Python. A ControllerSpec takes connection instances, named by their argument. odin_meta is left out, so it's discovered:

from fastcs.controllers import ControllerSpec
from fastcs.launch import FastCS
from fastcs.transports.epics.ca import EpicsCATransport

eiger = EigerConnection("http://10.0.0.1/detector/api/1.8.0")
odin = OdinConnection("http://10.0.0.2:8888/api/0.1").set_reconnect(
    attempts=10, period=2.0
)

FastCS(
    [ControllerSpec("EIGER", EigerController, eiger=eiger, odin=odin)],
    [EpicsCATransport()],
).run()

@coderabbitai

coderabbitai Bot commented Nov 16, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codecov

codecov Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.49550% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.85%. Comparing base (1d884b3) to head (fabab71).

Files with missing lines Patch % Lines
src/fastcs/runner.py 96.50% 5 Missing ⚠️
src/fastcs/connections/serial_connection.py 50.00% 4 Missing ⚠️
src/fastcs/control_system.py 94.28% 2 Missing ⚠️
src/fastcs/controllers/base_controller.py 93.93% 2 Missing ⚠️
src/fastcs/connections/ip_connection.py 85.71% 1 Missing ⚠️
src/fastcs/launch.py 98.07% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #459      +/-   ##
==========================================
+ Coverage   91.37%   91.85%   +0.48%     
==========================================
  Files          72       75       +3     
  Lines        2922     3034     +112     
==========================================
+ Hits         2670     2787     +117     
+ Misses        252      247       -5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Connections Rework

1 participant