NETWORK.md

Network Layer

The networking subsystem authenticates peers in NetworkProcessor, manages their WebSocket sessions in NodeConnectionManager, and counts traffic in Guardian. These owners communicate through direct queries and asynchronous network-message and lifecycle events.

Transport Ownership

The peer server uses Reactor Netty directly on websocket.port (default 8082). It owns its I/O event loops and virtual-thread application dispatcher. Spring WebFlux controllers remain on server.port (default 8080), with management on 8081 by default.

PeerNettyClient owns the outgoing Reactor Netty WebSocket and HTTP callback clients. They use private event loops, a virtual-thread application dispatcher, separate connection providers and an asynchronous bridge to the existing JVM DNS resolver. Neither peer transport uses Spring's shared Reactor resources. Request hostnames remain available for HTTP Host, TLS SNI and certificate hostname verification. Callback POSTs do not follow redirects or retry automatically; WebSocket redirects retain the existing secure-downgrade restriction and request limit.

PeerSession exposes complete binary messages, ordered enqueue, graceful close and cancellation to the connection manager. The Reactor session owns transport framing and cleanup for inbound and outbound connections, including enabling frame aggregation synchronously in the upgrade callback, before application dispatch. Both directions retain a 300-byte receive limit, including cumulative fragmented messages, and bounded incoming/outgoing queues with close-on-overflow. These transport limits belong to the session contract. Text messages are ignored after admission to the incoming queue; ping/pong and close frames remain transport concerns.

Peer Authentication

NetworkProcessor starts its private peer server with two endpoints:

• POST /handshakes accepts a CounterChallengeResponse. Guardian admits the remote address, the processor consumes a one-time challenge, checks the expected public URI, genesis hash, signature and counter-challenge prefix, then signs a ChallengeResponse. Successful authentication completes the waiting outbound attempt before the HTTP response is sent.
• WS / accepts a WebSocket with Atto-Public-Uri and Atto-Http-Challenge headers. Guardian admits the remote address, the processor validates the headers and challenge prefix, then posts a counter-challenge to the peer's /handshakes endpoint. It checks the returned URI and signature before handing the session to NodeConnectionManager.manage(). Authentication owns its attempt reservation: failure or cancellation removes that attempt conditionally by identity; success retains the existing five-second expiry window.

An inbound HTTP handshake has a five-second preparation deadline covering admission, body reading, verification and signing. A separate semaphore limits concurrent HTTP handshakes to 64. The callback service applies a five-second deadline covering URI validation, request construction/signing and the callback exchange. Both handshake payload directions have bounded body readers in HandshakePayloadLimit. Cancellation propagates to ongoing handshake work. Early HTTP rejection closes the connection after writing its status, so unread request content cannot leave a reusable connection stalled. WebSocket authentication runs after upgrade; a rejected attempt terminates without registering a peer.

Remote Signing

When atto.signer.backend=REMOTE, NettyRemoteSigner calls the existing signer's public-keys, blocks, votes and challenges endpoints. It preserves the configured authorization token, lazy public-key cache, signature checks, one-second retry delay and socket timeouts (five seconds for the public key, one second for signing). HTTP 408 and 429 are retried, as are transport and decoding failures; other HTTP errors terminate the operation. Cancellation stops retrying.

The signer owns JSON, retries and signature validation. Its HTTP transport owns connection and operation admission, shutdown, and separate atto-signer-io event loops, connection provider and atto-signer-work virtual-thread dispatcher. The whole signing operation runs on that dispatcher, including codecs, retry waits and signature validation. Peer server/client I/O uses separate atto-peer-server-io and atto-peer-client-io groups. Each transport closes only its owned resources. Virtual-thread executors have separate lifecycles but share the JVM's carrier scheduler. Local signing and signer metrics retain their existing interfaces. The node runtime does not depend on Ktor. Cucumber's live neighbour mode checks compatibility with the actual released node image, including connections initiated in either direction and reconnection.

Outbound Connections

bootstrap() runs every second and tries the configured default nodes. An inbound AttoKeepAlive can also supply a neighbour URI for discovery. connectAsync() reserves the URI in connectingMap before starting a connection coroutine; non-default outbound work also needs a permit from a pool of 64.

connection() validates the peer URI, checks whether it is already connected, and opens a WebSocket with the challenge headers. It waits for its OutboundConnectionAttempt to produce the authenticated node. A timeout task expires an unauthenticated attempt and cancels the connection coroutine after five seconds. Authentication and expiration compete through the same CompletableDeferred; expiration cannot win after authentication. After authentication, the timeout is cancelled and manage() owns the established session. The coroutine's finally block releases its reservation and any outbound permit.

Handshake State

• connectingMap is a Caffeine cache of URI → OutboundConnectionAttempt. It reserves pending peer work and lets an HTTP callback complete the outbound attempt's deferred node. Entries expire after five seconds.
• ChallengeStore generates random one-time challenges prefixed with the target URI. Entries expire after five seconds; consuming a challenge removes it.
• PeerUriValidator owns callback/outbound URI policy, including configured default-node exceptions and address checks. NetworkDnsResolver owns the dispatcher used for blocking DNS calls.

Active Connections and Peer Queries

NodeConnectionManager is the sole owner of active connections. Its Caffeine map stores URI → NodeConnection, with a 60-second expiry refreshed by keepalives from the current session. Registering another connection for the same URI replaces and disconnects the previous session: the last registration wins.

The manager implements ConnectedPeers, exposing snapshot() and isConnected(publicUri). A snapshot contains detached ConnectedPeer(node, socketAddress) metadata, without transport objects. These queries describe current local state; a peer may disconnect immediately after a query. They do not reserve a connection or promise delivery.

Consumers query this owner instead of reconstructing membership from connection events:

Consumer	Query and feature-owned policy
TransactionNetworkProvider	Checks membership before transaction lookup/streaming and between streamed responses. Historical-node eligibility, concurrent lookup permits, stream serialization and request deadlines remain with the provider.
GapDiscoverer	Selects connected historical peers and excludes peers with active gap sessions. It owns session budgets and cancellation; a disconnect event releases sessions only when the URI is currently disconnected.
SendDiscoverer	Finds a connected representative from the supplied votes, preserving vote order; otherwise broadcasts the missing-send request.
VoteKeeper	Finds a connected historical representative for missing votes. Local vote signing and missing-vote tracking remain with the keeper.
Guardian	Derives the current voter socket/public-key view, then applies its own current-weight eligibility check.
NetworkInfoContributor	Builds the paired diagnostic output from current connection metadata.

Session Lifecycle

manage() registers the connection and consumes complete binary messages from its peer session. It deserializes each binary message through NetworkSerializer, which also checks protocol validity. Invalid messages remove the connection. Valid messages become InboundNetworkMessage events with the peer URI, socket address and WEBSOCKET source.

A current session's keepalive renews its map entry without replacing a newer session. Flow completion, decoding rejection and send failures remove entries conditionally using the session identity, so an old session cannot remove its replacement. The finally block initiates transport close. Cache removal also schedules close and then publishes NodeDisconnected; expiry, explicit removal and replacement use this same cleanup path.

Lifecycle events run through independent asynchronous listener tasks. Their publication order does not guarantee consumer completion order, and an old session's disconnect event can arrive after its replacement is registered. Current peer queries avoid relying on that order. GapDiscoverer retains a disconnect reaction because cancelling its feature-owned work is a real side effect; its connectivity check prevents a delayed disconnect from releasing sessions solely because the URI reconnected. Gap sessions identify peers by URI and retain their existing expiry; this is not connection-generation isolation.

clear() removes connections and waits for already scheduled cleanup jobs. stop() clears the map and cancels the manager's scope. NetworkProcessor.stop() stops admission, cancels outbound work, closes the server and client transports, then clears managed connections. Closing the transports first releases stalled writes before manager cleanup waits for disconnect jobs. PeerNettyClient cancels active operations and closes sockets before disposing its connection providers, resolver work and event loops. Transport closure is idempotent. Neither shutdown path disposes Spring's HTTP resources.

Sending and Keepalives

• send(publicUri, message) serializes and sends to the current session, returning if the URI is absent. Failure removes that session conditionally.
• DirectNetworkMessage listeners send to one URI. BroadcastNetworkMessage selects all connected peers matching its EVERYONE or VOTERS strategy and optional URI exceptions.
• NodeBanned disconnects active connections using the banned IP address.
• Every ten seconds, keepAlive() broadcasts AttoKeepAlive with a random connected neighbour URI, if one is available. Recipients refresh their connection lifetime and may discover that neighbour.

Traffic Admission and Banning

Guardian owns traffic statistics, threshold decisions and inbound connection request counters. PeerBans owns active bans and their expiry; ExpectedResponses owns per-URI response credits. NetworkProcessor calls Guardian's requestInboundConnection() directly for HTTP handshake and WebSocket admission. Guardian queries current ban state before counting; other attempts contribute to statistics and the configured per-minute connection limit.

Message counting and window rotation share a lock, so each counted hit belongs to one window. Every 15 seconds, guard() swaps out the current window and retains it as the latest diagnostic snapshot. It queries connected voters, applies current minimal rebroadcast weight, and includes zero for idle eligible voters. If the median meets minimalMedian, addresses whose merged count reaches median × toleranceMultiplier are banned through PeerBans. The owner updates admission state before publishing NodeBanned; its asynchronous consumers cannot delay that state transition.

NetworkMessagePublisher records positive expected-response counts synchronously in ExpectedResponses before dispatching an outbound DirectNetworkMessage. A transport listener can therefore send a request and receive a fast response without racing a separate credit-allocation listener. Guardian consumes credits instead of counting solicited inbound responses; consuming the final credit immediately removes the entry. Only outbound request timestamps extend credit expiry. Inbound responses do not, and delayed publication retains the request's original deadline. Expired credits are removed during response counting or the next guard sweep. Credit lifetime is independent of peer metadata and survives reconnects until consumption or expiry. Credits retain the existing publication-based contract: publication does not promise successful transport delivery.

PeerBans reviews active bans every minute and removes them after one hour from the latest ban. Renewal, expiry, admission queries and snapshots share its lock, so an expiry cannot remove a concurrent renewal. NodeBanned and NodeUnbanned report completed state changes; they do not update the owner's state. Diagnostic banned output queries this same owner directly, so delayed lifecycle listeners cannot leave admission and diagnostics with conflicting ban projections.

Events

Network messages use the asynchronous application event dispatcher. Their publisher returns without waiting for every transport, feature, traffic-accounting or metrics listener to finish.

Event	Producer	Current lifecycle consumers
NodeDisconnected	NodeConnectionManager	GapDiscoverer session cleanup.
NodeBanned	PeerBans, called by Guardian	NodeConnectionManager disconnects affected sessions.
NodeUnbanned	PeerBans expiry review	No current consumers.