[Composite] Reliable mesh replication and repair-cycle grace for health monitors #41

Summary

Mesh replication still fails while TCP and libp2p ping look healthy, so repair cannot drain its backlog. That freezes health checkpoints and turns GET /api/node/health state into stale within health.maxCheckpointAgeMs (default three minutes). Monitors that alert on state != ready then fire far sooner than a few repair cycles.

This issue covers three linked pieces of work:

1. Expand the public health response so a monitor can wait for at least three complete repair attempts (about 1.5 hours at the default replication.repairSchedule of every 30 minutes) before treating a non-zero repair backlog as an alert
2. Explain and fix Replication stream ended before a complete message arrived and Too many outbound protocol streams for protocol "/ipfs/ping/1.0.0" - 2/1
3. Make replication and session reset between configured peers reliable, so a live ping no longer keeps a dead bitswap or replication session in place

Details

Observed on the two-node testnet (2026-09-21)

After #39 / PR 40, both nodes stayed up, attested the required peer, and served local reads. Cross-node placement still dropped copies:

• Repair walked the confirmed registry (1441 files) and finished with a non-zero backlog (7 on one node, 4 on the other)
• checks.repairFresh stayed false because a required repair cycle is healthy only when lastCompletedSuccessfully is true and backlog === 0
• Checkpoints stopped advancing; after maxCheckpointAgeMs (180000) both nodes reported state: "stale"
• Zabbix, which keys off state != ready, raised "state is not ready"

Helia, startup reconciliation, storage freshness, disk reserve, clock consistency, and peer attestations were all true. The only failing checks were repairFresh and then checkpointFresh.

Repair logs during that sweep:

• Replication stream ended before a complete message arrived (repeated, both directions)
• peer_session_recovery_skipped / "Peer answered a liveness ping; keeping the existing libp2p session"
• Too many outbound protocol streams for protocol "/ipfs/ping/1.0.0" - 2/1
• Occasional The stream has been reset

A later batch on one node restored only 2 of 6 under-replicated files. The remaining files are a 2-node mesh shortfall (the other node does not hold a live copy), not desiredCopies: 4 vs two machines: placeFile already caps holders at network size and treats networkTooSmall as "every node holds the file".

Why health screams so quickly

evaluateHealth will not complete a checkpoint while repairFresh is false. A complete repair cycle that still has stillMissing or unrecoverable records publishes lastCompletedSuccessfully: false and a non-zero lastCompletedBacklog. Mid-cycle, getRepairHealthEvidence already takes max(lastCompletedBacklog, stillMissing + unrecoverable), so the first unrepaired CID in a sweep can freeze checkpoints before the cycle ends.

docs/operations/monitoring.md already says to alert when replication.backlog stays non-zero across more than one repair cycle, not on the first miss. The public health body does not currently expose consecutive unsuccessful cycles, last-cycle counts, or a grace signal, so a Zabbix item on $.state cannot follow that rule.

Do not solve this by pretending the node is ready while Helia, disk reserve, the clock, startup, or required peer attestations fail. Grace applies only to a complete repair cycle that still reports a durability shortfall.

Why #39 did not finish the job

PR 40 added scheduled ping liveness, hangUp and redial on ping failure, and reactive recovery after replication stream errors. pingPeer and peering.cron.ts already document that ping is not proof that bitswap or replication work. Reactive recovery still pings first and returns without hangUp when the ping succeeds (src/peering/peerSessionRecovery.ts). That is exactly the skip seen on the testnet: replication frames die, ping succeeds, the session stays.

@libp2p/ping is registered as ping() with library defaults (src/ipfs-node.ts). The default outbound stream limit for /ipfs/ping/1.0.0 is 1. Scheduled peering (dropUnhealthyConnectedPeers) and reactive recovery (recoverPeerSession) can ping the same peer at once. The second attempt fails with Too many outbound protocol streams ... 2/1. That error has also been logged through the replication onError path as Replication request failed: ... ping ... 2/1.

Replication stream ended before a complete message arrived is thrown by readMessage in src/storage/replicationProtocol.ts when the framed JSON is incomplete and the stream closes. That can be a dead muxer session, a reset from a concurrent hangUp, a timeout abort, or a peer that stopped writing. The investigation must distinguish those cases with tests, not guess from log substrings.

Proposed direction

1. Health: monitorable repair grace

Expand GET /api/node/health (and keep GET /api/node/details aligned) so an operator can configure Zabbix without waiting three minutes for stale.

Add public, non-secret fields under replication, for example:

• consecutiveUnsuccessfulCycles — complete cycles in a row with backlog or lastCompletedSuccessfully: false
• last-cycle summary: examined, still missing, unrecoverable, completed at, successful
• schedule or the configured repair interval, so a monitor can express "three attempts" in terms of this node rather than a hardcoded 90 minutes

Add optional configuration, defaulting to current strict behaviour (0 extra cycles):

• health.repairBacklogGraceCycles — number of consecutive complete unsuccessful repair cycles during which a non-zero backlog does not fail repairFresh

At the default replication.repairSchedule of 0 */30 * * * *, repairBacklogGraceCycles: 3 is about 1.5 hours. A cycle that never completes still fails repairFresh through health.repairMaxAgeMs. Helia, storage, clock, startup, and peer-attestation checks stay immediate.

Document the Zabbix (and Docker healthcheck) split:

• Immediate: checks.helia, storageReserve, clockConsistent, startupReconciliation, peerAttestations
• Repair durability: replication.consecutiveUnsuccessfulCycles >= health.repairBacklogGraceCycles (or checks.repairFresh once grace is in the checkpoint math), not state != ready on the first backlog CID

Keep state honest: after grace expires, backlog still blocks checkpoints and the node becomes degraded then stale as today.

2. Replication and reconnect

Treat a replication or bitswap transfer failure on a configured peer as session-stale even when ping still answers. Ping remains a cheap scheduled liveness probe, not a veto on hangUp after an application-protocol failure.

Raise or serialize ping so two legitimate pings cannot fail the mesh with 2/1 (for example maxOutboundStreams / maxInboundStreams on ping(), and a single in-flight ping per peer).

Reset must actually produce a new session: hangUp, wait until the peer is no longer in getPeers(), then redial, with cooldown so a stream error during the reset itself does not loop.

Repair of a CID that failed because the stream died should be retried after the new session is up, not counted as a durable stillMissing until a live have/store on the fresh connection fails.

Record enough structured fields (event, protocol, peer, whether ping was used as a veto) to tell a dead application session from a peer that truly lacks the file. Do not log secrets, raw peer errors that identify users, or file payloads.

Checklist

• Add consecutive unsuccessful repair-cycle evidence to GET /api/node/health and the OpenAPI Health schema
• Add optional health.repairBacklogGraceCycles (default 0) and apply it only to complete cycles with backlog, not to a missing or overdue cycle
• Keep Helia, disk reserve, clock, startup, and peer-attestation failures outside that grace
• Document monitor and Docker healthcheck rules so Zabbix can wait three repair attempts (~1.5 hours at the default schedule) without ignoring a frozen checkpoint from other causes
• Determine why readMessage sees an incomplete replication frame (stale muxer, reset, timeout, or peer abort) and cover the cases with tests
• Stop using a successful ping as a veto on hangUp after a replication or retrieval stream failure
• Fix /ipfs/ping/1.0.0 2/1 by raising ping stream limits and/or serializing ping per peer
• Make hang-up plus redial replace the session, then re-attempt the failed replication on the new connection
• Confirm a two-node mesh with networkTooSmall can reach backlog === 0 and state: ready without a process restart when copies are repairable

Notes

Follows #39. Ping liveness was the right first step for a TCP session that is fully dead; it does not detect a session whose ping protocol still works.

Related: #22 (replication and repair), #23 (checkpoint semantics), #29 (repair backoff so a permanently missing copy does not hammer the peer). This issue does not implement monthly traffic caps.

GET /api/node/health stays HTTP 200. Readiness remains in state and checks. The health protocol between nodes is separate from the replication protocol; do not broaden stale-session substring matching across both.

Verification

• Two-node mesh: inject a replication stream close while ping still succeeds; the session is reset and the copy is placed without restarting either process
• Overlapping peering ping and reactive recovery ping do not log /ipfs/ping/1.0.0 2/1, and ping still fails a truly dead TCP session
• A complete repair cycle with a temporary shortfall does not flip state to stale before repairBacklogGraceCycles consecutive unsuccessful cycles
• After that many unsuccessful complete cycles, repairFresh is false and the node becomes degraded then stale on the existing checkpoint-age rule
• A node with Helia down, reserve exhausted, or a clock rollback still leaves state other than ready immediately, even when grace is set to 3
• Public health JSON, OpenAPI, monitoring, configuration, and troubleshooting docs describe the new fields and the Zabbix-oriented repair delay
• Existing health and peering tests still pass; add coverage for grace math, ping serialization, and recovery that hang-ups after a replication frame error despite a live ping