[Bug] Stale libp2p session breaks cross-node retrieval (408 timeout) #39

Summary

On the ADAMANT IPFS testnet (ipfs1test / ipfs2test, v0.1.0), clients saw HTTP 408 on GET /api/file/:cid when the file existed on the peer that received the upload but not on the node serving the read.

Investigation showed:

• The API returns 408 when ifs.stat(cid) hits findFileTimeout (20s) and raises FileNotFoundError
• A single libp2p TCP session between nodes can remain established while bitswap and replication streams fail in one direction (upload replication from ipfs2→ipfs1 failed; cross-node GET ipfs1←ipfs2 timed out; reverse path worked)
• The scheduled peering job only dials missing peers (getPeers()), so a zombie connection is never torn down or redialed
• Logs included repeated Replication request failed: Cannot write to a stream that is closed on the receiving node during inbound replication from the holder

Restarting the IPFS process on ipfs2 immediately restored replication and cross-node retrieval for previously failing CIDs.

Impact

• Cross-node reads fail with 408 until an operator restarts a node or the libp2p session happens to recover
• Upload replication from one testnet node to the other can fail silently (attempts: [{ ok: false }]) while the upload still returns 200 on the origin node
• Affects any multi-node deployment relying on bitswap between configured peers, not only the testnet

Expected behavior

Configured peers should exchange blocks over bitswap and replication without requiring process restarts when TCP still appears connected.

Actual behavior

Stale libp2p sessions pass the "connected" check in peering but do not deliver blocks; reads time out at 20s.

Reproduction (testnet, 2026-09-20)

1. Ensure ipfs1test and ipfs2test run IPFS Node 0.1.0 with a stable libp2p session (may require waiting after upgrade or under load)
2. Upload a small file to https://ipfs2test.adamant.im/api/file/upload
3. GET https://ipfs1test.adamant.im/api/file/<cid> → 408 after ~20s
4. Same CID on ipfs2 → 200
5. Upload on ipfs1 → GET on ipfs2 → 200 (asymmetric while session is degraded)

Proposed direction

• Periodic libp2p ping liveness checks for connected configured peers; hangUp and redial on failure
• Reactive session reset after replication stream errors and after retrieval timeout (FileNotFoundError), with cooldown to avoid storms

Environment

• IPFS Node 0.1.0 (dev build on testnet)
• Two-node testnet mesh, replication enabled, findFileTimeout: 20000