From fe17e189be8ce7523ab6b4788fe0b15621dee8cc Mon Sep 17 00:00:00 2001 From: murat <155527010+kejan2514@users.noreply.github.com> Date: Wed, 12 Aug 2026 22:47:01 +0300 Subject: [PATCH 1/6] docs: add RPC forwarder troubleshooting guide --- docs/troubleshooting-rpc-forwarder.md | 122 ++++++++++++++++++++++++++ 1 file changed, 122 insertions(+) create mode 100644 docs/troubleshooting-rpc-forwarder.md diff --git a/docs/troubleshooting-rpc-forwarder.md b/docs/troubleshooting-rpc-forwarder.md new file mode 100644 index 00000000..d38e7d25 --- /dev/null +++ b/docs/troubleshooting-rpc-forwarder.md @@ -0,0 +1,122 @@ +# Troubleshooting RPC transaction forwarding + +Follower nodes use `--rpc.forwarder` to submit transactions that cannot be propagated locally. A healthy follower can serve read-only JSON-RPC calls while transaction submission still fails if the configured forwarder cannot be reached by the execution-layer process. + +This guide focuses on failures such as: + +```text +error code -32603: error sending request for url (...) +``` + +and log messages around `eth_sendRawTransaction` forwarding. + +## 1. Confirm the forwarder URL used by the node + +The standard testnet configuration uses: + +```sh +--rpc.forwarder https://rpc.quicknode.testnet.arc.network/ +``` + +Check the actual process arguments rather than assuming the service file was reloaded: + +```sh +ps -ef | grep '[a]rc-node-execution' +``` + +For systemd installations, also inspect the effective unit: + +```sh +systemctl cat arc-execution +systemctl show arc-execution -p ExecStart +``` + +After changing a unit file, run `systemctl daemon-reload` before restarting the service. + +## 2. Test JSON-RPC from the same host + +A successful TCP or TLS connection alone does not prove that the upstream accepts JSON-RPC. Send a real request from the node host: + +```sh +curl -sS -X POST https://rpc.quicknode.testnet.arc.network/ \ + -H 'content-type: application/json' \ + --data '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}' +``` + +For Arc Testnet the response should report chain ID `5042002` (`0x4cef52`). + +If this request fails, fix DNS, routing, TLS, proxy, or firewall policy before debugging `arc-node-execution`. + +## 3. Test from the execution process environment + +Environment differences are common when `curl` works in an interactive shell but the systemd service fails. Inspect the service environment and proxy variables: + +```sh +systemctl show arc-execution -p Environment +systemctl show arc-execution -p User -p Group +``` + +Pay particular attention to `HTTP_PROXY`, `HTTPS_PROXY`, `NO_PROXY`, `SSL_CERT_FILE`, and `SSL_CERT_DIR`. A service can have a different certificate store or proxy configuration than the login shell. + +To reproduce the request under the same service account, run: + +```sh +sudo -u "$(systemctl show -p User --value arc-execution)" \ + curl -sS -X POST https://rpc.quicknode.testnet.arc.network/ \ + -H 'content-type: application/json' \ + --data '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}' +``` + +If the unit does not set `User=`, run the check as the account that starts the node. + +## 4. Enable targeted debug logs + +Run the execution layer with debug logging for the RPC client while reproducing a single transaction submission: + +```sh +RUST_LOG='info,rpc::eth=debug,alloy_rpc_client=debug' arc-node-execution node ... +``` + +With systemd, temporarily add the same `RUST_LOG` value to the service environment and restart it. Avoid enabling broad trace logging on a busy public RPC endpoint because it can produce a large amount of output. + +Then submit one transaction and capture the log lines immediately before and after the forwarding failure: + +```sh +journalctl -u arc-execution --since '2 minutes ago' --no-pager +``` + +## 5. Separate local RPC health from forwarding health + +These checks answer different questions: + +```sh +# Local execution-layer RPC is alive +curl -sS -X POST http://127.0.0.1:8545 \ + -H 'content-type: application/json' \ + --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' + +# Upstream forwarder is reachable and speaks the expected chain +curl -sS -X POST https://rpc.quicknode.testnet.arc.network/ \ + -H 'content-type: application/json' \ + --data '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}' +``` + +A follower can pass the first check while failing the second. In that state, reads from the local verified chain can work even though `eth_sendRawTransaction` cannot be forwarded. + +## 6. Information to include in a bug report + +When the generic forwarding error remains after the checks above, include: + +- `arc-node-execution --version` and the exact commit or release tag; +- operating system and architecture; +- the `--rpc.forwarder` URL with credentials or API keys removed; +- whether the JSON-RPC `eth_chainId` curl succeeds under the service account; +- the relevant `RUST_LOG` lines from `rpc::eth` and `alloy_rpc_client`; +- whether the node is started interactively, through systemd, or in Docker; +- proxy and custom CA usage, without including secrets. + +Do not post private keys, bearer tokens, API keys, JWT secrets, or complete environment dumps containing credentials. + +## Related documentation + +See [Running an Arc Node](running-an-arc-node.md) for the standard follower-node configuration and the current testnet forwarder example. From d0aaa949db8768ab98dde0bd41c0b5c3ce8cbb98 Mon Sep 17 00:00:00 2001 From: murat <155527010+kejan2514@users.noreply.github.com> Date: Wed, 12 Aug 2026 22:47:15 +0300 Subject: [PATCH 2/6] chore: mark RPC forwarder troubleshooting contribution --- docs/.forwarder-troubleshooting-note | 1 + 1 file changed, 1 insertion(+) create mode 100644 docs/.forwarder-troubleshooting-note diff --git a/docs/.forwarder-troubleshooting-note b/docs/.forwarder-troubleshooting-note new file mode 100644 index 00000000..e1cc3206 --- /dev/null +++ b/docs/.forwarder-troubleshooting-note @@ -0,0 +1 @@ +This file intentionally left as a marker for the RPC forwarder troubleshooting contribution. From f79b7c796140eb1fcaf5ad885fec4fc8a65183b0 Mon Sep 17 00:00:00 2001 From: murat <155527010+kejan2514@users.noreply.github.com> Date: Wed, 12 Aug 2026 22:47:45 +0300 Subject: [PATCH 3/6] chore: remove temporary troubleshooting marker --- docs/.forwarder-troubleshooting-note | 1 - 1 file changed, 1 deletion(-) delete mode 100644 docs/.forwarder-troubleshooting-note diff --git a/docs/.forwarder-troubleshooting-note b/docs/.forwarder-troubleshooting-note deleted file mode 100644 index e1cc3206..00000000 --- a/docs/.forwarder-troubleshooting-note +++ /dev/null @@ -1 +0,0 @@ -This file intentionally left as a marker for the RPC forwarder troubleshooting contribution. From 194559e66126026f73390e4f21a94175c6aabdc6 Mon Sep 17 00:00:00 2001 From: murat <155527010+kejan2514@users.noreply.github.com> Date: Wed, 12 Aug 2026 22:47:56 +0300 Subject: [PATCH 4/6] tmp --- docs/.keep2 | 1 + 1 file changed, 1 insertion(+) create mode 100644 docs/.keep2 diff --git a/docs/.keep2 b/docs/.keep2 new file mode 100644 index 00000000..587be6b4 --- /dev/null +++ b/docs/.keep2 @@ -0,0 +1 @@ +x From 74514a346b2bb6e5087c14e9262e1a2d7a37dc3a Mon Sep 17 00:00:00 2001 From: murat <155527010+kejan2514@users.noreply.github.com> Date: Wed, 12 Aug 2026 22:48:18 +0300 Subject: [PATCH 5/6] chore: remove temporary file --- docs/.keep2 | 1 - 1 file changed, 1 deletion(-) delete mode 100644 docs/.keep2 diff --git a/docs/.keep2 b/docs/.keep2 deleted file mode 100644 index 587be6b4..00000000 --- a/docs/.keep2 +++ /dev/null @@ -1 +0,0 @@ -x From c9ea98bb5e72dc9983627149b6ed25bc03edeb46 Mon Sep 17 00:00:00 2001 From: murat <155527010+kejan2514@users.noreply.github.com> Date: Wed, 12 Aug 2026 23:12:35 +0300 Subject: [PATCH 6/6] docs: add idle-gap forwarder diagnostics --- docs/troubleshooting-rpc-forwarder.md | 33 ++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/docs/troubleshooting-rpc-forwarder.md b/docs/troubleshooting-rpc-forwarder.md index d38e7d25..5061f36a 100644 --- a/docs/troubleshooting-rpc-forwarder.md +++ b/docs/troubleshooting-rpc-forwarder.md @@ -103,7 +103,34 @@ curl -sS -X POST https://rpc.quicknode.testnet.arc.network/ \ A follower can pass the first check while failing the second. In that state, reads from the local verified chain can work even though `eth_sendRawTransaction` cannot be forwarded. -## 6. Information to include in a bug report +## 6. Check for idle connection-pool failures + +If the reachability, TLS, DNS, proxy, and service-environment checks above all pass, test whether the failure correlates with an idle gap before the forwarded write. + +One useful comparison is: + +1. Poll a read-only RPC such as `eth_blockNumber` every few seconds, then submit a transaction immediately. +2. Repeat the same test, but leave the node idle for roughly 30–90 seconds before submitting the transaction. + +If the immediate write succeeds while the post-idle write fails, the symptom is more consistent with reuse of a stale pooled connection than with basic upstream reachability. This behavior is being investigated in [#59](https://github.com/circlefin/arc-node/issues/59). + +When testing the upstream directly, also compare the default curl behavior with an explicit HTTP/1.1 request: + +```sh +# Default protocol negotiation +curl -sS -X POST https://rpc.quicknode.testnet.arc.network/ \ + -H 'content-type: application/json' \ + --data '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}' + +# Force HTTP/1.1 as a negative control +curl --http1.1 -sS -X POST https://rpc.quicknode.testnet.arc.network/ \ + -H 'content-type: application/json' \ + --data '{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}' +``` + +If both variants succeed while the follower still fails only after an idle period, capture the timing and debug logs and include them in the issue report. That evidence helps distinguish protocol-negotiation problems from connection-pool staleness. + +## 7. Information to include in a bug report When the generic forwarding error remains after the checks above, include: @@ -111,6 +138,8 @@ When the generic forwarding error remains after the checks above, include: - operating system and architecture; - the `--rpc.forwarder` URL with credentials or API keys removed; - whether the JSON-RPC `eth_chainId` curl succeeds under the service account; +- whether the failure changes after a 30–90 second idle gap; +- whether default curl and `curl --http1.1` behave differently; - the relevant `RUST_LOG` lines from `rpc::eth` and `alloy_rpc_client`; - whether the node is started interactively, through systemd, or in Docker; - proxy and custom CA usage, without including secrets. @@ -120,3 +149,5 @@ Do not post private keys, bearer tokens, API keys, JWT secrets, or complete envi ## Related documentation See [Running an Arc Node](running-an-arc-node.md) for the standard follower-node configuration and the current testnet forwarder example. + +For the live investigation into idle-time forwarding failures and connection-pool behavior, see [issue #59](https://github.com/circlefin/arc-node/issues/59).