Repository navigation
Conversation
6b2ba02 to
1478850
Compare
cdecker
left a comment
There was a problem hiding this comment.
Two minor things that I'd like to see before merging this:
- Blocklist any channel that we got an
unknown_next_peerfrom, otherwise we risk immediately retrying it, causing the tolerance to run through quicker than necessary. - Implement the
fee_insufficienterrorchannel_updateextraction, and add it togossip_store. This could already resolve the issue of repeatedfee_insufficienterrors, as we now get the real feerate right away, and can try it immediately again if the MCF solver wants to use it. Also that'd remove one "weird" error
All in all I feel like the weird error mechanism can stay, but the two examples that are being used here may not be good matches (unknown_next_peer is an indication that we haven't cleaned the gossip_store using block spends, and fee_insufficient may very well be just the result of discarding the channel_update rather than persisting it).
| /* One hit can be a stale scid, so disable that channel. | ||
| * The same node returning it again is excluded, or we | ||
| * walk every other channel through it. */ | ||
| add_result_summary(attempt, LOG_DBG, | ||
| "We got %s for %s: disabling it for this payment", | ||
| errmsg, | ||
| describe_scidd(attempt, index)); | ||
| maybe_exclude_unknown_next_peer(aux_cmd, attempt, index); | ||
| goto disable_channel; | ||
|
|
There was a problem hiding this comment.
We still need to absolutely avoid this channel at all costs, the node told us it's no longer there. By biasing the node rather than banning the channel we risk reusing it and causing the node to be banned on the next attempt due to a repeated error.
There was a problem hiding this comment.
I tested a payment to Phoenix wallet:
- got an invoice inv1 from phoenix, routehint is some LSP only scid (16Mx...x...) not revealing the onchain channel,
- turned off the wallet,
- called xpay on core-lightning, payment failed with "unknown_next_peer" at scid,
- created a layer "disable-unknown-next-peer",
- disabled scid on that layer,
- opened up Phoenix,
- retried xpay invoice=inv1 and layer=["disable-unknown-next-peer"], got failure "All 1 channels to the destination are disabled",
- asked Phoenix for a second invoice inv2, same scid in the routehint,
- retried xpay invoice=inv2 and layer=["disable-unknown-next-peer"], got failure "All 1 channels to the destination are disabled",
Lesson learned: If we permanently disable channels that have given us "unknown_next_peer" (who knows about other other permanent errors),
we will shoot ourselves in the foot.
I get the point of doing something permanent if we get a "perm" error message. But that has to be carefully crafted for every message
considering the behavior of the other implementations. Can we do that in a follow up PR?
nGoline
left a comment
There was a problem hiding this comment.
CI is red because of the PR itself, not flakes:
test_pay_exclude_nodefails in 4 jobs because the new "adding a global bias to channel ..." line goes throughadd_result_summary, so it ends up in the error string returned to the user. I would log those two bias messages withattempt_log(..., LOG_DBG, ...)instead; they are internal bookkeeping, not something apaycaller needs to read. That also keeps the second commit bisectable.check-doc-exampleswantsxpay-failing-node-tolerancebeforexpay-handle-payinlistconfigs.json;make update-doc-examplesshould sort it (and the msggen schema).
On behaviour:
- The default tolerance of 4 undoes the #9590 fix that is on master but not released yet: that one excluded the node on the second
unknown_next_peer. The #9590 test only passes now because it sets the tolerance to 2; with the default, xpay walks all four exits of the hub again, which is what the test docstring calls the bug. I would either default to 2, or keep the #9590 test on defaults and make sure it passes. - Agree with @cdecker that a final-only error (
incorrect_or_unknown_payment_details,mpp_timeout,final_incorrect_*) from an intermediate node should disable the node right away. The NODE-bit codes (temporary_node_failure,permanent_node_failure,required_node_feature_missing) fall in the same bucket: BOLT #4 says to remove all channels of the erring node, and today they need 4 distinct scids first. - The node bias threshold is hardcoded to 2 while the disable threshold is configurable, and a tolerance of 0 disables on the first error. Worth tying them together, or at least documenting it.
- Minor: in the second commit,
test_xpay_fee_insufficient_biassays "we should not need to disable the hub" but asserts that it was disabled.
On the two points in the review summary: for fee_insufficient, xpay already extracts the channel_update (process_channel_update_from_onion_error) and applies it to the payment's private layer, so only errors without a usable update reach this path. Putting it into gossip_store would go against BOLT #4, which says the sender MUST NOT apply an onion error channel_update to its local network graph. For unknown_next_peer, the channel is already disabled for the rest of the payment; your Phoenix experiment is a good argument against making that persistent, and I am fine with handling per-code permanent bans in a follow-up.
🤖 Generated with Claude Code
880df4a to
de36c81
Compare
The assertion is: |
This test wants to prove that we are able to identify repeated "unknown_next_peer" errors from a hub and disable it. I don't like to set the default tolerance to 2 because we might be disabling a bridge node to the destination, so first adding a bias to it I do agree that we could have a default tolerance of 3. |
Worth tying and documenting it. |
Agree. Will do. |
de36c81 to
c5c61ae
Compare
We already have a branch that detects a channel update in the onion error. If present this channel update will be used to update the |
|
Ah, ok, I had missed that is already being added to the gossip store, if we have one, and mixed the two cases with and without the update 👍 |
|
A drawback could be the fact that we don't actually add the channel update to the gossip store, but to askrene instead. I think the rationale is that we will eventually learn of that gossip message. |
c5c61ae to
5b3cfb9
Compare
Though, I am not sure if that branch is covered by the integration tests. |
nGoline
left a comment
There was a problem hiding this comment.
One thing in the last commit I would change before merge:
invalid_onion_version,invalid_onion_hmacandinvalid_onion_keyare BADONION codes. A forwarding node does not produce them about itself: it receivesupdate_fail_malformed_htlcfrom its next peer and turns it into anupdate_fail_htlcwith that code (BOLT #2), so we decrypt it at the forwarding node's index while the node that failed to parse the onion ishops[index].next_node. Before this commit we disabledhops[index], the channel into that node, which is the right target. Now the forwarding node is disabled for the payment and also gets a persistent -10 node bias in thexpaylayer, so any peer of a hub can push us away from that hub by returning a malformed-onion error. I would keep those three codes on the plain channel-disable path (or attribute them to the next node).invalid_onion_payloadis about the erring node's own payload, so disabling it right away is fine.
Smaller things, not blocking:
- You mentioned a default tolerance of 3 would be fine with you, so I would take that (doc, listconfigs example, and
test_xpay_fee_insufficient_disableasserting 4). With the #9590 test pinned at 2, nothing runs that scenario at the default. - The "adding a global bias to ..." lines still go through
add_result_summary, so they end up in the error returned bypay/xpay, which is whytest_pay_exclude_nodeneeded the.*.attempt_log(..., LOG_DBG, ...)would keep them out of the user-facing string. Sinceprocess_channel_update_from_onion_errornow uses the same helper for its private-layer -1 bias, that message also says "global" for a bias that only lives for this payment. - The node-bias threshold of
min(2, tolerance)and the meaning of tolerance 0 are explained in the commit message; a sentence inlightningd-config.5.mdwould help users who tune it.
We could do this a follow up PR. I suppose that for invalid_onion_key and invalid_onion_payload the culprits reduce to: either A messing up the onion (again unlikely), or C trolling us, because at this point B cannot be blamed because he deliver the encrypted payload with the correct HMAC. |
nGoline
left a comment
There was a problem hiding this comment.
Agreed that pair tracking is the better model and fine as a follow-up. One correction on the split: invalid_onion_key belongs with version and hmac, not with payload. The ephemeral key in the update_add_htlc onion is computed by the forwarding node, so the node before the one that reports it can produce an unparsable key just as well as a bad HMAC. All three BADONION codes reach us through update_fail_malformed_htlc from the next peer, so the suspects are always the pair. invalid_onion_payload is the only one the erring node sends about itself, so disabling that node right away is right.
For this PR I'd keep invalid_onion_version, invalid_onion_hmac and invalid_onion_key on the path master uses today (disable the channel, no node disable and no persistent node bias), so the follow-up adds pair tracking without this PR making it easier for a peer to push us away from a hub in the meantime. It also needs a rebase, doc/schemas/listconfigs.json and tests/test_pay.py conflict with master.
🤖 Generated with Claude Code
Node and channel bias add up but they never should exceed the [-100, 100] range Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
5b3cfb9 to
bcc711e
Compare
bcc711e to
660f249
Compare
|
Thanks for the quick rebase, it merges cleanly and CI is green. Sorry, I made it sound like a rebase was all that was left, but one point from the earlier review is still open. It's the three onion-error codes: If that's in, I'm happy to approve and we can get it into the next RC. Optional: the askrene clamp commit has no Changelog line. Something like 🤖 Generated with Claude Code |
We have had reports of payment failures by xpay timeout after multiple routes have been tried over the same node with weird errors like unknown_next_peer and fee_insufficient. These nodes are usually well connected hubs with hundreds of channels and they tend to be preferred by getroutes even after the first erring channels are disabled. Naturally, we only disable channels for the current payment session, so next time a second payment prefers that route it will fail as well. To fix this we add a negative bias on the failing channels that persist across payments and if we detect a pattern in the failing node we disable it completely on the current session and add a persistent bias to it. When receiving one of the three payment errors invalid_onion_version, invalid_onion_key or invalid_onion_hmac we do not penalize the node sending the error, since it is sent on behalf of the next peer, we just disable the channel. Changelog-Fixed: xpay: avoid channel and nodes that fail too often with weird errors Reported-by: @blockdyor and @ErikDeSmedt Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Changelog-None Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
We disable a node for the current payment session if the number of weird failures is equal to the xpay-failing-node-tolerance. For repeated failing nodes we also apply a global negative bias to avoid that node on other payments. The bias is by default applied after the first 2 failures and it accumulates. If xpay-failing-node-tolerance is less than 2 we apply the bias the moment we disable the node. Changelog-Added: xpay: add xpay-failing-node-tolerance option to set the number of weird failures that trigger a node to be disabled during the current payment session. Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
Changelog-Fixed: xpay: instantly disable nodes that return plain wrong or impossible to recover error codes, eg. incorrect_or_unknown_payment_details from an intermediate node or permanent_node_failure, the disable operation applies to the current payment session and in addition a negative bias is applied to this node for all subsequent payments. Signed-off-by: Lagrang3 <lagrang3@protonmail.com>
4b66db2 to
4beb81c
Compare
This is a follow-up PR from #9592.
We keep a record of rare errors from nodes during payment attempts.
Repeated strange errors trigger a negative bias that is applied to the node and if the errors keep coming
above a certain tolerance counter we disable the node for the current payment session.
It addresses a known issue of xpay retrying payment attempts through the same node even after
repeated unlikely error messages: #8600, #9590