lightningd: don't force-close when fulfilled HTLC removal is in progress#8940
Open
vincenzopalazzo wants to merge 2 commits intoElementsProject:masterfrom
Open
Conversation
vincenzopalazzo
force-pushed
the
test/reproduce-issue-8899-fulfilled-htlc-deadline
branch
from
d6d8693 to
e316362
Compare
vincenzopalazzo
changed the title
…VE_HTLC deadline force-close Add test_fulfilled_htlc_deadline_no_force_close to reproduce the bug where CLN force-closes a channel with "Fulfilled HTLC SENT_REMOVE_HTLC cltv hit deadline" even though it has the preimage and just needs to reconnect to send update_fulfill_htlc upstream. The test sets up l1->l2->l3, sends a payment, and disconnects l2 from l1 right before update_fulfill_htlc is sent (-WIRE_UPDATE_FULFILL_HTLC). This leaves the incoming HTLC on the l1-l2 channel stuck in SENT_REMOVE_HTLC (or SENT_REMOVE_COMMIT under Valgrind). Mining blocks to the deadline triggers the buggy force-close. Reproduces: ElementsProject#8899 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
vincenzopalazzo
force-pushed
the
test/reproduce-issue-8899-fulfilled-htlc-deadline
branch
from
3ad64fb to
cbcc032
Compare
vincenzopalazzo
changed the title
vincenzopalazzo
force-pushed
the
test/reproduce-issue-8899-fulfilled-htlc-deadline
branch
from
4cf3776 to
066b3c7
Compare
When an incoming HTLC has been fulfilled (preimage known) and is in SENT_REMOVE_HTLC or later state, the removal is already in progress: channeld has been told to send update_fulfill_htlc upstream (or will be told on reconnect). Force-closing the channel is counterproductive because: 1. The preimage is persisted to DB and onchaind can claim on-chain 2. The cooperative path (reconnect + fulfill) is cheaper and faster 3. If the peer goes on-chain themselves, onchaind handles it Instead of force-closing, log a warning and let the removal complete through the normal state machine. CI run of the reproducer test (cbcc032) confirms the bug exists. Before this fix, the force-close fires: lightningd: Peer permanent failure in CHANNELD_NORMAL: Fulfilled HTLC 0 SENT_REMOVE_HTLC cltv 119 hit deadline After this fix, the force-close is skipped and a warning is logged: lightningd: UNUSUAL: Fulfilled HTLC 0 SENT_REMOVE_HTLC cltv 119 hit deadline, but removal already in progress Also updates test_htlc_no_force_close and test_htlc_in_timeout which depended on the old force-close behavior: l3/l2 no longer force-close for fulfilled HTLCs past deadline; instead the offering peer (l2/l1) force-closes for the offered HTLC timeout, and the fulfilling node claims on-chain via onchaind using the preimage. Changelog-Fixed: Don't force-close channel when fulfilled HTLC hits deadline but removal is already in progress. Fixes: ElementsProject#8899 Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
vincenzopalazzo
force-pushed
the
test/reproduce-issue-8899-fulfilled-htlc-deadline
branch
from
5ec99ea to
c95850f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #8899: CLN force-closes a channel with
Fulfilled HTLC SENT_REMOVE_HTLC cltv hit deadlineeven though the node has the preimage and the fulfill is already in progress.The bug
In
htlcs_notify_new_block(), when a fulfilled incoming HTLC hits the cltv deadline, CLN force-closes the channel without checking whether the HTLC removal is already in progress. The HTLC state machine shows the fulfill was queued to channeld (SENT_REMOVE_HTLC), but the force-close fires before the upstream peer receivesupdate_fulfill_htlc.This was reported in the wild: the upstream peer was connected and exchanging pings, but CLN force-closed instead of sending the fulfill message.
The fix
When the HTLC is in
SENT_REMOVE_HTLCor later state (>= SENT_REMOVE_HTLC), skip the force-close and log a warning instead. This is safe because:onchaindcan claim on-chain if neededupdate_fulfill_htlconchaindhandles it with the known preimageCommits
Fulfilled HTLC $ID SENT_REMOVE_HTLC cltv $CLTV hit deadlinewithout attempt to claim #8899 — Test that triggers the buggy force-close (l2 disconnects from l1 before sendingupdate_fulfill_htlc, then mining blocks to the deadline)hin->hstate >= SENT_REMOVE_HTLC, loglog_unusualinsteadChangelog-Fixed: Don't force-close channel when fulfilled HTLC hits deadline but removal is already in progress.
Test plan
test_fulfilled_htlc_deadline_no_force_closereproduces the bug (first commit)🤖 Generated with Claude Code