Core Lightning, send always fails

paulscode · October 18, 2024, 4:25pm

Hello, I am using the latest version of the Core Lightning service, and it has worked on the past (both sending and receiving). For the last couple of weeks or so, however, I have been unable to send to any invoice, even small amounts. I have active channels and plenty of outbound liquidity.

The send always runs out of routes after a few dozen attempts of so, the payment deadline expires, and it stops retrying. I’ve tried opening new channels with multiple popular nodes (Boltz, Voltage, ACINQ, 1ML, etc), closing channels and opening new ones, and deleting the gossip store. Nothing seems to get it into a state where I can send.

The log shows lots of errors like: (not sure if these are relevant, or expecting in normal operation)

htlc XX failed from Xth node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)

and

BROKEN plugin-topology: DEPRECATED API USED: listchannels.include_private by "pay:listchannels#XXX/cln:listchannels#XXXX"

My next step will probably be to close all channels and clear the balance, so that I can do a full uninstall and reinstall of the service from scratch, but thought I’d see if anyone has some less drastic things I might test, or if anyone has encountered this before.

h0mer · October 18, 2024, 8:44pm

Hey paulscode!

Sorry to hear you’re running into this issue, and thanks for the detailed info! Looks like its a known issue and fix is incoming , and it seems like it’s a known problem that will likely be fixed in an upcoming version. I would advice to wait before nuking your CLN.

paulscode · October 19, 2024, 2:40am

Thanks, I noticed a new version just dropped. I’ll try it out when I get back home. Fingers crossed

m1k_3 · October 21, 2024, 12:38pm

I am having a similar issue. Core Lightning is failing to send. I just setup the node yesterday and am running version 24.8.0.1. I am not seeing the listchannls error that paulscode is seeing. I have a channel open with bolz and in my last attempt I see:

2024-10-21T08:02:54-04:00  2024-10-21T12:02:54.883Z INFO 026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2-chan#2: htlc 11 failed from 1th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
<snip>
2024-10-21T08:03:49-04:00  2024-10-21T12:03:49.210Z INFO plugin-clnrest: RPC Error: {'code': 210, 'message': 'Ran out of routes to try after 8 attempts: see `paystatus`', 'attempts': [{'status': 'failed', 'failreason': 'failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)', 'partid': 0, 'amount_msat': 10000000}, {'status': 'failed', 'failreason': 'failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)', 'partid': 1, 'amount_msat': 10000000, 'parent_partid': 0}, {'status': 'failed', 'failreason': 'failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)', 'partid': 2, 'amount_msat': 10000000, 'parent_partid': 1}, {'status': 'failed', 'failreason': 'failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)', 'partid': 3, 'amount_msat': 10000000, 'parent_partid': 2}, {'status': 'failed', 'failreason': 'failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)', 'partid': 4, 'amount_msat': 10000000, 'parent_partid': 3}, {'status': 'failed', 'failreason': 'failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)', 'partid': 5, 'amount_msat': 10000000, 'parent_partid': 4}, {'status': 'failed', 'failreason': 'failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)', 'partid': 6, 'amount_msat': 10000000, 'parent_partid': 5}, {'status': 'failed', 'failreason': 'failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)', 'partid': 7, 'amount_msat': 10000000, 'parent_partid': 6}]}

paulscode · October 21, 2024, 10:32pm

Sadly, I am also still not able to send even after updating to 24.08.1. The update resolved the deprecated API used message, but that doesn’t seem to have been the cause. I’m still consistently getting the code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE) in the logs and eventually runs out of routes to try, payment deadline expires, and it stops retrying.

m1k_3 · October 21, 2024, 11:37pm

Someone on NOSTR told me Boltz does not route payments. That is who I had my channel open to as well. I am not sure if this valid or not.

paulscode · October 23, 2024, 12:27am

User @annie just sent me a physhing DM referencing this topic. Not sure how to report it, so mentioning it here.

paulscode · October 23, 2024, 11:13am

Good catch. By chance, I had closed all of my channels except for Boltz prior to upgrading to 24.08.1 while trying different combinations of channels with different nodes. After reading your comment, I closed my channel with Boltz and opened one with Voltage, and I am able to successfully send payments now.

TLDR; Upgrade to 24.08.1 to fix DEPRECATED API error, and open a channel with a node which routes payments.

m1k_3 · October 24, 2024, 1:41am

My issues persist. I think its related to this

github.com/ElementsProject/lightning

pay: Remember and update channel_hints across payments

ElementsProject:master ← ElementsProject:202407-pay-channel-hint-update

opened 02:47PM - 25 Jul 24 UTC

cdecker

+656 -261

Builds on top of #7487. This PR takes the `channel_hint`s that were previousl…y created, updated and then discarded as part of a payment, and instead we are making them long-lived. This means that we share what we learn from one payment with all later payments, allowing us to skip attempts that will likely fail anyway, and improve the overall performance (time-to-completion as well as success rate of payments). The downside of this is that we are now keeping observations for prolonged periods, meaning that these observations are no longer valid when we use them. If used unchanged these earlier observations could lead us to skip routes that we deem unavailable due to a prior observation, but which is now back to being usable, which in turn impacts our ability to complete a payment successfully. The core contribution in this PR therefore is the time-based relaxation of learnt constraints. It is based on a leaky-bucket approach, with a linear refill-rate for depleted channels, such that after 2h we consider any observation outdated. ## Example If at time `t` we observe a channel `c` with `cap_c` as overall capacity, and `bal_c = 0.5 * cap_c` the currently available balance (i.e., we observed a payment of `bal_c + 1` fail to generate this hint), then we have a refill rate `r = cap_c / 7200`, and after 1h the channel is considered to be fully available again, i.e., we don't have a more restrictive observation, and it is old enough that it does not add to the structural information we already have from the gossip. ```mermaid graph TD; CLN7487 --> CLN7494 ; click CLN7487 "https://github.com/ElementsProject/lightning/pull/7487" click CLN7494 "https://github.com/ElementsProject/lightning/pull/7494" style CLN7494 fill:#AAFFAA ```

There are some pay issues fixed in v24.08.2.

When should we expect v24.08.2 of core lighnting to be available in the marketplace?

paulscode · October 24, 2024, 2:49am

Thanks for linking that. It led me to this issue, which is still open (based on dates in the conversation, I suspect 24.08.2 doesn’t actually resolve it, sadly). This is a long-standing issue (last working version supposedly being 23.11.2 from last December, based on the conversation), so it must be intermittent enough not to be reported very frequently, given that me posting this thread didn’t trigger an instant link to the Github issue

github.com/ElementsProject/lightning

Core Lightning is incompatible with the Lightning Network - lacks support for payments

opened 03:06PM - 27 Mar 24 UTC

nakoshi-satamoto

I am not sure if it is just me or others also, but I have noticed that ever sinc…e updating to version 24 most payments fail. I get alot of these messages, the htlc and node number varies though. `htlc 16 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)` then payment deadline expires. I have even attempted to pay various invoices even with small payments, I also have some large channels (millions of sats) open with large and well connected nodes. Posting this for visibility in case if others are experiencing this also. If there is something I can check to confirm if it is an issue with my node or corelightning I can do so.

m1k_3 · October 28, 2024, 9:04pm

I have had some success getting payments to work with cli and using the renepay plugin available in CLN. Can anyone provide hints how I can install a plugin in the c-lightning.emabassy container which makes renepay the default pay method.

paulscode · October 28, 2024, 10:23pm

There is probably an easier process, but one potential option would be to fork the project and build a custom s9pk. You’d need to edite the Dockerfile to copy over the renepay Python file and give it execute permission. Then you’d either update the lightningd call or add an additional lightning-cli call (depending on how that plugin is loaded)