Hello, I am using the latest version of the Core Lightning service, and it has worked on the past (both sending and receiving). For the last couple of weeks or so, however, I have been unable to send to any invoice, even small amounts. I have active channels and plenty of outbound liquidity.
The send always runs out of routes after a few dozen attempts of so, the payment deadline expires, and it stops retrying. I’ve tried opening new channels with multiple popular nodes (Boltz, Voltage, ACINQ, 1ML, etc), closing channels and opening new ones, and deleting the gossip store. Nothing seems to get it into a state where I can send.
The log shows lots of errors like: (not sure if these are relevant, or expecting in normal operation)
htlc XX failed from Xth node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
and
BROKEN plugin-topology: DEPRECATED API USED: listchannels.include_private by "pay:listchannels#XXX/cln:listchannels#XXXX"
My next step will probably be to close all channels and clear the balance, so that I can do a full uninstall and reinstall of the service from scratch, but thought I’d see if anyone has some less drastic things I might test, or if anyone has encountered this before.
Sorry to hear you’re running into this issue, and thanks for the detailed info! Looks like its a known issue and fix is incoming , and it seems like it’s a known problem that will likely be fixed in an upcoming version. I would advice to wait before nuking your CLN.
I am having a similar issue. Core Lightning is failing to send. I just setup the node yesterday and am running version 24.8.0.1. I am not seeing the listchannls error that paulscode is seeing. I have a channel open with bolz and in my last attempt I see:
2024-10-21T08:02:54-04:00 2024-10-21T12:02:54.883Z INFO 026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2-chan#2: htlc 11 failed from 1th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
<snip>
2024-10-21T08:03:49-04:00 2024-10-21T12:03:49.210Z INFO plugin-clnrest: RPC Error: {'code': 210, 'message': 'Ran out of routes to try after 8 attempts: see `paystatus`', 'attempts': [{'status': 'failed', 'failreason': 'failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)', 'partid': 0, 'amount_msat': 10000000}, {'status': 'failed', 'failreason': 'failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)', 'partid': 1, 'amount_msat': 10000000, 'parent_partid': 0}, {'status': 'failed', 'failreason': 'failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)', 'partid': 2, 'amount_msat': 10000000, 'parent_partid': 1}, {'status': 'failed', 'failreason': 'failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)', 'partid': 3, 'amount_msat': 10000000, 'parent_partid': 2}, {'status': 'failed', 'failreason': 'failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)', 'partid': 4, 'amount_msat': 10000000, 'parent_partid': 3}, {'status': 'failed', 'failreason': 'failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)', 'partid': 5, 'amount_msat': 10000000, 'parent_partid': 4}, {'status': 'failed', 'failreason': 'failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)', 'partid': 6, 'amount_msat': 10000000, 'parent_partid': 5}, {'status': 'failed', 'failreason': 'failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)', 'partid': 7, 'amount_msat': 10000000, 'parent_partid': 6}]}
Sadly, I am also still not able to send even after updating to 24.08.1. The update resolved the deprecated API used message, but that doesn’t seem to have been the cause. I’m still consistently getting the code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE) in the logs and eventually runs out of routes to try, payment deadline expires, and it stops retrying.
Good catch. By chance, I had closed all of my channels except for Boltz prior to upgrading to 24.08.1 while trying different combinations of channels with different nodes. After reading your comment, I closed my channel with Boltz and opened one with Voltage, and I am able to successfully send payments now.
TLDR; Upgrade to 24.08.1 to fix DEPRECATED API error, and open a channel with a node which routes payments.
Thanks for linking that. It led me to this issue, which is still open (based on dates in the conversation, I suspect 24.08.2 doesn’t actually resolve it, sadly). This is a long-standing issue (last working version supposedly being 23.11.2 from last December, based on the conversation), so it must be intermittent enough not to be reported very frequently, given that me posting this thread didn’t trigger an instant link to the Github issue
I have had some success getting payments to work with cli and using the renepay plugin available in CLN. Can anyone provide hints how I can install a plugin in the c-lightning.emabassy container which makes renepay the default pay method.
There is probably an easier process, but one potential option would be to fork the project and build a custom s9pk. You’d need to edite the Dockerfile to copy over the renepay Python file and give it execute permission. Then you’d either update the lightningd call or add an additional lightning-cli call (depending on how that plugin is loaded)
Closing all channels and starting from scratch did not resolve the issue. After that, I started with only one channel with Boltz, and it was having the same error in the logs. After closing that one channel, and opening channels with other nodes (initially ACINQ, and since a number of others) I have not seen the issue. That said, I use this node now only for mining on Ocean and the occasional Robosats purchase. Most of my Lightning usage I have moved to LND and various services integrated with it. LND has been considerably more reliable that Core Lightning in my experience, and after moving away from only interacting with it via the atrocious Ride the Lightning interface, I’ve found the inferior user experience compared to Core Lightning to be acceptable.
Not sure who to reach out to for help. I have been using your Paulscode as my active channel as part of the ocean / datum setup with my bitaxe units. Unfortunately, I wasn’t using core lightning much these past few months. After reopening core lightning, it took a while to resync to the active channel status. The inbound liquidity was very low, so I was not receiving payments. I tried to send some stats to another wallet but that was unsuccessful. I then ended up setting up two new active channels with Paulscode. Still unable to send payments to another wallet to correct the inbound liquidity issue.
Do you have any suggestions? Is there a better way to chat with you? Thanks so much!
Yes, that node is unfortunately plagued by Core Lightning’s continuing issues with routing (the situation is improved, but I still have problems where I need to shut down, clear the gossip store, start back up, and wait for it to re-sync). I recommend opening a channel with my LND node (called PaulsCode LND). I’m keeping the other one up, just because I need it for Ocean payouts, but the LND node is now my primary one (and I’ve not seen the routing issues with it) If you need the funds back that you locked up in the other two channels and don’t want to wait for Core Lightning to improve, go ahead and close them (unfortunately it will cost you 2 base layer transaction fees, so if you are not in a hurry, I would time it for when the backlog is low and fees are 1 sat/vB)
Do note that when you first set up your first channel, or clear your gossip store, you will have to wait for the gossip to sync again, which can take a few hours depending on various factors.
If you still, after connecting to PaulsCode LND and waiting for the gossip store to sync, are not able to send, just let me know the name of your node, and I’ll open a channel back to you so you have some inbound liquidity for Ocean. That way you can just leave Core Lighting for your Ocean payouts, and move to LND if you want to get more serious with Lightning while waiting for Core Lightning to improve their solution.
Wow, Paul, thanks so much for your prompt response! As you can tell, I’m relatively new to all of this even though I successfully setup Datum thanks to your YouTube video. As it currently stands, I have the first active channel I set up with Paulscode (for my Ocean payouts) which has minimal inbound liquidity and 2 new Active channels with Paulscode with zero inbound liquidity.
Questions:
What is the gossip store?
If I close the 2 new channels, where are the existing funds sent? Or are they lost?
Since my first active channel is no longer receiving Ocean payouts due to the inbound liquidity issue, should I close it also? If I do close it, what will happen to the pending Ocean payouts and current fund balance?
When I open a new channel with PaulsCode LND, is that done through Core Lightning or do I have to set up LND on my Start9?
After the new channel setup, will I need to redirect the Bitaxes?
My understanding (I’m by no means an expert on this) is that this is information about the Lightning network’s topology which is forwarded around the network among the nodes, and used by your node to construct a route when you send a payment.
The funds land back in the base layer wallet in your node. From there you can send them on the base layer to somewhere else, or use them to open different channels.
For this, let me know the name of your node, and I’ll open a channel to you so you have inbound liquidity. Then you can close the current one too, along with the others.
Ocean will keep retrying, and they will hold onto your balance in the mean time until it reaches their minimum threshold for a base layer transaction. If they still can’t route your payout over lightning at that point, then they will send your balance to your base layer address (the one which is your Ocean account name). If I open a channel to your node, then you will have inbound liquidity again, and you’ll start receiving your payouts once again.
I think rather than doing this (since your goal is inbound liquidity), the easiest way would be for me to open a channel to you. However, if you don’t want to share your node name (though I could probably guess since you mentioned having 3 channels to my node…) then the idea would be from Core Lightning to open a channel with PaulsCode LND and send some sats out (hopefully unblocked this time) to create inbound liquidity for your Ocean payouts.
No, they connect to your Datum Gateway service, which wouldn’t be changing in either scenario.
