CLN recovery being invoked

RustyRussell · April 12, 2024, 7:29am

Hi! I have a CLN bug report for stuck funds, and the user supplied logs. Looks like the recover command is being invoked? Where do I look at to see how you run CLN, so I can see how (and why?) this is being called?

I’m hoping we can recover the funds, but I’m not sure why it would be in this state…

StuPleb · April 13, 2024, 5:42pm

Hi Rusty,

We have a user recently in our Telegram who specifically added disable-plugin=recover to their config, in an attempt to follow:

github.com/ElementsProject/lightning

Upgrading BROKEN wallet/wallet.c:2594: INSERT INTO channels

opened 06:02PM - 28 Mar 24 UTC

JavierRSobrino

in diagnostic

After upgrading from 23.11.2 to 24.02.1, I obtain the following error: ``` l…ightningd_build | 2024-03-28T17:27:39.815321629Z Error executing statement: wallet/wallet.c:2594: INSERT INTO channels ( peer_id, first_blocknum, id, revocation_basepoint_local, payment_basepoint_local, htlc_basepoint_local, delayed_payment_basepoint_local, funding_pubkey_local, require_confirm_inputs_remote, require_confirm_inputs_local) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?);: UNIQUE constraint failed: channels.id lightningd_build | 2024-03-28T17:27:39.815333462Z 2024-03-28T17:27:39.815Z **BROKEN** lightningd: Error executing statement: wallet/wallet.c:2594: INSERT INTO channels ( peer_id, first_blocknum, id, revocation_basepoint_local, payment_basepoint_local, htlc_basepoint_local, delayed_payment_basepoint_local, funding_pubkey_local, require_confirm_inputs_remote, require_confirm_inputs_local) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?);: UNIQUE constraint failed: channels.id ``` I had to downgrade and restore the backup I made just before updating in order for my node to continue operating.

github.com/ElementsProject/lightning

recover: The recovery plugin crashes if a channel has not fully been forgotten

opened 12:16AM - 21 Mar 24 UTC

nakoshi-satamoto

plugin::recovery

## New description The `recover` plugin appears to crash if we had dataloss, …but the database still contains a trace of the channel, causing the insert to fail during the recovery. ## Original issue ### Issue and Steps to Reproduce My system to froze up and was non-responsive. I had to do a forced reboot of the computer. lightningd won't even start up now. I'm on version 24.02, I even tried reverting back to the old version 23.11.2 and still no luck with starting up this node. I even re-extracted clightning-v24.02-Ubuntu-22.04.tar.xz and tried running v24.02 again and yet no luck. I cannot run the newer version 24.0.2.1 because that is not signed with any valid known signature. So I am on 24.0.2 as the latest version. Yes I was mirroring the sqlite database but that is pointless if the mirror gets corrupted also, if this is a case of database corruption. Below is the output I get when running in debug mode. While trying to keep starting up core lightning I noticed one of my channels with alot of funds got forced closed. The chan that got forced closed was 831793x2527x0 (boltz) with closure transaction ID being e6df69daa0eefd910d871601bbf0e78e838b6396c79601a28dbbf31bb8312591 I am scared that these funds have been lost forever, if this was a penalty closure, or if this was a force closure and my lightning node lost the information needed to claim the funds after the wait time. At this point, I'm done with lightning. Too many issues over my time with running core lightning. I just want to recover my funds. I'm okay if I have to force close all my chans if it means I get my funds back. **1. How can I get core lightning working again, or recover my funds?** **2. Assuming the database is corrupted and that chan was indeed forced closed, am I able to claim the funds from the forced close script after the wait time?** ### Output from lightningd in debug mode ``` 2024-03-21T00:02:54.885Z INFO lightningd: v24.02e6df69daa0eefd910d871601bbf0e78e838b6396c79601a28dbbf31bb8312591 2024-03-21T00:02:54.941Z DEBUG lightningd: Opened log file /home/user/.lightning/ln-log.txt 2024-03-21T00:02:54.947Z DEBUG plugin-manager: started(32757) /home/user/apps/clightning/clightning-24.02/usr/libexec/c-lightning/plugins/autoclean 2024-03-21T00:02:54.951Z DEBUG plugin-manager: started(32758) /home/user/apps/clightning/clightning-24.02/usr/libexec/c-lightning/plugins/chanbackup 2024-03-21T00:02:54.956Z DEBUG plugin-manager: started(32759) /home/user/apps/clightning/clightning-24.02/usr/libexec/c-lightning/plugins/bcli 2024-03-21T00:02:54.960Z DEBUG plugin-manager: started(32760) /home/user/apps/clightning/clightning-24.02/usr/libexec/c-lightning/plugins/commando 2024-03-21T00:02:54.965Z DEBUG plugin-manager: started(32761) /home/user/apps/clightning/clightning-24.02/usr/libexec/c-lightning/plugins/fetchinvoice 2024-03-21T00:02:54.970Z DEBUG plugin-manager: started(32762) /home/user/apps/clightning/clightning-24.02/usr/libexec/c-lightning/plugins/funder 2024-03-21T00:02:54.976Z DEBUG plugin-manager: started(32763) /home/user/apps/clightning/clightning-24.02/usr/libexec/c-lightning/plugins/topology 2024-03-21T00:02:54.979Z DEBUG plugin-manager: started(32764) /home/user/apps/clightning/clightning-24.02/usr/libexec/c-lightning/plugins/keysend 2024-03-21T00:02:54.983Z DEBUG plugin-manager: started(32765) /home/user/apps/clightning/clightning-24.02/usr/libexec/c-lightning/plugins/offers 2024-03-21T00:02:54.985Z DEBUG plugin-manager: started(32766) /home/user/apps/clightning/clightning-24.02/usr/libexec/c-lightning/plugins/pay 2024-03-21T00:02:54.988Z DEBUG plugin-manager: started(32767) /home/user/apps/clightning/clightning-24.02/usr/libexec/c-lightning/plugins/recover 2024-03-21T00:02:54.992Z DEBUG plugin-manager: started(32768) /home/user/apps/clightning/clightning-24.02/usr/libexec/c-lightning/plugins/txprepare 2024-03-21T00:02:54.995Z DEBUG plugin-manager: started(32769) /home/user/apps/clightning/clightning-24.02/usr/libexec/c-lightning/plugins/cln-renepay 2024-03-21T00:02:55.001Z DEBUG plugin-manager: started(32770) /home/user/apps/clightning/clightning-24.02/usr/libexec/c-lightning/plugins/spenderp 2024-03-21T00:02:55.007Z DEBUG plugin-manager: started(32771) /home/user/apps/clightning/clightning-24.02/usr/libexec/c-lightning/plugins/sql 2024-03-21T00:02:55.011Z DEBUG plugin-manager: started(32772) /home/user/apps/clightning/clightning-24.02/usr/libexec/c-lightning/plugins/cln-grpc 2024-03-21T00:02:55.015Z DEBUG plugin-manager: started(32774) /home/user/apps/clightning/clightning-24.02/usr/libexec/c-lightning/plugins/bookkeeper 2024-03-21T00:02:55.019Z DEBUG plugin-manager: started(32775) /home/user/apps/clightning/clightning-24.02/usr/libexec/c-lightning/plugins/clnrest/clnrest.py 2024-03-21T00:02:55.051Z INFO plugin-clnrest.py: Killing plugin: disabled itself: No module named 'gevent' 2024-03-21T00:02:55.051Z DEBUG lightningd: io_break: check_plugins_manifests 2024-03-21T00:02:55.051Z DEBUG lightningd: io_loop_with_timers: plugins_init 2024-03-21T00:02:55.052Z DEBUG lightningd: testing /home/user/apps/clightning/clightning-24.02/usr/libexec/c-lightning/lightning_channeld 2024-03-21T00:02:55.053Z DEBUG lightningd: testing /home/user/apps/clightning/clightning-24.02/usr/libexec/c-lightning/lightning_closingd 2024-03-21T00:02:55.054Z DEBUG lightningd: testing /home/user/apps/clightning/clightning-24.02/usr/libexec/c-lightning/lightning_connectd 2024-03-21T00:02:55.055Z DEBUG lightningd: testing /home/user/apps/clightning/clightning-24.02/usr/libexec/c-lightning/lightning_gossipd 2024-03-21T00:02:55.056Z DEBUG lightningd: testing /home/user/apps/clightning/clightning-24.02/usr/libexec/c-lightning/lightning_hsmd 2024-03-21T00:02:55.057Z DEBUG lightningd: testing /home/user/apps/clightning/clightning-24.02/usr/libexec/c-lightning/lightning_onchaind 2024-03-21T00:02:55.058Z DEBUG lightningd: testing /home/user/apps/clightning/clightning-24.02/usr/libexec/c-lightning/lightning_openingd 2024-03-21T00:02:55.059Z DEBUG hsmd: pid 32783, msgfd 52 2024-03-21T00:02:55.060Z DEBUG hsmd: capability +WIRE_HSMD_CHECK_PUBKEY 2024-03-21T00:02:55.060Z DEBUG hsmd: capability +WIRE_HSMD_SIGN_ANY_DELAYED_PAYMENT_TO_US 2024-03-21T00:02:55.060Z DEBUG hsmd: capability +WIRE_HSMD_SIGN_ANCHORSPEND 2024-03-21T00:02:55.060Z DEBUG hsmd: capability +WIRE_HSMD_SIGN_HTLC_TX_MINGLE 2024-03-21T00:02:55.060Z DEBUG hsmd: capability +WIRE_HSMD_SIGN_SPLICE_TX 2024-03-21T00:02:55.060Z DEBUG hsmd: capability +WIRE_HSMD_CHECK_OUTPOINT 2024-03-21T00:02:55.060Z DEBUG hsmd: capability +WIRE_HSMD_FORGET_CHANNEL 2024-03-21T00:02:55.060Z DEBUG hsmd: capability +WIRE_HSMD_REVOKE_COMMITMENT_TX 2024-03-21T00:02:56.333Z DEBUG connectd: pid 32784, msgfd 56 2024-03-21T00:02:56.334Z DEBUG hsmd: Client: Received message 27 from client 2024-03-21T00:02:56.334Z DEBUG hsmd: Client: Received message 27 from client 2024-03-21T00:02:56.334Z DEBUG hsmd: new_client: 0 2024-03-21T00:02:56.335Z DEBUG 038e5677dc7d1ec4e9a82bd004d524b2469fedb0867e85bd1e2f5d3352957e03b7-hsmd: Got WIRE_HSMD_ECDH_REQ 2024-03-21T00:02:56.335Z DEBUG hsmd: Client: Received message 1 from client 2024-03-21T00:02:56.582Z DEBUG connectd: Proxy address: 127.0.0.1:9050 2024-03-21T00:02:56.582Z DEBUG connectd: Created listener on 127.0.0.1:9735 2024-03-21T00:02:56.582Z INFO connectd: Static Tor service onion address: "d62gy2sc4tspad44resx6ylanagxo23sismn3wkczodl2aorn24y6pyd.onion:9735,127.0.0.1:9735" bound from extern port 9735 2024-03-21T00:02:56.582Z DEBUG connectd: REPLY WIRE_CONNECTD_INIT_REPLY with 0 fds 2024-03-21T00:02:56.582Z DEBUG lightningd: io_break: connect_init_done 2024-03-21T00:02:56.582Z DEBUG lightningd: io_loop: connectd_init 2024-03-21T00:02:56.607Z INFO plugin-bcli: bitcoin-cli initialized and connected to bitcoind. 2024-03-21T00:02:56.608Z DEBUG lightningd: io_break: plugin_config_cb 2024-03-21T00:02:56.608Z DEBUG lightningd: io_loop_with_timers: config_plugin 2024-03-21T00:02:56.608Z DEBUG lightningd: All Bitcoin plugin commands registered 2024-03-21T00:02:56.894Z DEBUG lightningd: Adding block 835386: 000000000000000000011645ca433b9b720096ab39de982f85bee18c9416fedf 2024-03-21T00:02:56.894Z DEBUG lightningd: io_break: maybe_completed_init 2024-03-21T00:02:56.901Z DEBUG lightningd: io_loop_with_timers: setup_topology 2024-03-21T00:02:56.901Z DEBUG wallet: Loaded 0 HTLC signatures from DB 2024-03-21T00:02:56.902Z DEBUG wallet: Loaded 0 HTLC signatures from DB 2024-03-21T00:02:56.902Z DEBUG wallet: Loaded 0 HTLC signatures from DB 2024-03-21T00:02:56.902Z DEBUG wallet: Loaded 0 HTLC signatures from DB 2024-03-21T00:02:56.903Z DEBUG wallet: Loaded 0 HTLC signatures from DB 2024-03-21T00:02:56.903Z DEBUG wallet: Loaded 0 HTLC signatures from DB 2024-03-21T00:02:56.904Z DEBUG wallet: Loaded 0 HTLC signatures from DB 2024-03-21T00:02:56.904Z DEBUG wallet: Loaded 0 HTLC signatures from DB 2024-03-21T00:02:56.905Z DEBUG wallet: Loaded 0 HTLC signatures from DB 2024-03-21T00:02:56.905Z DEBUG wallet: Loaded 0 HTLC signatures from DB 2024-03-21T00:02:56.906Z DEBUG wallet: Loaded 0 HTLC signatures from DB 2024-03-21T00:02:56.907Z DEBUG wallet: Loaded 1 HTLC signatures from DB 2024-03-21T00:02:56.908Z DEBUG wallet: Loaded 12 channels from DB 2024-03-21T00:02:56.908Z DEBUG wallet: Loading in HTLCs for channel 1 2024-03-21T00:02:56.910Z DEBUG wallet: Restored 0 incoming HTLCS 2024-03-21T00:02:56.910Z DEBUG wallet: Loading in HTLCs for channel 21 2024-03-21T00:02:56.910Z DEBUG wallet: Restored 0 incoming HTLCS 2024-03-21T00:02:56.910Z DEBUG wallet: Loading in HTLCs for channel 9 2024-03-21T00:02:56.914Z DEBUG wallet: Restored 0 incoming HTLCS 2024-03-21T00:02:56.914Z DEBUG wallet: Loading in HTLCs for channel 22 2024-03-21T00:02:56.915Z DEBUG wallet: Restored 0 incoming HTLCS 2024-03-21T00:02:56.915Z DEBUG wallet: Loading in HTLCs for channel 17 2024-03-21T00:02:56.932Z DEBUG wallet: Restored 0 incoming HTLCS 2024-03-21T00:02:56.932Z DEBUG wallet: Loading in HTLCs for channel 15 2024-03-21T00:02:56.932Z DEBUG wallet: Restored 0 incoming HTLCS 2024-03-21T00:02:56.932Z DEBUG wallet: Loading in HTLCs for channel 4 2024-03-21T00:02:56.937Z DEBUG wallet: Restored 0 incoming HTLCS 2024-03-21T00:02:56.938Z DEBUG wallet: Loading in HTLCs for channel 16 2024-03-21T00:02:56.938Z DEBUG wallet: Restored 0 incoming HTLCS 2024-03-21T00:02:56.938Z DEBUG wallet: Loading in HTLCs for channel 6 2024-03-21T00:02:56.941Z DEBUG wallet: Restored 0 incoming HTLCS 2024-03-21T00:02:56.941Z DEBUG wallet: Loading in HTLCs for channel 29 2024-03-21T00:02:56.963Z DEBUG wallet: Restored 1 incoming HTLCS 2024-03-21T00:02:56.963Z DEBUG wallet: Loading in HTLCs for channel 14 2024-03-21T00:02:56.966Z DEBUG wallet: Restored 0 incoming HTLCS 2024-03-21T00:02:56.966Z DEBUG wallet: Loading in HTLCs for channel 7 2024-03-21T00:02:56.966Z DEBUG wallet: Restored 0 incoming HTLCS 2024-03-21T00:02:56.971Z DEBUG wallet: Restored 0 outgoing HTLCS 2024-03-21T00:02:56.971Z DEBUG wallet: Restored 0 outgoing HTLCS 2024-03-21T00:02:56.973Z DEBUG wallet: Restored 0 outgoing HTLCS 2024-03-21T00:02:56.976Z DEBUG wallet: Restored 0 outgoing HTLCS 2024-03-21T00:02:56.980Z DEBUG wallet: Restored 0 outgoing HTLCS 2024-03-21T00:02:56.980Z DEBUG wallet: Restored 0 outgoing HTLCS 2024-03-21T00:02:56.981Z DEBUG wallet: Restored 0 outgoing HTLCS 2024-03-21T00:02:56.981Z DEBUG wallet: Restored 0 outgoing HTLCS 2024-03-21T00:02:56.982Z DEBUG wallet: Restored 0 outgoing HTLCS 2024-03-21T00:02:56.999Z DEBUG wallet: Restored 0 outgoing HTLCS 2024-03-21T00:02:57.002Z DEBUG wallet: Restored 0 outgoing HTLCS 2024-03-21T00:02:57.002Z DEBUG wallet: Restored 0 outgoing HTLCS 2024-03-21T00:02:57.008Z DEBUG gossipd: pid 32794, msgfd 55 2024-03-21T00:02:57.009Z DEBUG hsmd: new_client: 0 2024-03-21T00:02:57.010Z DEBUG gossipd: Store compact time: 0 msec 2024-03-21T00:02:57.010Z DEBUG gossipd: gossip_store: Read 12/12/1/0 cannounce/cupdate/nannounce/delete from store in 7614 bytes, now 7592 bytes (populated=true) 2024-03-21T00:02:57.010Z DEBUG gossipd: seeker: state = STARTING_UP New seeker 2024-03-21T00:02:57.010Z DEBUG gossipd: REPLY WIRE_GOSSIPD_INIT_REPLY with 0 fds 2024-03-21T00:02:57.010Z DEBUG lightningd: io_break: gossipd_init_done 2024-03-21T00:02:57.010Z DEBUG lightningd: io_loop: gossip_init 2024-03-21T00:02:57.011Z DEBUG lightningd: Looking for [autoclean,succeededforwards,num] 2024-03-21T00:02:57.013Z DEBUG plugin-cln-grpc: Killing plugin: disabled itself at init: Missing 'grpc-port' option 2024-03-21T00:02:57.031Z DEBUG lightningd: Looking for [autoclean,failedforwards,num] 2024-03-21T00:02:57.038Z DEBUG lightningd: Looking for [autoclean,succeededpays,num] 2024-03-21T00:02:57.044Z DEBUG plugin-fetchinvoice: Killing plugin: disabled itself at init: offers not enabled in config 2024-03-21T00:02:57.053Z DEBUG hsmd: Client: Received message 27 from client 2024-03-21T00:02:57.053Z DEBUG lightningd: Looking for [autoclean,failedpays,num] 2024-03-21T00:02:57.067Z DEBUG lightningd: Looking for [autoclean,paidinvoices,num] 2024-03-21T00:02:57.068Z DEBUG plugin-chanbackup: Chanbackup Initialised! 2024-03-21T00:02:57.074Z DEBUG plugin-recover: Recover Plugin Initialised! 2024-03-21T00:02:57.074Z DEBUG plugin-recover: Gossmap loaded! 2024-03-21T00:02:57.074Z DEBUG lightningd: Looking for [autoclean,expiredinvoices,num] 2024-03-21T00:02:57.076Z DEBUG hsmd: Client: Received message 27 from client 2024-03-21T00:02:57.077Z DEBUG plugin-bookkeeper: Setting up database at sqlite3://accounts.sqlite3 2024-03-21T00:02:57.077Z DEBUG connectd: REPLY WIRE_CONNECTD_ACTIVATE_REPLY with 0 fds 2024-03-21T00:02:57.077Z DEBUG lightningd: io_break: connect_activate_done 2024-03-21T00:02:57.077Z DEBUG lightningd: io_loop: connectd_activate 2024-03-21T00:02:57.077Z INFO lightningd: -------------------------------------------------- 2024-03-21T00:02:57.077Z INFO lightningd: Server started with public key 038e5677dc7d1ec4e9a82bd004d524b2469fedb0867e85bd1e2f5d3352957e03b7, alias nakoshi-satamoto@posteo.net (color #f7931a) and lightningd v24.02 2024-03-21T00:02:57.077Z DEBUG 026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2-chan#29: Adding funding_spend_watch 2024-03-21T00:02:57.077Z **BROKEN** 026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2-chan#29: Cannot broadcast our commitment tx: they have a future one 2024-03-21T00:02:57.077Z DEBUG 03ceeaec6cb017d1ea8ad04a5dfb3facb24a28399d24624ecce8f319973de361d9-chan#1: Watching for funding txid: 02008407990a213aa17d60e2c8954c59fbcefa312372b93897dc77dca17d23ad 2024-03-21T00:02:57.077Z DEBUG 03ceeaec6cb017d1ea8ad04a5dfb3facb24a28399d24624ecce8f319973de361d9-lightningd: Will try reconnect in 1 seconds 2024-03-21T00:02:57.077Z DEBUG 025c644f64160761662483c24aedba90f47f81f74292c7678640cf688a4223caeb-chan#21: Watching for funding txid: 2b0b9f5d365ff8dd5eecfda901789191be44b04fc90b713ceda70dfef331ee6a 2024-03-21T00:02:57.077Z DEBUG 025c644f64160761662483c24aedba90f47f81f74292c7678640cf688a4223caeb-lightningd: Will try reconnect in 1 seconds 2024-03-21T00:02:57.077Z DEBUG 0256ab9b7d257b30dc7308b7a9556b4793c9c3824989feaf0bfdafdb867278f7ee-chan#9: Watching for funding txid: fdd3b833fc60d37c97286c022920b7601d7cf77f5317940c7b25bcee8ddcdefb 2024-03-21T00:02:57.077Z DEBUG 0256ab9b7d257b30dc7308b7a9556b4793c9c3824989feaf0bfdafdb867278f7ee-lightningd: Will try reconnect in 1 seconds 2024-03-21T00:02:57.077Z DEBUG 020a128145c54dec1c1d72e9bf0ec26759d1e16b9a7bba1453e1fc2b4cec6a6fbd-chan#22: Watching for funding txid: 61f9908bfc066e6b7d0e737956b9782c6aba173e06bf76c38e6fd414d5a15556 2024-03-21T00:02:57.077Z DEBUG 020a128145c54dec1c1d72e9bf0ec26759d1e16b9a7bba1453e1fc2b4cec6a6fbd-lightningd: Will try reconnect in 1 seconds 2024-03-21T00:02:57.077Z DEBUG 035cf526f4c01ed693bacd7434461b9dc53d65b910fb3006110d8b42a574858fb2-chan#17: Watching for funding txid: 5ba6f3df167362ab540b09020ba15d757387bc0998ae7fd783cef43fc54d6627 2024-03-21T00:02:57.077Z DEBUG 035cf526f4c01ed693bacd7434461b9dc53d65b910fb3006110d8b42a574858fb2-lightningd: Will try reconnect in 1 seconds 2024-03-21T00:02:57.077Z DEBUG 03c1c8eef8e86c4bdf61067cf7d33e82532dc371f703e24207101e70989d3034a8-chan#15: Watching for funding txid: 055608f9ac3d2d3069a19ad07a4dcb31d3bd3dd925173f1a6c5350b18dcefa84 2024-03-21T00:02:57.077Z DEBUG 03c1c8eef8e86c4bdf61067cf7d33e82532dc371f703e24207101e70989d3034a8-lightningd: Will try reconnect in 1 seconds 2024-03-21T00:02:57.077Z DEBUG 02f1a8c87607f415c8f22c00593002775941dea48869ce23096af27b0cfdcc0b69-chan#4: Watching for funding txid: f18949a8a6985ffb6a1ffa4ba7bba62ee891ef15356c13064e9b5f8296a52c79 2024-03-21T00:02:57.077Z DEBUG 02f1a8c87607f415c8f22c00593002775941dea48869ce23096af27b0cfdcc0b69-lightningd: Will try reconnect in 2 seconds 2024-03-21T00:02:57.078Z DEBUG 03b279522ced2ccd9d2c7d1fc500d24f567129fc12b9495a076e30a2e861fffb1d-chan#16: Watching for funding txid: f736c8be9c6a6f8134b075f5b76016422462654cd3d8747a6ee15925ecb5a15a 2024-03-21T00:02:57.078Z DEBUG 03b279522ced2ccd9d2c7d1fc500d24f567129fc12b9495a076e30a2e861fffb1d-lightningd: Will try reconnect in 4 seconds 2024-03-21T00:02:57.078Z DEBUG 034cb28d896ef2ff25e2fbf5f6b524af3b5d9dc462272bf0620173d7d91b6559c9-chan#6: Watching for funding txid: 3953a2d07a99435e9ed03011e2da4c306bf473ee322cdad3af1304e4570c318c 2024-03-21T00:02:57.078Z DEBUG 034cb28d896ef2ff25e2fbf5f6b524af3b5d9dc462272bf0620173d7d91b6559c9-lightningd: Will try reconnect in 6 seconds 2024-03-21T00:02:57.078Z DEBUG 026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2-chan#29: Watching for funding txid: 86ef7d15b6117e07537a7cadf26dd17d48e79c4c5cbd1ca4a0944d3566a25355 2024-03-21T00:02:57.078Z DEBUG 036af7ab4e7a929065414e52de5ca4c4d88b1dddaf8e1be261b1530e69d0f3693e-chan#14: Watching for funding txid: 499b3f31b62a9c79ba32c2805992a0dc54867a58065bc9b36d2f5e8eac249d40 2024-03-21T00:02:57.078Z DEBUG 036af7ab4e7a929065414e52de5ca4c4d88b1dddaf8e1be261b1530e69d0f3693e-lightningd: Will try reconnect in 10 seconds 2024-03-21T00:02:57.078Z DEBUG 03eaf4f94ad680855a7818c2f156ae4a86482dea2f396320c336989ce5f49da880-chan#7: Watching for funding txid: 4b3a864968697f2625a48f7115bf265c9f84699997d2d673d18964ac807fdf5f 2024-03-21T00:02:57.078Z DEBUG 03eaf4f94ad680855a7818c2f156ae4a86482dea2f396320c336989ce5f49da880-lightningd: Will try reconnect in 12 seconds 2024-03-21T00:02:57.272Z DEBUG lightningd: Adding block 835387: 00000000000000000000e85174c561fd042c961d1e2f7ef241b217a9cfb4ead6 2024-03-21T00:02:57.526Z DEBUG lightningd: Adding block 835388: 000000000000000000028ad7c164e44b5bfd29422027026a22aff95a58957cba 2024-03-21T00:02:57.662Z DEBUG plugin-bcli: Log pruned 433 entries (mem 10492225 -> 4380503) 2024-03-21T00:02:57.742Z DEBUG lightningd: Adding block 835389: 0000000000000000000156b78efca111c021778e0a58f11c561d0e731947edee 2024-03-21T00:02:57.997Z DEBUG lightningd: Adding block 835390: 000000000000000000005d55276bdab0044e25ae890422972d34f2e456634b8d 2024-03-21T00:02:58.131Z DEBUG gossipd: REPLY WIRE_GOSSIPD_GET_ADDRS_REPLY with 0 fds 2024-03-21T00:02:58.166Z DEBUG plugin-bcli: Log pruned 203 entries (mem 10490366 -> 2596313) 2024-03-21T00:02:58.253Z DEBUG lightningd: Adding block 835391: 00000000000000000002e2002164799a1b6b96081c67002397c28bcb370a9800 2024-03-21T00:02:58.307Z DEBUG gossipd: REPLY WIRE_GOSSIPD_GET_ADDRS_REPLY with 0 fds 2024-03-21T00:02:58.495Z DEBUG lightningd: Adding block 835392: 00000000000000000001c86c1e42b72c93b405c67ed6bab305ab4cd23c7ca796 2024-03-21T00:02:58.544Z DEBUG gossipd: REPLY WIRE_GOSSIPD_GET_ADDRS_REPLY with 0 fds 2024-03-21T00:02:58.605Z DEBUG gossipd: REPLY WIRE_GOSSIPD_GET_ADDRS_REPLY with 0 fds 2024-03-21T00:02:58.663Z DEBUG plugin-bcli: Log pruned 151 entries (mem 10534526 -> 2454817) 2024-03-21T00:02:58.751Z DEBUG lightningd: Adding block 835393: 0000000000000000000274ab2523cae8073e4c86cb2442acf9c500e54e471411 2024-03-21T00:02:58.799Z DEBUG gossipd: REPLY WIRE_GOSSIPD_GET_ADDRS_REPLY with 0 fds 2024-03-21T00:02:58.992Z DEBUG lightningd: Adding block 835394: 000000000000000000004c11c8083e1e815b76d8ec4c98b5442e2bb8a0a3c8ca 2024-03-21T00:02:59.038Z DEBUG gossipd: REPLY WIRE_GOSSIPD_GET_ADDRS_REPLY with 0 fds 2024-03-21T00:02:59.038Z DEBUG plugin-recover: Listpeerchannels called 2024-03-21T00:02:59.038Z DEBUG plugin-recover: Detected a channel with lost state, Entering Recovery mode! 2024-03-21T00:02:59.039Z DEBUG lightningd: channel 6aee31f3fe0da7ed3c710bc94fb044be91917801a9fdec5eddf85f365d9f0b2b already exists! 2024-03-21T00:02:59.039Z DEBUG lightningd: channel 27664dc53ff4ce83d77fae9809bc8773755da10b02090b54ab627316dff3a65b already exists! 2024-03-21T00:02:59.039Z DEBUG lightningd: channel fbdedc8deebc257b0c9417537ff77c1d60b72029026c28977cd360fc33b8d3fd already exists! 2024-03-21T00:02:59.041Z **BROKEN** lightningd: Error executing statement: wallet/wallet.c:2594: INSERT INTO channels ( peer_id, first_blocknum, id, revocation_basepoint_local, payment_basepoint_local, htlc_basepoint_local, delayed_payment_basepoint_local, funding_pubkey_local, require_confirm_inputs_remote, require_confirm_inputs_local) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?);: UNIQUE constraint failed: channels.id 2024-03-21T00:02:59.041Z **BROKEN** lightningd: Error executing statement: wallet/wallet.c:2594: INSERT INTO channels ( peer_id, first_blocknum, id, revocation_basepoint_local, payment_basepoint_local, htlc_basepoint_local, delayed_payment_basepoint_local, funding_pubkey_local, require_confirm_inputs_remote, require_confirm_inputs_local) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?);: UNIQUE constraint failed: channels.id 2024-03-21T00:02:59.075Z **BROKEN** lightningd: FATAL SIGNAL 6 (version v24.02) 2024-03-21T00:02:59.075Z **BROKEN** lightningd: backtrace: common/daemon.c:38 (send_backtrace) 0x556ec94f9fe6 2024-03-21T00:02:59.075Z **BROKEN** lightningd: backtrace: common/daemon.c:75 (crashdump) 0x556ec94fa035 2024-03-21T00:02:59.075Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f5276ca004f 2024-03-21T00:02:59.075Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f5276ceee2c 2024-03-21T00:02:59.075Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f5276c9ffb1 2024-03-21T00:02:59.075Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f5276c8a471 2024-03-21T00:02:59.075Z **BROKEN** lightningd: backtrace: lightningd/log.c:1025 (fatal_vfmt) 0x556ec94a7c37 2024-03-21T00:02:59.075Z **BROKEN** lightningd: backtrace: wallet/db.c:1097 (db_error) 0x556ec94d9e55 2024-03-21T00:02:59.075Z **BROKEN** lightningd: backtrace: db/utils.c:316 (db_fatal) 0x556ec9548efd 2024-03-21T00:02:59.075Z **BROKEN** lightningd: backtrace: db/utils.c:190 (db_exec_prepared_v2) 0x556ec95492c1 2024-03-21T00:02:59.075Z **BROKEN** lightningd: backtrace: wallet/wallet.c:2619 (wallet_channel_insert) 0x556ec94e424b 2024-03-21T00:02:59.075Z **BROKEN** lightningd: backtrace: lightningd/opening_control.c:1581 (json_recoverchannel) 0x556ec94aed17 2024-03-21T00:02:59.075Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:839 (command_exec) 0x556ec949fe0c 2024-03-21T00:02:59.075Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:982 (rpc_command_hook_final) 0x556ec94a298b 2024-03-21T00:02:59.075Z **BROKEN** lightningd: backtrace: lightningd/plugin_hook.c:247 (plugin_hook_call_) 0x556ec94cf162 2024-03-21T00:02:59.075Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:1070 (plugin_hook_call_rpc_command) 0x556ec94a2174 2024-03-21T00:02:59.075Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:1189 (parse_request) 0x556ec94a2174 2024-03-21T00:02:59.075Z **BROKEN** lightningd: backtrace: lightningd/jsonrpc.c:1295 (read_json) 0x556ec94a259e 2024-03-21T00:02:59.075Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x556ec966a557 2024-03-21T00:02:59.075Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:407 (do_plan) 0x556ec966aa32 2024-03-21T00:02:59.075Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:417 (io_ready) 0x556ec966aacf 2024-03-21T00:02:59.075Z **BROKEN** lightningd: backtrace: ccan/ccan/io/poll.c:453 (io_loop) 0x556ec966c46c 2024-03-21T00:02:59.075Z **BROKEN** lightningd: backtrace: lightningd/io_loop_with_timers.c:22 (io_loop_with_timers) 0x556ec949fd88 2024-03-21T00:02:59.075Z **BROKEN** lightningd: backtrace: lightningd/lightningd.c:1425 (main) 0x556ec94a5456 2024-03-21T00:02:59.076Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f5276c8b249 2024-03-21T00:02:59.076Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f5276c8b304 2024-03-21T00:02:59.076Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x556ec9479d64 2024-03-21T00:02:59.076Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0xffffffffffffffff ```

Perhaps it’s this? If so, feel free to DM me in TG at Telegram: Contact @StuPleb and I’ll pass you any relevant pieces that might help.

Otherwise you should be able to get what you want out of this

github.com

Start9Labs/cln-startos/blob/master/docker_entrypoint.sh

#!/bin/sh

set -ea

_term() {
  echo "Caught SIGTERM signal!"
  kill -TERM "$lightningd_child" 2>/dev/null
  kill -TERM "$teosd_child" 2>/dev/null
  kill -TERM "$wtclient_child" 2>/dev/null
  kill -TERM "$wtserver_child" 2>/dev/null
}

_chld() {
  echo "Caught SIGCHLD signal!"
  kill -TERM "$lightningd_child" 2>/dev/null
  kill -TERM "$teosd_child" 2>/dev/null
  kill -TERM "$wtclient_child" 2>/dev/null
  kill -TERM "$wtserver_child" 2>/dev/null
  kill -TERM "$ui_child" 2>/dev/null
}

This file has been truncated. show original