Mempool - Lightning stats not working

Services Support
,
1

Hello amaizing Start9 community!

First of all thank you so much for this amaizing software, I came to it thanks to a YT video by Mattew Kratter.

I recently installed StartOS on my home lab and this is the stack :

  • PROXMOX VE v8.4.1

  • VM 16 cores, 32Gb RAM and 2Tb disk space

  • Bitcoin Knots + Electrs + LND + Mempool

  • Lightning software : ThunderHub, Ride The Lightning, LNBit and Lightning Terminal

Almost everything is fine, all services are synced and all dependencies satisfied. I keep the system and all services always up to date.

There’s just one thing I’m not able to make it work => the lightning stats on Mempool.
=> On mempool configuration I selected LND and activated Electrs.

Here is the result when I visit the page :

mempool1
mempool11862×875 84.2 KB

I already tried “System Rebuild”, uninstall/reinstall Mempool, restart LND in various combination with no success.

Am I the only one experiencing this ?
Could please someone help me to understand what causes this problem ?

Here the errors/failed I could spot on the log files (not all occurrencies but all relevant typologies of errros).

Bitcoin Knots :

2025-05-23T16:56:10+02:00  2025-05-23T14:56:10Z Socks5() connect to 5uhtgf74selv6yv5ot7tardaaonpyk7vqbjjczfzbp7bykfaw5p7mxqd.onion:8333 failed: InterruptibleRecv() timeout or other failure
2025-05-23T16:56:11+02:00  2025-05-23T14:56:11Z New outbound-full-relay v2 peer connected: version: 70016, blocks=898020, peer=3
2025-05-23T16:56:31+02:00  2025-05-23T14:56:31Z Socks5() connect to pupvkwohewa2g5rxzwnj6arkcehpjk5iu6k5d4d7qqp5e6mrojh3zsyd.onion:8333 failed: InterruptibleRecv() timeout or other failure
2025-05-23T16:56:32+02:00  2025-05-23T14:56:32Z UpdateTip: new best=00000000000000000001438bf6e698cc2d4cc26ec849f64a961d0697a0de33e3 height=898011 version=0x206a0000 log2_work=95.624335 tx=1194391421 date='2025-05-23T14:12:45Z' progress=0.999985 cache=3.8MiB(26923txo) warning='Miner violated version bit protocol'
2025-05-23T16:56:45+02:00  2025-05-23T14:56:45Z UpdateTip: new best=0000000000000000000213f5516c895a6395051d6724be2ac45ef17712543292 height=898012 version=0x24918000 log2_work=95.624347 tx=1194395609 date='2025-05-23T14:16:40Z' progress=0.999986 cache=4.9MiB(37075txo) warning='Miner violated version bit protocol'
2025-05-23T16:56:53+02:00  2025-05-23T14:56:53Z Progress loading mempool transactions from file: 80% (tried 3260, 814 remaining)
2025-05-23T16:57:04+02:00  2025-05-23T14:57:04Z UpdateTip: new best=00000000000000000001a4afaa6fed36122948924a0c89cdedd74ce754ef1f97 height=898013 version=0x2c224000 log2_work=95.624359 tx=1194399195 date='2025-05-23T14:26:14Z' progress=0.999990 cache=7.0MiB(49828txo) warning='Miner violated version bit protocol'
2025-05-23T16:57:06+02:00  2025-05-23T14:57:06Z Progress loading mempool transactions from file: 90% (tried 3667, 407 remaining)
2025-05-23T16:57:09+02:00  2025-05-23T14:57:09Z UpdateTip: new best=0000000000000000000143ace901594f0b44e3b6f7ed6936c02365a1fee43b27 height=898014 version=0x34af0000 log2_work=95.624372 tx=1194404032 date='2025-05-23T14:29:32Z' progress=0.999991 cache=8.1MiB(57862txo) warning='Miner violated version bit protocol'
2025-05-23T16:57:10+02:00  2025-05-23T14:57:10Z Imported mempool transactions from file: 1558 succeeded, 2516 failed, 0 expired, 0 already there, 0 waiting for initial broadcast
2025-05-23T16:57:10+02:00  2025-05-23T14:57:10Z initload thread exit
....
2025-05-23T22:28:29+02:00  2025-05-23T20:28:29Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications
2025-05-23T22:35:51+02:00  2025-05-23T20:35:51Z ERROR: ContextualCheckBlockHeader: forked chain older than last checkpoint (height 1)
2025-05-23T22:35:53+02:00  2025-05-23T20:35:53Z New block-relay-only v1 peer connected: version: 70016, blocks=898056, peer=199

LND :

2025-05-23T17:06:58+02:00  2025-05-23 15:06:58.959 [ERR] GRPH: process network updates got: unable to locate funding tx: tx_index=#56 is out of range (max_index=0), network_chan_id=500:56:21610
2025-05-23T17:06:59+02:00  2025-05-23 15:06:59.134 [ERR] GRPH: process network updates got: unable to locate funding tx: tx_index=#56 is out of range (max_index=0), network_chan_id=500:56:22287
2025-05-23T17:06:59+02:00  2025-05-23 15:06:59.274 [ERR] GRPH: process network updates got: unable to locate funding tx: tx_index=#56 is out of range (max_index=0), network_chan_id=500:56:23496
2025-05-23T17:06:59+02:00  2025-05-23 15:06:59.387 [ERR] GRPH: process network updates got: unable to locate funding tx: tx_index=#56 is out of range (max_index=0), network_chan_id=500:56:31375
2025-05-23T17:06:59+02:00  2025-05-23 15:06:59.467 [ERR] GRPH: process network updates got: unable to locate funding tx: tx_index=#56 is out of range (max_index=0), network_chan_id=500:56:45174
2025-05-23T17:06:59+02:00  2025-05-23 15:06:59.539 [ERR] GRPH: process network updates got: unable to locate funding tx: tx_index=#56 is out of range (max_index=0), network_chan_id=500:56:52140
2025-05-23T17:06:59+02:00  2025-05-23 15:06:59.853 [ERR] GRPH: process network updates got: unable to locate funding tx: tx_index=#56 is out of range (max_index=0), network_chan_id=500:56:61851
2025-05-23T17:06:59+02:00  2025-05-23 15:06:59.925 [ERR] GRPH: process network updates got: unable to locate funding tx: tx_index=#56 is out of range (max_index=0), network_chan_id=500:56:52358
2025-05-23T17:07:00+02:00  2025-05-23 15:07:00.024 [ERR] GRPH: process network updates got: unable to locate funding tx: tx_index=#57 is out of range (max_index=0), network_chan_id=500:57:15903
2025-05-23T17:07:00+02:00  2025-05-23 15:07:00.106 [ERR] GRPH: process network updates got: unable to locate funding tx: tx_index=#57 is out of range (max_index=0), network_chan_id=500:57:23628
2025-05-23T17:07:00+02:00  2025-05-23 15:07:00.278 [ERR] GRPH: process network updates got: unable to locate funding tx: tx_index=#57 is out of range (max_index=0), network_chan_id=500:57:23151
2025-05-23T17:07:00+02:00  2025-05-23 15:07:00.405 [ERR] GRPH: process network updates got: unable to locate funding tx: tx_index=#57 is out of range (max_index=0), network_chan_id=500:57:24103
2025-05-23T17:07:00+02:00  2025-05-23 15:07:00.519 [ERR] GRPH: process network updates got: unable to locate funding tx: tx_index=#57 is out of range (max_index=0), network_chan_id=500:57:32209
2025-05-23T17:07:00+02:00  2025-05-23 15:07:00.614 [ERR] GRPH: process network updates got: unable to locate funding tx: tx_index=#57 is out of range (max_index=0), network_chan_id=500:57:29426
2025-05-23T17:07:00+02:00  2025-05-23 15:07:00.791 [ERR] GRPH: process network updates got: unable to locate funding tx: tx_index=#57 is out of range (max_index=0), network_chan_id=500:57:35112
......
2025-05-23T17:19:33+02:00  2025-05-23 15:19:33.673 [ERR] GRPH: out of order block: expecting height=898025, got height=898024
2025-05-23T17:19:33+02:00  2025-05-23 15:19:33.674 [INF] GRPH: Skipping channel pruning since received block height 898024 was already processed.
2025-05-23T17:19:34+02:00  2025-05-23 15:19:34.289 [ERR] GRPH: out of order block: expecting height=898025, got height=898024
2025-05-23T17:19:34+02:00  2025-05-23 15:19:34.290 [INF] GRPH: Skipping channel pruning since received block height 898024 was already processed.
2025-05-23T17:19:34+02:00  2025-05-23 15:19:34.831 [ERR] GRPH: out of order block: expecting height=898025, got height=898024
2025-05-23T17:19:34+02:00  2025-05-23 15:19:34.831 [INF] GRPH: Skipping channel pruning since received block height 898024 was already processed.
2025-05-23T17:19:35+02:00  2025-05-23 15:19:35.470 [ERR] GRPH: out of order block: expecting height=898025, got height=898024
....
2025-05-23T17:28:56+02:00  2025-05-23 15:28:56.699 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=598459781126029313, chan_point=e4f9bccad30aebe64676219a96d3d6228e5c34efaaa7b8f8f4625a249144df67:1: target output has been spent
2025-05-23T17:28:56+02:00  2025-05-23 15:28:56.857 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=598444387880337409, chan_point=0fe81990cc07449c0ef8f1bdd32e6b75f1465f033560b909efeb4d0321a51446:1: target output has been spent
2025-05-23T17:28:57+02:00  2025-05-23 15:28:57.117 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=599492222468751360, chan_point=289f1562c0f0dec9a17589a5a46c97437fe48299037322b2440e3793252381ed:0: target output has been spent
2025-05-23T17:28:57+02:00  2025-05-23 15:28:57.426 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=598704972168298497, chan_point=8acc3d7400ab44d2711c6c42dde607cd5fb7f7d8de79a3391149d1666c742bfa:1: target output has been spent
2025-05-23T17:28:57+02:00  2025-05-23 15:28:57.750 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=599077706615619584, chan_point=acb759f0e72bde8f480ed5524cef715b61c067490046ba0add32b5c698bbff0a:0: target output has been spent
2025-05-23T17:28:58+02:00  2025-05-23 15:28:58.054 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=599306405016174592, chan_point=ade4ce235666c23fb039541571b5cc137ae98830873443a212a5f24a24b854a2:0: target output has been spent
2025-05-23T17:28:58+02:00  2025-05-23 15:28:58.210 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=599535103417122816, chan_point=81b6e11712d164d3ef39ebce95f5f190fbd92f64f8765e768e507436dd5676c5:0: target output has been spent
2025-05-23T17:28:58+02:00  2025-05-23 15:28:58.349 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=599787991054745601, chan_point=ab70756cf2521cb72f26e55e6e58c4c6ecacc6ddc7d9f7439d7fd181f34a5b0f:1: target output has been spent
2025-05-23T17:28:58+02:00  2025-05-23 15:28:58.584 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=600213502044340225, chan_point=e3649a395ba5b4b4342c1fe585dc29d0726b2a02f3ea25236f2731ab7840d19f:1: target output has been spent
2025-05-23T17:28:58+02:00  2025-05-23 15:28:58.961 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=599759403732041728, chan_point=1f778386e57c28249b7a7822b749a9adb704ff07225db78fdc9b09d098ae08ee:0: target output has been spent
2025-05-23T17:28:59+02:00  2025-05-23 15:28:59.175 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=600360836569300993, chan_point=8bfc2faf9630ca791ce713d18b170dc4684b99653cacc30be0b6436ac65ee438:1: target output has been spent
2025-05-23T17:28:59+02:00  2025-05-23 15:28:59.446 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=599794588087812096, chan_point=b8e2ad15e0fa7ab0c2b7e83be99b83e0a458c1acfea9b597026fab02698f026d:0: target output has been spent
2025-05-23T17:29:00+02:00  2025-05-23 15:29:00.313 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=599794588088008705, chan_point=9afd00e6660360dd6f0bc929c00c7f826e4aefbcf67556eff656a5b90ab734ce:1: target output has been spent
2025-05-23T17:29:00+02:00  2025-05-23 15:29:00.478 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=600678595486482433, chan_point=497886806d69b66ac7f4b666e8bdf22fb07d72996b0b3b2b004b40f63369ea12:1: target output has been spent
2025-05-23T17:29:00+02:00  2025-05-23 15:29:00.650 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=600839124188987392, chan_point=bd0fcb3cbf633ba3834f761c0c08a95cbdbe71b7669eae4c4bedec435d958b6b:0: target output has been spent
2025-05-23T17:29:01+02:00  2025-05-23 15:29:01.495 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=600935881197158400, chan_point=dfa89f8f02a8520c9b8a68ead0f7299cf0156ca66c59377d47e064546ab914fc:0: target output has been spent
2025-05-23T17:29:02+02:00  2025-05-23 15:29:02.427 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=601061225575219200, chan_point=4bb0da8afed3f328ddf241e150b4f9347bc72f5775aad936a4abbd066c88bdff:0: target output has been spent
2025-05-23T17:29:02+02:00  2025-05-23 15:29:02.599 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=600983160228610048, chan_point=390e356521af61004ba5d36f8aee00501aa436040140910fcdecd13b502c944a:0: target output has been spent
2025-05-23T17:29:02+02:00  2025-05-23 15:29:02.744 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=601322909392240641, chan_point=b5f2c41749e7e6d5e1f933fc063e055417c520d8f920d19840fb4f9f05f590ac:1: target output has been spent
2025-05-23T17:29:02+02:00  2025-05-23 15:29:02.863 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=601337202976817152, chan_point=7666cb0c20f5ed3a5097cf96310706c7add942cde13bf448de49fe73ffa8781c:0: target output has been spent
2025-05-23T17:29:03+02:00  2025-05-23 15:29:03.001 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=601758315996381184, chan_point=200a4a3d866529536731ca982823459590e6feb036da14025d086c269ff52c75:0: target output has been spent
2025-05-23T17:29:03+02:00  2025-05-23 15:29:03.156 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=601624175520972801, chan_point=95100421bdd5546c22088f5e247b0e02e21b0737842fc9922cbe59ecfd284a6e:1: target output has been spent
2025-05-23T17:29:03+02:00  2025-05-23 15:29:03.310 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=601735226168901632, chan_point=a0818904896966167cba010597f85be59fd7da8f4018a315075c1eb3f75457c8:0: target output has been spent
2025-05-23T17:29:03+02:00  2025-05-23 15:29:03.440 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=601745121819099136, chan_point=cf6b338c81a973ce5a4af34f5285700243619b50b432cedf46b5f62761ba9fcf:0: target output has been spent
2025-05-23T17:29:03+02:00  2025-05-23 15:29:03.555 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=601779206618218497, chan_point=86093648160fe8b5509dd10701a9972a1d2aae37b028db7870b9521a13f056cf:1: target output has been spent
2025-05-23T17:29:03+02:00  2025-05-23 15:29:03.668 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=601792400772628481, chan_point=370cd6cdba55f15150d9ab223c26bebff4651cb7e89965714d4c321f0c59cd94:1: target output has been spent
.....
2025-05-24T00:07:43+02:00  2025-05-23 22:07:43.098 [ERR] DISC: ignoring closed channel 897470:4614:0
2025-05-24T00:07:43+02:00  2025-05-23 22:07:43.102 [ERR] DISC: ignoring closed channel 748428:999:1
2025-05-24T00:07:43+02:00  2025-05-23 22:07:43.104 [ERR] DISC: ignoring closed channel 748429:784:1
2025-05-24T00:07:43+02:00  2025-05-23 22:07:43.105 [ERR] DISC: ignoring closed channel 748448:1467:1
2025-05-24T00:07:43+02:00  2025-05-23 22:07:43.105 [ERR] DISC: ignoring closed channel 748453:1076:0
2025-05-24T00:07:43+02:00  2025-05-23 22:07:43.105 [ERR] DISC: ignoring closed channel 748533:1352:0
2025-05-24T00:07:43+02:00  2025-05-23 22:07:43.106 [ERR] DISC: ignoring closed channel 748536:1979:0
2025-05-24T00:07:43+02:00  2025-05-23 22:07:43.106 [ERR] DISC: ignoring closed channel 748536:1979:2
2025-05-24T00:07:43+02:00  2025-05-23 22:07:43.107 [ERR] DISC: ignoring closed channel 748537:2536:1
2025-05-24T00:07:43+02:00  2025-05-23 22:07:43.107 [ERR] DISC: ignoring closed channel 897438:3028:1
2025-05-24T00:07:43+02:00  2025-05-23 22:07:43.108 [ERR] DISC: ignoring closed channel 748401:978:2
2025-05-24T00:07:43+02:00  2025-05-23 22:07:43.110 [ERR] DISC: ignoring closed channel 748414:914:1
2025-05-24T00:07:43+02:00  2025-05-23 22:07:43.110 [ERR] DISC: ignoring closed channel 748279:2027:2
2025-05-24T00:07:43+02:00  2025-05-23 22:07:43.110 [ERR] DISC: ignoring closed channel 748291:412:1
2025-05-24T00:07:43+02:00  2025-05-23 22:07:43.111 [ERR] DISC: ignoring closed channel 748279:2027:3
2025-05-24T00:07:43+02:00  2025-05-23 22:07:43.111 [ERR] DISC: ignoring closed channel 748279:2027:12
2025-05-24T00:07:43+02:00  2025-05-23 22:07:43.112 [ERR] DISC: ignoring closed channel 748287:522:0
2025-05-24T00:07:43+02:00  2025-05-23 22:07:43.112 [ERR] DISC: ignoring closed channel 748279:2027:14
2025-05-24T00:07:45+02:00  2025-05-23 22:07:45.278 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=920526528022511617, chan_point=e6c4c201124e7b04bb524f252be0a0aa1d0a0775587de3d0300435811549746e:1: target output has been spent
2025-05-24T00:07:45+02:00  2025-05-23 22:07:45.710 [ERR] GRPH: process network updates got: unable to fetch utxo for chan_id=920293431623548929, chan_point=31c3d18cdd5c7c7772f39d2a2a9b419a393851108127cc62b108d32eed4dc67d:1: target output has been spent
.....
2025-05-24T00:37:13+02:00  2025-05-23 22:37:13.410 [ERR] DISC: Skewed timestamp (1901351336) for edge policy of short_chan_id(742658531969269760), timestamp too far in the future: peer=02ae9c3713f1041b751f7fde24945a1d255d34461df72418ee7a3510528d54bae4@cuylzxreo6ozwxqaji2bqj2koi3l5hlfwcufnotglgop3tlgzdanafad.onion:9735, msg=ChannelUpdate, is_remote=true
2025-05-24T00:37:13+02:00  2025-05-23 22:37:13.411 [ERR] DISC: Skewed timestamp (1901350892) for edge policy of short_chan_id(771851665200054273), timestamp too far in the future: peer=02ae9c3713f1041b751f7fde24945a1d255d34461df72418ee7a3510528d54bae4@cuylzxreo6ozwxqaji2bqj2koi3l5hlfwcufnotglgop3tlgzdanafad.onion:9735, msg=ChannelUpdate, is_remote=true
2025-05-24T00:37:13+02:00  2025-05-23 22:37:13.413 [ERR] DISC: Skewed timestamp (1901350634) for edge policy of short_chan_id(717789778064113665), timestamp too far in the future: peer=02ae9c3713f1041b751f7fde24945a1d255d34461df72418ee7a3510528d54bae4@cuylzxreo6ozwxqaji2bqj2koi3l5hlfwcufnotglgop3tlgzdanafad.onion:9735, msg=ChannelUpdate, is_remote=true
2025-05-24T00:37:13+02:00  2025-05-23 22:37:13.414 [ERR] DISC: Skewed timestamp (1901351091) for edge policy of short_chan_id(770826920361984001), timestamp too far in the future: peer=02ae9c3713f1041b751f7fde24945a1d255d34461df72418ee7a3510528d54bae4@cuylzxreo6ozwxqaji2bqj2koi3l5hlfwcufnotglgop3tlgzdanafad.onion:9735, msg=ChannelUpdate, is_remote=true
2025-05-24T00:37:13+02:00  2025-05-23 22:37:13.414 [ERR] DISC: Skewed timestamp (1901350681) for edge policy of short_chan_id(775767026135728128), timestamp too far in the future: peer=02ae9c3713f1041b751f7fde24945a1d255d34461df72418ee7a3510528d54bae4@cuylzxreo6ozwxqaji2bqj2koi3l5hlfwcufnotglgop3tlgzdanafad.onion:9735, msg=ChannelUpdate, is_remote=true
2025-05-24T00:37:13+02:00  2025-05-23 22:37:13.414 [ERR] DISC: Skewed timestamp (1901350637) for edge policy of short_chan_id(772875310597931008), timestamp too far in the future: peer=02ae9c3713f1041b751f7fde24945a1d255d34461df72418ee7a3510528d54bae4@cuylzxreo6ozwxqaji2bqj2koi3l5hlfwcufnotglgop3tlgzdanafad.onion:9735, msg=ChannelUpdate, is_remote=true
2025-05-24T00:37:13+02:00  2025-05-23 22:37:13.415 [ERR] DISC: Skewed timestamp (1901350653) for edge policy of short_chan_id(753758101838102532), timestamp too far in the future: peer=02ae9c3713f1041b751f7fde24945a1d255d34461df72418ee7a3510528d54bae4@cuylzxreo6ozwxqaji2bqj2koi3l5hlfwcufnotglgop3tlgzdanafad.onion:9735, msg=ChannelUpdate, is_remote=true

Electrs :

2025-05-24T00:49:14+02:00  [2025-05-23T22:49:14.822Z WARN electrs::mempool] mempool sync failed: batch getrawtransaction request failed: transport error: Couldn't connect to host: Resource temporarily unavailable (os error 11)
2025-05-24T00:49:45+02:00  [2025-05-23T22:49:45.538Z WARN electrs::mempool] mempool sync failed: batch getrawtransaction request failed: transport error: Couldn't connect to host: Resource temporarily unavailable (os error 11)

Mempool :

2025-05-23T18:47:13+02:00  May 23 16:47:13 [358] ERR: <lightning> Exception in $logStatsDaily. Reason: timeout of 10000ms exceeded
2025-05-23T18:47:13+02:00  May 23 16:47:13 [358] INFO: <lightning> Starting lightning network forensics service
2025-05-23T18:47:14+02:00  May 23 16:47:14 [358] INFO: <lightning> [Lightning] Caching funding txs for currently existing channels
2025-05-23T18:47:14+02:00  May 23 16:47:14 [358] INFO: <lightning> Lightning topology folder is not set. Not importing historical LN stats
2025-05-23T18:47:46+02:00  May 23 16:47:46 [358] INFO: <lightning> [Mining] Inserted 608 Kraken EUR, USD, GBP, JPY, CAD, CHF and AUD weekly price history into db
2025-05-23T18:48:54+02:00  May 23 16:48:54 [358] ERR: <lightning> $updateBlocks stalled at "getting block data for 898028"
.....
2025-05-23T19:47:03+02:00  May 23 17:47:03 [358] ERR: <lightning> [Lightning] $runTasks() error: timeout of 10000ms exceeded
2025-05-23T19:47:24+02:00  May 23 17:47:24 [358] ERR: <lightning> Exception in $logStatsDaily. Reason: timeout of 10000ms exceeded
2025-05-23T19:51:44+02:00  May 23 17:51:44 [358] ERR: <lightning> Cannot fetch tx b57dfc99fd54bf4368f63ed514a3dbd7459df9af4205a6739ac8ca915af09e0a. Reason: ETIMEDOUT
2025-05-23T19:52:02+02:00  May 23 17:52:02 [358] ERR: <lightning> Cannot fetch tx 1b83838dbf7647e4396df6a42abaa1f35934dac6e3b5b6db2ac685965afe1d83. Reason: ETIMEDOUT
2025-05-23T20:00:10+02:00  May 23 18:00:10 [358] INFO: <lightning> Latest BTC fiat averaged price: {"time":1748023210,"USD":109450,"EUR":96319,"GBP":80876,"CAD":150259,"CHF":89990,"AUD":168316,"JPY":15628500,"BGN":-1,"BRL":-1,"CNY":-1,"CZK":-1,"DKK":-1,"HKD":-1,"HRK":-1,"HUF":-1,"IDR":-1,"ILS":-1,"INR":-1,"ISK":-1,"KRW":-1,"MXN":-1,"MYR":-1,"NOK":-1,"NZD":-1,"PHP":-1,"PLN":-1,"RON":-1,"RUB":-1,"SEK":-1,"SGD":-1,"THB":-1,"TRY":-1,"ZAR":-1}
2025-05-23T20:35:37+02:00  May 23 18:35:37 [358] ERR: <lightning> Cannot fetch tx 0e3da161daa34015cbb6266d96e0f360301169549ef4e1f541192c860861dbe6. Reason: ETIMEDOUT
2025-05-23T20:35:50+02:00  May 23 18:35:50 [358] ERR: <lightning> Cannot fetch tx 7504ad293c6bdf43ed7e3ef5e8c92de89559a4b1de0b4315a53a7ed17faa679a. Reason: ETIMEDOUT
2025-05-23T20:47:03+02:00  May 23 18:47:03 [358] ERR: <lightning> [Lightning] $runTasks() error: timeout of 10000ms exceeded
2025-05-23T20:47:34+02:00  May 23 18:47:34 [358] ERR: <lightning> Exception in $logStatsDaily. Reason: timeout of 10000ms exceeded
2025-05-23T21:00:19+02:00  May 23 19:00:19 [358] INFO: <lightning> Latest BTC fiat averaged price: {"time":1748026820,"USD":108822,"EUR":95796,"GBP":80396,"CAD":149362,"CHF":89342,"AUD":167510,"JPY":15517500,"BGN":-1,"BRL":-1,"CNY":-1,"CZK":-1,"DKK":-1,"HKD":-1,"HRK":-1,"HUF":-1,"IDR":-1,"ILS":-1,"INR":-1,"ISK":-1,"KRW":-1,"MXN":-1,"MYR":-1,"NOK":-1,"NZD":-1,"PHP":-1,"PLN":-1,"RON":-1,"RUB":-1,"SEK":-1,"SGD":-1,"THB":-1,"TRY":-1,"ZAR":-1}
2025-05-23T21:04:11+02:00  May 23 19:04:11 [358] ERR: <lightning> Cannot fetch tx f808f1758e9c72e254a279fb9e927396b6e7b169b057a546083bfed35b7cfe37. Reason: ETIMEDOUT
2025-05-23T21:04:21+02:00  May 23 19:04:21 [358] ERR: <lightning> Cannot fetch tx d6f8f1023ba5cfe1a568121d0a5c19c8848f39d229cd0874cb581e16b65b2b31. Reason: ETIMEDOUT
2025-05-23T21:26:10+02:00  May 23 19:26:10 [358] ERR: <lightning> Cannot fetch tx f603591d96e4135bde08669bddbc62ab14ed487941b8c40465c311694e2f0427. Reason: ETIMEDOUT
2025-05-23T21:26:35+02:00  May 23 19:26:35 [358] ERR: <lightning> Cannot fetch tx 790bdf68106b3f73a467700b9a45a714918d232e88873ae041c62e30409a1047. Reason: ETIMEDOUT
2025-05-23T21:27:10+02:00  May 23 19:27:10 [358] ERR: <lightning> Cannot fetch coinbase tx f603591d96e4135bde08669bddbc62ab14ed487941b8c40465c311694e2f0427. Reason: ETIMEDOUT
2025-05-23T21:27:10+02:00  May 23 19:27:10 [358] ERR: <lightning> [Mining] Block indexing failed. Trying again in 10 seconds. Reason: Cannot fetch coinbase tx f603591d96e4135bde08669bddbc62ab14ed487941b8c40465c311694e2f0427. Reason: ETIMEDOUT
2025-05-23T21:27:10+02:00  May 23 19:27:10 [358] ERR: <lightning> Indexer failed, trying again in 10 seconds. Reason: Cannot fetch coinbase tx f603591d96e4135bde08669bddbc62ab14ed487941b8c40465c311694e2f0427. Reason: ETIMEDOUT
2025-05-23T21:27:28+02:00  May 23 19:27:28 [358] ERR: <lightning> $updateBlocks stalled at "getting block data for 898045"
2025-05-23T21:27:35+02:00  May 23 19:27:35 [358] ERR: <lightning> Cannot fetch tx 790bdf68106b3f73a467700b9a45a714918d232e88873ae041c62e30409a1047. Reason: ETIMEDOUT
2025-05-23T21:28:27+02:00  May 23 19:28:27 [358] INFO: <lightning> Core index 'txindex' is synced. Best block height is 898045
2025-05-23T21:28:27+02:00  May 23 19:28:27 [358] INFO: <lightning> Core index 'coinstatsindex' is synced. Best block height is 898045
2025-05-23T21:28:27+02:00  May 23 19:28:27 [358] INFO: <lightning> Core index 'basic block filter index' is synced. Best block height is 898045
2025-05-23T21:28:27+02:00  May 23 19:28:27 [358] INFO: <lightning> [Mining] Indexing 3127 blocks from #898045 to #888046
2025-05-23T21:28:32+02:00  May 23 19:28:32 [358] INFO: <lightning> [Mining] Indexing coinstatsindex data for block #898039. Indexed 7 blocks.
2025-05-23T21:28:33+02:00  May 23 19:28:33 [358] INFO: <lightning> [Mining] Indexing missing coinstatsindex data completed. Indexed 13
2025-05-23T21:38:50+02:00  May 23 19:38:50 [358] ERR: <lightning> [Mining] Block indexing failed. Trying again in 10 seconds. Reason: ETIMEDOUT
2025-05-23T21:38:50+02:00  May 23 19:38:50 [358] ERR: <lightning> Indexer failed, trying again in 10 seconds. Reason: ETIMEDOUT
2025-05-23T21:39:15+02:00  May 23 19:39:15 [358] ERR: <lightning> Cannot fetch tx 08da9276511d3f220f8cebcee44b070b108098ba24cd4320b43a23b62ac4b4cc. Reason: ETIMEDOUT
.....
2025-05-23T18:47:03+02:00  May 23 16:47:03 [358] ERR: <lightning> [Lightning] $runTasks() error: timeout of 10000ms exceeded
2025-05-23T18:47:03+02:00  May 23 16:47:03 [358] INFO: <lightning> [Lightning] Starting Lightning Stats service
2025-05-23T18:47:08+02:00  May 23 16:47:08 [358] NOTICE: <lightning> [Mining] Inserted 168 MtGox USD weekly price history into db
2025-05-23T18:47:13+02:00  May 23 16:47:13 [358] ERR: <lightning> Exception in $logStatsDaily. Reason: timeout of 10000ms exceeded
2025-05-23T18:47:13+02:00  May 23 16:47:13 [358] INFO: <lightning> Starting lightning network forensics service
2025-05-23T18:47:14+02:00  May 23 16:47:14 [358] INFO: <lightning> [Lightning] Caching funding txs for currently existing channels
2025-05-23T18:47:14+02:00  May 23 16:47:14 [358] INFO: <lightning> Lightning topology folder is not set. Not importing historical LN stats
2025-05-23T18:47:46+02:00  May 23 16:47:46 [358] INFO: <lightning> [Mining] Inserted 608 Kraken EUR, USD, GBP, JPY, CAD, CHF and AUD weekly price history into db
2025-05-23T18:48:54+02:00  May 23 16:48:54 [358] ERR: <lightning> $updateBlocks stalled at "getting block data for 898028"
....
2025-05-23T18:46:52+02:00  May 23 16:46:52 [358] WARN: <lightning> Failed to parse rbf cache. Skipping. Reason: ENOENT: no such file or directory, open './cache/rbfcache.json'
....
2025-05-23T22:10:49+02:00  May 23 20:10:49 [358] ERR: <lightning> [Mining] Block indexing failed. Trying again in 10 seconds. Reason: ETIMEDOUT
2025-05-23T22:10:49+02:00  May 23 20:10:49 [358] ERR: <lightning> Indexer failed, trying again in 10 seconds. Reason: ETIMEDOUT

If you read until here, thank you so much for your patience! :stuck_out_tongue_winking_eye:

2

Welcome.

You’re never going to be seen LN graph data if you install LND and then never use it. No peers, no channels = no data.

And hopefully you weren’t using LND considering you’ve been uninstalling it and reinstalling it – not something used do with LND, unless I suppose you are fabulously wealthy.

The rest of the logs, from all the services, show issues everywhere. I’m not sure exactly what’s up, but from your description of your actions, you’ve undertake a process of button-smashing in order to see what happens. With everything well and truly smashed, I suppose the main question I’d have for you is… are there funds involved here?

If not, let’s start over.

  1. Make sure you have a stable connection. Make sure you’re VM has it’s own IP and no weird caching settings are enabled. I’m concerned by several instances in your logs where time seems to be going backwards. Also check Tor logs to make sure things are working right and bootstrapping gets to 100% and doesn’t start over.

  2. Make sure Bitcoin is stable. Check the logs. Make sure it is synced to 100% make sure it sees each new block.

  3. Nuke and reinstall Electrs, then LND. I don’t know what to make of your logs, it can only be a proxmox setting, but I’d start over once you fix that.

  4. When Electrs is at 100%, install Mempool. Obviously Mempool will show nothing on the lightning tab, since you’re not using it.

Basically, in the above, we’re going in the dependency order, from the foundation and building up. None of this should be installed in random orders or all at the same time.

3

Hello StuPleb,

Thank you for your answer.

Here is my reaction :

  1. Ok I will do as you say and reinstall everything from scratch.

  2. I don’t agree about the correlation between Mempool statistics and LND usage because I have the same setup on Umbrel and it works like a charm out of the box without opening one single channel.

  3. There’s not fundings involved at the time, I’m in the process of evaluating different solutions. I will start using funds when I will make the choice based on stability, reliability, performance and community support.

I’ll post here how rebase has worked.

4

Ok I start to update on the process :

  • Bitcoin Knots synced
  • Electrs synced
  • LND block synced but syncing to graph for more than 10 hours => I checked the reason and it seems there was no peers connected, I connect it manually to popular nodes with the following commands executed from lnd.embassy container runtime environment :
lncli --rpcserver=lnd.embassy connect 03c436af41160c4d063db628cd6c38a19be0da2f15cafa6b5085dc13c157d49df3@35.196.132.34:9735 --perm
lncli --rpcserver=lnd.embassy connect 030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f@54.77.250.40:9735 --perm
lncli --rpcserver=lnd.embassy connect 03abf6f44c355dec0d5aa155bdbdd6e0c8fefe318eff402de65c6eb2e1be55dc3e@3.132.230.42:9735 --perm
lncli --rpcserver=lnd.embassy connect 035e4ff418fc8b5554c5d9eea66396c227bd429a3251c8cbc711002ba215bfc226@170.75.163.209:9735 --perm
lncli --rpcserver=lnd.embassy connect 03864ef025fde8fb587d989186ce6a4a186895ee44a926bfc370e2c366597a3f8f@3.33.236.230:9735 --perm
lncli --rpcserver=lnd.embassy connect 0217890e3aad8d35bc054f43acc00084b25229ecff0ab68debd82883ad65ee8266@66.109.24.41:9735 --perm
lncli --rpcserver=lnd.embassy connect 02f1a8c87607f415c8f22c00593002775941dea48869ce23096af27b0cfdcc0b69@193.118.169.108:9735 --perm
lncli --rpcserver=lnd.embassy connect 0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3@3.124.63.44:9735 --perm
lncli --rpcserver=lnd.embassy connect 03864ef025fde8fb587d989186ce6a4a186895ee44a926bfc370e2c366597a3f8f@3.33.236.230:9735 --perm
lncli --rpcserver=lnd.embassy connect 03cde60a6323f7122d5178255766e38114b4722ede08f7c9e0c5df9b912cc201d6@34.65.85.39:9745 --perm
lncli --rpcserver=lnd.embassy connect 0294ac3e099def03c12a37e30fe5364b1223fd60069869142ef96580c8439c2e0a@8.210.134.135:26658 --perm

Sometimes peers get disconnected :

2025-05-29T20:01:23+02:00  2025-05-29 18:01:23.261 [WRN] PEER: Peer(0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3): pong response failure for 0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3@3.124.63.44:9735: timeout while waiting for pong response -- disconnecting
2025-05-29T20:01:23+02:00  2025-05-29 18:01:23.261 [INF] PEER: Peer(0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3): disconnecting 0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3@3.124.63.44:9735, reason: pong response failure for 0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3@3.124.63.44:9735: timeout while waiting for pong response -- disconnecting

But connecting to them with “–perm” parameters makes my node reconect to them just few seconds later.

LND syncing started again let’s see if it will go through

5

Ok LND is perfectly synced on both blocks and graph but the problem on mempool persists :

2025-05-29T22:09:50+02:00  May 29 20:09:50 [359] ERR: <lightning> [Lightning] $runTasks() error: timeout of 10000ms exceeded
2025-05-29T22:09:50+02:00  May 29 20:09:50 [359] INFO: <lightning> [Lightning] Starting Lightning Stats service
2025-05-29T22:10:00+02:00  May 29 20:10:00 [359] ERR: <lightning> Exception in $logStatsDaily. Reason: timeout of 10000ms exceeded
2025-05-29T22:10:00+02:00  May 29 20:10:00 [359] INFO: <lightning> Starting lightning network forensics service
2025-05-29T22:10:00+02:00  May 29 20:10:00 [359] INFO: <lightning> [Lightning] Caching funding txs for currently existing channels
2025-05-29T22:10:00+02:00  May 29 20:10:00 [359] INFO: <lightning> Lightning topology folder is not set. Not importing historical LN stats

Mempool doesn’t display any Lightning statistics.

Because of the packaging system implemented on Start9 it’s quite hard to debug and tweak configuration files.

This is a StartOS problem, on my Umbrel stack on the same proxmox cluster with the VM having the same settings I can visualise lightning stats without any issue without the need to open any channel (see below) :

umbrel-lightning-stats
umbrel-lightning-stats1331×892 224 KB

6

I’m not entirely convinced its a StartOS problem – if it was, other users would experience the same issue. That said, I haven’t met anyone using LND without channels or peers. I’d be interested to know if you do every open a channel if the issue goes away.

I can’t speak to what exactly those two Mempool log errors mean… that’s purely a Mempool thing. Perhaps the next step is to contact them.

7

Hello StuPleb,

I just realized that Mempool on Umbrel doesn’t have LND as a dependency service, just Bitcoin Knots and Electrs.
I suppose they pull the graph stats from an external source like mempool.space API or whatever.
This behavior could be configured on the mempool application.

I will try to open a channel on Start9 and keep you up to date.

8

Hello @StuPleb,

I just opened a Lightning network channel with my node.
The problem with Mempool persists, who’s in charge of the package of mempool for Start9 Registry ?

9

Looks like someone else is having the same issue: