I’m new to Start9, but I’ve run Casa and Umbrel nodes. I’ve installed Start9 on an older computer with a new SSD and it seems to be working; however, after a few hours when I go to the Services panel, electrs has the spinning icon and is constantly “Starting…”. I’ve rebooted a few of times and even installed a recent Start9 OS update, but I’m still seeing this behavior. FWIW, Bitcoin Core, LND, Mempool, and RTL all report “Running” and seem to work; however, my testing is limited since I haven’t added any BTC yet. Any ideas what the problem with electrs might be? Thanks!
It might be best for you to head over to Telegram support so we can get a live look at your logs… but…
-
I’d check whether your Bitcoin Core service is healthy and sync’d. It may not be.
-
You may still actually be in the initial indexing phase of Electrs, and this can take some time to complete. It wouldn’t account for it to be complete, and then after some time incomplete, it’s just that you might have assume it had completed before when it hadn’t. If you’re saying it’s always Starting though, this may be the case.
-
Your hardware may just not be up to spec to run what you’re trying to run, and things may be falling behind. Particularly, if Electrs and Core and falling behind, it might be a memory issue.
-
Looks good. I haven’t seen Bitcoin fall behind the block height at all and it’s been running for over a week.
-
No. Indexing was complete. My basic approach to building this box has been to install each component, wait for it to finish syncing, and then install the next component.
-
It’s older hardware, but I think it’s sufficient. It’s an AMD Phenom II, running on an ASRock 970 Extreme 4, with 16Gb RAM, and 2Tb SSD.
Thanks for the feedback! It did prompt me to check the logs. Electrs’ logs weren’t doing anything. Mempool was logging its failure to connect to electrs. As a result, I stopped everything, uninstalled electrs, reinstalled it, accepted the default config for it, and rebooted. Right now, electrs is logging its sync progress.
I’ll let you know if that fixes it.
Sadly, that didn’t fix it. Both electrs and mempool seemed to run reliably for 6 to 12 hours, but when I checked the next day, electrs was back to “Starting…”. I shutdown and uninstalled both. Then I reinstalled electrs. It’s been running reliably for 24+ hours. Of course, with nothing hitting it, I kinda expected it would be stable.
It’s just not expected behavior, or anything I’ve ever heard of before.
That the logs for Electrs don’t even start being recorded? It simply has to mean electrs isn’t running at all at that point.
Typically a System Rebuild (System → scroll to bottom) would run roughshod over whatever is causing it and just get things back to normal.
If/when this happens next, you’ll need to look at the OS and Kernel logs for some other troubleshooting starting point. But from a troubleshooting point of view, I’d also personally consider backing up the blockchain, wiping everything and starting again. There’s some unknown gremlin as work, nuking from space can be easier.
So electrs went back to “Starting…” after a day or two. I suspect the issue is with electrs or my hardware and that mempool, at worse, precipitates the problem.
The electrs logs reported something like trying to connect to bitcoind and failing. It would retry every few minutes or so. I deleted electrs before I saw your message, so I don’t have the logs.
I’m going to try the system rebuild. Can you help me understand the extent of that rebuild? I assume my ( 0 balance) Bitcoin and lightning nodes will come back with different pass phrases, the tor links will all be reset to something else, and I’ll have to redownload the CA file. Anything else?
A rebuild only rebuilds the containers, and will not touch any user data. You won’t have to do any recover work following a rebuild. It sounds to me like you are most likely experiencing a hardware problem. You seem to have sufficient resources, so I’m not sure exactly what might cause this. As Stu mentioned, it may be time for the nuclear option.
Ok. Reinstalling electrs now. I’ll let it run for a few days and then let you know if it flakes out again.
Seems stable after 3 days-- no “Starting…”. I’m installing mempool now. I’ll check back in, in a few days.
And electrs is back to “Starting…”. Here’s the end of the electrs logs:
2023-12-08T07:53:18-06:00 [2023-12-08T13:53:18.872Z INFO electrs::chain] chain updated: tip=00000000000000000002063fb5808f12c1976d83857cb2ad54e5a4ef42c2b480, height=820266
2023-12-08T08:24:54-06:00 [2023-12-08T14:24:51.358Z INFO electrs::index] indexing 1 blocks: [820267…820267]
2023-12-08T08:24:54-06:00 [2023-12-08T14:24:53.683Z INFO electrs::chain] chain updated: tip=0000000000000000000014369cc36413bae798ec01586ee3776847dbbbb2edd3, height=820267
2023-12-08T08:25:36-06:00 [2023-12-08T14:25:36.924Z INFO electrs::index] indexing 1 blocks: [820268…820268]
2023-12-08T08:25:39-06:00 [2023-12-08T14:25:39.365Z INFO electrs::chain] chain updated: tip=000000000000000000011648e38dffb71b38aefe7683ce79f6ff299125a3fb4c, height=820268
2023-12-08T08:26:43-06:00 thread ‘main’ panicked at src/db.rs:286:34:
2023-12-08T08:26:43-06:00 CF flush failed: Error { message: “IO error: While fdatasync: ./db/bitcoin/003199.sst: Input/output error” }
2023-12-08T08:26:43-06:00 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
2023-12-08T08:26:43-06:00 [2023-12-08T14:26:39.221Z INFO electrs::db] closing DB at ./db/bitcoin
IO errors are typically related to your disk and/or data connection
The OS log is reporting:
ERROR startos::manager::manager_container: The service electrs has crashed with the following exit code: 126
Google has informed me that:
Exit Code 126 means that a command used in the container specification could not be invoked
If this is not hardware related, then you could have binary corruption. In this case, try a System → System Rebuild. Even then, hardware may be the root of the problem.
1 Like
try running without mempool.space
I’ve noticed in my own experience that mempool.space is putting a lot of strain on electrs. Your hardware (and in my case, a Pi4 node) may not be able to handle it.
Not sure if this is related or will solve this specific case, but electrs is working on optimizing the handling of mempool transactions here: Async mempool scanning by conduition · Pull Request #970 · romanz/electrs · GitHub
2 Likes