I’m running a start9 node. Electrs doesn’t run properly most of the time:
rum RPC Interface
Success : The electrum RPC interface is alive and responding to basic commands.
Sync Status
The electrs Prometheus RPC is not yet returning the sync status
Thus mempool won’t work properly. I ran some logs though AI and it told me this:
This is a genuine RocksDB corruption/disk I/O problem, not a normal sync-in-progress state — that’s why the Prometheus sync-status metric never appears (electrs is crash-looping before it ever finishes initializing).
Key evidence:
Input/output error on 1528699.sst → the OS/filesystem returned a hardware-level read error on that file.
Corruption: block checksum mismatch on 1528691.sst → a different SST file’s on-disk bytes no longer match their stored checksum.
Two different corrupted files in the same DB, both underlying storage-layer issues, strongly points to failing disk/storage rather than an electrs bug.
It gave some steps to resolve or go further, but I’d like advice from here. Is there something wrong with the memory in my device? Is this fixable? Thank you.
It could be a memory issue or a disk issue or a disk controller issue, or just a transient error.
Try uninstalling electrs via Services > Electrs > Actions & Config > Uninstall.
Then re-install it from the marketplace. Do you encounter I/O errors or block checksum mismatches when the index is built?
If so, we can initiate an RMA if you let me know your order number. We’d do a return and get it diagnosed and any defective hardware replaced and send back to you.
Ok, almost 2 days on the ‘finishing database compaction step’ and there were issues in the logs. I ran it through AI and here is what it said:
Yes — this log tells us something important that the previous one didn’t: it’s now the exact same file, at the exact same byte offset, every single time (000661.sst, offset 15388672, len 2511), across 6 consecutive crash/restart cycles over several hours.
That’s a meaningfully different signature from your earlier log (which hit two different files/offsets). A few implications:
This points to a genuinely unrecoverable, fixed bad sector/block, not transient I/O flakiness. If it were a flaky network mount or intermittent driver issue, you’d expect the failure location to vary or occasionally succeed. Hitting the identical offset in the identical file 6/6 times means that specific data on disk is truly damaged/unreadable — either a physical bad sector, a filesystem-level corruption at that block, or (if this is on network/cloud storage) a persistent backend read failure for that exact page.
It’s always failing during “spending compaction” — same stage each time. Combined with #1, this confirms it’s not a race condition in electrs; it’s deterministically trying to read the same corrupted data whenever it reaches that compaction phase.
Notice the DB keeps growing every restart cycle (3234 → 3236 → 3239 → 3240 → 3242 SST files, size creeping up ~262.36xx GB) because electrs successfully re-indexes a handful of new blocks (indexing N blocks) before crashing again at the same compaction step. So it’s making forward progress on new block ingestion but never actually completing compaction — meaning disk usage will keep climbing with each crash-restart loop while the underlying corruption never gets resolved.
I’m guessing an RMA might be the right step at this time? My order number is X480759.
Can you please look at your Bitcoin logs? Is it healthy?
If it seems so, there might still be an issue with one of the files, or potentially the disk. But the best thing to do I think, so far, is a reindex of the blockchain via Services > Bitcoin Core > Actions & Config > Reindex Blockchain
Does that succeed or not? If it finds corruption in the blockchain, it should resolve it, and that may subsequently resolve your electrs issue. LMK how it goes.
It never finished, and I had an internet service disruption so had to start over. It’s been a couple of days and it’s still trying. Goes from syncing progress back to the start again. I put a portion of a log into AI and it said:
ere’s what’s happening in that log:
1. Fatal corruption event (first block of lines)
Bitcoin Core hit a LevelDB checksum mismatch in chainstate/030642.ldb — a corrupted chainstate file.
This is treated as a fatal, unrecoverable error by design (LevelDB corruption can’t be trusted), so Core called abort() and the process died via SIGABRT.
The RPC calls right after (Could not connect to 127.0.0.1:8332) are just Start9 health-checks failing because bitcoind was down at that moment — expected, not a separate problem.
It re-opened /root/.bitcoin/chainstate with LevelDB and this time succeeded — note the obfuscation key 26ca29e029cca789 (non-zero), meaning it’s using an existing chainstate DB, not one freshly wiped.
LoadBlockIndexDB shows the block index still references block files up to 5774 (heights ~968,396–968,420), confirming raw block data on disk is intact and near current chain tip.
3. The concerning part
After loading, it reports: Loaded best chain: ... height=696216 ... date=2021-08-17 ... progress=0.477036
That means the chainstate (UTXO set) it recovered only reflects height 696,216 (August 2021), even though the block files on disk go up to ~height 968,420 (current day).
It’s now replaying/connecting blocks one-by-one from 696,217 onward (UpdateTip lines climbing sequentially), rebuilding the UTXO set as it goes.
This is essentially the LevelDB corruption forcing a rollback to whatever last-good chainstate snapshot LevelDB could recover — roughly 272,000 blocks behind, not just a handful.
Bottom line / what to expect
Your node is not synced — it’s re-validating/rebuilding the UTXO set from mid-2021 forward. This is CPU/IO-heavy and will take a long time (many hours) since it’s connecting every block sequentially, not fast-syncing.
No wallet/fund loss is implied by this — it’s purely a local database integrity issue, and Core is self-healing by replaying blocks from disk.
Recommendations:
Let it run uninterrupted; interrupting again mid-replay risks another corruption.
Check disk health (SMART status) — checksum mismatches in LevelDB are often caused by underlying disk errors, bad sectors, or an unclean shutdown (power loss, OOM-kill, forced container restart).
Once it catches back up to the disk block-file height (~968,420), monitor for repeat corruption; if it recurs, consider a full -reindex-chainstate or reindex, and verify the storage medium.
Please advise. I just upgraded to 1gig internet speed but that doesn’t seem to make a difference.
Hi Bert, I’d stop the reindex here. It’s very unlikely to finish on this hardware, and each attempt costs you days.
Three separate databases on your server have now failed their integrity checks: electrs’ index twice, and now Bitcoin Core’s copy of the blockchain state. One of the electrs failures hit exactly the same spot in the same file six times in a row. Electrs and Bitcoin Core don’t share any database code, so what they do share is the drive and the memory underneath them. One of those is corrupting data, and anything you rebuild on top of it will keep breaking. Your internet connection has nothing to do with it, so the speed upgrade won’t change anything, and the outage wasn’t what made it start over. Bitcoin Core crashed when it read a corrupted file, and that crash is what restarted it.
The AI summary of your last log got one thing wrong. Your node didn’t roll back to a 2021 snapshot. Reindex Blockchain throws away that state on purpose and rebuilds it from the block files on disk, so the height it showed is just how far the rebuild had got before it hit a bad file and crashed. Nothing was saved from before.
It’s still unclear whether the drive or the memory is at fault, and these two commands would help narrow it down for the RMA. Connect to your server over SSH, run them, and paste what they print here:
The first shows the drive’s own health record. The second pulls any storage errors from the last week of system logs. If they come back clean, that points more toward the memory.
Before the server goes back to us, if you run Lightning or anything else with data you can’t simply download again, make a backup to an external drive now, from System → Create Backup. You don’t need to back up Bitcoin or electrs, since they rebuild from the network.
Definitely time to be emailing support@start9.com with your support number and a link to this post.
I’ve emailed support. I could not figure out how to connect via SSH (and/or the server didn’t recognize what I think is my servername). I don’t run lightning or anything else so no need to backup anything I believe.