I’m running on a Lenovo thinkpad, 8gb ram, 2TB SSD with lcache. I’ve only installed btc knots and electrs. It keeps crashing, showing over and over the same few lines, the most prominent is: “failed to start avahi-daemon.service start9”. Couldn’t find a solution specific to start9 or btc.
I’d initially assume there’s something else going on rather than the one line of many you’d honed in on in the logs. If a service on linux doesn’t start, it should not crash the whole system. Instead, something else is crashing the system and while crashing a service fails to start.
What are the symptoms of “it” “crashing”?
Is Knots ok? What is it say under the health check at Services > Bitcoin Knots? If that’s what keeps crashing, check its log @ Services > Bitcoin Knots > Logs
I think I figured out the problem. For anyone else in the future, what happened, I think, was my /etc/systemd/logind.conf was reset to default settings when I last updated either knots or electrs, since I updated both at the same time. So I ssh into start9 and edited the logind.conf so the laptop wouldn’t suspend when I shut the lid. I think the conf file was reset because I had no problem with the lid being closed before I updated. Hope this helps someone in the future, and fingers crossed that this was the problem!
“Well, it looks like my changes to the logind.conf file isn’t sticking. After I change the file, I save and exit. Then I run sudo systemctl restart systemd-logind.service and the screen crashes on the server, but somehow I’m still able to connect remotely… until it crashes altogether. When I reboot, logind.conf is back to the defaults. I’m guessing I have to do something else before restarting the systemd-logind.service?”
Nope. That is wrong, it’s still crashing. The following is what shows on the screen over and over again after the crash:
[7559.968387] BTRFS error (device sda3): bdev /dev/sda3 errs: wr 0, rd 3590, flush 0, corrupt 0, gen 0
three of those lines with increasing numbers between brackets and after rd, then:
[7559.972459] systemd[1]: Failed to start avahi-daemon.service - Avahi mDNS/DNS-SD Stack.
“The server shows line after line of the same error messages. I’m assuming the problem is when I close the lid, it tries to hibernate. Knots is running fine. Health checks are fine, both say success.”
Nope. That is wrong, it’s still crashing. The following is what shows on the screen over and over again after the crash:
[7559.968387] BTRFS error (device sda3): bdev /dev/sda3 errs: wr 0, rd 3590, flush 0, corrupt 0, gen 0
three of those lines with increasing numbers between brackets and after rd, then:
[7559.972459] systemd[1]: Failed to start avahi-daemon.service - Avahi mDNS/DNS-SD Stack.
As far as I know, StartOS won’t listen to any hibernation signals from the hardware, but if you want to try to save your logind.conf settings use this shell:
/usr/lib/startos/scripts/chroot-and-upgrade
Setting changes there will actually have them stick upon reboot.
If that fixes it, let us know, but other than that, I’d say you’re looking at disk errors on sda3.
1 Like
Well I’m at a loss now. I tried running all kinds of scans on the SSD via a live linux boot up and they all come back successful without any errors. And I no longer think it is a hibernation type problem because I’m leaving the laptop open and it still crashes. Not right away mind you, but after a few hours up to a day. I even tried a system rebuild from the start9 menu, and it is still crashing. Any other suggestions?
Can you post your dmesg output?
[7559.968387] BTRFS error (device sda3): bdev /dev/sda3 errs: wr 0, rd 3590, flush 0, corrupt 0, gen 0
Means you’re having thousands of read errors from the disk. Could be the disk, filesystem, disk controller, etc. Do you have another disk you can try? Do you have a backup of your services?
I can make a backup of my services, but it doesn’t backup the actual blockchain, right? I’d have to buy another disk…I’ve linked the dmesg log file here: Download 2025-12-20 - StartOS - BTC node - error log - dmesg.odt | LimeWire
It doesn’t backup the Blockchain but it’s just good practice to use a hefty hard drive. Especially if you’re using file storage, nextcloud etc
No, it’s a dedicated drive for a btc node. I’ll be using a separate setup for things like nextcloud
404
Content not found
It will only stay on the site for 7 days…I just re uploaded the file: Download 2025-12-20 - StartOS - BTC node - error log - dmesg.pdf | LimeWire
BTRFS is reporting tons of errors reading from sda3.
Can you try another disk? Looks like a disk issue. It may be dying/dead.
Hopefully you can either do a transfer with the new disk inside the unit and the old disk in a USB enclosure attached by USB:
https://docs.start9.com/0.3.5.x/misc-guides/transfer-data
Or you can restore from backup onto the new disk if you have a backup.
1 Like
Thanks for the quick reply.
Also just uploaded some screenshots with errors when trying to make an image backup of my entire SSD with clonezilla…maybe it is corrupt and I need to replace it?
Cool, I’ve never seen that page on start9.com before, even though I tried searching for it. Thanks again!
Tried doing a transfer with a new disk. Unfortunately, I got this error:
Error
RPC ERROR: Filesystem I/O Error cp /media/embassy/migrate/package-data/volumes/bitcoind/data/main/blocks/blk02543.dat → /embassy-data/package-data/volumes/bitcoind/data/main/blocks/blk02543.dat: Input/output error (os error 5)
t looks very likely that your SSD is failing.
The most important thing now is to avoid writing anything more to it and try to copy data off as safely as possible. If you can, remove the drive from the ThinkPad and connect it to another machine using a SATA/USB or NVMe/USB adapter or enclosure, then:
-
See if the disk is detected at all in the OS or BIOS.
-
If it is, try to make a full disk image (clone) to another healthy drive first, and then work from that image rather than the failing disk itself.
-
If the disk cannot be read reliably or you hear strange noises, further attempts may make recovery worse and you may want to consider a professional data recovery service if the data is critical.
If you have Start9 backups, restoring them onto a new SSD will usually be much faster and safer than fighting with a dying disk. Once you have a new, known‑good drive installed, you can either transfer data from backup or start from a clean install and resync Bitcoin.
I have another method here you can try. It’s very involved but possible it may work:
Seems you have disk issues.
If you lack a backup, you may have luck recovering some of your data via this long process:
- Boot up your server with the new disk and StartOS flashed onto it
- Visit http://embassy.local
- Choose “Fresh Install” and select the fresh SSD as the target of installation in the web setup wizard
- Log in for the first time and go to System > Shutdown
- Download Linux Mint: Linux Mint 22.2 "Zara" - Linux Mint
- Burn the iso to a USB thumb drive with balena or your favorite flashing tool and boot it on your StartOS server
- While booted into mint, insert your old ssd from your server with the data issues into a usb port using a m.2<->usb adapter of some kind and go into the file manager, you should see the disk
- Click on it and mount it using the password ‘password’
- Mount the internal ssd in the server the same way – you should see an encrypted partition available to mount which will have an empty volumes directory since you’ve installed no services on the disk
- Copy the contents of the old disk’s ‘volumes’ folders to the new disk’s empty volumes folder (excluding the bitcoind folder since we know the disk has errors retrieving some of the block files there based on the error message that displayed when you attempted the Transfer opteration).
- Hopefully they all copy ok
- Reboot and as it does, remove the Mint USB thumb drive and the old disk’s USB adapter to disconnect them both, and reboot to the StartOS on the new internal disk
- Install the services you had beforehand on the old disk, anew, via the Marketplace
- Each new service should pick up the old data
If you had copying issues along the way, your data just may be unrecoverable.
Thanks for the advice. I ended up buying a new 2TB ssd and using gddrescue to clone it. Had to execute the command twice because it got stuck the first time. So far so good…