# StartOS v0.4.0.3 update — Read-only filesystem, I/O errors, services failing to start **URL:** \ **Category:** StartOS Support\ **Created:** [October 25, 2026, 4:06pm UTC](https://community.start9.com/t/startos-v0-4-0-3-update-read-only-filesystem-i-o-errors-services-failing-to-start/5887 "2026-10-25T16:06:58Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![comfyslippers](https://community.start9.com/letter_avatar_proxy/v4/letter/c/e47774/32.png) [@comfyslippers](https://community.start9.com/u/comfyslippers)\ **Post date:** [October 25, 2026, 4:06pm UTC](https://community.start9.com/t/startos-v0-4-0-3-update-read-only-filesystem-i-o-errors-services-failing-to-start/5887/1 "2026-10-25T16:06:58Z")
**Hardware:** Start9 Server One (2026), 16 GB RAM, 2 TB NVMe **StartOS version:** v0.4.0.3 **Date:** October 25, 2026 **Previous version:** Unknown ## Problem summary Immediately after updating StartOS to v0.4.0.3, the server became inaccessible through both its web interface and SSH. After waiting well over an hour, I performed a forced power cycle because the server would not respond to a normal short press of its power button. The server subsequently booted, and both SSH and the web interface became accessible. However, the system now exhibits serious filesystem errors, missing or inaccessible executables, failed services, and database write failures. **The most significant error is:** ```auto RPC ERROR: Database Error IO Error: Read-only file system (os error 30) ``` This occurs when attempting to access notifications in the StartOS web interface. ## Timeline and troubleshooting 1. Successfully completed a StartOS backup to a network SMB share immediately before installing the update. 2. Installed v0.4.0.3 through the StartOS web interface. 3. I don’t think the system ever rebooted; the server responded to ping, but ports 80 and 443 refused connections. 4. SSH port 22 was open, but connections reset before the SSH identification/key exchange. 5. Waited more than an hour without change. 6. A short press of the physical power button did not initiate shutdown. 7. Performed a forced power cycle using the power button. 8. SSH and the web interface returned, but system and application errors persisted. ## SSH and operating-system errors The StartOS management daemon reports as running: ```auto startd.service - StartOS Daemon Active: active (running) ``` However, basic commands that previously worked are now failing. Attempting to access standard executables: ```auto ls -l /usr/bin/clear /usr/bin/date /bin/clear /bin/date ``` Returns: ```auto Input/output error ``` Other observations: - `clear` and `date` are no longer available through the shell. - `cat /proc/mounts` produces `Segmentation fault`. - `timedatectl status` fails after 25 seconds with: ```auto Failed to query server: Failed to activate service 'org.freedesktop.timedate1': timed out (service_start_timeout=25000ms) ``` - `dbus.service` reports `active (running)`. - Unprivileged `journalctl -k -b` reports no journal files. ## StartOS web interface and services The web interface is accessible but reports **Clock sync failure**. Application status is abnormal: - **Bitcoin Core:** Remains in Starting state. - **Tor:** Initially stopped, later began starting and connecting to peers, with progress stuck at around 5%. - **Fulcrum:** Reported as running. - **DATUM:** Reports it cannot run because its Fulcrum dependency is unavailable. - **Am I Exposed?:** Waiting on Tor. Both the OS Logs and Kernel Logs viewers display: ```auto Disconnected Reconnecting ``` The notification interface produces the read-only filesystem/database error noted above. ## System metrics From the StartOS dashboard approximately 17 minutes after the forced restart: | Metric | Value | | --- | --- | | CPU utilization | 5.0% | | I/O wait | 18.6% | | Memory usage | 8.4% | | Storage usage | 63.5% | | Storage available | 723.48 GB | | Temperature | 54.9°C | There is no indication that the storage volume is full. ## Current status The machine is powered on and accessible through SSH and the web interface, but many services are not functioning. No manual filesystem repairs, remounts, package installations, or factory resets have been attempted. A successful pre-update backup is available on a separate machine. ## Questions 1. Is this a known issue with StartOS v0.4.0.3 or its update/migration process? 2. Could the update have left the root filesystem or system image in a read-only or corrupted state? 3. Is there a supported diagnostic or recovery procedure that can be performed through SSH without reinstalling? 4. How can I distinguish an update-related filesystem issue from an underlying NVMe hardware failure? 5. If reinstalling is necessary, can the existing application data be preserved, or is a full restore from backup required? I’d prefer to avoid a factory reset or reinstall unless necessary, especially since the server contains a synchronized Bitcoin Core node. Any guidance on safe diagnostics or recovery would be appreciated. ---
**Author:** ![simberety372](https://community.start9.com/letter_avatar_proxy/v4/letter/s/ecb155/32.png) [@simberety372](https://community.start9.com/u/simberety372)\ **Post date:** [October 25, 2026, 4:13pm UTC](https://community.start9.com/t/startos-v0-4-0-3-update-read-only-filesystem-i-o-errors-services-failing-to-start/5887/2 "2026-10-25T16:13:56Z")
I had the exact same error, to solve it I just powered off the machine for a day and then started to work properly. Also there were another person with the same error: > [@URGENT - VAULTWARDEN DOWN. UPGRADE FROM 1.37.4:0 to 1.37.4:1 FAILS](https://community.start9.com/t/urgent-vaultwarden-down-upgrade-from-1-37-4-0-to-1-37-4-1-fails/5869): > > This has me in a bit of a panic as ALL my passwords are in my Vaultwarden server which no longer recognizes my master password from either Brave or Firefox. StartOS notified me there’s a Vaultwarden upgrade available which I thought might solve the problem, but when I attempted to update Vaultwarden, I get an RPC error and a server connection pop-up that comes and goes. Me: > [@My Start9 is cooked? Can't log in](https://community.start9.com/t/my-start9-is-cooked-cant-log-in/5863): > > Just tried to log in and there shows an error. That’s the correct password, if I try with a wrong one, it shows “incorrect password”. In fact the server isn’t working in the background, I am trying to connect to the fulcrum on sparrow and It shows “unrecognized name”. A recent thing that I have done is to upgrade to 0.4.0.2. But that upgrade was successful that time. After reboot the server (physically), I was able to log in but the server was OK for 1 day, then started experiencing the same is… ---
**Author:** ![comfyslippers](https://community.start9.com/letter_avatar_proxy/v4/letter/c/e47774/32.png) [@comfyslippers](https://community.start9.com/u/comfyslippers)\ **Post date:** [October 25, 2026, 5:13pm UTC](https://community.start9.com/t/startos-v0-4-0-3-update-read-only-filesystem-i-o-errors-services-failing-to-start/5887/3 "2026-10-25T17:13:26Z")
## Update: Possible recurrence of previously resolved NVMe/PCIe power-management failure I’ve remembered some important history that may explain the current problem. This server experienced an almost identical series of failures shortly after its initial setup on October 1, 2026. At that time, the problem was traced to the NVMe controller becoming unresponsive, apparently related to PCIe/NVMe power-state transitions. ### Original failure — October 1 During the initial Bitcoin Core blockchain synchronization, the server repeatedly experienced storage failures that eventually left the filesystem read-only and caused StartOS services to malfunction. Kernel diagnostics at the time showed: - The NVMe controller entering the **D3cold** PCIe power state. - Failed attempts to transition the controller back from D3cold to D0. - NVMe controller reset failures. - PCI configuration reads returning `ffff`, consistent with the controller becoming inaccessible. - The NVMe namespace effectively disappearing or being reported as a zero-byte device. - Btrfs reporting missing-device and write errors. - The filesystem becoming read-only. The system would temporarily recover after a hard restart, but the problem returned under load. Interestingly, SMART diagnostics showed the NVMe drive as healthy, with zero reported media errors. This led me to investigate PCIe and NVMe power management rather than immediately suspecting defective NAND storage. ### Original resolution On October 1, I modified `/etc/default/grub` to include these kernel parameters: ```auto nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off ``` I then regenerated the GRUB configuration using: ```auto sudo /usr/sbin/update-grub ``` I verified that the parameters were present in `grub.cfg` for both normal and recovery boot entries. After rebooting, I also verified that all three parameters appeared in `/proc/cmdline`, confirming that the kernel was actually receiving them. The result was a dramatic improvement in stability. Bitcoin Core subsequently ran for several hours under synchronization load without another NVMe failure. The server survived overnight, eventually completed its initial blockchain download, and continued operating normally for approximately three and a half weeks. ### Connection to the October 25 update failure Immediately after updating StartOS to v0.4.0.3, the server began exhibiting symptoms remarkably similar to those original failures. The initial post-update boot left the machine responding to ping but without a functioning web interface or usable SSH connection. After waiting more than an hour, a forced power cycle restored access temporarily. However, the system then exhibited: - `Input/output error` when accessing executables under `/usr/bin`. - `cat /proc/mounts` producing a segmentation fault. - `timedatectl` timing out while attempting to activate `systemd-timedated`. - StartOS reporting `Read-only file system (os error 30)` when accessing notifications. - Bitcoin Core remaining stuck in its Starting state. - Other services failing to start or satisfy dependencies. - OS and kernel log viewers repeatedly disconnecting. - Elevated I/O wait despite low overall CPU utilization and plenty of available storage. Since my original post, the situation has deteriorated further. SSH access has failed again, and the web interface has become unresponsive. I am now shutting the server down. ### Current hypothesis **I’m wondering whether the v0.4.0.3 operating-system update regenerated or replaced the GRUB configuration, removing the custom kernel parameters that resolved the original NVMe failures.** If those parameters were lost, the NVMe controller could once again be exposed to the same problematic power-state transitions. That would provide a plausible explanation for the recurrence of nearly identical symptoms immediately following the OS update. I realize that similar symptoms do not prove the same underlying cause, and I have not yet obtained kernel logs confirming another D3cold failure. However, the previous diagnosis, the verified configuration changes, and the subsequent weeks of stability make this possibility worth investigating. **My next step, once the server is accessible again, is to inspect `/proc/cmdline` and verify whether those three kernel parameters survived the update.** I’ll report back with the results. In the meantime, I’d be interested to know whether StartOS operating-system updates are expected to preserve user modifications to `/etc/default/grub` and the generated GRUB configuration. @simberety372 Thanks for the quick reply. Those do seem related or, at least part of the same larger issue. ---
**Author:** ![comfyslippers](https://community.start9.com/letter_avatar_proxy/v4/letter/c/e47774/32.png) [@comfyslippers](https://community.start9.com/u/comfyslippers)\ **Post date:** [October 25, 2026, 6:02pm UTC](https://community.start9.com/t/startos-v0-4-0-3-update-read-only-filesystem-i-o-errors-services-failing-to-start/5887/4 "2026-10-25T18:02:10Z")
## Update #2 — GRUB Configuration Restored, Server Recovered Following up on my previous post about the possibility that the StartOS v0.4.0.3 update had overwritten the custom GRUB configuration implemented on October 1. After getting the server running again, I was able to regain SSH access and inspect the active kernel command line: ```auto cat /proc/cmdline ``` All three of the NVMe/PCIe power-management parameters I had previously added were missing. I then checked `/etc/default/grub` and found that those parameters had also been removed from the configuration itself. ### Recovery steps I restored the following kernel parameters, which had originally resolved the NVMe controller failures experienced during the server’s initial setup: ```auto nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off ``` Specifically, I edited `/etc/default/grub` so that the relevant line reads: ```auto GRUB_CMDLINE_LINUX="boot=startos console=ttyS0,115200n8 console=tty0 nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" ``` I preserved the existing StartOS boot arguments, then regenerated the GRUB configuration: ```auto sudo /usr/sbin/update-grub ``` This completed successfully. I subsequently inspected `/boot/grub/grub.cfg` and verified that all three parameters were present in the generated boot entries for both installed kernels, including the recovery entries. After a controlled reboot, I checked `/proc/cmdline` again and confirmed that the running kernel (`7.2.9+deb13-amd64`) was receiving all three parameters. ### Results The server has now recovered substantially: - All seven installed StartOS services are running normally. - Bitcoin Core, Fulcrum, DATUM Gateway, Core Lightning, Mempool, Tor, and Am I Exposed? are operational according to StartOS. - All four connected miners have automatically reconnected to the local DATUM gateway and resumed mining through their primary pool configuration. - SSH access is functioning. - Commands that previously failed with I/O errors (`date`, `clear`, etc.) are working normally. - The root filesystem was confirmed mounted read-write during recovery. - I/O wait has dropped from approximately 18.6% during the earlier failure to approximately 1% after recovery. - NTP synchronization initially reported a failure but subsequently completed successfully. There was also an unrelated-looking SSH access issue following the update: the StartOS SSH Keys page contained no registered keys. Re-adding my existing public key immediately restored SSH access. **I cannot yet establish whether the update itself removed that registration** , but it is worth mentioning. ### Concern regarding Start9-supplied hardware One aspect of this situation concerns me beyond the immediate software recovery: **This is a brand-new Start9 Server One, purchased as a complete, preconfigured hardware/software system directly from Start9.** The original failures began during initial setup, while Bitcoin Core was performing its initial blockchain download. Kernel diagnostics at that time showed the NVMe controller entering D3cold, failing to return to D0, unsuccessful controller resets, and eventual Btrfs read-only filesystem errors. SMART diagnostics showed no media errors, and the three kernel parameters above appeared to resolve the problem completely for approximately three and a half weeks. I would expect a turnkey server sold by the operating-system developer to have its specific motherboard, NVMe controller, firmware, and power-management configuration validated against the supplied operating system. I recognize that hardware and firmware interactions can be complicated, but a newly purchased, supported hardware platform should not require undocumented kernel boot parameters to prevent its storage controller from disappearing under normal workloads, and if such parameters are necessary for this hardware revision, they should ideally be part of the supported default configuration—or at least preserved across operating-system updates. The loss of those parameters during an otherwise routine OS update apparently exposed the machine to the same failure mode that had already been resolved weeks earlier. ### Remaining questions I’d appreciate clarification on whether: 1. StartOS updates are expected to regenerate `/etc/default/grub` and discard custom kernel parameters. 2. These NVMe/PCIe power-management parameters are known to be necessary on any Server One hardware revisions. 3. There is a supported, update-persistent method for configuring kernel boot parameters. 4. The hardware/firmware combination in this Server One should be investigated for a compatibility issue or possible defect. **For clarity, the causal connection is still a hypothesis:** I’ve confirmed that the update removed the previously successful workaround, but I have not captured fresh kernel logs proving that another D3cold transition caused the October 25 failure. Nevertheless, the matching symptoms, the confirmed configuration change, and the successful recovery after restoring the parameters make that explanation compelling. ### Next steps I’m leaving the server running under its normal workload, including Bitcoin Core, Fulcrum, DATUM, and four connected miners. **I’ll report back after approximately 24 hours of observation** to confirm whether the system remains stable with the restored kernel parameters. Hopefully this information will also help anyone else experiencing unexplained NVMe controller resets or read-only filesystem failures after a StartOS update.