StartOS v0.4.0.3 update — Read-only filesystem, I/O errors, services failing to start

StartOS Support
1

Hardware: Start9 Server One (2026), 16 GB RAM, 2 TB NVMe
StartOS version: v0.4.0.3
Date: October 25, 2026
Previous version: Unknown

Problem summary

Immediately after updating StartOS to v0.4.0.3, the server became inaccessible through both its web interface and SSH.

After waiting well over an hour, I performed a forced power cycle because the server would not respond to a normal short press of its power button.

The server subsequently booted, and both SSH and the web interface became accessible. However, the system now exhibits serious filesystem errors, missing or inaccessible executables, failed services, and database write failures.

The most significant error is:

RPC ERROR: Database Error
IO Error: Read-only file system (os error 30)

This occurs when attempting to access notifications in the StartOS web interface.

Timeline and troubleshooting

  1. Successfully completed a StartOS backup to a network SMB share immediately before installing the update.

  2. Installed v0.4.0.3 through the StartOS web interface.

  3. I don’t think the system ever rebooted; the server responded to ping, but ports 80 and 443 refused connections.

  4. SSH port 22 was open, but connections reset before the SSH identification/key exchange.

  5. Waited more than an hour without change.

  6. A short press of the physical power button did not initiate shutdown.

  7. Performed a forced power cycle using the power button.

  8. SSH and the web interface returned, but system and application errors persisted.

SSH and operating-system errors

The StartOS management daemon reports as running:

startd.service - StartOS Daemon
Active: active (running)

However, basic commands that previously worked are now failing.

Attempting to access standard executables:

ls -l /usr/bin/clear /usr/bin/date /bin/clear /bin/date

Returns:

Input/output error

Other observations:

  • clear and date are no longer available through the shell.

  • cat /proc/mounts produces Segmentation fault.

  • timedatectl status fails after 25 seconds with:

Failed to query server:
Failed to activate service 'org.freedesktop.timedate1':
timed out (service_start_timeout=25000ms)

  • dbus.service reports active (running).

  • Unprivileged journalctl -k -b reports no journal files.

StartOS web interface and services

The web interface is accessible but reports Clock sync failure.

Application status is abnormal:

  • Bitcoin Core: Remains in Starting state.

  • Tor: Initially stopped, later began starting and connecting to peers, with progress stuck at around 5%.

  • Fulcrum: Reported as running.

  • DATUM: Reports it cannot run because its Fulcrum dependency is unavailable.

  • Am I Exposed?: Waiting on Tor.

Both the OS Logs and Kernel Logs viewers display:

Disconnected
Reconnecting

The notification interface produces the read-only filesystem/database error noted above.

System metrics

From the StartOS dashboard approximately 17 minutes after the forced restart:

Metric Value
CPU utilization 5.0%
I/O wait 18.6%
Memory usage 8.4%
Storage usage 63.5%
Storage available 723.48 GB
Temperature 54.9°C

There is no indication that the storage volume is full.

Current status

The machine is powered on and accessible through SSH and the web interface, but many services are not functioning.

No manual filesystem repairs, remounts, package installations, or factory resets have been attempted.

A successful pre-update backup is available on a separate machine.

Questions

  1. Is this a known issue with StartOS v0.4.0.3 or its update/migration process?

  2. Could the update have left the root filesystem or system image in a read-only or corrupted state?

  3. Is there a supported diagnostic or recovery procedure that can be performed through SSH without reinstalling?

  4. How can I distinguish an update-related filesystem issue from an underlying NVMe hardware failure?

  5. If reinstalling is necessary, can the existing application data be preserved, or is a full restore from backup required?

I’d prefer to avoid a factory reset or reinstall unless necessary, especially since the server contains a synchronized Bitcoin Core node.

Any guidance on safe diagnostics or recovery would be appreciated.

2

I had the exact same error, to solve it I just powered off the machine for a day and then started to work properly.

Also there were another person with the same error:

Me:

3

Update: Possible recurrence of previously resolved NVMe/PCIe power-management failure

I’ve remembered some important history that may explain the current problem. This server experienced an almost identical series of failures shortly after its initial setup on October 1, 2026.

At that time, the problem was traced to the NVMe controller becoming unresponsive, apparently related to PCIe/NVMe power-state transitions.

Original failure — October 1

During the initial Bitcoin Core blockchain synchronization, the server repeatedly experienced storage failures that eventually left the filesystem read-only and caused StartOS services to malfunction.

Kernel diagnostics at the time showed:

  • The NVMe controller entering the D3cold PCIe power state.

  • Failed attempts to transition the controller back from D3cold to D0.

  • NVMe controller reset failures.

  • PCI configuration reads returning ffff, consistent with the controller becoming inaccessible.

  • The NVMe namespace effectively disappearing or being reported as a zero-byte device.

  • Btrfs reporting missing-device and write errors.

  • The filesystem becoming read-only.

The system would temporarily recover after a hard restart, but the problem returned under load.

Interestingly, SMART diagnostics showed the NVMe drive as healthy, with zero reported media errors. This led me to investigate PCIe and NVMe power management rather than immediately suspecting defective NAND storage.

Original resolution

On October 1, I modified /etc/default/grub to include these kernel parameters:

nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off

I then regenerated the GRUB configuration using:

sudo /usr/sbin/update-grub

I verified that the parameters were present in grub.cfg for both normal and recovery boot entries.

After rebooting, I also verified that all three parameters appeared in /proc/cmdline, confirming that the kernel was actually receiving them.

The result was a dramatic improvement in stability. Bitcoin Core subsequently ran for several hours under synchronization load without another NVMe failure. The server survived overnight, eventually completed its initial blockchain download, and continued operating normally for approximately three and a half weeks.

Connection to the October 25 update failure

Immediately after updating StartOS to v0.4.0.3, the server began exhibiting symptoms remarkably similar to those original failures.

The initial post-update boot left the machine responding to ping but without a functioning web interface or usable SSH connection.

After waiting more than an hour, a forced power cycle restored access temporarily.

However, the system then exhibited:

  • Input/output error when accessing executables under /usr/bin.

  • cat /proc/mounts producing a segmentation fault.

  • timedatectl timing out while attempting to activate systemd-timedated.

  • StartOS reporting Read-only file system (os error 30) when accessing notifications.

  • Bitcoin Core remaining stuck in its Starting state.

  • Other services failing to start or satisfy dependencies.

  • OS and kernel log viewers repeatedly disconnecting.

  • Elevated I/O wait despite low overall CPU utilization and plenty of available storage.

Since my original post, the situation has deteriorated further. SSH access has failed again, and the web interface has become unresponsive.

I am now shutting the server down.

Current hypothesis

I’m wondering whether the v0.4.0.3 operating-system update regenerated or replaced the GRUB configuration, removing the custom kernel parameters that resolved the original NVMe failures.

If those parameters were lost, the NVMe controller could once again be exposed to the same problematic power-state transitions. That would provide a plausible explanation for the recurrence of nearly identical symptoms immediately following the OS update. I realize that similar symptoms do not prove the same underlying cause, and I have not yet obtained kernel logs confirming another D3cold failure.

However, the previous diagnosis, the verified configuration changes, and the subsequent weeks of stability make this possibility worth investigating.

My next step, once the server is accessible again, is to inspect /proc/cmdline and verify whether those three kernel parameters survived the update.

I’ll report back with the results. In the meantime, I’d be interested to know whether StartOS operating-system updates are expected to preserve user modifications to /etc/default/grub and the generated GRUB configuration.

@simberety372 Thanks for the quick reply. Those do seem related or, at least part of the same larger issue.

4

Update #2 — GRUB Configuration Restored, Server Recovered

Following up on my previous post about the possibility that the StartOS v0.4.0.3 update had overwritten the custom GRUB configuration implemented on October 1.

After getting the server running again, I was able to regain SSH access and inspect the active kernel command line:

cat /proc/cmdline

All three of the NVMe/PCIe power-management parameters I had previously added were missing. I then checked /etc/default/grub and found that those parameters had also been removed from the configuration itself.

Recovery steps

I restored the following kernel parameters, which had originally resolved the NVMe controller failures experienced during the server’s initial setup:

nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off

Specifically, I edited /etc/default/grub so that the relevant line reads:

GRUB_CMDLINE_LINUX="boot=startos console=ttyS0,115200n8 console=tty0 nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off"

I preserved the existing StartOS boot arguments, then regenerated the GRUB configuration:

sudo /usr/sbin/update-grub

This completed successfully. I subsequently inspected /boot/grub/grub.cfg and verified that all three parameters were present in the generated boot entries for both installed kernels, including the recovery entries. After a controlled reboot, I checked /proc/cmdline again and confirmed that the running kernel (7.2.9+deb13-amd64) was receiving all three parameters.

Results

The server has now recovered substantially:

  • All seven installed StartOS services are running normally.

  • Bitcoin Core, Fulcrum, DATUM Gateway, Core Lightning, Mempool, Tor, and Am I Exposed? are operational according to StartOS.

  • All four connected miners have automatically reconnected to the local DATUM gateway and resumed mining through their primary pool configuration.

  • SSH access is functioning.

  • Commands that previously failed with I/O errors (date, clear, etc.) are working normally.

  • The root filesystem was confirmed mounted read-write during recovery.

  • I/O wait has dropped from approximately 18.6% during the earlier failure to approximately 1% after recovery.

  • NTP synchronization initially reported a failure but subsequently completed successfully.

There was also an unrelated-looking SSH access issue following the update: the StartOS SSH Keys page contained no registered keys. Re-adding my existing public key immediately restored SSH access. I cannot yet establish whether the update itself removed that registration, but it is worth mentioning.

Concern regarding Start9-supplied hardware

One aspect of this situation concerns me beyond the immediate software recovery: This is a brand-new Start9 Server One, purchased as a complete, preconfigured hardware/software system directly from Start9.

The original failures began during initial setup, while Bitcoin Core was performing its initial blockchain download. Kernel diagnostics at that time showed the NVMe controller entering D3cold, failing to return to D0, unsuccessful controller resets, and eventual Btrfs read-only filesystem errors.

SMART diagnostics showed no media errors, and the three kernel parameters above appeared to resolve the problem completely for approximately three and a half weeks.

I would expect a turnkey server sold by the operating-system developer to have its specific motherboard, NVMe controller, firmware, and power-management configuration validated against the supplied operating system. I recognize that hardware and firmware interactions can be complicated, but a newly purchased, supported hardware platform should not require undocumented kernel boot parameters to prevent its storage controller from disappearing under normal workloads, and if such parameters are necessary for this hardware revision, they should ideally be part of the supported default configuration—or at least preserved across operating-system updates.

The loss of those parameters during an otherwise routine OS update apparently exposed the machine to the same failure mode that had already been resolved weeks earlier.

Remaining questions

I’d appreciate clarification on whether:

  1. StartOS updates are expected to regenerate /etc/default/grub and discard custom kernel parameters.

  2. These NVMe/PCIe power-management parameters are known to be necessary on any Server One hardware revisions.

  3. There is a supported, update-persistent method for configuring kernel boot parameters.

  4. The hardware/firmware combination in this Server One should be investigated for a compatibility issue or possible defect.

For clarity, the causal connection is still a hypothesis: I’ve confirmed that the update removed the previously successful workaround, but I have not captured fresh kernel logs proving that another D3cold transition caused the October 25 failure.

Nevertheless, the matching symptoms, the confirmed configuration change, and the successful recovery after restoring the parameters make that explanation compelling.

Next steps

I’m leaving the server running under its normal workload, including Bitcoin Core, Fulcrum, DATUM, and four connected miners. I’ll report back after approximately 24 hours of observation to confirm whether the system remains stable with the restored kernel parameters.

Hopefully this information will also help anyone else experiencing unexplained NVMe controller resets or read-only filesystem failures after a StartOS update.