StartOS Launch Error: Filesystem I/O Error

StartOS Support
1

Continuing the discussion from Filesystem I/O Error: File exists (os error 17):

I’ve been having these errors for a few weeks at least, typically with some updates. I found this comment just recommending a System Restart, so I tried that. But now I get a StartOS Launch Error which I’ve never seen before. I’m hoping to be able to recover remotely because the server is in a different location. Thankfully I can at least see this error screen remotely.

The given Possible Solution is just “Contact support” so here I am. I have two buttons available to me: Restart Server, Repair Drive. But Repair Drive is in red so I’m afraid to go that route for now.

Multiple Errors

Multiple Errors: Filesystem I/O Error: rm /var/lib/lxc/SMBLEBWNXX35TLPTKJXFQE3BKIH4U2S6/rootfs/media/startos/volumes/main: No such file or directory (os error 2): No such file or directory (os error 2); Filesystem I/O Error: rm /var/lib/lxc/SMBLEBWNXX35TLPTKJXFQE3BKIH4U2S6/rootfs/media/startos/assets: No such file or directory (os error 2): No such file or directory (os error 2); Filesystem I/O Error: rm /var/lib/lxc/SMBLEBWNXX35TLPTKJXFQE3BKIH4U2S6/rootfs/usr/lib/startos/package: No such file or directory (os error 2): No such file or directory (os error 2); Multiple Errors: Filesystem I/O Error: rm /var/lib/lxc/CDCZ3Z2PF2ALSBKUJZ3XOYFY476H3MXV/rootfs/media/startos/volumes/grafana: No such file or directory (os error 2): No such file or directory (os error 2); Filesystem I/O Error: rm /var/lib/lxc/CDCZ3Z2PF2ALSBKUJZ3XOYFY476H3MXV/rootfs/media/startos/volumes/prometheus: No such file or directory (os error 2): No such file or directory (os error 2); Filesystem I/O Error: rm /var/lib/lxc/CDCZ3Z2PF2ALSBKUJZ3XOYFY476H3MXV/rootfs/media/startos/assets: No such file or directory (os error 2): No such file or directory (os error 2); Filesystem I/O Error: rm /var/lib/lxc/CDCZ3Z2PF2ALSBKUJZ3XOYFY476H3MXV/rootfs/usr/lib/startos/package: No such file or directory (os error 2): No such file or directory (os error 2); Multiple Errors: Filesystem I/O Error: rm /var/lib/lxc/QI53Q7NLSEGF36ETN53WVZARMG2C6NJH/rootfs/media/startos/volumes/db: No such file or directory (os error 2): No such file or directory (os error 2); Filesystem I/O Error: rm /var/lib/lxc/QI53Q7NLSEGF36ETN53WVZARMG2C6NJH/rootfs/media/startos/volumes/main: No such file or directory (os error 2): No such file or directory (os error 2); Filesystem I/O Error: rm /var/lib/lxc/QI53Q7NLSEGF36ETN53WVZARMG2C6NJH/rootfs/media/startos/assets: No such file or directory (os error 2): No such file or directory (os error 2); Filesystem I/O Error: rm /var/lib/lxc/QI53Q7NLSEGF36ETN53WVZARMG2C6NJH/rootfs/usr/lib/startos/package: No such file or directory (os error 2): No such file or directory (os error 2); Multiple Errors: Filesystem I/O Error: rm /var/lib/lxc/TPLWETIWRZSLBPZFMPVCKCAALUA7SXEI/rootfs/media/startos/volumes/main: No such file or directory (os error 2): No such file or directory (os error 2); Filesystem I/O Error: rm /var/lib/lxc/TPLWETIWRZSLBPZFMPVCKCAALUA7SXEI/rootfs/media/startos/assets: No such file or directory (os error 2): No such file or directory (os error 2); Filesystem I/O Error: rm /var/lib/lxc/TPLWETIWRZSLBPZFMPVCKCAALUA7SXEI/rootfs/usr/lib/startos/package: No such file or directory (os error 2): No such file or directory (os error 2); Multiple Errors: Filesystem I/O Error: rm /var/lib/lxc/7SLUGCG67SRHZ6UBPOE44GX3MV5KOSG3/rootfs/media/startos/volumes/main: No such file or directory (os error 2): No such file or directory (os error 2); Filesystem I/O Error: rm /var/lib/lxc/7SLUGCG67SRHZ6UBPOE44GX3MV5KOSG3/rootfs/media/startos/assets: No such file or directory (os error 2): No such file or directory (os error 2); Filesystem I/O Error: rm /var/lib/lxc/7SLUGCG67SRHZ6UBPOE44GX3MV5KOSG3/rootfs/usr/lib/startos/package: No such file or directory (os error 2): No such file or directory (os error 2); Multiple Errors: Filesystem I/O Error: rm /var/lib/lxc/UPL3V2HPQMATMAPKVMEVGFPLWWMYJLPM/rootfs/media/startos/volumes/embassy: No such file or directory (os error 2): No such file or directory (os error 2); Filesystem I/O Error: rm /var/lib/lxc/UPL3V2HPQMATMAPKVMEVGFPLWWMYJLPM/rootfs/media/startos/volumes/main: No such file or directory (os error 2): No such file or directory (os error 2); Filesystem I/O Error: rm /var/lib/lxc/UPL3V2HPQMATMAPKVMEVGFPLWWMYJLPM/rootfs/media/startos/assets: No such file or directory (os error 2): No such file or directory (os error 2); Filesystem I/O Error: rm /var/lib/lxc/UPL3V2HPQMATMAPKVMEVGFPLWWMYJLPM/rootfs/usr/lib/startos/package: No such file or directory (os error 2): No such file or directory (os error 2); Multiple Errors: Filesystem I/O Error: rm /var/lib/lxc/DKZ3XWCBCSSQQWSXTY7SQLWSN6YMMNCU/rootfs/media/startos/volumes/main: No such file or directory (os error 2): No such file or directory (os error 2); Filesystem I/O Error: rm /var/lib/lxc/DKZ3XWCBCSSQQWSXTY7SQLWSN6YMMNCU/rootfs/media/startos/assets: No such file or directory (os error 2): No such file or directory (os error 2); Filesystem I/O Error: rm /var/lib/lxc/DKZ3XWCBCSSQQWSXTY7SQLWSN6YMMNCU/rootfs/usr/lib/startos/package: No such file or directory (os error 2): No such file or directory (os error 2); Multiple Errors: Filesystem I/O Error: rm /var/lib/lxc/ONY3HI3SJF63T6E2MMGSE3ZVWBNP7THJ/rootfs/media/startos/volumes/main: No such file or directory (os error 2): No such file or directory (os error 2); Filesystem I/O Error: rm /var/lib/lxc/ONY3HI3SJF63T6E2MMGSE3ZVWBNP7THJ/rootfs/media/startos/assets: No such file or directory (os error 2): No such file or directory (os error 2); Filesystem I/O Error: rm /var/lib/lxc/ONY3HI3SJF63T6E2MMGSE3ZVWBNP7THJ/rootfs/usr/lib/startos/package: No such file or directory (os error 2): No such file or directory (os error 2)

2

Looks like I’ve lost connection to the Diagnostic Mode screen

3

That screen comes from the restart itself, not from your drive, and your data isn’t affected.

It’s a known StartOS bug. While stopping services for a restart, StartOS removes a few temporary mount folders inside each service’s container. If one of them is already gone, it treats that as fatal and stops on this screen instead of rebooting. Every line in your error is that same cleanup, once per service. It’s fixed in StartOS’s code (#3937), but the fix isn’t in a release yet.

Don’t use Repair drive. It runs an aggressive filesystem repair on the next boot. That’s meant for a damaged drive, and nothing is wrong with yours.

Press Restart server. The server never restarts by itself from that screen; it waits for that button. Those temporary folders don’t survive a reboot, so it should come back up normally. Give it about 10 minutes, then open your usual address.

About losing the connection:

  • If you’d already pressed a button, the server is restarting and the diagnostic page won’t come back. Wait, then try your usual address. If it was Repair drive, the repair runs first and takes longer. Don’t cut the power while it runs.
  • If you hadn’t, the server is still waiting on that screen and the problem is your remote connection. Press Restart server once you can reach it again. If you can’t, someone at the location can shut it down cleanly with a short press of the power button, then press it again to turn it back on.

Until the fix ships, a restart can land on this screen again. Restart server is the way out each time.

Two questions:

  1. Which StartOS version are you on?
  2. What exactly did the update errors over the last few weeks say? Updating a service stops the old version first, and this same bug can make an update fail. If yours said File exists (os error 17) instead, that’s something else, and I’d like to see the full text.
4
  1. 0.4.0.1
  2. Haven’t seen os error 17
5

Thanks. 0.4.0.1 is the newest StartOS release, and it still has this bug. The fix is listed for the next release, 0.4.0.2, which isn’t out yet. Once you’re on that, restarts and updates stop tripping over it.

Since you haven’t seen os error 17, the update errors were most likely this same bug. To update a service, StartOS stops the old version first, and on 0.4.0.1 that stop can fail on the same missing folders. That shows up as an Updating Failed notification mentioning rm /var/lib/lxc/… and No such file or directory (os error 2). When it happens, StartOS puts the old version back, so you can just run the update again. If one of those notifications is still in your list and says something different, please paste it here.

Until 0.4.0.2 is out, if a restart lands you on the red screen again, Restart server is still the way out.