Console (tty1) unresponsive + SSH unreachable after update/restart — resolved via full power cycle

StartOS Support
,
1

Node: brave-crybaby, StartOS v0.4.0.2 DIY - I have been troubleshooting this for some time it could be also a corrupt leftover disk from Apple will remove it but was wondering if anyone else have had a similar behavior as per below…

What happened:
Ran a StartOS update via SSH; server restarted to apply it. After restart:

  • Physical console appeared stuck on what looked like emergency mode:
You are in emergency mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, or "exit" to continue bootup.
sulogin: tcgetattr failed: Input/output error
Cannot open access to console, the root account is locked.
  • A subsequent SSH attempt timed out: ssh: connect to host 192.168.188.108 port 22: Operation timed out
  • System was unreachable both locally and remotely for roughly 1.5 hours (boot ID 01494c0aa67b423e8e344384362ece22, 15:50:49–17:20:54 UTC)

Confirmed root cause:
getty@tty1.service failed continuously throughout the boot:

Sep 08 15:47:37 brave-crybaby agetty[4402]: could not get terminal name: -22
Sep 08 15:47:37 brave-crybaby agetty[4402]: -: failed to get terminal attributes: Input/output error

(repeating every ~10s for the full duration). journalctl -xb -1 -p err/warning and mount/fstab/emergency keyword searches returned nothing else — no fstab or mount failures were logged for this boot. startd itself reported a fully successful init sequence (disk, database, SSH keys, network controller all completed).

Ruled out:

  • Disk/partition health — checked SMART status, GPT partition tables, and LVM/LUKS integrity on all three attached drives. The active StartOS disk (sdb) and a secondary unused drive (sdc) are healthy; a third unused drive (sda, the original Apple HDD) failed its SMART health check (imminent failure warning, 1815 reallocated sectors) but showed no boot-time errors or bus delays in either boot examined — not implicated in this incident, but worth replacing/removing on its own merits.
  • Memory — checked for OOM/ECC/MCE events and current usage; no issues found.
  • Early boot-time kernel/ATA logs for the failed boot were not fully retrievable (journald’s log storage switches location partway through StartOS boot), so an early SATA-bus stall couldn’t be 100% excluded, but no direct evidence supports it.

Other hardware observations (unrelated, chronic, not tied to this incident):

  • Recurring clocksource: Watchdog remote CPU X read timed out warnings, seen in both this failed boot and a subsequent healthy boot.
  • Recurring b43-phy0: Loading firmware version 784.2 messages every ~7 minutes continuously across multiple boots — suggests the Broadcom WiFi chip is periodically resetting. Known flaky combination (old Apple b43 WiFi + Linux).

Resolution:
Two standard restarts (systemctl reboot / SSH-triggered restart) did not resolve the issue. The system only recovered on the third attempt, which involved a full power-down (hardware off for ~20 seconds) followed by a cold boot, with a monitor also connected at that point. It’s unclear whether the monitor, the hard power cycle, or both together were the actual fix — a full power-off clears more hardware/firmware state than a warm reboot, so the cold boot itself may have been the deciding factor.

Open questions for support:

  1. Is it expected/by design that tty1 fails to initialize entirely if no monitor is detected at boot, even though the OS and SSH are otherwise fully functional?
  2. Why might SSH also become unreachable in this scenario, given startd reports the network controller and SSH service as loaded successfully?
  3. Is there a known difference in how tty1/console initialization behaves between a warm reboot (systemctl reboot) and a full power cycle, that would explain why only the latter resolved this?
  4. Separately: the Kiosk Mode toggle in System settings does nothing when clicking “Enable” (no error, no state change) — possibly related, possibly a separate bug.
2

Excellent write-up — that is a lot more legwork than most reports arrive with, and it’s what made this diagnosable. The conclusion needs correcting though: getty@tty1 is a dead end, and I think two different boots got merged into one story.

The two observations can’t belong to the same boot

  • In emergency mode systemd never reaches getty.target, so getty@tty1 never starts at all — it can’t log an agetty error every 10 s for 90 minutes.
  • startd is WantedBy=multi-user.target. If it logged a clean init sequence, that boot was not in emergency mode.

Your own timestamps line up with that: the agetty lines are stamped 15:47:37, but the boot ID you analysed runs 15:50:49–17:20:54. Those agetty errors are from the previous boot. The boot you examined was a healthy one.

What actually happened

  1. Something failed local-fs.target, so systemd dropped to emergency.target.
  2. emergency.service ran sulogin on a console it couldn’t open (tcgetattr failed: Input/output error), and StartOS locks the root account — so sulogin exited immediately.
  3. Nothing past sysinit.target ever ran. No multi-user.target means no sshd and no startd. That’s your question 2 — SSH wasn’t unreachable, it was never started.
  4. StartOS runs its root filesystem as a squashfs image with a tmpfs overlay. With persistent storage unmounted, journald stayed in /run/log/journal, which is RAM. The failed boot’s logs were destroyed by the reboot — which is exactly why the early boot logs “weren’t retrievable”, and why journalctl -b -1 handed you a healthy boot instead.

That last point matters: you ruled out the failing Apple HDD using logs from a boot that wasn’t the failure. It goes back on the suspect list.

The likely trigger

StartOS’s installer writes /etc/fstab using kernel device names — /dev/sdb1, /dev/sdb2, /dev/sdb3 — rather than UUIDs, and with no nofail on any entry. /dev/sdX assignment follows SATA probe order, which is not stable. On a three-disk box where one drive has 1815 reallocated sectors and is slow to respond, probe order is precisely the thing that drifts.

If the OS disk comes up under a different letter for one boot, the /boot device never appears, the mount times out after 90 s, local-fs.target fails, and you get exactly the console you photographed.

It also explains the fix. A warm systemctl reboot largely preserves SATA link state, so the bad enumeration reproduces. A cold power-off re-runs full link training from scratch, which reshuffles it. The monitor was almost certainly incidental.

Could you post the output of:

cat /etc/fstab
lsblk -o NAME,PATH,SIZE,UUID,PARTUUID,MOUNTPOINT

If fstab names /dev/sdX devices, that’s it — and it will happen again.

:warning: Before you pull the failing drive

This is the important part. If your fstab names /dev/sdb* and you physically remove sda, the OS disk becomes /dev/sda — and fstab will then point at a device that no longer exists. You’d go straight into permanent emergency mode. Removing the bad disk is the right call, but change fstab to PARTUUID= first.

One more trap: StartOS’s root is read-only with a RAM overlay, so editing /etc/fstab directly does nothing — the change evaporates on reboot. The persistent copy is:

/media/startos/config/overlay/etc/fstab

That’s a manual, unsupported edit, so take a photo of the current file first and change one thing at a time.

Your four questions

  1. No, and it isn’t monitor-related. /dev/tty1 exists on a headless box. Note what the log actually says — agetty was invoked with - as its terminal argument (-: failed to get terminal attributes), which means systemd was handing it an already-open /dev/tty1 and that handle was broken. The Input/output error is the console genuinely dead at kernel level, the same reason sulogin failed. On old Apple boards that usually means the GPU/VT doesn’t re-initialise across a warm reboot.

  2. Answered above — the system never got far enough to start sshd.

  3. Yes, and it’s probably the deciding factor. A cold power-off resets GPU/VT and SATA link state; a warm reboot resets neither.

  4. Separate from the outage, and possibly not a bug at all — I need one more detail from you.

    First, the expected behaviour, since you didn’t say what you were looking for: enabling Kiosk Mode requires a restart to take effect. Nothing appears on the monitor until you reboot. What should change immediately is the UI — the badge beside “Kiosk Mode” flips from Disabled to Enabled, the button relabels to Disable, and a bar appears at the top of the page reading “Kiosk mode changed, restart to apply” with a Restart button.

    Also expected: if no keyboard layout has been set yet, clicking Enable opens a Select Keyboard Layout dialog first, and you have to choose one before kiosk is enabled. Closing that dialog cancels the whole thing silently — which looks exactly like “nothing happened”. Servers set up from a browser rather than from an attached monitor don’t get asked for a layout during setup, so this is the common path.

    So, when you click Enable:

    • Does a Select Keyboard Layout dialog appear?
    • Afterwards, does the badge still read Disabled, and is there no restart bar at the top?
    • Anything in your browser’s dev console (F12 → Console)?

    If the badge flips and you get the restart bar, it worked and you just needed to reboot. If a dialog appeared, that’s working as designed. If genuinely nothing happens and the console shows an error, that’s a real bug and I’ll file it.

I’ve filed the fstab issue. Thanks for the detail — the emergency-mode screenshot plus the boot IDs is what made it possible to tell the two boots apart.

3

HI Stu,

thank you for also responding so thouroughly, this issue has been pesting me now for some time…

Confirmed: /etc/fstab uses raw device names, no nofail:

/dev/sdb2  /boot       vfat    umask=0077  0   2
/dev/sdb1  /boot/efi   vfat    umask=0077  0   1
/dev/sdb3  /           ext4    defaults    0   1

Same content found in the persistent overlay copy at /media/startos/config/overlay/etc/fstab, confirming that’s the right file.

Full disk/partition map for reference:

sda    931.5G  (Apple HDD, SMART FAILED — 1815 reallocated sectors)
├─sda1 200M   PARTUUID=6f3bc03b-6142-437d-8a2f-dd918a993b9a
└─sda2 931.3G PARTUUID=726f2816-7022-4eb4-b89a-0a122d6ef09d

sdb    1.8T   (active StartOS disk, healthy)
├─sdb1 100M   PARTUUID=a7ecf3d5-14f6-4728-a95d-f60db1640351  /boot/efi
├─sdb2 1G     PARTUUID=115e3bc0-ce64-4705-b454-405088f5c431  /boot
├─sdb3 15G    PARTUUID=a827d536-4d46-4732-b321-829fb3d4a112  /
└─sdb4 1.8T   (LVM/LUKS — data)

sdc    931.5G (unused, healthy)
└─sdc1 931.5G PARTUUID=7b038126-6774-4e5f-9888-e2f2e7a22034

Backed up before touching anything:

/media/startos/config/overlay/etc/fstab.bak-20260910

Proposed replacement, using PARTUUID:

PARTUUID=a7ecf3d5-14f6-4728-a95d-f60db1640351  /boot/efi     vfat  umask=0077        0  1
PARTUUID=115e3bc0-ce64-4705-b454-405088f5c431  /boot         vfat  umask=0077        0  2
PARTUUID=a827d536-4d46-4732-b321-829fb3d4a112  /             ext4  defaults,nofail   0  1

Questions before I apply this:

  1. Does this look correct — any objection to the PARTUUIDs I pulled, or the nofail addition on /?
  2. Should I also add nofail to /boot and /boot/efi, or is that inadvisable for either of those mounts specifically?
  3. Does editing the overlay fstab file directly take effect on next boot as-is, or does StartOS need something else re-run (e.g. update-initramfs, a rebuild/regenerate step) for the change to actually apply at early boot?
  4. Given sda’s SMART failure, is it safe to physically remove that drive once fstab is switched to PARTUUID, or is there anything else on this box that might still reference it (e.g. leftover LVM PV signatures, old mdadm metadata) that I should check first?

Kiosk Mode: will test again for the Select Keyboard Layout dialog and dev console behavior as requested, and report back.

4

That’s the confirmation — thank you for pulling the PARTUUIDs, that’s exactly what was needed. I’ve filed it as #3923.

One detail in your output is worth noting: your root line says ext4, but the current StartOS installer writes btrfs there. So your fstab wasn’t written by the installer you’re running now — it’s been carried forward from an older install or through a migration. That widens the issue rather than narrowing it, and I’ve added it to the report.

1. Your PARTUUIDs are correct — two changes to the plan

The three PARTUUIDs map correctly to /boot/efi, /boot and / as mounted. Two things I’d change:

Drop nofail from /. It does nothing useful there and I’d rather not introduce an unknown on the one line that must not misbehave. On StartOS your real root filesystem isn’t that partition at all — the initramfs assembles root as an overlay, and the kernel finds it via root=PARTUUID=... on the boot command line, not via fstab. The fstab / line is close to vestigial; StartOS’s own code explicitly ignores every entry that isn’t /boot*. That, incidentally, is why you reached emergency mode rather than an initramfs panic — root resolved fine, it was the /boot mounts that failed.

Keep the original line order. You’ve moved /boot/efi above /boot. /boot/efi sits inside /boot, so the parent should be listed first. systemd works out the ordering itself these days, so this is unlikely to bite you, but there’s no reason to invert it.

So:

PARTUUID=115e3bc0-ce64-4705-b454-405088f5c431  /boot      vfat  umask=0077,nofail  0  2
PARTUUID=a7ecf3d5-14f6-4728-a95d-f60db1640351  /boot/efi  vfat  umask=0077,nofail  0  1
PARTUUID=a827d536-4d46-4732-b321-829fb3d4a112  /          ext4  defaults           0  1

If you want belt-and-braces, setting the last column to 0 on the two boot lines disables their boot-time fsck as well — one less thing that can fail early. Optional.

2. Yes — nofail on /boot and /boot/efi is the entire point

Neither is needed to reach a running system. The kernel and initramfs are already in memory by the time fstab is read; those partitions matter for OS updates and for GRUB. With nofail, a mount that fails degrades to “the next update won’t work” instead of “the server is unreachable and can’t be logged into”. That’s the trade you want.

The caveat, and please take this one seriously: with nofail, a failed /boot mount becomes silent. StartOS’s updater writes the new kernel into /boot without first checking that it’s actually mounted, so an update run while /boot is unmounted would write the kernel into RAM instead of the disk, and you’d reboot into a stale kernel against a new system image. Before you ever run an OS update, check:

mountpoint /boot && mountpoint /boot/efi

Both must say “is a mountpoint”. I’ve reported that missing check as part of #3923.

3. Yes — the overlay file is all you need, no rebuild step

Editing /media/startos/config/overlay/etc/fstab is sufficient and takes effect on the next boot. Nothing needs regenerating: the initramfs never reads fstab (it finds root from the kernel command line), and fstab is read by systemd in userspace, after the initramfs has already layered your overlay copy over the base image. No update-initramfs, no rebuild.

Two things to expect:

  • The running /etc/fstab will not change when you edit the overlay copy — that’s normal, the overlay is assembled fresh at boot and can’t be changed underneath a live mount. Verify your edit by reading the overlay file, not /etc/fstab.
  • Because the change only lands at boot, the reboot is the test. Have physical access available for that one, just in case.

Since nofail on the two boot mounts is what removes the brick risk, and the PARTUUID switch is what removes the recurrence, you can do both in one edit — but if you’d rather de-risk it, adding nofail alone first is a strictly safe change: even a wrong PARTUUID can’t drop you into emergency mode once those two lines can’t fail.

4. Probably safe to pull sda, but check four things first

Once fstab is on PARTUUIDs, re-lettering can’t hurt the mounts. What can still hurt you:

Firmware boot entries. Your sda1 is 200 MB, which is an EFI System Partition — the standard Apple layout. If your firmware is booting through that ESP rather than sdb1, pulling the disk makes the machine unbootable. This is the one that would actually ruin your day:

efibootmgr -v

Confirm the active entry references sdb1’s PARTUUID (a7ecf3d5-…) and not sda1’s (6f3bc03b-…).

The kernel’s root= parameter:

grep -o 'root=[^ ]*' /boot/grub/grub.cfg | sort -u

root=PARTUUID=… or root=UUID=… is fine. If it says root=/dev/sdb3, that’s a second landmine and needs fixing before you remove anything.

Leftover volume metadata:

sudo pvs; sudo vgs
cat /proc/mdstat
sudo blkid /dev/sda1 /dev/sda2

You’re looking for any sign that sda carries an LVM PV or mdadm superblock that the data volume group thinks it owns. Your lsblk suggests it doesn’t — sdb4 is the only LVM/LUKS member — but confirm it rather than assume.

Backup targets. StartOS records a disk backup target by its device name, not its UUID. If you’ve ever configured a backup to a physically attached drive, removing sda re-letters sdc and that target could quietly point somewhere else. Check your backup targets in the UI after the change and re-select the drive if it looks wrong. (Same root cause as #3923 — I’ve added it there.)

For what it’s worth: that drive is reporting 1815 reallocated sectors and a failed SMART health assessment. It should come out regardless. Do the fstab change first, reboot, confirm the box comes up clean, then pull it — one variable at a time.

Let me know how the reboot goes, and no rush on the Kiosk Mode test.

5

Thanks for the detailed breakdown — the boot-merge catch was the key thing I missed, and the explanation of why (RAM-only journald under the overlay setup) actually cleared up a few things about how StartOS boots that I didn’t get before.

Fix is confirmed working: rebooted with physical access ready just in case, /boot and /boot/efi both came up as proper mountpoints, no emergency mode.

Appreciate you digging into this as much as you did — flagging the ext4/btrfs mismatch and the silent-update risk on top of the fstab fix was more than I expected. Thanks for the help.