I’ve been testing my luck with a Pi and it crashed a couple of days ago… I have a Dell Optiplex 755 that I’m going to use until I find something better. StartOS (non-free iso) loaded up fine to the Dell, I selected Recover > Use existing drive (the 2TB used with the Pi), the OS prompts for new password then begins initializing. A few minutes later an error pops up, see attachment, RPC error Docker Error image platform (linux/arm64) does not match expected platform (linux/arm64).
The initializing screen is still going but no progress is being shown after an hour of running. I have BTC Core, Nextcloud, Vaultwarden, SyncThing and a few other small apps on the drive.
EDIT: this morning, no progress in syncing. Re-flashed os and tried to load from existing drive again, still no luck but was able to load a new instance of StartOS to the Dell. Wondering where to go from here, try to restore from the physical drive in the new OS or restore from backup? Hoping to be able to keep all the old app url’s and restore services quickly.
There’s no way to use an existing drive full of services originally installed on an ARM architecture on an x86 processor, It just won’t work, as per this error. These are different processor architectures.
The only way to achieve the porting of services on a drive that was connect to a Raspberry Pi to an x86 machine is via the transfer process, to a new drive entirely.
I started the transfer yesterday morning and it completed sometime thru the night. For some reason I had to restart a few times to get Vaultwarden to work. BTC Core has been syncing for several hours which seems kinda slow since it was just 6 days of downtime. Everything else seems good.
I’ll add this comp to the known good hardware list.
If this Dell Optiplex 755 is relatively slow, it would make sense that it would take a while to sync Bitcoin, similar, but not as as bad as a Pi. It’ll be something like 8GB of RAM or the slow external USB, or a combination of the two, that’ll be causing this Pi-like behavior. But it should sync eventually.
I’m getting some weirdness… The Dell finally synced and then crashed, I wasn’t able to connect to .local or .onion. I did a hard restart and the login screen appeared and froze.
I thought the Dell was maxed out in it’s capabilities so I spun up my frankenstein Lenovo and loaded it up using the hard drive from the Dell, this was amazingly easy btw. So BTC core was just about to finish syncing and I was about to run a backup and it crashed, I did a hard restart and was able to reach it on my laptop but not mobile. This morning I can’t reached it on any device just like the Dell and maybe the Pi at .local or .onion. After a hard restart I can reach it on my laptop and on mobile after restarting the phone.
Any idea what’s causing the lost connection and the need to hard restart to restore it?
There are a few different unrelated errors you mention there. Including some I haven’t seen before. Dealing specifically with the networking issues specifically…
There are two broad directions of troubleshooting you can take…
Work out if it’s StartOS going haywire. As things start of stop being reachable, look at the output of journalctl and look for clues. Look at top too, and see if startd is crashing
Work out it’s your networking. Does restarting your client and your router instead of your server help in the same way? What exactly is not working before a restart? .locals? .onions? individual services? the main UI? You mention it at the start, but I’m unsure what’s happening because you then move onto other unreleated issues.
Thanks Stu. For 1. Where can I find journalctl and top? It goes from working well, all services running with good health, BTC core and electrs syncing and reachable on all devices and url’s to not connected and unreachable with no warning or too quick for me to catch anyway.
And 2. Nothing has changed in my networking for a few months. When it goes from connected to disconnected, nothing works (no url’s, no services, main ui). It goes from green ‘connected’ banner to yellow ‘connecting’, I refresh the browser on laptop or mobile with no luck, then quit the browser and try to login but get url unreachable or not found, try again on .onion and .local. Then I go to the server and it’s on a blank screen that won’t wake up so I hard restart it. I was able to grab a couple of pics during the restart, maybe it’s a piece of the puzzle.
And it just happened again. Blockchain fully downloaded/synced to BTC Core and electrs and everything is unreachable, went from connected to connecting but never connects. I tried to get to the logs before leaving but it just spins.
I went to the comp that StartOS is installed on and the screen is blank, I hit a few keys and moved the mouse around but no luck. Doesn’t make sense that it quits working as soon as everything’s synced!
Any thoughts on this or ideas as to how to fix it or what to try next?
Sorry for all the replies! OK, it’s back online. I left it, went to chop some wood and came back. In the OS logs there’s some error info, I’ve trimmed it to remove the normal stuff but have left the last good line before error and the first line after error. Electrs fails quite often shown in the first block of errors. Then the OS fails, errors shown below.
Electrs: this one happens quite a bit but the OS continues:
2025-01-09T13:58:39-04:00 2025-01-09T17:58:39.920327Z DEBUG run_main:check: startos::manager::health: Checking health of nextcloud
2025-01-09T13:58:47-04:00 2025-01-09T17:58:47.939057Z ERROR startos::manager::manager_container: The service electrs has crashed with the following exit code: 1
2025-01-09T13:58:48-04:00 2025-01-09T17:58:48.550730Z DEBUG run_main:check: startos::manager::health: Checking health of mutiny-wallet
20
Then the OS fails a few times it looks like:
2025-01-09T14:05:12-04:00 > 2025-01-09T18:05:12.962360Z DEBUG run_main:check: startos::manager::health: Checking health of searxng
2025-01-09T14:05:13-04:00 2025-01-09T18:05:13.503878Z ERROR startos::db: WebSocket Closed: Network Error: WebSocket protocol error: Connection reset without closing handshake
2025-01-09T14:05:13-04:00 2025-01-09T18:05:13.504896Z DEBUG startos::db: Error { source:
2025-01-09T14:05:13-04:00 0: WebSocket protocol error: Connection reset without closing handshake
2025-01-09T14:05:13-04:00 1: Connection reset without closing handshake
2025-01-09T14:05:13-04:00 Location:
2025-01-09T14:05:13-04:00 /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/core/src/convert/mod.rs:716
2025-01-09T14:05:13-04:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2025-01-09T14:05:13-04:00 0: startos::db::deal_with_messages
2025-01-09T14:05:13-04:00 at startos/src/db/mod.rs:78
2025-01-09T14:05:13-04:00 1: startos::db::ws_handler
2025-01-09T14:05:13-04:00 at startos/src/db/mod.rs:32
2025-01-09T14:05:13-04:00 Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
2025-01-09T14:05:13-04:00 Run with RUST_BACKTRACE=full to include source snippets., kind: Network, revision: None }
2025-01-09T14:05:13-04:00 2025-01-09T18:05:13.583902Z DEBUG run_main:check: startos::manager::health: Checking health of syncthing
025-01-09T14:08:24-04:00 2025-01-09T18:08:24.180270Z DEBUG run_main:check: startos::manager::health: Checking health of bitcoind
2025-01-09T15:48:41-04:00 2025-01-09T19:34:38.455309Z ERROR startos::db: WebSocket Closed: Network Error: IO error: Broken pipe (os error 32)
2025-01-09T15:48:41-04:00 2025-01-09T19:48:41.227286Z DEBUG startos::db: Error { source:
2025-01-09T15:48:41-04:00 0: IO error: Broken pipe (os error 32)
2025-01-09T15:48:41-04:00 1: Broken pipe (os error 32)
2025-01-09T15:48:41-04:00 Location:
2025-01-09T15:48:41-04:00 /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/core/src/convert/mod.rs:716
2025-01-09T15:48:41-04:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2025-01-09T15:48:41-04:00 0: startos::db::deal_with_messages
2025-01-09T15:48:41-04:00 at startos/src/db/mod.rs:78
2025-01-09T15:48:41-04:00 1: startos::db::ws_handler
2025-01-09T15:48:41-04:00 at startos/src/db/mod.rs:32
2025-01-09T15:48:41-04:00 Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
2025-01-09T15:48:41-04:00 Run with RUST_BACKTRACE=full to include source snippets., kind: Network, revision: None }
2025-01-09T15:48:41-04:00 2025-01-09T19:48:41.266795Z DEBUG run_main:check: startos::manager::health: Checking health of searxng
2025-01-09T15:48:41-04:00 2025-01-09T19:48:41.272038Z DEBUG run_main:check: startos::manager::health: Checking health of filebrowser
2025-01-09T15:48:41-04:00 2025-01-09T19:48:41.272713Z ERROR startos::net::tor: Tor Daemon Error: Tor health check> failed for longer than current timeout (120s): Restarting tor
2025-01-09T15:48:41-04:00 2025-01-09T19:48:41.272771Z DEBUG run_main:check: startos::manager::health: Checking health of vaultwarden
2025-01-09T15:48:41-04:00 2025-01-09T19:48:41.272781Z DEBUG startos::net::tor: Error { source:
2025-01-09T15:48:41-04:00 0: Tor health check failed for longer than current timeout (120s)
2025-01-09T15:48:41-04:00 Location:
2025-01-09T15:48:41-04:00 startos/src/net/tor.rs:628
2025-01-09T15:48:41-04:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2025-01-09T15:48:41-04:00 0: startos::net::tor::torctl
2025-01-09T15:48:41-04:00 at startos/src/net/tor.rs:250
2025-01-09T15:48:41-04:00 Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
2025-01-09T15:48:41-04:00 Run with RUST_BACKTRACE=full to include source snippets., kind: Tor, revision: None }
2025-01-09T15:48:41-04:00 2025-01-09T19:48:41.752801Z DEBUG run_main:check: startos::manager::health: Checking health of syncthing
2025-01-09T15:48:41-04:00 2025-01-09T19:48:41.883897Z DEBUG run_main:check: startos::manager::health: Checking health of mutiny-wallet
2025-01-09T15:48:41-04:00 2025-01-09T19:48:41.888780Z DEBUG run_main:check: startos::manager::health: Checking health of nextcloud
2025-01-09T15:48:42-04:00 2025-01-09T19:48:42.059455Z ERROR startos::core::rpc_continuations: WebSocket Closed: Network Error: IO error: Connection reset by peer (os error 104)
2025-01-09T15:48:42-04:00 2025-01-09T19:48:42.059524Z DEBUG startos::core::rpc_continuations: Error { source:
2025-01-09T15:48:42-04:00 0: IO error: Connection reset by peer (os error 104)
2025-01-09T15:48:42-04:00 1: Connection reset by peer (os error 104)
2025-01-09T15:48:42-04:00 Location:
2025-01-09T15:48:42-04:00 /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/core/src/convert/mod.rs:716
2025-01-09T15:48:42-04:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2025-01-09T15:48:42-04:00 0: startos::logs::ws_handler
2025-01-09T15:48:42-04:00 at startos/src/logs.rs:67
2025-01-09T15:48:42-04:00 Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
2025-01-09T15:48:42-04:00 Run with RUST_BACKTRACE=full to include source snippets., kind: Network, revision: None }
2025-01-09T15:48:47-04:00 2025-01-09T19:48:47.767012Z ERROR startos::manager::manager_container: The service electrs has crashed with the following exit code: 1
2025-01-09T15:48:48-04:00 2025-01-09T19:48:48.052709Z INFO torctl: startos::net::tor: Tor is started
2025-01-09T15:48:56-04:00 2025-01-09T19:48:56.753597Z DEBUG run_main:check: startos::manager::health: Checking health of mempool
2025-01-09T14:08:24-04:00 2025-01-09T18:08:24.180270Z DEBUG run_main:check: startos::manager::health: Checking health of bitcoind
2025-01-09T15:48:41-04:00 2025-01-09T19:34:38.455309Z ERROR startos::db: WebSocket Closed: Network Error: IO error: Broken pipe (os error 32)
2025-01-09T15:48:41-04:00 2025-01-09T19:48:41.227286Z DEBUG startos::db: Error { source:
2025-01-09T15:48:41-04:00 0: IO error: Broken pipe (os error 32)
2025-01-09T15:48:41-04:00 1: Broken pipe (os error 32)
2025-01-09T15:48:41-04:00 Location:
2025-01-09T15:48:41-04:00 /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/core/src/convert/mod.rs:716
2025-01-09T15:48:41-04:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2025-01-09T15:48:41-04:00 0: startos::db::deal_with_messages
2025-01-09T15:48:41-04:00 at startos/src/db/mod.rs:78
2025-01-09T15:48:41-04:00 1: startos::db::ws_handler
2025-01-09T15:48:41-04:00 at startos/src/db/mod.rs:32
2025-01-09T15:48:41-04:00 Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
2025-01-09T15:48:41-04:00 Run with RUST_BACKTRACE=full to include source snippets., kind: Network, revision: None }
2025-01-09T15:48:41-04:00 2025-01-09T19:48:41.266795Z DEBUG run_main:check: startos::manager::health: Checking health of searxng
2025-01-09T15:48:41-04:00 2025-01-09T19:48:41.272038Z DEBUG run_main:check: startos::manager::health: Checking health of filebrowser
2025-01-09T15:48:41-04:00 2025-01-09T19:48:41.272713Z ERROR startos::net::tor: Tor Daemon Error: Tor health check failed for longer than current timeout (120s): Restarting tor
2025-01-09T15:48:41-04:00 2025-01-09T19:48:41.272771Z DEBUG run_main:check: startos::manager::health: Checking health of vaultwarden
2025-01-09T15:48:41-04:00 2025-01-09T19:48:41.272781Z DEBUG startos::net::tor: Error { source:
2025-01-09T15:48:41-04:00 0: Tor health check failed for longer than current timeout (120s)
2025-01-09T15:48:41-04:00 Location:
2025-01-09T15:48:41-04:00 startos/src/net/tor.rs:628
2025-01-09T15:48:41-04:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2025-01-09T15:48:41-04:00 0: startos::net::tor::torctl
2025-01-09T15:48:41-04:00 at startos/src/net/tor.rs:250
2025-01-09T15:48:41-04:00 Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
2025-01-09T15:48:41-04:00 Run with RUST_BACKTRACE=full to include source snippets., kind: Tor, revision: None }
2025-01-09T15:48:41-04:00 2025-01-09T19:48:41.752801Z DEBUG run_main:check: startos::manager::health: Checking health of syncthing
2025-01-09T15:48:41-04:00 2025-01-09T19:48:41.883897Z DEBUG run_main:check: startos::manager::health: Checking health of mutiny-wallet
2025-01-09T15:48:41-04:00 2025-01-09T19:48:41.888780Z DEBUG run_main:check: startos::manager::health: Checking health of nextcloud
2025-01-09T15:48:42-04:00 2025-01-09T19:48:42.059455Z ERROR startos::core::rpc_continuations: WebSocket Closed: Network Error: IO error: Connection reset by peer (os error 104)
2025-01-09T15:48:42-04:00 2025-01-09T19:48:42.059524Z DEBUG startos::core::rpc_continuations: Error { source:
2025-01-09T15:48:42-04:00 0: IO error: Connection reset by peer (os error 104)
2025-01-09T15:48:42-04:00 1: Connection reset by peer (os error 104)
2025-01-09T15:48:42-04:00 Location:
2025-01-09T15:48:42-04:00 /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/core/src/convert/mod.rs:716
2025-01-09T15:48:42-04:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2025-01-09T15:48:42-04:00 0: startos::logs::ws_handler
2025-01-09T15:48:42-04:00 at startos/src/logs.rs:67
2025-01-09T15:48:42-04:00 Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
2025-01-09T15:48:42-04:00 Run with RUST_BACKTRACE=full to include source snippets., kind: Network, revision: None }
2025-01-09T15:48:47-04:00 2025-01-09T19:48:47.767012Z ERROR startos::manager::manager_container: The service electrs has crashed with the following exit code: 1
2025-01-09T15:48:48-04:00 2025-01-09T19:48:48.052709Z INFO torctl: startos::net::tor: Tor is started
2025-01-09T15:48:56-04:00 2025-01-09T19:48:56.753597Z DEBUG run_main:check: startos::manager::health: Checking health of mempool
2
OK, when you first mentioned all this, my assumption was that it was being caused by some aspect of your DIY hardware.
From the later Electrs crash, I’m guessing that crashed because Bitcoin crashed first (most likely) or because Electrs crashed independently. Both of these are typically caused by issues with the disk… but to confirm you’d need to look at what is happened with the logs of Bitcoin Core and Electrs. If Core is crashing and bringing down the system, that might explain your connectivity issues, and I suppose it makes sense that you didn’t consider checking Core or Electrs.
I’d been checking the logs of Core and Electrs since the server has crashed a couple times after syncing and didn’t notice anything out of the ordinary. The server became unreachable sometime Thurs night to Fri morning on all devices and url’s, on the server itself the login screen was there but when I enter the pw, ‘Unknown Error’ came up. I double checked the pw and entered it a couple of time, no luck. I didn’t want to restart in case it came back online like the other day and we’d be able to see the logs.
I just restarted it and everything started ok, Core and Electrs are synced and running.
By issues with the disk do you mean the drive that the server OS is on or the drive with the services?
Is there anything I should I be looking for in the logs or any other ideas?
It’s tricky to know what to say without seeing all the logs. For example, that crash you saw on Electrs will have had something more detailed in the logs for Electrs itself, both at the same time and in the run up to happening.
You say the logs for Core are fine… which leaves me at a loss. I’d probably rather see them for myself to be sure. I’ll ask you to send me those in a moment.
And yes, when there’s nothing showing up on the software side of things, it makes sense to look at hardware, and Bitcoin in particular always exposes faults in any SSD or RAM since it’s so intensive.
Well, they were fine until they weren’t and then it all went down. I’ll get the logs open on diff tabs to hopefully preserve some info. Core, Electrs and OS logs, anything else?
Hi @StuPleb,
Did you see anything in the logs? I was looking at the OS logs a few days ago and noticed the server would crash on Searxng, Nextcloud and others too so I stopped all services except essential ones and it’s been running well so I’m guessing it’s a resources issue?
Is it possible to transfer certain services to a new comp (i.e. BTC and related on one server and everything else on another) without starting fresh?
You said you were sending them, but I never received anything, so I thought you hadn’t done it yet. Could you send me the email address you used by the open DM we have? I’ll see if it got filtered somewhere.
The Electrs logs we can ignore, because it just says it can’t reach Bitcoin.
The Bitcoin logs show that it can’t poll itself and then Bitcoin crashes/restarts.
I wonder if one of two things is involved…
Low spec hardware - when you do have the StartOS UI loading, does it show multiple timeouts and retrying on the web UI healthcheck for Bitcoin? For this machine to have worse bottlenecks that an Pi though, is probably unexpected. Tell me about the specs.
Did something go wrong with the flashing of the OS or the transfer of Bitcoin. I’d suggest maybe a semi-nuking of select things. For example, reflash the OS, and see if that helps. Then nuke Bitcoin by uninstalling and reinstalling. Your slow hardware might be an issue. You could always copy and the manually re-add the blockchain, or some of it. I would bet that hitting stop on Bitcoin will completely solve all the connection issues.
I’m thinking low spec hardware… I just opened the comps and they’re both 4GB RAM… My thinkcentre just arrived so I’ll swap everything over in the next few days and report back.
