Unable to access Web interface

m1k_3 · November 13, 2024, 1:34am

I noticed my miner was offline so I tried to login to my start 9 via the web interface but it is not responding. I can usually access it no problem. I have tried all the usual devices I typically access it from and none work. I am able to access it via ssh.
My services appear to be running.

tart9@deadly-lama:~$ sudo podman ps --all
CONTAINER ID  IMAGE                                            COMMAND         CREATED      STATUS      PORTS       NAMES
e049ae942fd9  docker.io/start9/x_system/utils:latest           sleep infinity  10 days ago  Up 10 days              netdummy
75672838934a  docker.io/start9/bitcoind/main:27.1.0                            10 days ago  Up 10 days              bitcoind.embassy
dcae6f76fc21  docker.io/start9/electrs/main:0.10.5                             10 days ago  Up 10 days              electrs.embassy
1a89b1d8e527  docker.io/start9/mempool/main:3.0.0.1                            10 days ago  Up 10 days              mempool.embassy
6deddcd62fd9  docker.io/start9/ride-the-lightning/main:0.15.1                  10 days ago  Up 10 days              ride-the-lightning.embassy
b2dd49ec8d5f  docker.io/start9/datum/main:0.1.0                                4 days ago   Up 4 days               datum.embassy

Rick · November 13, 2024, 1:44am

First thing to always try is hard refresh ctrl + shift + r (Windows/Linux) cmd + shift + R (Mac). If that doesn’t work try to restart the client device. You can also work through this list of common issues Start9 | Common Issues

m1k_3 · November 13, 2024, 1:57am

I hard reboot the server and now I can access the webinterface. I was hoping there was something I could look at from the command line. Maybe next time.

ekpapo · November 14, 2024, 2:31pm

I have the same issue every month or two. I think an update or something to one of the services may be the cause. It never happened before in the past few years. I just do a hard reset and it goes away for a month or two. I tried looking at the logs and don’t see anything. I suspect it is related to FreeGPT soaking up resources or something. I did notice if I don’t use FreeGPT it lasts longer. It could be coincidental but I have refrained from using it and will know for sure this month as it is at 43 days running right now without the web interface dying.

StuPleb · November 15, 2024, 10:10am

This doesn’t sound related. Difficultly reaching your server could mean a whole range of issues, and is probably worth starting your own thread over if you need support.

One of the key tests to carry out is to check if it’s a local networking issue on the client sit by seeing if you can reach the server in two ways…

Via your .onion in Tor Browser
From another device on the LAN (i.e. a phone).

m1k_3 · November 15, 2024, 1:32pm

In my case, I could login vi ssh. It was only the web interface that was not accessible from many devices that I could previously access from. Now that I am looking at the output of sudo podman ps --all that I provided above I see c-lightning was not running and should have been, so maybe it crashed and was causing my problem.

ekpapo · November 15, 2024, 4:12pm

i did try from a secondary device (phone) but didn’t try over TOR

StuPleb · November 16, 2024, 2:16pm

Let’s get the issue reproduced… and when you do, go into System in the StartOS UI and get me all the the logs downloaded. When you have those email them to stu@start9.com or send me a message here and I’ll show you how else you can get them to me.

m1k_3 · November 16, 2024, 3:03pm

Hi Stu,
Not sure if you are replying to me or ekpapo. Are the logs persistent across a hard reboot? If not, am I able to collect them via ssh access which is still accessible when the web ui is not?

StuPleb · November 16, 2024, 3:50pm

Either, you can do something like

sudo journalctl -ea > journal_output.txt

to create a log file

or

sudo journalctl -efa

to watch logs live

m1k_3 · May 29, 2025, 7:50pm

I am still hitting the issue where I am unable to access the web interface. A reboot always solves the issue. I was able to collect some more logs.

Below is a snip from sudo journalctl -u startd > journal_output.txt. The timeout error keeps repeating.

May 25 13:53:03 deadly-lama mempool.embassy[22667]: May 25 13:53:03 [610] INFO: Core index 'basic block filter index' is synced. Best block height is 898314
May 25 13:53:04 deadly-lama mempool.embassy[22667]: May 25 13:53:04 [610] INFO: migrated 0 audits to version 1
May 25 13:54:32 deadly-lama electrs.embassy[7954]: [2025-05-25T13:54:32.307Z INFO  electrs::index] indexing 1 blocks: [898315..898315]
May 25 13:54:32 deadly-lama electrs.embassy[7954]: [2025-05-25T13:54:32.706Z INFO  electrs::chain] chain updated: tip=000000000000000000005aa8f4d92c7793b78159315175f0b4390a36ea85bbe0, height=898315
May 25 14:00:02 deadly-lama mempool.embassy[22667]: May 25 14:00:02 [610] INFO: Latest BTC fiat averaged price: {"time":1748181603,"USD":107553,"EUR":94714,"GBP":79477,"CAD":147975,"CHF":88447,"AUD":166096,"JPY":15350400}
May 25 14:02:22 deadly-lama electrs.embassy[7954]: [2025-05-25T14:02:22.698Z INFO  electrs::index] indexing 1 blocks: [898316..898316]
May 25 14:02:23 deadly-lama electrs.embassy[7954]: [2025-05-25T14:02:23.274Z INFO  electrs::chain] chain updated: tip=00000000000000000001f7b16758dea16c5cde66eb7231282df365cb6381257f, height=898316
May 25 14:13:13 deadly-lama mempool.embassy[22667]: May 25 14:13:13 [610] ERR: $updateBlocks stalled at "begin $updateBlocks"
May 25 14:20:17 deadly-lama mempool.embassy[22667]: May 25 14:20:17 [610] WARN: Exception in runMainUpdateLoop() (count: 5). Retrying in 1 sec. Reason: ETIMEDOUT. Stack trace: Error: ETIMEDOUT
May 25 14:20:17 deadly-lama mempool.embassy[22667]:     at Timeout._onTimeout (/backend/package/rpc-api/jsonrpc.js:75:23)
May 25 14:20:17 deadly-lama mempool.embassy[22667]:     at listOnTimeout (node:internal/timers:594:17)
May 25 14:20:17 deadly-lama mempool.embassy[22667]:     at process.processTimers (node:internal/timers:529:7)
May 25 14:22:18 deadly-lama mempool.embassy[22667]: May 25 14:22:18 [610] WARN: Exception in runMainUpdateLoop() (count: 6). Retrying in 1 sec. Reason: ETIMEDOUT. Stack trace: Error: ETIMEDOUT
.....
May 26 02:16:13 deadly-lama mempool.embassy[22667]:     at Timeout._onTimeout (/backend/package/rpc-api/jsonrpc.js:75:23)
May 26 02:16:13 deadly-lama mempool.embassy[22667]:     at listOnTimeout (node:internal/timers:594:17)
May 26 02:16:13 deadly-lama mempool.embassy[22667]:     at process.processTimers (node:internal/timers:529:7)
May 26 02:18:14 deadly-lama mempool.embassy[22667]: May 26 02:18:14 [610] WARN: Exception in runMainUpdateLoop() (count: 361). Retrying in 1 sec. Reason: ETIMEDOUT. Stack trace: Error: ETIMEDOUT
May 26 02:18:14 deadly-lama mempool.embassy[22667]:     at Timeout._onTimeout (/backend/package/rpc-api/jsonrpc.js:75:23)
May 26 02:18:14 deadly-lama mempool.embassy[22667]:     at listOnTimeout (node:internal/timers:594:17)
May 26 02:18:14 deadly-lama mempool.embassy[22667]:     at process.processTimers (node:internal/timers:529:7)
May 26 02:20:15 deadly-lama mempool.embassy[22667]: May 26 02:20:15 [610] WARN: Exception in runMainUpdateLoop() (count: 362). Retrying in 1 sec. Reason: ETIMEDOUT. Stack trace: Error: ETIMEDOUT
May 26 02:20:15 deadly-lama mempool.embassy[22667]:     at Timeout._onTimeout (/backend/package/rpc-api/jsonrpc.js:75:23)
May 26 02:20:15 deadly-lama mempool.embassy[22667]:     at listOnTimeout (node:internal/timers:594:17)
May 26 02:20:15 deadly-lama mempool.embassy[22667]:     at process.processTimers (node:internal/timers:529:7)
May 26 02:22:16 deadly-lama mempool.embassy[22667]: May 26 02:22:16 [610] WARN: Exception in runMainUpdateLoop() (count: 363). Retrying in 1 sec. Reason: ETIMEDOUT. Stack trace: Error: ETIMEDOUT
May 26 02:22:16 deadly-lama mempool.embassy[22667]:     at Timeout._onTimeout (/backend/package/rpc-api/jsonrpc.js:75:23)
May 26 02:22:16 deadly-lama mempool.embassy[22667]:     at listOnTimeout (node:internal/timers:594:17)
May 26 02:22:16 deadly-lama mempool.embassy[22667]:     at process.processTimers (node:internal/timers:529:7)

The Receive Queue is stuck at 1025 for port 443

tart9@deadly-lama:/tmp$ sudo ss -ltnp             
State        Recv-Q       Send-Q               Local Address:Port                Peer Address:Port       Process                                          
LISTEN       0            4096                       0.0.0.0:9050                     0.0.0.0:*           users:(("tor",pid=3005375,fd=6))                
LISTEN       0            4096                    127.0.0.54:53                       0.0.0.0:*           users:(("systemd-resolve",pid=704,fd=20))       
LISTEN       0            4096                     127.0.0.1:9051                     0.0.0.0:*           users:(("tor",pid=3005375,fd=7))                
LISTEN       0            244                      127.0.0.1:5432                     0.0.0.0:*           users:(("postgres",pid=2161,fd=6))              
LISTEN       0            128                        0.0.0.0:22                       0.0.0.0:*           users:(("sshd",pid=836,fd=3))                   
LISTEN       0            5                          0.0.0.0:23334                    0.0.0.0:*           users:(("simpleproxy",pid=795,fd=3))            
LISTEN       0            4096                 127.0.0.53%lo:53                       0.0.0.0:*           users:(("systemd-resolve",pid=704,fd=18))       
LISTEN       0            4096                       0.0.0.0:5355                     0.0.0.0:*           users:(("systemd-resolve",pid=704,fd=12))       
LISTEN       0            1024                     127.0.0.1:53                       0.0.0.0:*           users:(("startd",pid=970,fd=12))                
LISTEN       0            1024                             *:9814                           *:*           users:(("startd",pid=970,fd=21))                
LISTEN       0            244                          [::1]:5432                        [::]:*           users:(("postgres",pid=2161,fd=5))              
LISTEN       0            1024                             *:3010                           *:*           users:(("startd",pid=970,fd=17))                
LISTEN       0            1024                             *:2106                           *:*           users:(("startd",pid=970,fd=18))                
LISTEN       0            128                           [::]:22                          [::]:*           users:(("sshd",pid=836,fd=4))                   
LISTEN       6            128                              *:80                             *:*           users:(("startd",pid=970,fd=25))                
LISTEN       1025         1024                             *:443                            *:*           users:(("startd",pid=970,fd=14))                
LISTEN       0            1024                             *:23335                          *:*           users:(("startd",pid=970,fd=23))                
LISTEN       0            1024                             *:8080                           *:*           users:(("startd",pid=970,fd=19))                
LISTEN       0            1024                             *:7272                           *:*           users:(("startd",pid=970,fd=22))                
LISTEN       0            4096                          [::]:5355                        [::]:*           users:(("systemd-resolve",pid=704,fd=14))

● startd.service - StartOS Daemon
     Loaded: loaded (/lib/systemd/system/startd.service; enabled; preset: enabled)
     Active: active (running) since Sat 2025-05-17 14:21:44 UTC; 1 week 5 days ago
   Main PID: 970 (startd)
      Tasks: 172 (limit: 34716)
     Memory: 1.1G
        CPU: 1d 21h 52min 57.724s
     CGroup: /system.slice/startd.service
             ├─    970 /usr/bin/startd
             ├─   3513 /usr/bin/fuse-overlayfs -o lowerdir=/var/lib/containers/storage/overlay/l/LAK27LHIC4HTUERRWOI35DXZXV:/var/lib/containers/storage/o>
             ├─   3567 /usr/bin/conmon --api-version 1 -c 01df7d70266cd46604a5e144aad43a609330cda43a6e995121e70b3572e14600 -u 01df7d70266cd46604a5e144aad>
             ├─   3596 /usr/bin/fuse-overlayfs -o lowerdir=/var/lib/containers/storage/overlay/l/6RXLUA5X3TQOTBE2PP7GA6DNID:/var/lib/containers/storage/o>
             ├─   4076 podman run --rm --network=start9 --add-host=embassy:172.18.0.1 --name ride-the-lightning.embassy --hostname=ride-the-lightning.emb>
             ├─   4440 podman run --rm --network=start9 --add-host=embassy:172.18.0.1 --name datum.embassy --hostname=datum.embassy --no-healthcheck --mo>
             ├─   5438 /usr/bin/fuse-overlayfs -o lowerdir=/var/lib/containers/storage/overlay/l/5CBTBUACCEFMDIJ6Z2AUFNT5YR:/var/lib/containers/storage/o>
             ├─   5461 /usr/bin/fuse-overlayfs -o lowerdir=/var/lib/containers/storage/overlay/l/6HOA7GCJDJNGDDRDFHSATDYE4N:/var/lib/containers/storage/o>
             ├─   5547 /usr/bin/conmon --api-version 1 -c 1117c0e73bcea008da699ca376ab07fcaa1dbe17f606121503852329e8acca4e -u 1117c0e73bcea008da699ca376a>
             ├─   5555 /usr/bin/conmon --api-version 1 -c 7bc0e45f54bea3d8e3e38caae0d0959c87b2387dc71b8183ec15516e84e2abb9 -u 7bc0e45f54bea3d8e3e38caae0d>
             ├─   6704 podman run --rm --network=start9 --add-host=embassy:172.18.0.1 --name bitcoind.embassy --hostname=bitcoind.embassy --no-healthchec>
             ├─   7186 /usr/bin/fuse-overlayfs -o lowerdir=/var/lib/containers/storage/overlay/l/A2DU6IAWZM42JFM2F5ZA4GEQIS:/var/lib/containers/storage/o>
             ├─   7201 /usr/bin/conmon --api-version 1 -c 78d988193dab5d036d9eb7f0d543b51c7b98a55217d3f0e6f181bfb36415454f -u 78d988193dab5d036d9eb7f0d54>
             ├─   7681 podman run --rm --network=start9 --add-host=embassy:172.18.0.1 --name electrs.embassy --hostname=electrs.embassy --no-healthcheck >
             ├─   7817 /usr/bin/fuse-overlayfs -o lowerdir=/var/lib/containers/storage/overlay/l/WXYVHAVW5KTITP2LBZVSZZZCJG:/var/lib/containers/storage/o>
             ├─   7954 /usr/bin/conmon --api-version 1 -c 4f29811a3ef73807e6f0d12d685348bd50844e1698756c36e6c09359d9d78b8e -u 4f29811a3ef73807e6f0d12d685>
             ├─  22106 podman run --rm --network=start9 --add-host=embassy:172.18.0.1 --name mempool.embassy --hostname=mempool.embassy --no-healthcheck >
             ├─  22455 /usr/bin/fuse-overlayfs -o lowerdir=/var/lib/containers/storage/overlay/l/EPWXLORYJUU65YF44AQBSGOQOU:/var/lib/containers/storage/o>
             ├─  22667 /usr/bin/conmon --api-version 1 -c 1ea15575dc4b63cdd65106eb2a75e538ee1c00ec00a40abcb5f82798bc4076d4 -u 1ea15575dc4b63cdd65106eb2a7>
             ├─ 198157 /usr/bin/conmon --api-version 1 -c 7bc0e45f54bea3d8e3e38caae0d0959c87b2387dc71b8183ec15516e84e2abb9 -u 0ce56c34ce2824277e585072e30>
             ├─2131859 /usr/bin/conmon --api-version 1 -c 7bc0e45f54bea3d8e3e38caae0d0959c87b2387dc71b8183ec15516e84e2abb9 -u b2f90c22003f419bd6759d1dfd8>
             ├─2762926 podman run --rm --network=start9 --add-host=embassy:172.18.0.1 --name c-lightning.embassy --hostname=c-lightning.embassy --no-heal>
             ├─2763404 /usr/bin/fuse-overlayfs -o lowerdir=/var/lib/containers/storage/overlay/l/G6YORJQ4DRP5WE5OGXL2T6VW2X:/var/lib/containers/storage/o>
             ├─2763412 /usr/bin/conmon --api-version 1 -c 34c6706ac221091c02b7b1b3695fe901b933bb1567cae2a4a275034df92ab3cf -u 34c6706ac221091c02b7b1b3695>
             ├─2763531 avahi-alias 2n3vljslj2o3glknw5j256l3etww5hzgj55k4f2336v57t52kqxbkhad eguckg3rmc7d64tr6lofqe5myqv6zkt5nrdjhpj3ll6lvfgvinbwkxyd ibeb>
             ├─2915444 /usr/bin/conmon --api-version 1 -c 7bc0e45f54bea3d8e3e38caae0d0959c87b2387dc71b8183ec15516e84e2abb9 -u 907b15d84ac5b1e580dd60e5042>
             ├─3005372 journalctl --output=json --output-fields=MESSAGE -n0 -u tor@default --follow
             └─3926546 /usr/bin/conmon --api-version 1 -c 1117c0e73bcea008da699ca376ab07fcaa1dbe17f606121503852329e8acca4e -u 0c2136c1f1a45121a1909e4191f>

May 26 02:18:14 deadly-lama mempool.embassy[22667]:     at listOnTimeout (node:internal/timers:594:17)
May 26 02:18:14 deadly-lama mempool.embassy[22667]:     at process.processTimers (node:internal/timers:529:7)
May 26 02:20:15 deadly-lama mempool.embassy[22667]: May 26 02:20:15 [610] WARN: Exception in runMainUpdateLoop() (count: 362). Retrying in 1 sec. Reason:>
May 26 02:20:15 deadly-lama mempool.embassy[22667]:     at Timeout._onTimeout (/backend/package/rpc-api/jsonrpc.js:75:23)
May 26 02:20:15 deadly-lama mempool.embassy[22667]:     at listOnTimeout (node:internal/timers:594:17)
May 26 02:20:15 deadly-lama mempool.embassy[22667]:     at process.processTimers (node:internal/timers:529:7)
May 26 02:22:16 deadly-lama mempool.embassy[22667]: May 26 02:22:16 [610] WARN: Exception in runMainUpdateLoop() (count: 363). Retrying in 1 sec. Reason:>
May 26 02:22:16 deadly-lama mempool.embassy[22667]:     at Timeout._onTimeout (/backend/package/rpc-api/jsonrpc.js:75:23)
May 26 02:22:16 deadly-lama mempool.embassy[22667]:     at listOnTimeout (node:internal/timers:594:17)
May 26 02:22:16 deadly-lama mempool.embassy[22667]:     at process.processTimers (node:internal/timers:529:7)