Hi,
My StartOS server is freezing at least once per day.
When server freezes, it doesn’t shut down and ethernet lights are blinking. Also, I have noticed that it isn’t listed in LAN devices list when checking on the router. Of course, I can’t reach server through browser or SSH (ssh start9@192.168.1.100
ssh: connect to host 192.168.1.100 port 22: No route to host
)
I have tried with connected monitor, mouse and keyboard, but screen is black. When restarted, everything works normal for a day or less.
As this was happening every day, yesterday I have opened SSH connection on my laptop and in terminal run sudo journalctl -xefa. As you can notice the last line in log is nextcloud.
I am not sure is it the nextcloud service a reason for server failure so will appreciate is someone can check the log and see it something else.
Feb 28 12:55:01 ironic-surface CRON[3432648]: pam_unix(cron:session): session closed for user root
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: Unhandled Error
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: Traceback (most recent call last):
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/python/log.py”, line 96, in callWithLogger
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: return callWithContext({“system”: lp}, func, *args, **kw)
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/python/log.py”, line 80, in callWithContext
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: return context.call({ILogContext: newCtx}, func, *args, **kw)
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/python/context.py”, line 117, in callWithContext
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: return self.currentContext().callWithContext(ctx, func, *args, **kw)
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/python/context.py”, line 82, in callWithContext
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: return func(*args, **kw)
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: — —
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/internet/posixbase.py”, line 482, in _doReadOrWrite
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: why = selectable.doRead()
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/internet/tcp.py”, line 248, in doRead
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: return self._dataReceived(data)
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/internet/tcp.py”, line 253, in _dataReceived
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: rval = self.protocol.dataReceived(data)
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/internet/endpoints.py”, line 151, in dataReceived
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: return self._wrappedProtocol.dataReceived(data)
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/txtorcon/socks.py”, line 529, in dataReceived
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: self._machine.feed_data(data)
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/txtorcon/socks.py”, line 136, in feed_data
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: self.got_data()
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/automat/_methodical.py”, line 284, in got_data
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: value = output(oself, *a, **k)
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/automat/_methodical.py”, line 322, in call
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: return self.method(oself, *args, **kwargs)
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/txtorcon/socks.py”, line 311, in _relay_data
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: self._sender.dataReceived(d)
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/protocols/tls.py”, line 324, in dataReceived
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: self._checkHandshakeStatus()
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/protocols/tls.py”, line 251, in _checkHandshakeStatus
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: self._tlsShutdownFinished(Failure())
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/protocols/tls.py”, line 377, in _tlsShutdownFinished
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: if _representsEOF(reason.value):
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/protocols/tls.py”, line 121, in _representsEOF
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: errorQueue = exceptionObject.args[0]
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: builtins.IndexError: tuple index out of range
Feb 28 12:55:02 ironic-surface jam.embassy[76886]:
Feb 28 12:55:02 ironic-surface jam.embassy[76886]: 2025-02-28 12:55:01,410 [INFO] Lost IRC connection to: darkirc6tqgpnwd3blln3yfv5ckl47eg7llfxkmtovrv7c7iwohhb6ad.onion . Should reconnect automatically soon.
Feb 28 12:55:04 ironic-surface startd[874]: 2025-02-28T12:55:04.190952Z DEBUG run_main:check: startos::manager::health: Checking health of lnd
Feb 28 12:55:04 ironic-surface podman[3432662]: 2025-02-28 12:55:04.249911471 +0000 UTC m=+0.052980541 container exec 6c96e90eb56be2ea0097693d4ee5aa2ee15e04304da05aa36a4e73625e831177 (image=docker.io/start9/lnd/main:0.18.5, name=lnd.embassy)
Feb 28 12:55:04 ironic-surface podman[3432662]: 2025-02-28 12:55:04.26093174 +0000 UTC m=+0.064000800 container exec_died 6c96e90eb56be2ea0097693d4ee5aa2ee15e04304da05aa36a4e73625e831177 (image=docker.io/start9/lnd/main:0.18.5, name=lnd.embassy)
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: Unhandled Error
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: Traceback (most recent call last):
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/python/log.py”, line 96, in callWithLogger
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: return callWithContext({“system”: lp}, func, *args, **kw)
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/python/log.py”, line 80, in callWithContext
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: return context.call({ILogContext: newCtx}, func, *args, **kw)
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/python/context.py”, line 117, in callWithContext
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: return self.currentContext().callWithContext(ctx, func, *args, **kw)
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/python/context.py”, line 82, in callWithContext
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: return func(*args, **kw)
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: — —
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/internet/posixbase.py”, line 482, in _doReadOrWrite
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: why = selectable.doRead()
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/internet/tcp.py”, line 248, in doRead
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: return self._dataReceived(data)
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/internet/tcp.py”, line 253, in _dataReceived
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: rval = self.protocol.dataReceived(data)
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/internet/endpoints.py”, line 151, in dataReceived
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: return self._wrappedProtocol.dataReceived(data)
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/txtorcon/socks.py”, line 529, in dataReceived
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: self._machine.feed_data(data)
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/txtorcon/socks.py”, line 136, in feed_data
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: self.got_data()
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/automat/_methodical.py”, line 284, in got_data
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: value = output(oself, *a, **k)
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/automat/_methodical.py”, line 322, in call
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: return self.method(oself, *args, **kwargs)
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/txtorcon/socks.py”, line 311, in _relay_data
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: self._sender.dataReceived(d)
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/protocols/tls.py”, line 324, in dataReceived
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: self._checkHandshakeStatus()
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/protocols/tls.py”, line 251, in _checkHandshakeStatus
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: self._tlsShutdownFinished(Failure())
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/protocols/tls.py”, line 377, in _tlsShutdownFinished
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: if _representsEOF(reason.value):
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/protocols/tls.py”, line 121, in _representsEOF
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: errorQueue = exceptionObject.args[0]
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: builtins.IndexError: tuple index out of range
Feb 28 12:55:04 ironic-surface jam.embassy[76886]:
Feb 28 12:55:04 ironic-surface jam.embassy[76886]: 2025-02-28 12:55:03,679 [INFO] Lost IRC connection to: darkirc6tqgpnwd3blln3yfv5ckl47eg7llfxkmtovrv7c7iwohhb6ad.onion . Should reconnect automatically soon.
Feb 28 12:55:04 ironic-surface startd[874]: 2025-02-28T12:55:04.729526Z DEBUG run_main:check: startos::manager::health: Checking health of c-lightning
Feb 28 12:55:04 ironic-surface podman[3432706]: 2025-02-28 12:55:04.79444114 +0000 UTC m=+0.056761423 container exec dfccc6623478eddee00b10949b66051e1aaadefd30e6bdfd287d1c9899bb4f89 (image=docker.io/start9/c-lightning/main:24.11.1.1, name=c-lightning.embassy)
Feb 28 12:55:04 ironic-surface podman[3432708]: 2025-02-28 12:55:04.832021481 +0000 UTC m=+0.094473665 container exec dfccc6623478eddee00b10949b66051e1aaadefd30e6bdfd287d1c9899bb4f89 (image=docker.io/start9/c-lightning/main:24.11.1.1, name=c-lightning.embassy)
Feb 28 12:55:04 ironic-surface podman[3432706]: 2025-02-28 12:55:04.835098711 +0000 UTC m=+0.097419014 container exec_died dfccc6623478eddee00b10949b66051e1aaadefd30e6bdfd287d1c9899bb4f89 (image=docker.io/start9/c-lightning/main:24.11.1.1, name=c-lightning.embassy)
Feb 28 12:55:04 ironic-surface podman[3432708]: 2025-02-28 12:55:04.848448304 +0000 UTC m=+0.110900467 container exec_died dfccc6623478eddee00b10949b66051e1aaadefd30e6bdfd287d1c9899bb4f89 (image=docker.io/start9/c-lightning/main:24.11.1.1, name=c-lightning.embassy)
Feb 28 12:55:04 ironic-surface podman[3432707]: 2025-02-28 12:55:04.881945519 +0000 UTC m=+0.144469949 container exec dfccc6623478eddee00b10949b66051e1aaadefd30e6bdfd287d1c9899bb4f89 (image=docker.io/start9/c-lightning/main:24.11.1.1, name=c-lightning.embassy)
Feb 28 12:55:04 ironic-surface podman[3432707]: 2025-02-28 12:55:04.906361889 +0000 UTC m=+0.168886319 container exec_died dfccc6623478eddee00b10949b66051e1aaadefd30e6bdfd287d1c9899bb4f89 (image=docker.io/start9/c-lightning/main:24.11.1.1, name=c-lightning.embassy)
Feb 28 12:55:06 ironic-surface startd[874]: 2025-02-28T12:55:06.517873Z DEBUG run_main:check: startos::manager::health: Checking health of simplex
Feb 28 12:55:06 ironic-surface podman[3432870]: 2025-02-28 12:55:06.573467974 +0000 UTC m=+0.050254004 container exec a48282460d60cf8a2624f443a2c62b328b7037839f14f9f6fd1460a62163add7 (image=docker.io/start9/simplex/main:6.1.3.1, name=simplex.embassy, org.opencontainers.image.ref.name=ubuntu, org.opencontainers.image.version=22.04)
Feb 28 12:55:06 ironic-surface podman[3432896]: 2025-02-28 12:55:06.636591711 +0000 UTC m=+0.056454561 container exec_died a48282460d60cf8a2624f443a2c62b328b7037839f14f9f6fd1460a62163add7 (image=docker.io/start9/simplex/main:6.1.3.1, name=simplex.embassy, org.opencontainers.image.ref.name=ubuntu, org.opencontainers.image.version=22.04)
Feb 28 12:55:06 ironic-surface podman[3432870]: 2025-02-28 12:55:06.641442511 +0000 UTC m=+0.118228650 container exec_died a48282460d60cf8a2624f443a2c62b328b7037839f14f9f6fd1460a62163add7 (image=docker.io/start9/simplex/main:6.1.3.1, name=simplex.embassy, org.opencontainers.image.ref.name=ubuntu, org.opencontainers.image.version=22.04)
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: Unhandled Error
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: Traceback (most recent call last):
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/python/log.py”, line 96, in callWithLogger
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: return callWithContext({“system”: lp}, func, *args, **kw)
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/python/log.py”, line 80, in callWithContext
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: return context.call({ILogContext: newCtx}, func, *args, **kw)
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/python/context.py”, line 117, in callWithContext
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: return self.currentContext().callWithContext(ctx, func, *args, **kw)
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/python/context.py”, line 82, in callWithContext
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: return func(*args, **kw)
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: — —
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/internet/posixbase.py”, line 482, in _doReadOrWrite
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: why = selectable.doRead()
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/internet/tcp.py”, line 248, in doRead
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: return self._dataReceived(data)
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/internet/tcp.py”, line 253, in _dataReceived
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: rval = self.protocol.dataReceived(data)
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/internet/endpoints.py”, line 151, in dataReceived
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: return self._wrappedProtocol.dataReceived(data)
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/txtorcon/socks.py”, line 529, in dataReceived
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: self._machine.feed_data(data)
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/txtorcon/socks.py”, line 136, in feed_data
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: self.got_data()
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/automat/_methodical.py”, line 284, in got_data
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: value = output(oself, *a, **k)
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/automat/_methodical.py”, line 322, in call
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: return self.method(oself, *args, **kwargs)
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/txtorcon/socks.py”, line 311, in _relay_data
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: self._sender.dataReceived(d)
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/protocols/tls.py”, line 324, in dataReceived
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: self._checkHandshakeStatus()
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/protocols/tls.py”, line 251, in _checkHandshakeStatus
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: self._tlsShutdownFinished(Failure())
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/protocols/tls.py”, line 377, in _tlsShutdownFinished
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: if _representsEOF(reason.value):
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: File “/usr/local/lib/python3.9/dist-packages/twisted/protocols/tls.py”, line 121, in _representsEOF
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: errorQueue = exceptionObject.args[0]
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: builtins.IndexError: tuple index out of range
Feb 28 12:55:07 ironic-surface jam.embassy[76886]:
Feb 28 12:55:07 ironic-surface jam.embassy[76886]: 2025-02-28 12:55:06,322 [INFO] Lost IRC connection to: darkirc6tqgpnwd3blln3yfv5ckl47eg7llfxkmtovrv7c7iwohhb6ad.onion . Should reconnect automatically soon.
Feb 28 12:55:08 ironic-surface startd[874]: 2025-02-28T12:55:08.780353Z DEBUG run_main:check: startos::manager::health: Checking health of nextcloud
Feb 28 12:55:08 ironic-surface podman[3432915]: 2025-02-28 12:55:08.838309597 +0000 UTC m=+0.052030920 container exec 082b57a0917f5fd4c579a286df1bf122852cb3434d4e74a4a2860d4b066a9e0b (image=docker.io/start9/nextcloud/main:27.1.7, name=nextcloud.embassy)
Feb 28 12:55:08 ironic-surface nextcloud.embassy[4480]: 127.0.0.1 - 28/Feb/2025:12:55:08 +0000 “GET /index.php” 200
Feb 28 12:55:08 ironic-surface podman[3432940]: 2025-02-28 12:55:08.925439065 +0000 UTC m=+0.045336547 container exec_died 082b57a0917f5fd4c579a286df1bf122852cb3434d4e74a4a2860d4b066a9e0b (image=docker.io/start9/nextcloud/main:27.1.7, name=nextcloud.embassy)
Feb 28 12:55:08 ironic-surface podman[3432915]: 2025-02-28 12:55:08.930193904 +0000 UTC m=+0.143915207 container exec_died 082b57a0917f5fd4c579a286df1bf122852cb3434d4e74a4a2860d4b066a9e0b (image=docker.io/start9/nextcloud/main:27.1.7, name=nextcloud.embassy)