Searching on this topic showed only one post that unfortunately wasn’t helpful.
Is there a way to see each services CPU percentage use?
My CPU only recently has been at 100% and suspect that it may be a service.
I usually have tabs open showing the different services that I want to keep an eye on including the StartOS monitor page itself, will this cause higher CPU usage?
Thank you!
Hi Baguette,
What do you mean by tabs open? Are you talking about your client device (Windows, Mac, Linux, Android, iOS device)?
Or have you connected a monitor to a server? A 100% CPU usage would suggest something is amiss. If you made the mistake of treating a home server as a PC and you’re running it what we call “kiosk mode”, then extensive use of the cut down Firefox might lead to higher resource use… but it’s unlikely to be heavy on just the processor.
To work out what’s going on, you’d need to switch over to your client device, set up SSH (Start9 | Using SSH) then you’ll be able to log in and run standard linux commands. htop for example.
Hey Stu, thanks for the help.
Using Librewolf on my Linux desktop, I’m just at the .local address to access my services so I have several tabs open to show them.
My StartOS is built on a mini PC with 32Gb ram and 4TB SSD.
I did finally do a restart and it fixed the CPU usage for now, so it’s not the tabs but I would still like to see each services CPU usage just to see if it’s any one in particular that is causing an issue if it happens again.
I’ll follow the SSH link you gave and move forward from there, if you have any other suggestions, I appreciate it!
If it happens again before you finish setting up SSH (which is EASY on Linux) and could also just stop each service one by one to work out which it is. Then check the logs for that service to see what’s going wrong.
Excellent.
So far so good but I’ll keep an eye on it.
Thanks again!
So I didn’t have a 100% CPU event but I did have another crash which has been happening about every few days. I suspect they’re related.
I cannot access my dashboard via .local or Onion but I was able to access one of my services via TOR which means the server itself is working.
I did follow through on your SSH suggestion and now I’m sitting here looking at a connected terminal window with no idea what to do next since this is my first time SSHing into anything.
Is there a document somewhere showing the different commands or process to check my different services to narrow down and shut down so I can gain access to my dashboard again?
We would be fishing for clues, rather than doing anything specific.
i.e.
checking to see the uptime on your containers with
sudo podman ps - did any crash?
looking at the processes that are running
htop - does anything look out of place?
looking at serious errors in the logs
sudo dmesg | less - particularly at the exact same time things stopped working
… then restarting the device with sudo reboot now and seeing if anything looks different with the above now after a fresh boot.
Okie Dokie,
So the only major thing that I found was under HTOP, no CPU activity and every service showed “sleeping” except for three lines showing “Zombie”.
After restart there was again CPU activity and varying status’ of “Running”, “Disk Sleep”, "and “Sleeping” with everything except Fulcrum (explained below), functioning properly as far as I can tell.
The questions are, what would put every function to “Sleeping” mode and
why was I able to access some services if they all showed sleeping?
Specific to your questions-
Podman- did any crash?
No
htop- does anything look out of place?
As mentioned above, no CPU usage and everything “Sleeping”.
dmesg- serious errors in the logs
This is the oddest one. The only thing that shows in the logs is a thousand lines of “UFW BLOCK” and something about my VPN. Maybe filtering because of my VPN and only showing info from that?
The only other clearly broken thing is that Fulcrum will not start. It cycles quickly from “Starting” to “Running” with the health checks and back to running over and over and over.
I only just saw that Fulcrum has Bitcoin Knots as a dependency so that could be the issue here? I do not run Knots and so may have to abandon Fulcrum for now and go back to electrs. Am I on the right track here?
Bump
Had to reboot the system yesterday and everything ran great until just now when it locked up.
Checking HTOP showed everything in sleep mode.
But I just checked and I can access one of my services and it’s still running.
I am really confused on the trouble shooting here.
Tried to “Sudo Reboot Now” and got "Connection to *.local closed by remote host.
Connection to *.local closed.
A process in sleep mode just means it’s waiting for an event, such as an input from the user, or a response from another process or device.
Fulcrum should work with Core, just as “poorly” as it works with Knots. If you want a reliably functioning server however, you shouldn’t be using software from the Beta registries. Certainly not while you are having other issues with your server. The beta registries are for experimenting, ans testing only.
If your server appears to be on, but not responding to it’s adjective-noun.local address, it may have dropped into diag mode. Try start.local
Thanks for the help!
Understood on the sleep mode.
I figured that was the case on the Beta registry services but that’s the only place that Fulcrum was available and I was trying to solve an issue with Electrs. But I’ll uninstall Fulcrum and see if the system stays up from then.
I only knew of the adjective-noun.local and the .onion address, are you saying that just start.local should work? (it didn’t)
The sequence seems to be that I can’t access at the .local or .onion address’, then after a few minutes, I can’t SSH in or if I had SSH’d in it will kick me. Then about that time or a few minutes later, all services will have stopped and I will have to hard re-boot and then it works fine for a day or three.
Thanks again for the help!
Yea, Fulcrum is not quite ready for production on SartOS. Definitely remove that until you’ve been able to get your server to function properly otherwise.
The start.local address is only accessible during initial setup, or if the system drops into diag mode. If the server won’t respond to the adjective-noun.local address, the .onion address, or start.local, then either the server has become unresponsive. Or you have a separate issue with the client machine, or network, not allowing you to reach the server. You might want to connect a mouse monitor, and keyboard to access your server through what we call kiosk mode. The interface likely won’t show up if the server is already on. The monitor needs to be connected before you power on the server.
Copy that.
So far after removing Fulcrum we are good.