I have a local limitless AI-like software here on my laptop. I was wondering if since my software needs an AI model to help with transcribing or inferring some things that may not make sense from the transcript. I could run a local model on my Start9 server and really just not have anything blow up or overheat. Do you think I could run a powerful enough model for this use case and still not have a really hot server all the time? Let me know, and any input on how the process of going about setting it all up would be greatly appreciated.
You can certainly run models on your server, and in a variety of ways and also provide API access to whatever other software you’re running.
Check out the AI category here: https://marketplace.start9.com/
Your stated use case has no specific quantitative metrics associated with it. And you don’t say what your server hardware is. I can’t say for sure if your server will become “really hot”, but it probably will. But that’s what the fan is for. As for capability, if you’re comparing your home server with the $1bn datacenter of a large company, yes there’s going to be a difference. That large company is running thousands of the equivalent of a high end gaming machine all connected together. With StartOS v040 and GPU support, we might very well get into “comfortable for general use” with large models though.
1 Like
Thanks for your help