Ollama Performance Report: CPU-only on DGX Spark (GB10

StartOS Support
1

Hi, StuPleb,

I’d like to answer your last question (better late than never):

“Can you tell us anything about performance?”.

The NVIDIA GB10 GPU is detected but not utilized. The logs show:
skipping CUDA device — compute capability not in compiled architectures
device=“NVIDIA GB10” cc=1210

System Details
library=CUDA compute=12.1 name=CUDA0
description=“NVIDIA GB10”
total=“121.7 GiB” available=“118.3 GiB”

  • GPU: NVIDIA GB10 (Blackwell, compute capability 1210)
  • VRAM: 121.7 GiB total, 118.3 GiB available
  • CUDA driver: 13.0
  • OLLAMA_VULKAN: true (no CUDA backend present)

Performance Notes
GPU utilization: Not available (compute capability 1210 not in compiled architectures)

  • ollama-logs.html

Baseline: CPU-only inference (estimated 20 tokens/sec)
Context size: Default 262,144 tokens based on VRAM

Model Info (Llama)

  • Architecture: llama
  • Embedding dimension: 4096
  • Layers: 32
  • Heads: 32
  • Key-value heads: 8
  • Rotary embedding size: 128
  • Training context size: 131072
  • Token cache: special tokens = 256; token to piece = 0.7999 MB

Links:

Original thread with GPU detection issue:

(closed)

Note:
This appears to be an Ollama-specific limitation. Start9OS currently offers Ollama, llama.cpp, vLLM, and Open WebUI.

On this NVIDIA GB10 hardware with CUDA driver 13.0 and compute capability 1210, the following applications can utilize GPU acceleration:

CUDA-based AI/ML Applications

PyTorch with CUDA for deep learning and model execution
TensorFlow with CUDA for ML workloads and models
CUDA-based compiler nvcc for CUDA programs (when available)
cuDNN and other CUDA tools (if installed via Start9 or as ARM64 ports)

Vulkan-based Applications

Game Engines: Unity (with Vulkan), Godot, Unreal (via Vulkan plugins)
Vulkan-based rendering/compute applications:
vulkan-tools and vulkan-validation-layers
Vulkan-based AI tools such as GPU computer software

once again:
the logs confirm that Ollama skips the GPU due to compute capability 1210 not being listed in the compiled architectures, while the system correctly detects the NVIDIA GB10 with driver 13.0 and CUDA support.

2

Correction / Follow-up

I need to slightly correct my earlier report. The GPU limitation is indeed only a Ollama-specific issue.

What I changed:
In Open WebUI I replaced Ollama with llama.cpp as the OpenAI-compatible backend.

Result:

Metric|Value

|Model|Llama-3.2-1B (Q4_K_M)|
|Token rate| 182 t/s**|
|GPU SM utilization | 0% idle → 96% under load Daten_nvidia-smi_dmon_01.txt|
|GPU power draw | 11W idle → 44W under load Daten_nvidia-smi_dmon_01.txt|

The nvidia-smi dmon output confirms GPU activity during inference — SM jumps to 96% and power consumption quadruples. llama.cpp logs show CUDA support for ARCH=1200 (Blackwell) and GPU-layer offloading.

Summary for DGX Spark (GB10) users:

Ollama → CPU-only (compute capability 1210 not compiled) ollama-logs (1).html
llama.cpp (via Open WebUI) → GPU fully utilized

And that may well be only the beginning…

Best

3

I ran this by the developers and one reported that this used to work and so it seems to be a regression in Ollama. You might search for or make an issue on their github’s issues page:

Looking at the number of open tickets though, I’m not sure if you’ll get a timely answer:

4

Comprehensive GPU Issues on DGX Spark (GB10) with StartOS v0.4.0.1: Ollama, llama.cpp, and vLLM

Hi everyone,

Building on my earlier reports about GPU detection on the DGX Spark (GB10), I want to provide a comprehensive update on the current state of AI inference services on this hardware. If you’re considering running StartOS on a GB10, please be aware of the following limitations.
Current Situation (as of 2026-08-05)
Hardware: NVIDIA DGX Spark (GB10, ARM64, Blackwell sm_121, 128GB unified memory)StartOS: v0.4.0.1 with NVIDIA drivers installedGPU Status: Detected by nvidia-smi, CUDA 13.0 available
Service-Specific Issues

  1. Ollama - CPU-only
    Problem: Ollama does not utilize the GB10 GPU at all.
    Root cause: The bundled llama.cpp in Ollama was not compiled with compute capability 1210 (Blackwell), causing it to skip the CUDA device entirely.
    Symptoms:
    • Logs show: skipping CUDA device — compute capability not in compiled architectures
    • Performance: ~5 tokens/sec on 70B models (CPU-only)
    • Expected with GPU: 20-30+ tokens/sec
    Workaround: None currently available within StartOS Ollama package.

  2. llama.cpp - Works with GPU (when configured correctly)
    Good news: llama.cpp does support the GB10 GPU when properly configured.
    Verified performance:
    • Llama-3.2-1B-Instruct-Q4_K_M: 182 tokens/sec
    • GPU SM utilization: 96%
    • Power draw: 44W (vs 11W idle)
    Configuration: Requires -ngl 999 flag and proper CUDA setup.

  3. vLLM - Unusable on StartOS
    Problem: vLLM consistently fails to start due to StartOS health check timeout (35 minutes).
    Symptoms:
    • Health check fails after 35 minutes with: “Die vLLM-API ist nicht innerhalb von 35 Minuten gestartet”
    • Process is killed and restarted repeatedly
    • Logs show GPU initialization is happening (FlashAttention, NCCL backend active)
    • Model loading takes >35 minutes even for cached models
    Tested models:
    • RedHatAI/Llama-3.3-70B-Instruct-NVFP4: Failed (too slow to initialize)
    • RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-NVFP4: Crashed (PixtralProcessor bug)
    • RedHatAI/Qwen3-30B-A3B-NVFP4: Still testing (in progress)
    Root cause: The 35-minute health check timeout is too short for vLLM’s initialization process, especially on Blackwell architecture. This appears to be a StartOS limitation, not a vLLM issue.
    Attempted workarounds:
    • Reduced --max-model-len from 131072 to 8192 (didn’t help)
    • HuggingFace token for faster downloads (no effect - models are cached anyway)
    • Smaller models (still hit timeout)
    Recommendations for GB10 Users

  4. Use llama.cpp - It’s the only service that reliably uses the GPU on this hardware. Expect 180+ tokens/sec on small models.

  5. Avoid Ollama - CPU-only until the package is rebuilt with Blackwell support.

  6. Don’t use vLLM - The health check timeout makes it unusable unless StartOS increases the timeout.

  7. Manual model downloads may be necessary - The automatic HF Hub downloads are unreliable and slow. Consider downloading models locally and copying them to the StartOS volumes.
    Questions for Start9 Team

  8. Is there a way to increase the health check timeout for services like vLLM?

  9. Are there plans to rebuild Ollama with Blackwell compute capability support?

  10. Could the llama.cpp package be updated to ensure GPU support is always included on aarch64-nvidia builds?
    Conclusion

The DGX Spark is a powerful piece of hardware, but StartOS currently cannot fully leverage its GPU capabilities. Only llama.cpp works reliably with GPU acceleration. Ollama and vLLM are either CPU-only or fail to start.

If you’re buying a GB10 specifically for local AI inference on StartOS, be prepared for these limitations. The hardware is excellent, but the software support is still catching up.

I’ll continue testing and update this thread if the situation improves.

Tags: #dgx-spark #gb10 #gpu #ollama #vllm #llama-cpp #blackwell #aarch64-nvidia

5

Hi Walther and thanks. I just want to make sure, are you on Ollama 0.32.5:0?