Locai Link vs Ollama
How does Locai Link (llama.cpp build b9789) compare to Ollama (ollama-cuda) when both run the same Gemma 4 (Q4_K_M) model on the same CUDA hardware? Identical sampling, token-identical outputs, single-stream load. Measured on Linux; Mac and Windows results coming soon.
Test environment
Identical hardware, identical model file, identical sampling parameters across both backends.
- ·NVIDIA RTX 4070 · 12 GiB VRAM
- ·Arch Linux
- ·CUDA 13.3 · driver 610.43.02
- ·Compiler: nvcc / g++-15
- ·Gemma 4 E2B-it
- ·4.6B params · Q4_K_M
- ·GGUF · 2.9 GB
- ·Sliding-window attention, shared KV layers
- ·temperature = 0
- ·top_p = 1
- ·seed = 42
- ·streaming = true
Token-identical replies at temperature=0 confirmed for all prompts across both backends.
Results by scenario
Two workloads were measured: a short conversational prompt and a long-context document analysis.
Conversation. Short conversational prompt (~30 tokens), mid-length reply (~300 tokens). 20 runs plus 3 warmup, 0 errors.
| Metric | Locai Link | Ollama | Result |
|---|---|---|---|
| Time to first token (median) | 31 ms | 326 ms | Locai Link 10.5x faster |
| Time to first token (p95) | 132 ms | 338 ms | Locai Link 2.6x faster |
| Prefill throughput | 936 tok/s | 92 tok/s | Locai Link 10.2x faster |
| Decode throughput (median) | 196 tok/s | 166 tok/s | Locai Link 1.18x faster |
| Decode throughput (p95) | 198 tok/s | 169 tok/s | Locai Link 1.17x faster |
| End-to-end latency (median) | 1.6 s | 2.1 s | Locai Link 1.3x faster |
| Peak VRAM | 4,244 MiB | 1,990 MiB | Ollama uses 2.2 GiB less |
| Peak host memory (RSS) | 5,799 MB | 3,022 MB | Ollama uses 2.8 GB less |
Document analysis. Long-context document prompt (~9,500 tokens) from “From the Earth to the Moon”, reply capped at 500 tokens. 10 runs plus 3 warmup, 0 errors.
| Metric | Locai Link | Ollama | Result |
|---|---|---|---|
| Time to first token (median) | 56 ms | 385 ms | Locai Link 6.9x faster |
| Prefill throughput | 169,466 tok/s | 24,728 tok/s | Locai Link 6.9x faster |
| Decode throughput (median) | 174 tok/s | 158 tok/s | Locai Link 1.10x faster |
| Decode throughput (p95) | 178 tok/s | 160 tok/s | Locai Link 1.11x faster |
| End-to-end latency (median) | 1.2 s | 1.6 s | Locai Link 1.3x faster |
| Peak VRAM | 4,266 MiB | 2,006 MiB | Ollama uses 2.2 GiB less |
| Peak host memory (RSS) | 6,003 MB | 3,091 MB | Ollama uses 2.9 GB less |
What each metric means
- ·How long before the model starts replying.
- ·Speed of ingesting prompt tokens.
- ·Speed of generating reply tokens.
- ·Resident GPU and host memory at peak.
- ·Full request lifecycle: send to final token.
How to read this
Locai Link wins decisively on time to first token and prefill throughput, the phases that dominate perceived latency for interactive UIs and long-context document workflows. Decode throughput (token generation speed once the reply has started) is also faster, though the gap is narrower because both backends share the same underlying CUDA kernels for the generation loop.
Ollama uses meaningfully less peak VRAM and host RSS. That trade-off is by design: Ollama defers KV-cache and weight allocations more aggressively, while Locai Link pre-allocates to remove allocation cost from the hot path.
Caveats
- Single-stream measurements only. One request in flight at a time per backend. Concurrent load was not measured. Results are specific to Arch Linux and the ollama-cuda Arch package; build steps and library paths differ on Ubuntu and RHEL.
- These numbers describe a single in-flight request. Throughput under concurrent load is a separate test we will publish next.