Local versus hosted models
Compare hosted APIs with open-weight models you run yourself, size the hardware honestly, and route between the two when neither answer fits everywhere.
The trade-off
| Dimension | Hosted API | Local open-weight |
|---|---|---|
| Capability ceiling | Highest available | Depends on the size you can run |
| Cost model | Per token, scales with use | Fixed hardware, near-zero marginal cost |
| Latency | Network plus queueing | Low and predictable after the model is loaded |
| Privacy | Data leaves your boundary | Data never leaves the machine |
| Availability | Depends on a third party | Depends on your own hardware |
| Maintenance | None | Drivers, quantisation, upgrades, capacity |
| Reproducibility | Model can change under you | Pinned weights are truly pinned |
- Hosted wins on capability and on the absence of operational work; local wins on privacy, per-request cost at high volume, and determinism.
- The break-even is arithmetic, not ideology: compare tokens per month against the cost of the GPU and the engineer time to run it.
- Privacy requirements are often the deciding factor long before cost is.
Sizing local hardware
def vram_gb(params_b, bits_per_weight=4, overhead=1.2):
"""Rough weight memory for a quantised model, with room for the KV cache."""
return params_b * bits_per_weight / 8 * overhead
for p in (3, 8, 14, 32, 70):
print(f"{p}B at 4-bit: about {vram_gb(p):.1f} GB")| Model size | 4-bit weights | Realistic GPU |
|---|---|---|
| 3B | about 2 GB | Any modern laptop GPU or CPU |
| 8B | about 5 GB | 8 GB GPU |
| 14B | about 9 GB | 12 to 16 GB GPU |
| 32B | about 19 GB | 24 GB GPU |
| 70B | about 42 GB | 48 GB or two 24 GB GPUs |
Add memory for the key-value cache, which grows with context length and concurrent requests, and expect throughput to fall sharply if the model does not fit entirely in VRAM and spills to system memory.
Routing between them
def route(prompt, tier, sensitive):
if sensitive:
return local_model(prompt) # data must not leave
if tier == "cheap" or is_simple(prompt):
out = local_model(prompt)
if is_confident(out):
return out
return hosted_model(prompt) # escalate for quality
def with_fallback(prompt):
try:
return local_model(prompt)
except LocalUnavailable:
return hosted_model(prompt) # never leave the user with nothing💡
A hybrid setup concentrates the operational complexity: two prompt formats, two latency profiles, two evaluation results. Only take it on when there is a concrete reason, such as a hard data-residency rule or a volume large enough to make the hardware cheaper.
FAQ
Is a local model cheaper?
Only at volume. A GPU that sits idle costs the same as one that is busy, so compare the monthly hardware cost against tokens times price per token. Below a few million tokens a month, a hosted API is usually cheaper once you count the engineer time.
How does local inference affect quality?
A 7B model is not a small version of a frontier model; it fails differently, especially on multi-step reasoning and unusual instructions. Evaluate it on your own cases rather than extrapolating from benchmarks.
Related
Using a model API Choosing between prompting, RAG and fine-tuning
Last refreshed 2026-09-18.