How models scale under concurrent slots · V1
GPU Parallel Scaling
When a GPU runs multiple inference slots at once, the per-slot throughput drops but the aggregate (sum across slots) might rise. This observation maps the trade for the GTX 1070 8GB: tiny models scale near-linearly to 8 concurrent slots; mid-size 3B models gain modest aggregate throughput; large 14B models actively lose throughput under any concurrency.
- smollm2:135m at 8 slots
- 30.8×
- cogito:3b at 8 slots
- 2.3×
- cogito:14b at 4 slots
- 0.19×
Aggregate throughput across slots
Each row: model, single-slot tok/s, peak aggregate across 1-8 slots, peak aggregate / single-slot ratio, per-slot efficiency at the peak.
- qwen2.5:0.5b
- 189
- cogito:3b
- 62
- smollm2:135m
- 16.6
- smollm2:360m
- 13.6
- cogito:14b
- 4.8
5 entries · ordered by Solo tok/s as observed in this run · not a ranking
What this means for fleet design
Tiny model fleet
smollm2:135m scales to 511 tok/s
Eight concurrent slots, 96% per-slot efficiency. If your workload is short-context routing or compression, a single GPU can serve dozens of concurrent users.
3B sweet spot for aggregation
cogito:3b at 8 slots = 142 tok/s
Single-slot only 62 tok/s, but aggregate 142 — 2.3× lift. Per-slot efficiency drops to 26%, but the GPU is better utilized.
14B is solo-only
cogito:14b drops from 4.8 to 0.9 tok/s
At 4 concurrent slots, throughput collapses. Don't try to parallelize large models on a single mid-tier GPU.
Rule of thumb
Parallel slots favor small models
If you can split the workload across small workers, you can serve high concurrency on a GTX 1070 GPU. If you need a single large-model response, no amount of concurrency helps.