Ollama deployment composite observation · V1

Ollama Deployment Observation

August 26, 2025 — the earliest preserved Sigma5C observation. Twenty-seven Ollama models, 184 runs, scored on a composite of quality, average latency, success rate, and "Web Integration" (the model's ability to consume retrieved web content as part of an answer). Only four models cleared the Professional tier. Average success rate across the field was 14.5%, meaning most models failed the multi-source web-integration probe entirely.

Professional tier: 4
deployment-ready tier: 0
avg success rate: 14.5%
chunking ablation files: 15

What 27 deployment observations recorded

The composite score blends quality, latency, success rate, and web-integration ability into a single number. Most models failed the web-integration probe entirely — only four cleared the Professional tier. The four highlights below summarize the chart.

Observed group

Three Dolphin Qwen 2 7B variants + Gemma 2 2B

Three Cognitive Computations Dolphin-Qwen2-7B quantizations and Google's Gemma 2 2B were the only four models to clear the Professional tier composite.

Headline number

14.5% mean success rate

Across all 27 models and 184 runs, only ~14.5% of attempts cleared the multi-source web-integration probe. The probe is a hard filter — most models fail it entirely.

Top observation band

Zero models in the top band

No model reached the top composite band in this observation. Even the four Professional-tier models recorded quality 0.77-0.78 and ~50s latency, not the >0.85 plus sub-30s threshold.

Companion ablation

15-file chunking study

Same observation captured a 15-file embedding-chunking matrix (256/512/1024 tokens × 5 corpora). Recording per-corpus chunk-size observations that feed downstream retrieval.

The full Professional observed score table

Composite score = blend of quality × latency × success × web-integration. Models below the cut-off (the remaining 23) had quality < 0.7 or success rate near zero.

Composite

dolphin-2.9.2-qwen2-7b:Q3_K_S: 0.857
dolphin-2.9.2-qwen2-7b:Q2_K: 0.851
dolphin-2.9.2-qwen2-7b:Q3_K_M: 0.849
gemma2:2b: 0.832

4 entries · ordered by Composite as observed in this run · not a ranking

15-file chunking ablation matrix

Alongside the model observation, the same observation included a chunking-strategy ablation across five corpora and three chunk sizes. Each cell is one file; the matrix has 5 × 3 = 15 files.

5 corpora

Per-corpus chunking

Python code · API documentation · Sigma5C internal standards · technical content · mixed content.

3 chunk sizes

256 / 512 / 1024 tokens

Each corpus chunked at every size. Output: 5 corpora × 3 sizes = 15 files of pre-chunked retrieval input.

Used downstream

Feeds web-integration probe

The chunked content was retrievable input to the model's web-integration tests. The 14.5% baseline was conditional on getting the right chunks back from retrieval.