Aider — 169 runs across 83 models · V1

Aider Deep Research

A campaign-scale observation of the Aider coding wrapper. 161 numbered runs plus 7 control artifacts plus 1 generic. Each run captures the full chat history, exit code, git diff, observatory session ID, and an ollama-journal probe tagging each HTTP status to its run.

completed: 78
validation_failed: 53
invalid_diff: 13
truly "ok" (T2): 5
raw payload: 192 MB

What was observed

Track A · Tier 1 (T1)

lru_cache decorator from scratch

Implement lru_cache(max_size) in lru.py passing a 7-test pytest suite, ruff=0, mypy=0. Threading test with 8 concurrent threads. Strict gating: 7 acceptance criteria.

Track A · Tier 2 (T2)

Token bucket rate limiter

Implement TokenBucket with capacity/refill, concurrent acquire, optional blocking. Validated via test_bucket.py + test_rate_limiter.py.

Track B · paper_review

arxiv preprint review

39 runs. Review papers like Understanding Human Actions through the Lens of Executable Models, produce JSON-schema-validated paper_review.json + self-contained report.html with 12 H2 sections, scored claims with evidence quotes, taxonomy-controlled enums.

smoke / longtask / cloud_probe / judge_run

Plumbing and judging

40 smoke runs for plumbing health, 1 multi-task longtask, 1 cloud reachability probe, plus the 108 blinded judging runs (gpt-5.4 high reasoning).

Five patterns that surfaced

Diff-format dependency

13 invalid_diff + 53 validation_failed

Many Ollama-cloud models could not produce valid <<<<<<< SEARCH … ======= … >>>>>>> REPLACE blocks. Aider's diff format is a hard gate; producing valid pytest-passing code but failing ruff/mypy or the diff format counts as validation_failed.

Ollama 500-storm

9 consecutive 500s, then recovery

Paper_review with kimi-k2:1t-cloud captured a 69-second window of consecutive 500s on /api/chat. Aider's litellm retry ladder (0.2s, 0.5s, 1.0s, 2.0s, 4.0s, 8.0s, 16.0s, 32.0s) was logged. Final status ok_recovered via JSON-coercion of a malformed response.

Strict gating

Only 5 runs truly "ok"

The 5 cells with pytest_passed=7/7, ruff=0, mypy=0 were all Tier 2: qwen3.5:397b-cloud, gemini-3-flash-preview:cloud, glm-5:cloud, qwen3.5:cloud, ministral-3:3b-cloud. The 3B Ministral hit clean — size-quality decoupling again.

Blinded judging

gpt-5.4 high with name redaction

Each agentic run gets a judge.json from gpt-5.4 high reasoning with blinding_state.model_name_hidden=true and redacted_fields=["model","model_slug","aider_command"]. Scored on 5 axes: correctness, code_quality, type_discipline, robustness, minimality.