Does scaffolding help a cloud model? · V1

Scaffolded Observations

Two thousand scaffolded runs across seven cloud models, five layer combinations, ten atomic pipeline rules, and two prompt formats — designed to settle whether the standard scaffolding playbook (long guidelines + multi-agent review + pre-commit hooks + TDD) improves cloud-model code generation. The headline answer is uncomfortable: guidelines and multi-agent review usually hurt; only reflection and TDD reliably help.

total observations: 1,934
retained · 5 errors: 240
layer combinations: 32
pipeline orderings: 60

Layers · rules · format

Axis 1 — Layer System

L1 to L5

Five binary scaffolds toggled independently — guidelines, self-reflection, TDD, pre-commit hooks, multi-agent review. 32 combinations.

Axis 2 — Pipeline Rules

10 atomic rules × 6 orderings

typing · docstrings · error_handling · naming · security · validation · edge_cases · performance · imports · dry. Ordered six ways: correctness-first, style-first, security-first, priority, top-3, minimal.

Axis 3 — Prompt Format

v1 flat prose · v2 xml

Same instructions, two presentations: paragraph-style prose vs structured XML tags. A/B tested across all models and layers.

The scaffolded observed score table

For each model: Base (no scaffold), L1 (Guidelines), L2 (Reflection), L3 (TDD), L4 (Pre-commit), L5 (Multi-Agent), All (all five enabled). The Lift column is "All" minus "Base".

Base

kimi-k2:1t-cloud: 8.8
gpt-oss:120b-cloud: 8.2
qwen3-coder:480b-cloud: 6.8
devstral-small-2:24b-cloud: 6.4
gemma3:12b-cloud: 6.4
ministral-3:8b-cloud: 6.4
ministral-3:3b-cloud: 5

7 entries · ordered by Base as observed in this run · not a ranking

Scaffold changes observed in this run

Per-layer average score change vs Base, averaged across all seven cloud models in the study. Two layers reliably help; two reliably hurt; one is a mild positive. This is the actionable summary.

Layer	What it adds	Δ vs Base	Verdict
L1 — Guidelines	Wall-of-text style and convention rules	−1.21	HURT
L2 — Self-Reflection	Ask the model to review its own output once before finalizing	+0.52	HELP (best)
L3 — Test-Driven	Generate test, then implementation; run tests before finalizing	+0.46	HELP
L4 — Pre-commit Hooks	Run ruff / mypy / format before finalizing	+0.26	MILD HELP
L5 — Multi-Agent Review	Spawn a second agent to review the first agent's output	−0.54	HURT

Layer observation. In this run, large guideline blocks recorded −1.21 and multi-agent review recorded −0.54 versus the base condition. Self-reflection recorded +0.52 and test-driven generation recorded +0.46. The result is scoped to this protocol and observation window.

What to do with this

Reflection is the best lever

+0.52 average lift, zero cost

Asking the model to review its own output once before finalizing is the single highest-yield scaffold. No new tools, no second model — just "now check your work". Cheap to implement, reliable across all 7 cloud models tested.

TDD is the runner-up

+0.46 average lift, modest cost

Generate tests first, then implementation, then run the tests before finalizing. Adds a test-generation step but pays for itself in caught errors and tightened scope.

Stop writing long guidelines

L1 averages −1.21 across models

The wall-of-text guidelines block is a net negative for most cloud models. Replace with a short principle list (3-5 lines max) plus per-task validation gates.

Multi-agent review needs context-equity

L5 averages −0.54

The reviewer agent sees the output but not the implementer's working context. Asymmetric context produces noisy reviews. Either give the reviewer equal context (expensive) or just use single-agent reflection.