The LLM-as-a-Judge literature · V1
Research synthesis
A literature synthesis records the judging ideas behind Sigma5C’s observation design, including bias controls and role-weighted panels. Judge reliability remains an observation-specific question.
- Prior systems
- 8
Judging design
The recorded design combines generalist and specialist judges, with category weights and an additional judge for close consensus scores.
The synthesis describes randomized presentation order, response-length normalization, thinking-tag removal and family self-judging exclusions as bias controls.
Literature contributions
| System | Recorded contribution |
|---|---|
| MT-Bench | Multi-turn judging |
| G-Eval | Structured scoring criteria |
| Length-Controlled AlpacaEval | Length-bias controls |
| Arena-Hard | Difficult prompt selection |
| LLMBar | Adversarial judge tests |
| JudgeLM | Fine-tuned judges |
| Prometheus 2 | Open-weight judges |
| Sage | Long-form judging |
Current judging method
Design scope
These are recorded design choices, rather than measured guarantees of judge agreement or correctness. Observation-specific protocols and judge disagreement remain necessary context.
Recorded thresholds
- Median divergence
- 3.0
- Close consensus delta
- 1.5
The design excludes a judge vote above the median-divergence threshold and invokes an additional judge within the close-consensus threshold. Bar lengths are relative to the larger threshold; these are design settings, rather than measured model scores.