Stage 1 — Cloud Reachability and Local Command Generation (2026-02-03) · V1
Archive intake · Stage 1
- models in roster
- 29
- reported reached
- 28
- command prompts
- 60
- local models
- 6
About this intake. This observation was produced before the Observatory existed, in the research repository that preceded it. It is admitted here because the conditions and denominators were recorded at the time and can still be checked against the retained roster files. It is published as an archive intake: the measurements are reported as they were taken, and the limitations below are part of the record, not caveats added afterwards.
1. What was observed
Two distinct measurements were taken on 2026-02-02 and written up the following day.
- Cloud reachability and latency. Each model on the cloud roster was called and its response time recorded. The original write-up reports 28 of 28 calls returning, with a mean response time of 4.79 seconds.
- Local command generation. A fixed battery of 60 command-generation prompts was put to six locally served models. The highest-scoring model produced a correct command on 90% of prompts, with a mean score of 86 out of 100. Other models scored materially lower, and at least one returned close to zero correct commands on this task type.
A third measurement — a response cache, cold against warm, over 15 repeated queries — is reported below because its denominator was recorded. It measures the harness, not any model.
2. Cloud response latency by size band
The original report grouped the roster into four size bands and gave a mean response time for each. The bands are reproduced exactly as they were written, including the problem noted underneath.
- Small (3B-14B)
- 0.61
- Large (20B-120B)
- 1.31
- XLarge (235B-1T)
- 5.37
- Medium (4B-30B)
- 7.62
4 entries · ordered by seconds (lower is better) as observed in this run · not a ranking
The size bands overlap, and the result is not monotonic. A model of 12B parameters falls inside three of the four declared bands (3B-14B, 4B-30B, 20B-120B is the only one it misses). Because the bands are not disjoint, a given model's band assignment is not determined by the definitions as written, and the per-band means cannot be reproduced from the roster without the original assignment table, which was not retained. The band means in declared size order are 0.61s, 7.62s, 1.31s, 5.37s, which do not rise with declared size — the pattern an overlapping partition tends to produce. These four numbers are reported as recorded and should not be read as a relationship between model size and latency. They are published so the flaw is on the record rather than lost with the file.
3. Response cache, cold against warm
Fifteen repeated queries were run against a cold cache and then a warm one. This measures the harness around the models, not the models.
- Warm cache
- 10.8
- Cold cache
- 29.6
2 entries · ordered by seconds (lower is better) as observed in this run · not a ranking
A cache hit itself returned in roughly 0.4 to 0.5 milliseconds. The residual 10.8 seconds on a warm cache is therefore almost entirely work that was not cached, which is the more useful reading of this pair than the speedup figure.
4. The roster, and a discrepancy in the count
The cloud roster captured on 2026-02-03 lists 29 entries. The report states that 28 models were called. The two numbers were never reconciled at the time.
One entry accounts for the difference on inspection: every model on the roster carries a -cloud suffix except gemini-3-pro-preview:latest, which is tagged in the local style. The likeliest explanation is that the run selected on that suffix and skipped the odd entry. That is an inference from the roster file, not something the record states, and it is reported as an inference. The consequence for the headline is small but real: the reported success rate of 28 of 28 describes the models that were called, not the roster as captured.
5. What this intake deliberately leaves out
The source report is a five-part pack. Most of it is engineering work on the harness rather than observation of models, and is not part of this record:
- Test coverage figures (reported between 10% and 14%), test-suite stability, and CI next steps. These describe the observing software, not anything observed.
- Security findings and hardening proposals for the harness.
- Refactoring and scalability proposals.
- Multi-agent audit summaries — five teams and fifteen agents reviewing the repository. Internal review of the tooling.
The distinction applied throughout: a measurement of a model under declared conditions is an observation; a measurement of our own code is not.
6. Limitations of this record
These are properties of the archived evidence and cannot be repaired after the fact. They bound what the numbers above can support.
- Model identities were stripped from the summary. The source pack was sanitized for external distribution, and its redaction note states that all model identifiers were removed. The roster files were retained separately, so the set of models is known; the mapping from an individual result to an individual model is not. This is why no model is named as the 90% performer. That figure is real; the attribution is not recoverable.
- Raw responses were not retained. Only aggregates survive. Nothing in this record can be re-scored, and no output can be re-read.
- Single run, no repetition. Each figure comes from one pass on one day. No variance is available, so none is claimed.
- Scoring method for the command battery is not recorded. A mean score of 86 out of 100 is reported alongside a 90% correct rate, but the rubric that produced the 100-point score was not retained.
- Conditions were not held constant across the two measurements. Cloud models were reached over a network; local models ran on the observing host. Latency figures from the two are not comparable.
What this observation supports. That on 2026-02-02, under the conditions recorded, a 29-model cloud roster was reachable and returned within single-digit seconds on average, and that six locally served models varied widely on a fixed 60-prompt command-generation battery — from 90% correct down to near zero. What it does not support: any relationship between model size and latency, any attribution of a score to a named model, and any claim about which model is preferable for any purpose.