We are publishing our AI SRE evaluations: the scenarios, the results, and the mistakes

Until this month, the only evidence we could point to for Berth’s AI SRE was “5 of 5 clean answers” for one local model, and that is not evidence. So we built a scored evaluation, ran it repeatedly on local models, changed the product because of what it showed, and found problems in our own scoring along the way. This post releases the material and explains how to read it.

What is published

In the public repository under evals/:

• 17 scenarios as JSON: a fake cluster, a user question and explicit pass criteria, covering diagnosis, safety, coverage, availability, capacity, custom-resource discovery, multi-turn follow-up, a large cluster and a control with nothing wrong.
• Results for every run set: the recorded tool calls, token counts, latency, the final answer and every check, plus the unmodified original files next to the rescored ones.
• An adjudication log: for each run we quote, what the strict scorer said, what a person reading the answer said, and why they differ.
• METHODOLOGY.md and CLAIMS.md, which maps every public number to a run directory.

The data is licensed CC BY 4.0.

What is not published, and why that matters

The harness that runs the scenarios is private. It imports internal packages of the Berth application, so it cannot run outside our repository as it stands, and the application source is not public. That means you can audit every run and criterion, but you cannot re-run the evaluation yourself yet. We would rather say that plainly than imply a reproducibility we do not have. Two things would change it: a standalone runner that scores any Berth instance through its chat API (which needs a fixture-cluster mode in the product), or opening the harness. Neither is decided.

We are also keeping a held-out set of scenarios private. We tune the prompt and tools against the public scenarios, so over time public scores will drift upward for reasons that have nothing to do with real quality. Holding some back is how we notice.

How to read it

“Pass rate” here means a run passed every critical check on a synthetic scenario. It is not diagnostic accuracy on real incidents. Every figure carries a run count and a 95% interval. With three runs per scenario, the intervals are wide, and most differences between arms are inside them.

Local model (original 9 scenarios, 3 runs each)	Passed	Mean latency
qwen3:8b	19/27 (52 to 84%)	58 s
berth-sre (our tuned qwen3:8b)	18/27 (48 to 81%)	74 s
qwen3.5:9b-mlx, thinking on	21/27 (59 to 89%)	34 s
qwen3.5:9b-mlx, thinking off	19/27 (52 to 84%)	14 s

Apple M4 Pro, 24 GiB, Ollama 0.35.1, 16k context.

What the data supports: qwen3.5 was 2 to 4 times faster on this machine; pass rates do not separate the four arms; our tuned model did not beat the stock one it is built on. What it suggests but does not prove: the qwen3 models followed injected instructions more often than qwen3.5 (6 of 54 runs against 0 of 54), a result with caveats we cover in a follow-up post.

Our own mistakes, in the open

We wrote the pass criteria before running any model, then changed some of them after reading answers. We did it twice. In total, three regexes were widened after the first run and four changes were made after the second, which flipped four stored runs from fail to pass and none the other way. We kept the original result files, published the rescored ones beside them, and wrote down each change and why. We also keep finding false failures in our own scorer: a regex that matched the word “crashloop” in an answer saying there were none, and a “password” match on an answer that said the password was redacted. That is why the adjudication log exists.

Each correction, with the reason for it, is written up in the methodology file.

What we have not measured

• Claude or Bedrock models. We have not run a cloud matrix, so we say nothing about cloud quality or cost per solved scenario.
• Real clusters, large messy clusters beyond one 606-pod scenario, mean time to resolution.
• Resistance to prompt injection in any general sense. We measured named models on named synthetic scenarios.
• Any ranking of third-party tools.

Using it

If you run local models for Kubernetes work, the scenarios are a free starting point for your own checks. If you find a scenario that is wrong or a criterion that is unfair, tell us: we will publish the change and the reason.

Read the evaluation page, or get a free Community key to try Berth on a cluster.