---
title: "Berth AI SRE evaluation: open scenarios, results and claims ledger | CloudGuys Blog"
description: "What we measured about Berth's AI SRE, what we did not, and how to read it: 17 synthetic scenarios, repeated runs with intervals, criteria revisions disclosed, and a claims ledger."
url: "https://cloudguys.io/blog/berth-ai-sre-evaluation-open-data"
language: "en"
image: "https://cloudguys.io/_astro/cover-cloudguys.DBSgBsxx.jpg"
published: "2026-10-07T00:00:00.000Z"
modified: "2026-10-07T00:00:00.000Z"
---

ENGINEERING NOTES / BERTH

# We are publishing our AI SRE evaluations: the scenarios, the results, and the mistakes

Berth's AI SRE eval data is public: 17 synthetic Kubernetes scenarios, repeated local-model runs, the criteria we changed and why, and a claims ledger. The harness stays private; here is what that means.

Sabbir · · · October 7, 2026 · · · 7 min · · · About Berth

![We are publishing our AI SRE evaluations: the scenarios, the results, and the mistakes](https://cloudguys.io/_astro/cover-cloudguys.DBSgBsxx_Z2dsg7F.webp)

Until this month, the only evidence we could point to for Berth’s AI SRE was “5 of 5 clean answers” for one local model, and that is not evidence. So we built a scored evaluation, ran it repeatedly on local models, changed the product because of what it showed, and found problems in our own scoring along the way. This post releases the material and explains how to read it.

## What is published

In the public repository under `evals/`:

-   **17 scenarios** as JSON: a fake cluster, a user question and explicit pass criteria, covering diagnosis, safety, coverage, availability, capacity, custom-resource discovery, multi-turn follow-up, a large cluster and a control with nothing wrong.
-   **Results** for every run set: the recorded tool calls, token counts, latency, the final answer and every check, plus the unmodified original files next to the rescored ones.
-   **An adjudication log**: for each run we quote, what the strict scorer said, what a person reading the answer said, and why they differ.
-   **`METHODOLOGY.md`** and **`CLAIMS.md`**, which maps every public number to a run directory.

The data is licensed CC BY 4.0.

## What is not published, and why that matters

**The harness that runs the scenarios is private.** It imports internal packages of the Berth application, so it cannot run outside our repository as it stands, and the application source is not public. That means you can audit every run and criterion, but you cannot re-run the evaluation yourself yet. We would rather say that plainly than imply a reproducibility we do not have. Two things would change it: a standalone runner that scores any Berth instance through its chat API (which needs a fixture-cluster mode in the product), or opening the harness. Neither is decided.

We are also keeping a **held-out set** of scenarios private. We tune the prompt and tools against the public scenarios, so over time public scores will drift upward for reasons that have nothing to do with real quality. Holding some back is how we notice.

## How to read it

“Pass rate” here means a run passed every critical check on a synthetic scenario. It is not diagnostic accuracy on real incidents. Every figure carries a run count and a 95% interval. With three runs per scenario, the intervals are wide, and most differences between arms are inside them.

| Local model (original 9 scenarios, 3 runs each) | Passed | Mean latency |
| --- | --- | --- |
| `qwen3:8b` | 19/27 (52 to 84%) | 58 s |
| `berth-sre` (our tuned `qwen3:8b`) | 18/27 (48 to 81%) | 74 s |
| `qwen3.5:9b-mlx`, thinking on | 21/27 (59 to 89%) | 34 s |
| `qwen3.5:9b-mlx`, thinking off | 19/27 (52 to 84%) | 14 s |

Apple M4 Pro, 24 GiB, Ollama 0.35.1, 16k context.

What the data supports: `qwen3.5` was 2 to 4 times faster on this machine; pass rates do not separate the four arms; our tuned model did not beat the stock one it is built on. What it suggests but does not prove: the `qwen3` models followed injected instructions more often than `qwen3.5` (6 of 54 runs against 0 of 54), a result with caveats we cover in a follow-up post.

## Our own mistakes, in the open

We wrote the pass criteria before running any model, then changed some of them after reading answers. We did it twice. In total, three regexes were widened after the first run and four changes were made after the second, which flipped four stored runs from fail to pass and none the other way. We kept the original result files, published the rescored ones beside them, and wrote down each change and why. We also keep finding false failures in our own scorer: a regex that matched the word “crashloop” in an answer saying there were none, and a “password” match on an answer that said the password was redacted. That is why the adjudication log exists.

Each correction, with the reason for it, is written up in [the methodology file](https://github.com/unishsys/berth/blob/master/evals/METHODOLOGY.md).

## What we have not measured

-   Claude or Bedrock models. We have not run a cloud matrix, so we say nothing about cloud quality or cost per solved scenario.
-   Real clusters, large messy clusters beyond one 606-pod scenario, mean time to resolution.
-   Resistance to prompt injection in any general sense. We measured named models on named synthetic scenarios.
-   Any ranking of third-party tools.

## Using it

If you run local models for Kubernetes work, the scenarios are a free starting point for your own checks. If you find a scenario that is wrong or a criterion that is unfair, tell us: we will publish the change and the reason.

[Read the evaluation page](https://berth.cloudguys.io/evals/?utm_source=cloudguys_blog&utm_medium=referral&utm_campaign=berth-ai-sre-evaluation-open-data), or [get a free Community key](https://berth.cloudguys.io/?utm_source=cloudguys_blog&utm_medium=referral&utm_campaign=berth-ai-sre-evaluation-open-data) to try Berth on a cluster.

#Berth · #AI SRE · #Evaluation · #Kubernetes

Share this article

-   [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcloudguys.io%2Fblog%2Fberth-ai-sre-evaluation-open-data%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial "Share on LinkedIn")
-   [X](https://twitter.com/intent/tweet?text=We%20are%20publishing%20our%20AI%20SRE%20evaluations%3A%20the%20scenarios%2C%20the%20results%2C%20and%20the%20mistakes&url=https%3A%2F%2Fcloudguys.io%2Fblog%2Fberth-ai-sre-evaluation-open-data%3Futm_source%3Dx%26utm_medium%3Dsocial "Share on X")
-   [Reddit](https://www.reddit.com/submit?url=https%3A%2F%2Fcloudguys.io%2Fblog%2Fberth-ai-sre-evaluation-open-data%3Futm_source%3Dreddit%26utm_medium%3Dsocial&title=We%20are%20publishing%20our%20AI%20SRE%20evaluations%3A%20the%20scenarios%2C%20the%20results%2C%20and%20the%20mistakes "Share on Reddit")
-   [Hacker News](https://news.ycombinator.com/submitlink?u=https%3A%2F%2Fcloudguys.io%2Fblog%2Fberth-ai-sre-evaluation-open-data%3Futm_source%3Dhackernews%26utm_medium%3Dsocial&t=We%20are%20publishing%20our%20AI%20SRE%20evaluations%3A%20the%20scenarios%2C%20the%20results%2C%20and%20the%20mistakes "Share on Hacker News")
-   [Email](mailto:?subject=We%20are%20publishing%20our%20AI%20SRE%20evaluations%3A%20the%20scenarios%2C%20the%20results%2C%20and%20the%20mistakes&body=We%20are%20publishing%20our%20AI%20SRE%20evaluations%3A%20the%20scenarios%2C%20the%20results%2C%20and%20the%20mistakes%0A%0Ahttps%3A%2F%2Fcloudguys.io%2Fblog%2Fberth-ai-sre-evaluation-open-data%3Futm_source%3Demail%26utm_medium%3Dsocial "Share on Email")

SEE IT ON YOUR OWN CLUSTER

## Berth: a Kubernetes dashboard  
you run yourself.

Endpoint health, capacity planning, and a read-only AI SRE. Free for small clusters, license checked offline. Discuss your Kubernetes security and architecture with CloudGuys.

[Get a free Community key ↗](https://berth.cloudguys.io/?utm_source=cloudguys_blog&utm_medium=referral&utm_campaign=berth-ai-sre-evaluation-open-data) [Talk to CloudGuys ↗](https://cloudguys.io/#architecture-review)

## Structured data

```json
[
  {
    "@context": "https://schema.org",
    "@type": [
      "ProfessionalService",
      "Organization"
    ],
    "@id": "https://cloudguys.io/#organization",
    "name": "CloudGuys",
    "url": "https://cloudguys.io/",
    "logo": {
      "@type": "ImageObject",
      "url": "https://cloudguys.io/assets/logo.png",
      "width": 512,
      "height": 512
    },
    "image": "https://cloudguys.io/assets/og-image.png",
    "description": "Security assessments, cloud architecture reviews, and practical remediation across AWS, Google Cloud, Azure, Kubernetes, and infrastructure as code.",
    "email": "hello@cloudguys.io",
    "slogan": "Know your risks. Build a stronger cloud.",
    "areaServed": "Worldwide",
    "hasOfferCatalog": {
      "@type": "OfferCatalog",
      "name": "CloudGuys engagements",
      "itemListElement": [
        {
          "@type": "Offer",
          "itemOffered": {
            "@type": "Service",
            "name": "Cloud security assessment",
            "url": "https://cloudguys.io/services/cloud-security-assessment",
            "provider": {
              "@id": "https://cloudguys.io/#organization"
            }
          }
        },
        {
          "@type": "Offer",
          "itemOffered": {
            "@type": "Service",
            "name": "Cloud architecture review",
            "url": "https://cloudguys.io/services/cloud-architecture-review",
            "provider": {
              "@id": "https://cloudguys.io/#organization"
            }
          }
        },
        {
          "@type": "Offer",
          "itemOffered": {
            "@type": "Service",
            "name": "Security remediation",
            "url": "https://cloudguys.io/services/security-remediation",
            "provider": {
              "@id": "https://cloudguys.io/#organization"
            }
          }
        }
      ]
    },
    "knowsAbout": [
      "DevOps",
      "Kubernetes",
      "Terraform",
      "Cloud migration",
      "Cloud cost optimization",
      "GitOps",
      "Site reliability engineering",
      "Kubernetes dashboard",
      "AI SRE",
      "Kubernetes troubleshooting",
      "AWS",
      "Google Cloud",
      "Azure"
    ],
    "contactPoint": {
      "@type": "ContactPoint",
      "contactType": "sales",
      "email": "hello@cloudguys.io",
      "availableLanguage": [
        "English"
      ],
      "areaServed": "Worldwide"
    },
    "sameAs": [
      "https://www.linkedin.com/company/agrohitech",
      "https://berth.cloudguys.io",
      "https://tele.cloudguys.io"
    ],
    "owns": [
      {
        "@type": "SoftwareApplication",
        "@id": "https://berth.cloudguys.io/#software",
        "name": "Berth",
        "applicationCategory": "DeveloperApplication",
        "operatingSystem": "Kubernetes",
        "url": "https://berth.cloudguys.io",
        "description": "Berth is a self-hosted Kubernetes dashboard with a read-only AI SRE: endpoint health, capacity planning, guided app exposure, and evidence-based troubleshooting, run in your own cluster."
      },
      {
        "@type": "SoftwareApplication",
        "@id": "https://tele.cloudguys.io/#software",
        "name": "Tele",
        "applicationCategory": "CommunicationApplication",
        "operatingSystem": "Web",
        "url": "https://tele.cloudguys.io",
        "description": "Tele is real-time audio and video call infrastructure: a meeting API for embedding calls in your own application and a managed workspace for teams, deployable in your own cloud or private environment."
      }
    ]
  },
  {
    "@context": "https://schema.org",
    "@type": "WebSite",
    "@id": "https://cloudguys.io/#website",
    "url": "https://cloudguys.io/",
    "name": "CloudGuys",
    "description": "Security and cloud consultancy helping organizations assess risks, improve architecture, and verify remediation.",
    "publisher": {
      "@id": "https://cloudguys.io/#organization"
    },
    "inLanguage": "en"
  },
  {
    "@context": "https://schema.org",
    "@type": "BlogPosting",
    "@id": "https://cloudguys.io/blog/berth-ai-sre-evaluation-open-data#article",
    "headline": "We are publishing our AI SRE evaluations: the scenarios, the results, and the mistakes",
    "description": "What we measured about Berth's AI SRE, what we did not, and how to read it: 17 synthetic scenarios, repeated runs with intervals, criteria revisions disclosed, and a claims ledger.",
    "image": "https://cloudguys.io/_astro/cover-cloudguys.DBSgBsxx.jpg",
    "datePublished": "2026-10-07T00:00:00.000Z",
    "dateModified": "2026-10-07T00:00:00.000Z",
    "author": {
      "@type": "Organization",
      "name": "Sabbir",
      "url": "https://cloudguys.io/"
    },
    "publisher": {
      "@id": "https://cloudguys.io/#organization"
    },
    "mainEntityOfPage": {
      "@type": "WebPage",
      "@id": "https://cloudguys.io/blog/berth-ai-sre-evaluation-open-data"
    },
    "articleSection": "Berth",
    "keywords": "Berth, AI SRE, Evaluation, Kubernetes",
    "inLanguage": "en",
    "about": {
      "@type": "SoftwareApplication",
      "@id": "https://berth.cloudguys.io/#software",
      "name": "Berth",
      "url": "https://berth.cloudguys.io"
    }
  },
  {
    "@context": "https://schema.org",
    "@type": "BreadcrumbList",
    "itemListElement": [
      {
        "@type": "ListItem",
        "position": 1,
        "name": "Home",
        "item": "https://cloudguys.io/"
      },
      {
        "@type": "ListItem",
        "position": 2,
        "name": "Blog",
        "item": "https://cloudguys.io/blog"
      },
      {
        "@type": "ListItem",
        "position": 3,
        "name": "We are publishing our AI SRE evaluations: the scenarios, the results, and the mistakes",
        "item": "https://cloudguys.io/blog/berth-ai-sre-evaluation-open-data"
      }
    ]
  }
]
```