Fixing the AI SRE tool loop in Berth 0.5.0: 18 tools to 15, forced final answers, and what the numbers showed

Berth 0.5.0 is a release about one thing: what the AI SRE does when it runs out of tool calls, and how many tools it has to choose between. We found both problems by running the assistant repeatedly against synthetic clusters and reading what it did, then changed the code and measured again. This post walks through the mechanisms and the numbers. It is still a restricted beta for trusted cluster administrators on a single cluster; there is no SSO, per-user authorisation, audit viewer or multi-cluster support, and the assistant has no tool that writes to your cluster.

The full evidence, with its limits, is in our evaluation post. The list of changes is in the changelog.

The problem: an agent loop with a cap has to end somewhere

Berth’s assistant runs a tool loop: the model asks for a tool, Berth runs it against the read-only cluster provider, the result goes back, and the model either asks again or answers. The loop is capped at eight model calls. A cap is necessary (a confused model can otherwise call tools until your budget is gone), but what happens at the cap decides whether the user gets an answer.

We found two different failures, one per engine.

Claude (Anthropic API and Bedrock): the cap could end the run with no answer

The Claude engine uses the SDK’s tool runner. The runner stops after the last permitted model call, and if that call was itself asking for a tool, nobody answers it. The run ends with an empty or preamble-only reply and reports success.

The fix is to take the choice away on the last call. When the second-to-last call finishes, Berth sets tool_choice to none, so the final call has to answer from what is already in the conversation:

forbidTools := func() {
    none := anthropic.NewBetaToolChoiceNoneParam()
    runner.Params.ToolChoice = anthropic.BetaToolChoiceUnionParam{OfNone: &none}
}
if maxIter == 1 {
    forbidTools()
}
for msg, err := range runner.All(ctx) {
    // ... accumulate usage ...
    usage.Iterations++
    if usage.Iterations == maxIter-1 {
        forbidTools() // takes effect on the next (final) call
    }
}

Three further cases are handled on the way out, because returning an empty string as a success is the worst outcome for an operator who is trusting the answer:

Final stop reason	Before	Now
max_tokens or context window exceeded	Truncated text returned as if complete	Text returned with a visible “cut off” notice
tool_use (the model still wanted tools)	Empty or partial reply, reported as success	An error that says the run ended while the model was still calling tools
refusal	Handled	Unchanged

The per-reply output cap defaults to 8192 tokens when no limit is configured.

Ollama: the forced last call produced narration, not an answer

The Ollama engine is a hand-written loop, and it already made a last call with no tools when the cap was reached. The problem was what the model did with it. With nothing else in the prompt, a model in the middle of an investigation keeps narrating its next step. In our runs the “answer” was sometimes a fragment such as “Let me try getting the latest logs instead.”

The fix is a plain instruction appended as a user message before that last call:

const finalAnswerNudge = "No more tool calls are available. Answer the user's question now, " +
    "using only the evidence already gathered above. Lead with the outcome, and say plainly " +
    "what you could not check or confirm. Do not describe what you would do next."

The same release adds one retry for a 5xx from Ollama. Ollama returns HTTP 500 when the model emits a tool call it cannot parse (for example malformed XML in the call). That is transient far more often than not, because the next sample usually parses. The retry waits 300 ms and happens once. It applies only to a 5xx status from the server: connection failures and 4xx replies are not retried, and the number of retries used is reported in the run’s usage so you can see how often a model needs it.

The problem: overlapping tools

The previous toolset had 18 tools. Three overlapped with others:

• get_resource read any object through a generic path; describe_resource read built-in kinds through a richer path.
• list_api_resources discovered kinds; list_resources listed instances.
• list_nodes listed nodes; cluster_health summarised the cluster.

A smaller model can spend iterations choosing between near-duplicates, and every model call pays for all the tool definitions again. To see whether the overlap mattered in practice, we counted how get_resource was actually used across our earlier runs. There were 36 calls:

What the call asked for	Calls
A genuine custom-resource read (the tool’s real job)	10
A Secret or ConfigMap (Berth refuses these whatever the arguments)	17
A built-in kind that describe_resource already serves	9

Most of those mistakes came from the older qwen3 models, so this data motivated the merge but cannot show that the merge fixed a behaviour in qwen3.5.

How the merged tools route

Before	After
get_resource	describe_resource with an optional apiVersion
list_api_resources, list_resources	list_resources: no kind discovers API kinds; kind plus apiVersion lists instances
list_nodes	cluster_health includes a row per node, capped at 12 (analyze_capacity groups larger clusters by pool)

describe_resource takes kind, namespace, name and the optional apiVersion, and routes in a fixed order. The security-relevant case comes first:

switch {
case strings.EqualFold(in.Kind, "Secret") || strings.EqualFold(in.Kind, "ConfigMap"):
    // Same refusal the generic read gives, regardless of apiVersion.
    txt, err = d.Provider.GetResource("v1", in.Kind, in.Namespace, in.Name)
case strings.TrimSpace(in.APIVersion) != "":
    txt, err = d.Provider.GetResource(in.APIVersion, in.Kind, in.Namespace, in.Name)
default:
    txt, err = d.Provider.Describe(in.Kind, in.Namespace, in.Name)
    if err != nil && !builtinKinds[strings.ToLower(strings.TrimSpace(in.Kind))] {
        err = fmt.Errorf("%w; %q is not a built-in kind: for a custom resource pass its apiVersion "+
            "(find it with list_resources)", err, in.Kind)
    }
}

Two details matter for the security boundary and for small models:

• A Secret or ConfigMap is refused whatever apiVersion is passed, so adding the parameter did not open a path around the refusal. Without an apiVersion a read still goes to the describe path that is off by default and enabled only by AI_ALLOW_RAW_DIAGNOSTICS. We pinned these routes with tests so a later refactor cannot loosen them.
• The error message teaches the model the next step. When a read of a non-built-in kind fails, the error says to pass an apiVersion and how to find it. list_resources does the same: asking for a kind with no apiVersion returns an error that says to call it first with a filter and no kind.

The discover-then-read flow for a custom resource is therefore three calls. This is the path our crd-operator-discovery scenario scores for the question “Is the ledger database healthy? I don’t remember which operator manages it”:

list_resources(filter: "postgres")
  -> kinds served by the cluster: Backup, Cluster and Pooler, all postgresql.cnpg.io/v1
list_resources(kind: "Cluster", apiVersion: "postgresql.cnpg.io/v1", namespace: "data")
  -> ledger-db [instances ready 2/3, Waiting for the instances to become active]
describe_resource(kind: "Cluster", apiVersion: "postgresql.cnpg.io/v1", namespace: "data", name: "ledger-db")
  -> spec and status as YAML: readyInstances 2, with ledger-db-3 listed as failed

The 15 tools are: ten that read cluster state (describe_resource, list_pods, get_events, get_logs, analyze_workload_resources, cluster_health, analyze_capacity, check_availability, check_disruption_budgets, list_resources), two that read Berth’s own knowledge (lookup_runbook, recall_incidents), one that formats a proposed fix (propose_remediation), one that writes a note to Berth’s local memory (record_incident), and one inert stub (apply_remediation). We kept the stub on purpose: it is not an overlapping tool, and our tests use calls to it as a signal that a model obeyed an instruction it should have ignored.

A pricing bug that came from substring matching

The usage-cost estimate looks up a model’s list price by substring of the model id, so that Bedrock’s anthropic. and us. prefixes and date suffixes still match. The first match wins. The id claude-opus-5-5 contains claude-opus-5, so until this release it matched the older, more expensive row and the Opus 5.5 estimate came out 25% too high. The fix is ordering: a more specific id must sit before the id it extends, and a test pins that Opus 5.5 is priced at its own rate. Cache reads are billed at a fraction of the input price that differs by model (10% by default, 5% for Opus 5.5, 2.5% for Fable 5.1), so each row now carries its own multiplier. Everything derived from the table is shown as an estimate; the provider’s invoice is the truth.

Did it help? What we measured

Setup: the 17 synthetic scenarios, qwen3.5:9b-mlx through Ollama 0.40.0 with thinking on, three runs per scenario, identical pass criteria before and after.

Prompt size (certain, because it is a property of the definitions):

	Before	After	Change
Tool definitions (bytes)	11,613	8,579	-26%
System prompt plus tools (bytes)	17,112	14,151	-17%
Input tokens per model call	4,493	3,843	-14%

Outcomes (small samples, read with care):

	Before	After	p
Runs passing every critical check	45 of 51	44 of 51	1.0
Input tokens per run	24.2k	19.8k	0.08
Tool calls per run	5.3	4.4	0.21
Runs that used all 8 iterations	14	13	n/a

The merge made the prompt smaller and did not change the pass rate. The per-run token and tool-call reductions point the right way but are not distinguishable from noise at this sample size. Roughly a quarter of runs still use the whole eight-iteration loop, so merging tools did not cure over-exploration, and that remains the largest open problem.

The loop fixes are measured on the eight scenarios added later, thinking on, before and after:

	Before	After
Answers that were a “Let me try…” fragment	2 of 24	0 of 24
Runs that died on an Ollama 500	2 of 24	0 of 24
Runs that reached the forced final call and answered	n/a	8 of 8
Passing every critical check	19 of 24	22 of 24 (p = 0.42)

One retry recovered from a 500 and that run then passed. The pass-rate change is not significant, so we claim the mechanism (fragments and 500s are gone), not the number.

What did not change

• Restricted beta, single cluster, Community (free key, up to 10 nodes) and Enterprise (cloud AI under your own key) tiers as before.
• Read-only assistant. No tool writes to the cluster. record_incident writes a note to Berth’s own local memory, and apply_remediation does nothing.
• Where text goes. With Ollama, requests go to the server you run. With Claude, the text the tools return for that turn goes to your provider account. Budgets are admission controls, not a billing cap.
• The Bedrock default model and the EFFORT default are unchanged.
• The Anthropic API default model is now claude-sonnet-5-5 instead of claude-opus-5. A model saved in Settings or pinned in your values is kept.

Upgrading, and one compatibility note

Existing installs keep their saved AI settings, including a saved model. Only an unset model picks up the new default.

helm upgrade --install berth oci://ghcr.io/unishsys/charts/berth \
  --version 0.5.0 --namespace berth --create-namespace --reuse-values

If anything of yours reads the tool events streamed by the chat endpoint and matches on tool names, update it: get_resource is now describe_resource, list_api_resources is now list_resources (called with no kind), and list_nodes is now cluster_health.

What we still do not know

The evals are small, synthetic and run on one Apple silicon machine. They say nothing about real clusters, time to resolution, or cloud models (we have not run a cloud matrix yet). We also know of a flaw in our own scenarios: the canned discovery output in crd-operator-discovery still tells the model to “use get_resource”, a tool that no longer exists after the merge, so that scenario’s after-merge runs saw a misleading hint. We are fixing it and will publish the corrected scenario with a note rather than edit the published one silently. Follow-up posts in this series describe the model results and where they stop. If you try 0.5.0, check the assistant’s answers against the tool calls it shows you.

Get a free Community key and point it at a cluster, or read the v0.5.0 release.