When the agent runs out of tool calls: two small fixes that mattered

Every production agent loop has a cap on model round-trips. What happens at the cap is rarely tested, because well-behaved runs never reach it. Ours reached it more often than we expected, and what the user saw was not an error. It was a sentence of narration presented as the answer.

Constraints

Berth’s assistant runs a bounded loop: the model may call tools for up to eight round-trips, then must answer. It runs on a local Ollama model or a cloud model, through two different loops (a hand-written one for Ollama, the provider SDK’s tool runner for Claude). Runs must finish inside a time limit, and a failed run should say so rather than return nothing.

What we saw

In an evaluation of the assistant on a local model with thinking on, 2 of 24 runs ended like this:

The previous log doesn’t work since the pod crashed. Let me try getting the latest logs instead.

That’s odd - list_pods showed these pods exist but describe_resource can’t find them. This might be a limitation of the tool or the snapshot has partial data. […] Let me try a simpler approach - just get the pod descriptions using the exact names from list_pods output.

Both had used all eight tool iterations and then the forced final call. The Ollama loop already handled the cap: after the last tool round it made one more call with no tools advertised, so the model had to answer. But the model, mid-investigation, did not read “no tools” as “stop investigating”. It narrated its next step, and that narration was returned to the user as the answer.

Separately, 2 of 27 runs in an earlier arm and 2 of 24 here ended in an HTTP 500 from Ollama: the model had emitted a tool call whose XML Ollama could not parse (XML syntax error ... </function>). The engine treated any non-200 as fatal.

The fixes

1. Tell the model it is out of tools. The forced final call now ends with a user message:

No more tool calls are available. Answer the user's question now, using only the
evidence already gathered above. Lead with the outcome, and say plainly what you
could not check or confirm. Do not describe what you would do next.

It is added only on the forced call. Runs that finish inside the cap never see it.

2. Retry a server error once. A 500 from malformed tool-call output is a sampling accident: another sample usually parses. The engine now retries a 5xx once after a short pause, not 4xx, not connection failures, and it stops waiting if the run is cancelled. Crucially it counts retries and reports them with the run, so a model that fails repeatedly cannot hide behind the retry. A second failure returns the error as before.

3. On the cloud path, the same cap hid a different bug. From reading the provider SDK’s tool runner: when the iteration limit is reached it stops without executing the tool calls the last response asked for, and returns that last message. If that message is a tool request with a line of preamble, the user gets the preamble as the answer, with no error. We now make the last permitted call with tool use forbidden, mark an answer that was cut off at the output limit, and turn an empty answer, or a run that ends still asking for tools, into an error. We tested this against a fake API server, including a model that keeps asking for tools after tool use is forbidden. We have not yet run it against the live Bedrock endpoint.

Does it work?

We re-ran the same 24 runs (8 scenarios, three repeats, thinking on, same machine). Both sides are scored with the revised criteria. The “before” run scored 18/24 under the original criteria, and the revision was made after we had seen those results, so treat the 19 as slightly generous:

	Before	After
Passed	19/24	22/24
Ollama 500 errors	2	0
Fragment answers	2	0
Runs that used all 8 iterations	5	8
Mean latency	36.5 s	40.3 s

Read it carefully:

• The mechanism is verified. All 8 runs that reached the forced final call gave a real answer and passed. One 500 was retried, and that run then passed.
• The pass-rate change is not significant (19 to 22 of 24, Fisher p = 0.42), so we claim the mechanism, not the number.
• The fix does not stop the flailing. More runs now use the whole loop (8 against 5), because a run that would have ended in an error or a fragment now ends in an answer. Latency did not improve.
• One scenario got worse. rightsize-overprovisioned went from 3/3 to 1/3. Both failures are answers that recommend lowering requests in wording our check does not match. We left the check alone so the before and after scores stay comparable.

Where this still fails

• The model’s over-exploration is the real problem. Tasks that need a handful of tool calls took 10 to 14 on two scenarios (a blocked drain, with a budget of 4, and a poisoned-memory case, with a budget of 6). The cap is now safe, but we are still paying for the wander.
• A forced answer after a failed investigation is an answer from partial evidence. The instruction asks the model to say what it could not check; we have not measured how well it does.
• One retry cannot rescue a model that emits malformed calls every time.
• The cloud-path fix is covered by tests against a fake server, not by a live run.

The transferable part

If you build an agent loop: decide what the user gets when the cap is hit and test it on purpose, by setting the cap to 1 or 2 in a test. Do not return the last message unless you have checked it is an answer. Say “answer now” in words, because “no tools” is a constraint the model may not read as an instruction. Retry transient server errors, but count the retries.

The evaluation these runs came from is described in how we test an AI SRE, and the corrections to its scoring in the eval scoring we got wrong. The assistant is part of Berth, currently a restricted beta.