Fewer tools, better agent? What merging 18 tools into 15 did and did not buy

The folk wisdom says small models do better with fewer tools. We had eighteen, local models of eight and nine billion parameters that sometimes dithered between them, and a log of every tool call from 198 evaluation runs. So we looked at what the model actually called, merged what overlapped, and measured the result. It is a modest result and an honest one.

Constraints

Constraint	Consequence
The assistant has no tool that writes to the cluster	A merged tool cannot widen what any single read could reach
Raw describe output and logs are off unless an operator enables them	The merge must keep the gated and ungated paths separate
Secrets and ConfigMaps must never be readable	The refusal must hold whatever arguments are passed
Small models pay for every token of tool definition on every call	Descriptions and schemas count

What the models actually called

Across 198 runs and 725 tool calls from nine result sets:

Tool	Calls
describe_resource	157
list_pods	128
get_events	92
get_logs	88
cluster_health	38
get_resource	36
analyze_workload_resources	36
list_api_resources	30
list_resources	28
analyze_capacity	21
list_nodes	20
the rest	fewer than 20 each

The overlap is in the middle. describe_resource reads a built-in object; get_resource reads any object by API version and kind; list_api_resources finds which kinds exist; list_resources lists instances of a kind; list_nodes and cluster_health both answer questions about nodes. Of the 36 get_resource calls, 10 were genuine custom-resource reads, 17 asked for a Secret or ConfigMap (which are always refused), and 9 asked for built-in kinds (a Deployment, a Pod, a Service) that describe_resource already serves. The tool was mostly a place for models to make a mistake.

The merge

Before	After	Rule
describe_resource + get_resource	describe_resource with an optional apiVersion	No apiVersion: the gated describe path, as before. With one: the generic read, as before. Secret or ConfigMap: refused, whatever is passed
list_api_resources + list_resources	list_resources	No kind: discover API kinds, optionally filtered. kind plus apiVersion: list instances
list_nodes + cluster_health	cluster_health	Per-node rows appended, capped at 12 with a pointer to the pool-level tool

This shipped in Berth 0.5.0. The 15 tools are ten that read cluster state, two that read Berth’s own knowledge (runbooks and past incidents), one that formats a proposed fix, one that writes a note to Berth’s local memory, and the inert apply_remediation.

The design rule was delegation, not policy: the merged tool routes to exactly the code each old tool called, so the security posture cannot change by accident. Tests pin it: a built-in with no version reaches the describe path; a custom resource reaches the generic read; a Secret or ConfigMap request is sent to the refusing read path, with and without a version (the refusal itself lives in the Kubernetes layer); a custom kind passed without a version gets an error that says how to fix it. We also shortened every description and removed cross-references that the system prompt now carries.

We kept one tool we could have removed. apply_remediation does nothing: it returns a message that applying changes is not enabled, and nothing reaches the cluster. It is not an overlapping tool, and in our evaluation a call to it counts as a failure, because it means the model obeyed an instruction it should have ignored. Removing it would have deleted that tripwire.

The measurement

qwen3.5:9b-mlx, thinking on, three runs per scenario on 17 scripted scenarios against a fixture cluster (synthetic data, not a live one), same engine, same machine and the same criteria on both sides (the checks that name tools were renamed to the new tool names, with no change in leniency). Before: the old 18-tool set (27 runs on the original nine scenarios plus 24 on the newer eight). After: 15 tools, all 17 scenarios, 51 runs.

	Before (18 tools)	After (15 tools)	Reading
Passed	45/51	44/51	No change (p = 1.0)
Input tokens per model call	4,493	3,843	-14%, from the smaller definitions (not a sampling effect)
Input tokens per run	24.2k	19.8k	-18%, permutation p = 0.08
Tool calls per run	5.3	4.4	p = 0.21
Mean latency	39.5 s	35.6 s	-10%, p = 0.25
Runs that used all 8 iterations	14	13	No real change

The definitions went from 11,613 bytes to 8,579 (-26%); the system prompt plus tools from 17,112 to 14,151 bytes (-17%).

What we can say: the prompt is smaller and the pass rate did not move. The reductions in tool calls, tokens per run and latency point the right way, but at 51 runs per arm they are not distinguishable from noise. Some scenarios dropped a lot (a drain blocked by a PodDisruptionBudget went from 10 tool calls to 6; an availability question from 3.3 to 1.3); we do not know how much of that is the merge and how much is sampling. We adjudicated the failures only in the after arm, so the strict scores above are the like-for-like comparison.

What did not work

• The mistakes we merged away barely occurred in the run we measured. Of the 17 refused reads, 13 came from the older qwen3 models; the 9 misrouted calls were split 3 from qwen3 and 6 from qwen3.5, across several earlier result sets. In the qwen3.5 runs we used for the before-and-after comparison they were rare (2 refused reads and 1 misrouted call in 51 runs), so this comparison could not show a behaviour fix. The merge may be worth more for the older models, which we did not re-run.
• Over-exploration is untouched. About a quarter of runs still use the whole eight-iteration loop. Fewer tools to choose from did not stop the model wandering.
• Seven failures remain after the change. On a person’s reading, four are false fails of our scoring and three are real (two failed-rollout answers (bad image tag) that never say the old pods still serve, one node-removal answer that does not list the checks still needed).

Where this fails

We changed descriptions and tools together, so we cannot say which helped. We measured one model on one machine. A merged tool with an optional argument is a small ambiguity: a model can pass an API version for a built-in kind and take the generic path. It returns the projected spec-and-status read that get_resource already gave, not the gated describe, which is as before, but the tool’s behaviour now depends on a field the model may or may not set.

What we would do again

Count your tool calls before you redesign your tools. Merge by delegation so the security boundary is the thing that cannot move. Pin the boundary with tests before you refactor it. And keep a tripwire.

The method is in how we test an AI SRE. The agent is Berth’s assistant; Berth is a restricted beta.