Every LLM agent that operates real infrastructure eventually hits the same wall: the user asks for something no predefined tool covers. Most systems answer “I can’t do that.” In the Kubernetes framework I built and published in the Journal of Grid Computing, the agent does something else — it writes, tests, and registers a new tool at runtime, then uses it. On genuinely novel requests, that worked 81.8% of the time (63 of 77).

Here’s how it works, and — more importantly — how to do it without letting an LLM run arbitrary code against your cluster.

The problem with a fixed tool catalog

A typical agent is bound to a static set of tools: list pods, get logs, scale a deployment. That covers the common cases. But real operations have a long tail — “find every Job that failed more than twice in the last hour and show me which node it landed on” — and you cannot pre-write a tool for every one.

You have two options: give the agent one giant do-anything tool (a shell), which is a security disaster, or teach it to generate a narrow, purpose-built tool on demand and gate it. I went with the second.

The synthesis pipeline

When the supervisor decides no existing tool fits, it routes to a dedicated Code Generator Agent. The generated code goes through a pipeline before it’s ever allowed to run:

  1. Generate. The LLM writes a small, parameterized Python function — not a one-off script. Parameterization matters: a tool that takes namespace and threshold is reusable; a hardcoded one is throwaway.
  2. Static analysis (AST). The code is parsed and its abstract syntax tree is inspected. Anything that calls exec, eval, or open is rejected outright — before execution. This is the single most important guardrail: you block dangerous constructs structurally, not with a regex.
  3. Self-refine. If validation fails, the agent gets the error and revises — a bounded loop, not infinite.
  4. Sandboxed test. The candidate runs in a REPL with a 30-second timeout. If it throws or hangs, it never graduates.
  5. Checksum + persist. A passing tool is hashed (SHA-256) and written to persistent storage.
  6. Register. It’s recorded in a tool registry so the supervisor knows it exists.

Only after all six steps can the agent call it — and, because every mutating action in the system is behind a human-in-the-loop approval gate, a person still confirms anything that changes the cluster.

The part that compounds: reuse

The synthesized tool doesn’t disappear at the end of the session. It’s in the registry. The next time any session needs that capability, the supervisor routes straight to the existing tool and skips generation entirely.

That turns a one-time cost into a growing library. The agent effectively learns the shape of your cluster’s operational long tail over time — the tenth “weird Job query” is instant because the first one built the tool.

Why AST validation instead of a sandbox alone

A common instinct is “just run it in a container.” Containers help, but they’re a coarse boundary and they don’t stop the model from writing something that looks fine and quietly does damage within its permissions. Parsing the AST and refusing specific dangerous nodes gives you a cheap, deterministic, pre-execution check that composes with the sandbox rather than replacing it. Defense in depth: reject structurally, then test in isolation, then still require human approval to act.

What I’d tell anyone building this

  • Force parameterized functions, not scripts. Reuse is the whole payoff.
  • Validate the AST before you run anything. Block exec/eval/open at the tree level.
  • Timeout every generated execution. 30s was plenty for cluster queries; pick yours.
  • Persist and register. A synthesis system with no memory pays the cost every time.
  • Never let synthesis bypass your approval gate. Generation and execution are different privileges.

Runtime tool synthesis isn’t a party trick — it’s what let the system move from “answers the common 80%” to handling the operational long tail, with numbers that survived peer review.

The full architecture and evaluation are open and peer-reviewed — see the research page.