Tool calling: when you need Jev and when LLM caching is enough

Picture a support agent on turn eleven of a ticket. The customer was double charged. The agent has read the order, checked the payment log, and now it proposes a call: issue_refund(order_id, amount: 4999, reason: "duplicate_charge").

The request comes back fast. The tool definitions, the system prompt, and most of the conversation were already in the provider’s prompt cache, so this turn costs a fraction of turn one. Good.

Then someone asks who approved the refund. The same model that proposed it. Its reasoning says “clear duplicate, safe to refund.” That sentence was cheap and fast. It is still a sentence.

We keep hearing a reasonable question from teams building agents: “We already cache everything, so the main LLM can handle tool selection and approval cheaply. Why add another model?” Our short answer:

Prompt caching lowers the cost and latency of re-reading the same tools and instructions; it does not change the shape or calibration of the decision. If code must branch on the answer with a known confidence, use a typed decision model; if the LLM’s own tool choice is low stakes and reversible, cached LLM tool calling is usually enough.

The rest of the post is how we’d decide which side of that line a given tool sits on.

Two different problems wearing one name

“Make tool calling cheaper and safer” sounds like one problem. It is two.

The first is cost. An agent loop resends the same tool schemas and instructions on every step. Prompt caching for tool calling stops you paying full price to re-read a prefix the model has already processed.

The second is decision reliability. When the agent picks issue_refund over escalate_to_human, can your code tell a confident answer from a coin flip?

Caching is a fix for the first problem. A decision model like TypeSafe Jev is aimed at the second. If you want the primer on what Jev is and the job split between generating and deciding, we covered that in Decision model vs LLM: when agents should classify, route, and gate. This post assumes it and stays on the caching question.

What prompt caching actually buys you in a tool loop

A tool-calling request has a natural shape: a big stable part at the front and a small changing part at the end.

Everything below the line can be cached; everything above it is processed fresh. Most of the design work is keeping the base stable and pushing anything volatile above the boundary.

The provider docs are specific about how this works:

• Anthropic caches the prompt in a fixed order: tools, then system, then messages. Cache reads are priced at 0.1x the base input rate on most models (some newer models are cheaper still), and 5-minute cache writes cost 1.25x. The default lifetime is 5 minutes, refreshed each time the cache is hit. Their docs also state plainly that modifying tool definitions (names, descriptions, parameters) invalidates the entire cache. (Anthropic prompt caching docs)
• OpenAI enables prompt caching by default on supported models. It needs an exact prefix match, and tool names, descriptions, schemas, and ordering are all part of that prefix. Cached input is discounted, by up to 95% according to their docs. (OpenAI prompt caching guide)

What that means for agents: your tool list is the most expensive thing to churn. Add a tool mid-session, reorder the array, or tweak a description per user, and the cache goes with it.

How much does this matter? One study, Don’t Break the Cache, evaluated caching strategies across more than 500 agent sessions and reported API cost reductions of 41 to 80% and time-to-first-token improvements of 13 to 31% across providers. Those are their numbers, not ours, but the direction matches what we’d expect in long tool loops.

What caching does not change

This is the part teams tend to miss. Anthropic’s docs note that prompt caching has no effect on output generation, and the response is identical to what you’d get without caching. OpenAI’s FAQ says the same: caching does not change how the model generates output.

So after caching, the tool-call decision is exactly what it was before:

• The model still generates the tool call. issue_refund with amount: 4999 is a sampled output, not a lookup.
• You still get no calibrated confidence your policy can threshold. There is no number that says “this approval is 0.62 likely to be right, send it to a human.”
• The failure modes are unchanged. If the model is confidently wrong on a refund edge case without caching, it is confidently wrong with caching.

A confidently wrong approval is now a cheap confidently wrong approval. That is a better bill and the same risk.

Sidebar: prompt caching is not response caching

Prompt caching reuses the model’s processing of an identical input prefix. The model still runs and still produces a fresh answer for the current state.

Response caching (or semantic caching) returns a stored answer when a new request looks similar enough. Fine for FAQ-style lookups, risky for approvals, because the state that made the old answer correct (balance, permissions, order status) may have changed. Never let a semantic cache answer “should this refund go through?”

Where cached LLM tool calling is enough

In fairness, the simple setup covers most tools in most agents. We’d happily let the main LLM choose and call tools, with caching on, when:

• The tool is read-only or easily reversible. Search, fetch a document, list orders, draft (not send) an email.
• The blast radius is small. A wrong call wastes a step, not money or data.
• The user sees the result anyway and can redirect a wrong pick.
• The guardrails live underneath. Allowlists, sandboxes, scoped credentials, and rate limits do the real protecting, independent of what the model decides.
• The arguments are tightly typed. Strict schemas with enums and ranges for arguments remove whole classes of malformed calls.

Here a decision model is ceremony. Cache well, validate arguments, log calls, move on.

Where a Jev gate earns its place

A handful of tools are different. This is where we’d put a Jev call in front of the executor:

• Irreversible or money-moving actions. Refunds, payouts, deletes, sending external messages.
• Sensitive data. Citizen or customer personal data, where someone will later ask why access was granted.
• Production writes. apply_patch to a prod config, schema changes, feature flag flips.
• You need a threshold. The policy is “act when confident, ask a human when not.” That needs a confidence value, not a sentiment in prose.
• You need an audit row. A typed value (a Choice of clear or caution, with probabilities) is easy to store, query, and explain later.

TypeSafe’s docs describe Jev as taking state plus typed questions and returning typed values and probability distributions, with confidence on Choice and Score. They also push a design habit we like: ask atomic questions and compose them in code, so that “when priorities shift, change a coefficient in your code rather than rewriting a prompt.” For a refund gate: is the charge a true duplicate, does the amount match, are there abuse signals, combined by your own logic.

On cost and speed, we’ll only repeat what the docs claim: all questions are evaluated in parallel against the same state in one call, and “adding questions barely changes the response time.” (TypeSafe docs) We haven’t benchmarked it, so no number from us. Either way, the gate adds one call at a few dangerous tools, not a reasoning chain on every step.

In pseudo-code, it looks like this:

# main loop: cached LLM proposes the next tool call
proposal = llm.call(tools=STABLE_TOOLS, system=STABLE_SYSTEM,   # cached prefix
                    messages=history + [latest])                # fresh suffix

if proposal.tool in GATED_TOOLS:
    verdict = jev.ask(state=build_state(proposal, account, order),
                      questions=[
                        Choice("clear or caution for this refund?", ["clear", "caution"]),
                        Noul("the charge is a duplicate of an earlier settled charge"),
                      ])
    if verdict.choice != "clear" or verdict.confidence < POLICY_THRESHOLD:
        route_to_human(proposal, verdict)       # store the typed verdict
        return
execute(proposal)

Notice the main loop keeps its cache. The gate sits beside it, not instead of it.

The honest counterpoint: strict function calling

A fair objection: “Structured outputs with enums already give me a closed set. Make the LLM return {"decision": "approve" | "deny"} and I’m done.”

That buys something real: no prose parsing, a valid value every time, a clean switch in code. For many low-stakes routing decisions we’d start there.

What it does not give you is a calibrated distribution and a confidence value designed for branching. The enum tells you which option the model emitted, not how sure it should be. A “confidence: 0.9” field in the schema is just another generated token. If your policy depends on a threshold, that matters. If it doesn’t, strict function calling may be all you need.

A decision rule you can paste into a design doc

Two questions per tool: how bad is a wrong call, and does your code need to branch on confidence?

	Code does not need confidence	Code must threshold confidence
Reversible, low blast radius	Cached LLM tool calling. Strict schemas, allowlists, logs.	Cached LLM, plus a Jev check if you want calibrated routing (for example, to a human queue).
Irreversible, money, sensitive data, prod writes	Cached LLM proposes, strict schema plus human approval or hard policy rules.	Cached LLM proposes, Jev gate before the executor, human on low confidence.

The bottom-right cell is where both stack. Caching stays on for the main loop; the gate decides whether the proposal executes. They don’t compete.

FAQ

Does prompt caching reduce the cost of LLM tool calling?

Yes. Tool definitions and system prompts are usually the largest stable part of the request, and caching re-reads them at a reduced rate. Anthropic prices cache reads at 0.1x base input on most models, and OpenAI documents discounts of up to 95% on cached input.

Is prompt caching enough for agent tool approval?

Not on its own. Prompt caching lowers the cost and latency of re-reading the same tools and instructions; it does not change the shape or calibration of the decision. If code must branch on the answer with a known confidence, use a typed decision model; if the LLM’s own tool choice is low stakes and reversible, cached LLM tool calling is usually enough.

When do I need a separate decision model for tool calls?

When the tool is irreversible, moves money, touches sensitive data, or writes to production, and your policy needs a confidence threshold or a typed audit record.

What is the difference between prompt caching and response caching?

Prompt caching reuses processing of an identical prefix; the answer is still fresh. Response or semantic caching returns a stored answer, which is unsafe for approvals because state changes.

Can strict function calling or structured outputs replace a decision model?

For closed-set, low-stakes choices, often yes. They guarantee a valid option, not a calibrated probability or confidence your code can threshold.

Close

Caching is a bill fix. Jev is a control fix. Most teams want the first everywhere and the second at a handful of dangerous tools. Deciding which tools those are, and what happens when the gate says “caution,” is the policy layer. We wrote that up in AI agent approval gates.

Turn on caching this sprint. Then list your tools, mark the ones you’d be embarrassed to explain after a wrong call, and put the gate there.