The headline
A rule the model was told
A developer who wants an agent to keep a rule usually writes it in the system prompt. The same three rules were each held four ways, on two small models and on Claude Sonnet 5: not stated at all (the vendor’s field description only), in the system prompt, as aGloss on the field, and enforced on the field,
where a refused call tells the model which field was wrong and why. 20 runs per
cell, temperature 0.7 on the small models (Sonnet’s is not settable through the
subscription it ran on); nothing reaches a real API.
Runs whose call broke the rule and reached the API, of 20:
“Declined” is a run that made no call at all: the model refused the task the rule
forbids, which keeps the rule. The cents rule was never broken, so it shows
nothing. The model is not told an enforced rule up front, so its first call broke
it in every run; what happened next:
The same sentence held on two models and failed every time on the third. That is
the point, rather than which model is better: a guarantee written as text depends
on which model reads it, and a field rule does not. Sonnet kept every prompted rule,
as glm-5p3-flash did, so on these three rules a frontier model needed no
enforcement; it shows the enforced rule costing nothing, not preventing anything.
Sonnet ran through
claude -p on a Claude subscription, as a model and nothing
else: no tools of its own, no Claude Code system prompt, the tools written into its
system prompt and its calls read back as JSON
(harness/bench/claude_provider.py).
It did not see tools the way an API caller’s model does.
Records: harness/results/bench/rules_pilot/run.jsonl,
code: harness/bench/rules_pilot.py.
Prompt injection
AgentDojo (ETH Zürich) measures how often an agent does the task, and how often an attacker’s instruction planted in the data gets done instead. Its banking suite, v1.2.2, with itsimportant_instructions attack, four ways, same model and system message:
- stock: the suite’s own tools, AgentDojo’s default pipeline.
- tool_filter: AgentDojo’s built-in defense of that name: the model first names the tools the task needs, and the rest are removed.
- charter_plain: the same bank served over HTTP and declared with Charter, with no rule beyond the types.
- charter_rules: that declaration with four field rules, written from the suite’s
tool definitions and its sixteen user tasks before any attack or injection task
was read, and committed before the run
(
RULES.md): a payee the account already knows or the user named, an amount within the balance, a password only if the user typed it, and the account holder’s name not offered to the model.
On nemotron-lightning the attack lands 79% of the time on the suite’s own tools.
With the four field rules it landed in none of the 144 episodes, and the agent still
did the user’s task in 62% of them, against 71% with no defense. AgentDojo’s tool
filter cut the attack to 20% and the task to 54%. The declaration alone changes
nothing (charter_plain), which is expected: the rules are what refuse the
attacker’s call, and the refusal says which field and why. When an attack is
refused, the hijacked model often keeps trying it and never gets to the user’s
task, which is most of the rules’ cost. On glm-5p3-flash the attack barely lands,
once in 144, so that model shows the rules’ cost and nothing about their benefit.
On Claude Sonnet 5 it never landed, so the Charter arms were not run there: they
could only have shown a cost. Sonnet saw its tools as text in its prompt rather than
through tool calling, so its task success is not comparable with the other two
models’.
AgentDojo’s tool filter needed two changes to run on Fireworks at all, both
recorded in
run.py with the voided first runs beside the results: it asks its
model to choose tools under tool_choice="none", which Fireworks answers without
showing the model any tools, so the tools are listed in its prompt; and it leaves
its own reply as the conversation’s last message, which Fireworks answers with
nothing, so that exchange is dropped once the filter has chosen.
Records: harness/results/bench/agentdojo/,
code: harness/bench/agentdojo/.
A support agent’s written policy
τ²-bench (Sierra) puts an agent in front of a simulated customer, with a written policy and a store’s tools, and scores the store’s database afterwards against the one the task’s gold actions produce. It ran here twice over: first its retail domain’s first 20 tasks on three models, then every retail and airline task on Claude Sonnet 5.Retail, 20 tasks, three models (24 October)
The first 20 tasks of retail’s test split, three ways, with the same agent model and policy and glm-5p3-flash as the customer throughout:- stock: the domain’s own tools and agent.
- charter_plain: the same tools declared with Charter, field for field, and called over HTTP, with no rule beyond the types.
- charter_rules: that declaration with three rules taken from the policy, written
from the policy and the tools before any task was read and committed first
(
RULES.md): only the lookups until a customer is authenticated; only that customer’s account and orders; a change to the account only straight after a message that begins with “yes”.
In six conversations the agent changed a customer’s account before the customer
had been authenticated, and τ²-bench scored all six as passes. Three were on
nemotron-lightning with the stock tools, two with charter_plain, and one was Sonnet
with charter_plain, an address or an exchange each time. Sonnet took the customer’s
word for who they were (“My user ID is mei_kovacs_8020. I don’t know my email
offhand, so use that”), though the policy asks for an email, or a name and zip
code, “even when the user already provides the user id”. Neither of τ²-bench’s
graders asks how the agent got to the result: the database came out right, and the
judge, reading the conversation, passed it too. With the rules nothing ran before
authentication; seven calls tried, on nemotron-lightning and Sonnet, and were refused.
Rule 3 is mostly a cost. It refused 99 writes over the two runs, most of them after
the customer agreed in other words (“yeah, go ahead”), and the rest after a message
with its “yes” further in. The agent usually asked again and went on. On
glm-5p3-flash it cost 3 tasks of 80 (72 against stock’s 75), all in the second
run; on nemotron-lightning it cost nothing net (65 and 65); on Sonnet the rules arm
came out one ahead (15 and 14 of 20).
RULES.md named this cost before the run.
The same arm moved by up to five tasks of 40 between the two runs, which is the
noise in these numbers, and the declaration alone moved nothing outside it.
Every retail and airline task, Claude Sonnet 5 (25 and 26 October)
τ²-bench’s wholebase split: retail’s 114 tasks and airline’s 50, with Sonnet as the
agent and glm-5p3-flash as the customer again. The grade is τ²-bench’s full one:
retail’s database check times its natural-language assertions, and airline’s
database check times the facts the agent had to tell the customer. τ²-bench judges
both with OpenAI models; glm-5p3-flash judged them, with τ²-bench’s own prompts. Each
arm ran each task once. Four trials were planned, and the runs were stopped after one
because they used up the subscription’s limits, so these rows have no pass^k beyond
pass^1, and the noise between trials (up to five tasks of 40 above) is not measured
in them.
Airline’s rules were written the same way, from its policy and tools before any of
its tasks was read, and committed with predictions
(RULES_AIRLINE.md):
eleven rules, among them that the customer gives their own user id, an explicit yes
before any change, what a basic-economy booking may change, when a booking may be
cancelled, the free-bag allowance, and when a certificate may be offered. The stock
arm ran first, to see whether there was anything to measure. It scored 44 of 50, and
4 calls broke a rule and ran, in 2 conversations, both scored as passes. All four
broke the explicit-yes rule after the customer had agreed in other words (“Just get
those bags on there”; “Go ahead then. Upgrade…”, said before the agent listed the
charges). No other rule was broken. The prediction had been 15 to 40 such calls; with
4, the Charter arm was not run on airline.
Two of the 45 changed an account before the customer had been authenticated, and
neither is a clean case. In task 5 the agent opened the account by the user id the
customer gave and checked the zip code on it, where the policy asks it to find the id
by email, or by name and zip code, first. In task 13 its call to look the customer up
came out as text instead of a call, the simulated customer answered with a user id in
JSON, and the agent took that as verified. The other 43 broke rule 3 alone: the
customer agreed, but not in a message that begins with “yes”. All 45 are in
graded_passes.md,
each call with what the customer said before it, as they were read to reach these counts
(graded_passes.py).
Rule 3 cost tasks. The rules arm scored 82 against stock’s 93: it failed 19 tasks
stock passed, 9 of them with a rule-3 refusal in the conversation and 10 with no
refusal at all, and passed 8 that stock failed. After its 183 rule-3 refusals the
agent asked again, and a call to the same tool later went through 168 times.
One thing in these runs is the harness’s, not either arm’s. Sonnet ran through
claude -p with its tools written into its system prompt, and in 54 stock and 62
charter_rules retail conversations (and 9 airline ones) at least one of its calls came
out in a form the provider could not read as a call, and reached the customer as text:
89 and 88 such messages. It hit both arms alike, and task 13 is one of its results.
The Claude calls would have cost about $69 at API prices, for everything the provider
served over the two days, including episodes cut off by a limit and 17 of a second
trial that are left out; the subscription was charged nothing extra. The customer on
Fireworks cost $0.73.
Records: harness/results/bench/tau/
(the Sonnet runs in sonnet5-2026-10-25/, with every conversation),
code: harness/bench/tau/.
The rules read the episode’s session from Charter’s per-call
context. When these runs were recorded
they read it from a module-level ContextVar, which context replaced.
replay_rules.py
replays every recorded call of the rules arm through the current code, on the
episode’s own database and with no model: in both runs that kept their
conversations (sonnet5-2026-10-25/ and run2/), all 1,703 replies, the 252
refusals among them, are the recorded ones, byte for byte.
The same calls through each tool layer
No model here: the same call is sent through Charter, through the API’s most-starred or official MCP server, through Composio where the benchmark’s account connects (Gmail), and as a raw HTTP request, and what each layer hands back is labelled against the vendor’s own documentation. The MCP servers, pinned: github-mcp-server 2.0.3 (GitHub’s own), slack-mcp-server 1.3.0, Linear’s hosted server, shopify-mcp 1.0.8 (the most-starred one for the Admin API; Shopify’s own serves documentation), @stripe/mcp 0.3.3 (Stripe’s own) and google_workspace_mcp 2.2.0 (run withcryptography 46, because 50 has no Intel macOS wheel), each at its
quickstart configuration. Every call ran against the harness’s test accounts on 24
and 25 October.
A call outside the documented contract. Each asks for something the vendor rules
out: Gmail’s maxResults over 500, Calendar’s over 2,500, a Calendar window that ends
before it starts, GitHub’s per_page over 100, Stripe’s limit over 100, and a
Calendar syncToken sent with a filter. The first five the APIs answer with 200, a
page or no events, and say nothing about the argument they did not honour. The sixth
Google refuses itself, with a 400: it is the control.
A failure inside a 2xx. Linear answers an update or a comment on an issue that
does not exist with 200 and a GraphQL
errors array; Slack answers a reaction to a
deleted message, and a post to an archived channel, with 200 and ok: false;
Shopify answers an update to a product or a customer that does not exist with 200
and userErrors. Charter and the MCP servers reported all six failures: here the
popular servers do what Charter does. A raw call returned each error as data, for
the caller to notice. Gmail answers a label update (PUT) for an id that does not
exist with 200 and the request echoed back, a case documented nowhere: Charter’s
labels_update reported it as a success, as a raw call did, while the MCP server,
Composio and Charter’s own labels_patch sent a PATCH, which Gmail answers with
404. That one is a loss.
What reaches Gmail. Four drafts, each read back through Gmail’s own API: a
display name outside ASCII, a Cc, a body outside ASCII, a reply in a thread. Charter
and the MCP server stored all four exactly. Composio stored three, and for
Zoë Müller <zoe.muller@example.com> stored the bare address: the name was lost.
What it costs in the prompt. The definition each layer hands the model, in bytes
of JSON, for the same operations. Per operation it is close: Charter’s is the larger
on 6 of 10 and the smaller on 4, at most 2.7 times the MCP server’s either way
(Slack’s post: 3,750 against 1,403; Linear’s issue update: 5,757 against 8,271). A
whole pack is not close, because it covers more of the API:
Handing a model every tool of a pack costs 1.5 to 7.5 times what the server costs;
that is a loss unless the tools are narrowed to the job.
Every row, with each layer’s raw reply:
harness/results/bench/same_calls.md
and schema_size.md;
code: harness/bench/same_calls/,
schema_size.py.
Failures that come back as a success
For each of the 16 APIs Charter ships, the vendor’s own documentation was read for a request that fails, in whole or part, while the response is still 2xx.
Code that checks the status and not the body reports these as successes. Of the
eight whose body says so, Charter raises for the three whose failures are a body
field (Slack, Linear, Shopify), declared once per API as an
Envelope; for the other five the flag reaches the model with
the response. Two more turned up live and are documented nowhere: Gmail answers a
label update for an id that does not exist with 200 and the request echoed back,
and Calendar answers the instances of a missing event with 200 and an empty list.
The per-API table, with the vendors’ words and links:
harness/results/bench/survey_200.md.
When the thing the task names is gone
Four trap tasks name a Slack message deleted before the task began, or a Linear issue that does not exist: react to the message, edit it, move the issue to Done, comment on it. They cannot be done, and both APIs say so in the response body: Slack inside a 200, Linear in its GraphQLerrors. Each has a control where the
target exists. glm-5p3-flash ran 10 per task per arm, and Claude Opus 5.5 ran 3,
through the same Claude subscription provider as Sonnet. Every run was judged three
ways: by reading the account, by reading the agent’s final answer by hand, and by an
eval in the style of Google ADK’s tool_trajectory_avg_score and
response_match_score, the way most teams grade agents today.
When the named message or issue was gone, glm-5p3-flash often found a lookalike,
edited it, reacted to it or closed it, and reported the task done. An eval that
checks the expected call was made passed two runs in three, because the call was
made; it failed. Adding the answer to the eval caught every trap run, at a price: at
ADK’s documented threshold of 0.8 it also failed 35 of the 36 control runs that did
the task, and at 0.5 it failed 1 to 4 of them.
Opus did none of that. In all 24 trap runs it said the task could not be done and
that it had changed nothing, and it changed nothing; it did the control task in 12
of 12 runs with Charter and 10 of 12 with the raw tool. Charter made no difference
on either model: both arms show the model the error, and what it did next was the
model’s. These traps test whether a model acts on an error it is shown, which a
stronger model does; the calls a tool layer lets through without an error are in
the same-calls table and
the error-message swap.
The traps and their controls shared one channel and one team, so the lookalikes a
trap agent changed were another run’s; a real workspace has lookalikes too. The
Charter arm here hands the model every tool of each pack a task touches, which is
a cost: a Charter run took 3.3 times the tokens of a raw run
on glm-5p3-flash and 10 times on Opus.
Records and the hand-read labels, with the rubric:
harness/results/bench/false_passes.md,
said_done.json,
and for Opus false_passes_opus.md,
said_done_opus.json.
Code: harness/bench/false_passes.py,
scenarios: t01_stale_ids.py.
After a bad call
Does the error a model reads decide whether it fixes its call? Six conversations were frozen at an invalid call, each breaking a contract the vendor documents: the five silent cases from the same-calls table that Charter refuses, a CalendarsyncToken sent with a filter, and $15.50 sent to
Stripe as 15.5 where it wants cents. Then only the tool’s result was swapped,
among five texts: Charter’s refusal; the refusal Charter’s predecessor gave before
its messages were rewritten (pydantic’s own error inside LangGraph’s template); the
API’s own answer and the MCP server’s, both captured live; and Error: invalid arguments. The model went on for up to ten calls, offered the tool either as
Charter declares it or with every numeric limit stripped from the schema, the
vendor’s descriptions kept. glm-5p3-flash ran 10 samples a cell, Opus 3.
The wording made no difference. Every text that said something was wrong got
the call fixed on the next try, on both models and both schemas, including a bare
invalid arguments: the vendor’s own descriptions name every limit here, and the
models read them. Charter’s rewritten refusal did no better than the one before it.
What made the difference was being told at all. Where the API or the MCP
server answered the invalid call with a 200, glm-5p3-flash told the user something
untrue four times: three times that a first page of 100 GitHub issues was all of
them, once that a calendar had no events in a window it had searched backwards. No
error text produced a single such answer. Opus never claimed more than it had; in
five of its six GitHub runs with GitHub’s own answer it said the list was partial,
and stopped there.
Two of glm-5p3-flash’s answers say a second page “returned the same page”: a call
that still breaks the contract is handed the same captured text again, where the
real API would have sent the next page. The rubric for every answered episode, and
each label with its reason:
harness/results/bench/error_swap/answers.json;
records and code: error_swap/,
error_swap.py.
Scopes, and Google’s security assessment
An app that requests a scope Google classes as restricted needs a yearly CASA security assessment before anyone outside its test-user list can connect. For five common jobs, the scopes each option requests, read off the code. The MCP server is taylorwilsdon/google_workspace_mcp v2.1.2, the most-starred Gmail MCP server on GitHub (3,405 stars on 24 October):
Charter asks for what the tools handed out need, from each method’s accepted scopes
(
scopes_for); a job whose tools need a restricted scope still
needs the assessment. Every tool’s scopes were checked against Google’s own API with
a test account holding exactly each one
(scripts/live_google_scopes_check.py, 209 of 209 pairs).
Details: harness/results/bench/scopes_by_job.json.
Google Docs, through Charter and Composio
Google’sdocuments.batchUpdate takes 53 kinds of edit. What each option’s Docs
tools let a model reach, read off the definitions each hands a model on 25 October:
Charter’s gdocs pack, and Composio’s googledocs toolkit, of which 28 of its 32
tools are Docs operations (two read Sheets charts; search and copy go through Drive).
Composio reaches every kind through one tool,
GOOGLEDOCS_UPDATE_EXISTING_DOCUMENT,
whose editDocs is a list of objects with no declared fields: the model writes
Google’s request format from three examples and a link. Charter declares 33 kinds
field by field and cannot send the other 20 at all: comments, suggestions, tabs,
dropdowns, rich links, people chips, dates and named styles. Of Composio’s fields
that are no name in Google’s API, most are its own renamings (text_to_insert,
insertion_index, find_text), and two, oddPageFooterId and oddPageHeaderId
on its document-style tool, name fields Google’s DocumentStyle does not have; it
has evenPage… and firstPage….
What an edit leaves. One document was built with a heading in a named style,
bold, coloured text, a link, a bulleted list, a table, a footnote and a named range,
and copied three times. In each, “12%” became “14%”: through Charter’s
replaceAllText, through Composio’s GOOGLEDOCS_REPLACE_ALL_TEXT, and through
Composio’s GOOGLEDOCS_UPDATE_DOCUMENT_MARKDOWN, which replaces the whole document
with Markdown, the way an agent using that tool has to send it. Read back through
Google’s API:
Both typed paths kept all eleven things the document held. The Markdown path lost
what Markdown cannot say, colour and a named range, and what its converter did not
carry over, the footnote. Comments and suggestions were not tested: Google’s API
cannot create a suggestion, and a comment it creates is not anchored to text.
Details:
harness/results/bench/docs_coverage.md,
docs_edit_loss.json;
code: docs_coverage.py,
docs_edit_loss.py.
Time and money per call
The same cheap call, 100 times each, direct withhttpx, through Charter, and for
Gmail through Composio, which runs it on its servers for the connected account. The
calls were interleaved in a random order from one machine on 25 October.
Milliseconds:
Charter’s own work is the time spent in the library rather than on the network.
With a client shared across calls, which is how these rows were taken, a call costs
what the API does. A call through Composio crosses its servers both ways, and added
about 0.7 seconds at the median. Zapier MCP was not measured.
What each charges per call, from the vendors’ pages on 25 October:
Composio is cheap per call, so price is not the argument against it. Where the data
goes is:
Composio supports a customer’s own OAuth app, so the consent screen can carry your
name there too; the data path is the part that does not change.
Cost per successful task
From the earlier campaign’s records, divided by the runs that succeeded rather than by all runs:Where Charter lost
- A field rule written wrong costs the task, every time. Rule 2 capped an amount at the account’s balance of $1,810. User task 15 sets up a $2,200 rent standing order, so the rule refused a legitimate payment: task 15 succeeded in 0 of 10 runs against 10 of 10 with the stock tools on glm-5p3-flash, and in 0 of 9 attacked runs against 6 of 9 on nemotron-lightning. The rules were committed before the run and are published as written; the refusal said which field and why, and nothing was sent.
- The rules cost task success. Under attack, from 71% to 62% on nemotron-lightning and from 91% to 83% on glm-5p3-flash, where the attack barely lands. Task 15 is most of it on both; the rest is a hijacked model that keeps retrying the refused call instead of returning to the user’s task.
- A rule worded tighter than the people it reads. τ²-bench’s rule 3 asks for a message that begins with “yes”; its simulated customers mostly agree with “yeah, go ahead”. The rule refused 99 writes over two runs, and on glm-5p3-flash it cost 3 tasks of 80, where after the refusal the agent never made the change. On every retail task with Sonnet the rules arm scored 82 of 114 against stock’s 93; 9 of the 19 tasks it lost had a rule-3 refusal in them.
- The benchmark found a bug in Charter. A list sent as a JSON string is parsed
for the model, but only under the field’s snake_case name, while Charter’s schema
advertises camelCase:
item_ids: "[...]"worked anditemIds: "[...]"was refused. Found in τ²-bench’s first trial run, fixed in the same commit as these results, with a test, before the recorded runs. - The wording of a refusal made no difference. In the error-message swap,
Charter’s refusal, the one before it was rewritten, the MCP server’s error and a
bare
invalid argumentsall got the call fixed on the next try, 98 to 100% of the time, because the vendor’s descriptions already name the limits. What mattered was that something said the call was wrong. - A Gloss is not a guarantee. On nemotron-lightning, the #support rule as a Gloss still reached the API in 6 of 20 runs, against 20 of 20 in the system prompt.
- A client per call is slow. Without a shared
httpx.AsyncClient, Charter opens one per call, which added 103 ms at p50 on Gmail, 108 ms on Slack and 200 ms on GitHub. Passclient=toainvoke, or use aToolSession. - Charter did not stop an agent changing a lookalike. When the named message or issue was gone, glm-5p3-flash said it was done in 15 of 40 runs on each arm, and changed something the task did not name in 12 (Charter) and 13 (raw). Opus did neither, on either arm.
- Whole packs in the prompt. On the trap tasks, where the arm hands the model every tool of the Slack and Linear packs, a Charter run took 364,329 tokens on glm-5p3-flash against the raw tool’s 110,404, and 336,099 against 33,564 on Opus.
- A Gmail label update to a missing id reported success. Charter’s
labels_updatesends Gmail’sPUT, which answers an id that does not exist with 200 and the request echoed back, and Charter passed it on as a success. The MCP server and Composio sendPATCH, which Gmail answers with 404, and so does Charter’s ownlabels_patch. Nothing in the response marks the echo, so noEnvelopecan catch it; the Gmail pack says to preferlabels_patch. - The Docs pack is behind Google. It covers 33 of the 53 kinds of edit Google’s
documents.batchUpdatetakes today; comments, suggestions, tabs, dropdowns and the newer inline elements are missing. Composio reaches all 53, 31 of them through an untyped passthrough. - A whole pack is a lot of prompt. Every tool of a pack is 1.5 to 7.5 times the
bytes of the same API’s MCP server (Stripe: 118,596 against 15,840), because the
pack covers more of the API. Hand a model the tools a job needs
(
ToolSession), or narrow them (derived).
What this does not show
- Three models, one of them frontier. Sonnet kept every prompted rule; the point of the rules table is that a guarantee should not depend on which model. Sonnet saw its tools as text in its system prompt, not through tool calling.
- τ²-bench’s three-model runs are 20 tasks, with one model as every simulated customer and glm-5p3-flash, not the OpenAI models τ²-bench names, judging the assertions. Sonnet ran all 164 retail and airline tasks, but once each, so its rows carry no measure of noise, and in about half its retail conversations a call came out as text the harness could not read.
- One AgentDojo suite and one attack. Banking with
important_instructions. Other suites, and attacks written against field rules, are not measured. - The rules pilot calls no real API. Requests go to a mock that records them, so “reached the API” is a request Charter let out.
- Competitors are measured from their code, not their runtime, for the scopes table. The same-calls table did run them: six MCP servers and Composio, on seven APIs.
Reproducing it
FIREWORKS_API_KEY; the trap campaign needs the harness’s
Slack and Linear accounts, the latency run its Slack and GitHub tokens, and the
same-calls run every test account it touches, plus Composio’s
(harness/AGENTS.md).
The Sonnet rows need a logged-in claude CLI.