Whose time it was
ToolCall splits every call five ways rather than
reporting one duration:
overhead_ms is total_ms - upstream_ms: everything that was not the API. It is
the number that answers is this layer in my way, and a boundary layer should be
able to answer that from your own logs rather than from a benchmark someone else
ran.
In the ordinary case it is small and there is nothing on this page for you. When
it is not, it is almost always credential_ms — a provider that re-fetches a
token it could have cached — and that is your code, in your process, which is the
point of splitting it out.
Reuse the connection
ainvoke builds an httpx.AsyncClient per call when it has no other one to
use, which means a fresh TCP connection and a fresh TLS handshake against an API
you are about to call again. Measured against the APIs these packs cover, that
handshake is worth about 114ms:
client.py
dispatch takes the same
argument and hands it through:
dispatch.py
client is keyword-only and shadows a schema field literally named
client. Pass such a field in the positional dict.)
Through an adapter, it is not reached
An adapter hands your framework a coroutine and the framework decides when to call it. There is no call site in between, so a tool called by a LangChain agent or an MCP server opens its own client and pays the handshake every time. Passing one is reachable only from a loop you hold yourself. This is a known limitation rather than an oversight, and the reason is the arithmetic above. An agent run here makes 6.3 tool calls spread across two or three providers, so reuse has three or four handshakes to save — about 0.45 seconds of a 62.8-second run, or 0.7%. That is smaller than one retry, and smaller than the run-to-run variance on the same scenario. The case where connection reuse genuinely pays is a tight loop against one host — paginating two hundred messages, not calling four APIs once each. That case has an explicit call site, which is the snippet above. Somewhere between the two there is a deployment this matters for;upstream_ms against overhead_ms in
your own logs is what says whether yours is it.
The first call against a tool
schema_ms is the one field here that is zero on almost every call and large on
a handful. A tool’s view is derived on first use rather than when its pack is
imported, because a session exposes a few of a pack’s tools and deriving all
128 of Linear’s would be most of the import. So whichever call reaches a tool
first pays to build it. On a schema whose types refer to each other that is
seconds, not milliseconds:
validate_ms, which means what it
says again. What neither fixes is that a request paid for it. Build the tools a
process will expose while it is still starting up:
prepare is idempotent and thread-safe, and it
builds the JSON schema too. That part matters for a synchronous adapter:
to_openai_tools runs inside the turn loop and blocks the
thread it is on. After a startup pass, a non-zero schema_ms in production means
a tool nobody prepared.
The number not to chase
Wall time per run is the most quotable latency number and the least stable one. Across the two campaigns on the measured results page, the same two arms went in opposite directions:Related
- Observability — the record, and what it deliberately does not measure
- Getting it right first try — the retry loop, which is where the time usually is
Tool.ainvoke—client,headers, and what each is forTool.prepare- deriving a tool’s view at startup rather than in a request