Author
Sofia works on agent evaluation and reliability. She writes about measuring LLM systems before and after they reach production.
If your agent has a tool that takes a URL, the model chooses where your server connects. Measured end to end on Node 24 with no dependencies and no API key: the guard most people write fails four ways, including the IPv4-mapped IPv6 bypass behind CVE-2026-56266, which also defeated the first version of my own hardened guard.
ConditionalRequirement accepts onlyBefore, validates it five ways, and never consults it. A seven-arm harness measures the tool set a BeeAI RequirementAgent actually hands the model.
The prompt injection guardrail is third in the bundle and the HTML sanitizer is fourth, so markup the sanitizer strips rejoins a banned phrase after the detector has already passed it.
A misspelled path in a Mastra workflow branch predicate never errors. On @mastra/core 1.74.0 five of nine operators then skip the branch and four run it, so ne and not wrapping eq disagree on the same question. Four runnable scripts print the table, and the checker that would have caught it ships in the same package wired elsewhere.
On @openai/agents 0.18.0, the tool calls of one turn share a cancellation signal. One fatal failure aborts every other tool in the turn with SiblingCancellationError, and a tool that ignores details.signal keeps running, so its side effects land after the run has already thrown.
Measured on @openai/agents 0.18.0: the built-in client-side tool_search loader matches paths against tool names by exact string equality, and a miss returns zero tools with no error and no warning. The failure arrives one turn later as an error that blames the model.
Measured on ai 7.0.118 and zod 4.6.5: a .refine() on a tool input schema is stripped before the schema reaches the model, with no warning. z.string().startsWith('acct_') arrives as a pattern; the same rule written as a refinement arrives as nothing.
On ai 7.0.114, whatever your tool execute function throws is serialised and sent to the model provider in a tool_result block marked is_error. The documented toModelOutput hook is skipped on that path. Measured first-party with a local capture server, no API key.
An MCP stdio server that holds a timer or a pool open does not exit when its client dies. Measured on SDK 1.30.0 and Node 24.8.0: it is reparented to init and keeps running. StdioServerTransport registers no stdin end-of-file listener. Here is the reproduction and the four-line fix.
If your agent has a read_file tool, you probably guard it with path.resolve plus startsWith. It blocks the obvious traversal and is defeated by a symlink and by a sibling directory sharing the prefix. Both reproduced on Node 24.8.0, with the guard that passes all nine cases.
Mastra documents that the object key, not the tool id, becomes the name the model sees. It does not document that the key is rewritten first: invalid characters become underscores and the result is cut to 63. The tools array goes through that rewrite. toolChoice does not.
allowedTools auto-approves, tools restricts. That half is documented. What is not: an AgentDefinition has no allowedTools field, and the SDK forwards one verbatim to the CLI anyway, unvalidated.
