Author
Sofia works on agent evaluation and reliability. She writes about measuring LLM systems before and after they reach production.
allowedTools auto-approves, tools restricts. That half is documented. What is not: an AgentDefinition has no allowedTools field, and the SDK forwards one verbatim to the CLI anyway, unvalidated.
The skills option validates every name before the session starts, and its wildcard guard tests for exactly two endings. Eight of eleven wildcard shapes get through, become a Skill(name) rule that names no skill, and say nothing. Measured first-party on 0.3.272, with the bisect.
npm error notarget No matching version found for @anthropic-ai/sdk means you asked for a major that does not exist. Anthropic ships two SDKs on unrelated version lines: Python is on 1.5.0, npm has never left 0.x. Read from both registries, with the caret trap that follows.
ImportError: cannot import name 'streamablehttp_client' means pip resolved MCP Python SDK 2.x against v1 client code. The rename is one underscore. The part nobody mentions is that passing your own httpx2 client silently drops the 300 second read timeout to 5 (2026).
Claude Code v2.1.260 narrowed strict sandbox mode so it no longer covers commands you type at the ! prompt. Anthropic documents two exceptions the changelog never mentions, and the variable that closes the gap is filed under credential scrubbing.
keybindingFlavor is deprecated and inert in Claude Code v2.1.261 and later. The readline word-editing behaviour it gated is now on for everyone. Ctrl+W deletes back to whitespace while the Alt keys stop at punctuation, and neither can be rebound: the keybindings reference documents 115 actions and none of them is a word-editing action.
"TodoWrite no longer available" is not a bug. Claude Code v2.1.233 drops the five task-tracking tools on Sonnet 5 and newer unless you opt in. The gate follows the session, not the model, and the SDK opt-in line behaves differently in TypeScript and Python (2026).
MCP error -32000: Connection closed is generated inside the client SDK and is never sent by a server. It means your MCP server process died. Here is how to recover the real error, and a measured correction to the stdout advice (2026).
Agents fail quietly: a prompt tweak that fixes one task often breaks three others, and manual spot-checks never re-test what used to work. The fix is a frozen eval set scored on every change. This tutorial builds that harness and tracks five metrics that actually catch regressions, task success rate, tool-call accuracy, step efficiency, cost per task, and a safety/guardrail rate. You will assemble an eval set, write a runner that scores each metric, and turn the before/after diff into a regression gate so a change only ships when the numbers hold or improve.
