Evaluator-Optimizer: Build a Self-Correcting LLM Agent in Python (2026)

An evaluator-optimizer workflow puts a second model in the loop: one call generates a draft, a second call grades it against a fixed rubric, and the feedback feeds the next attempt until the draft passes or you hit a stop condition. It is the fifth of the five workflow patterns in Anthropic's "Building Effective Agents" (December 2024), and it is the one to reach for when a single generation is close but not reliably good enough. This tutorial builds a working self-correcting agent in Python, with a real rubric and the guardrails that keep the loop from spinning forever.

Quick answer

The evaluator-optimizer pattern is a two-role loop: a generator produces output, an evaluator scores it against explicit pass/fail criteria and returns actionable feedback, and the generator revises using that feedback. You repeat until the evaluator returns PASS or a guard trips (max iterations or a feedback plateau). It is worth the extra latency and token cost only when you have clear evaluation criteria and when iterative refinement measurably improves the result. Below is a runnable Python implementation using the Anthropic SDK and Claude Sonnet 5.

What the pattern actually is

Two roles, one loop.

• The generator takes the task, plus optional reviewer feedback, and returns a draft.
• The evaluator takes the task and the draft, applies a rubric, and returns a structured verdict: PASS, or NEEDS_WORK with specific feedback.

The loop runs generate, then evaluate. On PASS it returns the draft. On NEEDS_WORK it hands the feedback back to the generator and tries again. This is the same critic-in-the-loop idea people call a reflection agent or a self-critique loop; evaluator-optimizer is Anthropic's name for the version where the critic is a separate, rubric-driven call rather than an inline "now check your work" instruction.

It composes with the other patterns. Once you have a critic, it slots naturally into the orchestrator-workers pattern when individual worker outputs each need review before they are merged, and it is a strict upgrade over a bare prompt chain when one of the chain steps has a quality bar you can articulate.

When to reach for it, and when not to

Use it when both of these hold:

1. You can write down what "good" means as criteria, not vibes.
2. A first draft is often wrong in ways a reviewer can name, and naming them helps.

Skip it when either fails. If you cannot articulate the criteria, the evaluator has nothing to enforce and you just burn tokens. If the model already gets it right in one shot, the loop adds latency and cost for no gain. And if your criteria are fully mechanical, for example "the JSON parses" or "the tests pass", you do not need an LLM evaluator at all. More on that below.

Prerequisites

• Python 3.10 or newer.
• An Anthropic API key. As of July 2026, Claude Sonnet 5 is 3 USD per million input tokens and 15 USD per million output tokens, with introductory pricing of 2 USD / 10 USD per million tokens through August 31, 2026 (Anthropic pricing, 2026). A short evaluator-optimizer run is a handful of calls, so expect fractions of a cent per task.

Install the SDK and set your key:

pip install "anthropic>=0.40"
export ANTHROPIC_API_KEY="sk-ant-your-key-here"

The generator

The generator has one job: return an answer to the task. When feedback is present, it must treat that feedback as a revision checklist, not a suggestion.

from anthropic import Anthropic

client = Anthropic()  # reads ANTHROPIC_API_KEY from the environment
GEN_MODEL = "claude-sonnet-5"

def generate(task: str, feedback: str | None = None) -> str:
    system = "You are a precise technical writer. Return only the answer, no preamble."
    if feedback is None:
        user = task
    else:
        user = (
            f"{task}\n\n"
            "A reviewer returned the feedback below on your previous attempt. "
            "Produce a new version that resolves every point.\n\n"
            f"Feedback:\n{feedback}"
        )
    msg = client.messages.create(
        model=GEN_MODEL,
        max_tokens=1024,
        system=system,
        messages=[{"role": "user", "content": user}],
    )
    return msg.content[0].text.strip()

The evaluator

The evaluator is where the pattern lives or dies. Give it a rubric with named, pass/fail axes and force a structured output so the loop can branch on it. Ask for minified JSON and parse defensively, because models sometimes wrap JSON in a code fence.

import json

EVAL_MODEL = "claude-sonnet-5"

RUBRIC = """You grade a DRAFT against a TASK on three axes, each pass or fail:
1. accuracy: every factual claim is correct and current.
2. completeness: the draft fully satisfies the task with nothing missing.
3. clarity: a mid-level engineer can act on it without rereading.
Return ONLY minified JSON of the form:
{"verdict":"PASS","feedback":""}
or
{"verdict":"NEEDS_WORK","feedback":""}
Return PASS only if all three axes pass."""

def _parse_json(raw: str) -> dict:
    raw = raw.strip()
    if raw.startswith("```"):
        raw = raw.split("```")[1]
        if raw.startswith("json"):
            raw = raw[4:]
    return json.loads(raw.strip())

def evaluate(task: str, draft: str) -> dict:
    msg = client.messages.create(
        model=EVAL_MODEL,
        max_tokens=512,
        system=RUBRIC,
        messages=[{"role": "user", "content": f"TASK:\n{task}\n\nDRAFT:\n{draft}"}],
    )
    return _parse_json(msg.content[0].text)

The loop, with guardrails

A naive loop is while not passed: revise. That is how you get an agent that runs ten times, oscillates between two mediocre drafts, and bills you for the privilege. Two guards fix the common failure modes:

• A hard iteration cap bounds cost and latency no matter what.
• A plateau guard stops early when the evaluator returns the same feedback twice, which means the generator cannot act on it and more loops will not help.

def evaluator_optimizer(task: str, max_iters: int = 4) -> dict:
    draft = generate(task)
    last_feedback = None
    history = []
    for i in range(1, max_iters + 1):
        verdict = evaluate(task, draft)
        history.append({"iter": i, "verdict": verdict["verdict"]})
        if verdict["verdict"] == "PASS":
            return {"status": "passed", "iters": i, "output": draft, "history": history}
        if verdict["feedback"] == last_feedback:
            return {"status": "plateaued", "iters": i, "output": draft, "history": history}
        last_feedback = verdict["feedback"]
        draft = generate(task, feedback=verdict["feedback"])
    return {"status": "max_iters", "iters": max_iters, "output": draft, "history": history}

Run it end to end

if __name__ == "__main__":
    task = (
        "Explain, in under 120 words, why a serverless function that talks to "
        "Postgres should use a connection pooler. Name one concrete failure mode "
        "it prevents."
    )
    result = evaluator_optimizer(task)
    print(f"{result['status']} after {result['iters']} iteration(s)\n")
    print(result["output"])

A typical run prints passed after 2 iteration(s) followed by a tightened answer. The first draft usually explains pooling but forgets the "name one concrete failure mode" constraint; the evaluator flags the omission on the completeness axis; the second draft adds the connection-exhaustion failure and passes. That is the whole value of the pattern: the constraint you buried in the prompt gets enforced by a second pair of eyes instead of hoped for.

Designing a rubric that converges

The rubric is the product. Three rules keep it from thrashing:

• Make every axis pass/fail, not a 1 to 10 score. Numeric scores drift and the generator cannot tell what to fix. Binary axes give unambiguous targets.
• Keep the axes independent. If "accuracy" and "clarity" overlap, the evaluator double-counts and the feedback contradicts itself between runs.
• Write the axes down and freeze them. A rubric is only useful if two runs agree. The same discipline that makes a public benchmark reproducible applies to an in-loop evaluator: pin the axes, make each one binary, and version them. Open, community-editable scorecards like BuilderProof's benchmarks are a good model for what "reproducible pass/fail axes" looks like in practice.

If your rubric produces different verdicts on the same draft across runs, it is too subjective to drive a loop. Tighten the axes before you add iterations.

LLM evaluator vs. code evaluator

The core lesson from Anthropic's Building Effective Agents is that the evaluator-optimizer pattern earns its keep only when the evaluation criteria are clear. Sometimes those criteria are mechanical, and then the evaluator should be code, not a model call:

• The output must be valid JSON that matches a schema. Validate it. Feed the validation error back as feedback.
• The output is code that must pass tests. Run the tests. Feed failing test names and tracebacks back as feedback.
• The output must satisfy a regex, a length bound, or a linter. Check it directly.

A deterministic evaluator is cheaper, faster, and never flaky. Reserve the LLM evaluator for genuinely subjective axes: tone, factual nuance, whether an explanation is actually clear. Many production loops are hybrid: code checks the hard constraints, an LLM checks the soft ones, and the feedback strings are concatenated. Frameworks like LangGraph formalize this as a graph with a conditional edge back to the generator node, which is worth adopting once your loop grows more than one evaluator.

Verify your setup

Before you trust the loop, confirm the SDK is installed and the key is wired:

python -c "import anthropic; print('anthropic', anthropic.__version__)"
python -c "import os; assert os.environ.get('ANTHROPIC_API_KEY'), 'set ANTHROPIC_API_KEY'; print('key ok')"

Then run the script. If you see a status line and a revised answer, the loop is working. If _parse_json raises, print the raw evaluator text once and confirm the rubric is asking for JSON only; that is the single most common break.

Limitations and open questions

• The evaluator is a model and can be wrong. It can pass a bad draft or reject a good one. Treat PASS as "cleared the rubric", not "correct". For anything high-stakes, keep a human or a deterministic check on the final output.
• Cost and latency scale with iterations. Every loop is at least two calls. Cap iterations and prefer the smallest capable model for the evaluator; Claude Haiku 4.5 is often enough to grade a rubric even when the generator uses a stronger model.
• Shared blind spots. When generator and evaluator are the same model family, they can share a failure mode and agree on something wrong. Varying the evaluator model, or using a code evaluator for the parts you can, mitigates this.
• Open question: how to detect useful disagreement versus noise when the evaluator flip-flops. The plateau guard above handles identical feedback, but near-identical, reworded feedback still wastes a loop. Semantic similarity on successive feedback strings is a promising next guard.