Parallelization Pattern in Python: Fan-Out LLM Calls (2026)

Quick answer (July 28, 2026): The parallelization workflow runs independent LLM calls at the same time and then aggregates their results, instead of waiting for each call in turn. It shows up in two shapes: sectioning (split one task into independent subtasks and run each concurrently) and voting (run the same prompt several times and combine the answers). In Python you build it with asyncio.gather, cap it with a Semaphore, and make it survive rate limits with exponential backoff. Wall-clock time drops to roughly your slowest single call rather than the sum of all of them.

This is the parallelization spoke of our agentic-workflow-patterns series. If you have not read the pillar yet, start with agentic-workflow-patterns-python for the five-pattern map, then come back here.

We use three tools in the code below. Python 3.11+ for asyncio, the Anthropic Python SDK for the calls, and the same shape works with OpenAI's async client with one import swap. Nothing here is framework specific.

What is the parallelization pattern?

Parallelization is one of the five workflow patterns in Anthropic's Building Effective Agents (December 2024). The rule of thumb: if two LLM calls do not depend on each other's output, they should not run one after the other.

Two flavors cover almost every real case:

• Sectioning. Break a task into independent subtasks and run each in its own call. A code review that checks security, performance, and style is three prompts that never read each other's output. Fan them out.
• Voting. Run the same prompt several times, then reduce the answers to one. Useful for classification, guardrail checks, or anything where a single sample is noisy and a majority is more reliable.

Sequential code that ignores this pays for it in latency. Three calls at roughly three seconds each take about nine seconds in a row and about three seconds in parallel. The savings grow with fan-out width.

Prerequisites

• Python 3.11 or newer.
• pip install anthropic (or pip install openai; the pattern is identical).
• An API key in your environment: export ANTHROPIC_API_KEY=sk-ant-....
• Expected outcome: a runnable script whose wall-clock time equals your slowest call, not the sum, and that does not fall over when the API returns HTTP 429.

a runnable async fan-out

Start with one async helper for a single call, then fan out with asyncio.gather. gather schedules every coroutine on the event loop at once and resolves when all of them finish. This is the asyncio.gather primitive doing the work; the LLM SDK is incidental.

import asyncio
from anthropic import AsyncAnthropic

client = AsyncAnthropic()          # reads ANTHROPIC_API_KEY
MODEL = "claude-sonnet-4-5"        # set to your current model id

async def ask(prompt: str) -> str:
    msg = await client.messages.create(
        model=MODEL,
        max_tokens=512,
        messages=[{"role": "user", "content": prompt}],
    )
    return msg.content[0].text

async def review(diff: str) -> dict[str, str]:
    sections = {
        "security":    f"Review this diff for security issues ONLY:\n{diff}",
        "performance": f"Review this diff for performance issues ONLY:\n{diff}",
        "style":       f"Review this diff for style and readability ONLY:\n{diff}",
    }
    answers = await asyncio.gather(*(ask(p) for p in sections.values()))
    return dict(zip(sections.keys(), answers))

if __name__ == "__main__":
    out = asyncio.run(review("def f(x): return eval(x)"))
    for k, v in out.items():
        print(k, "->", v[:80])

Three prompts, one round trip in wall-clock terms. That is sectioning.

bound the concurrency with a Semaphore

asyncio.gather has no idea your provider has rate limits. Fan out fifty prompts and you will fire fifty requests in the same instant, which is the fastest way to get throttled. An asyncio.Semaphore caps how many run at once.

sem = asyncio.Semaphore(4)         # at most 4 concurrent requests

async def ask_bounded(prompt: str) -> str:
    async with sem:
        return await ask(prompt)

Now gather can hold a thousand coroutines and only four touch the API at a time. Pick the cap from your account's requests-per-minute limit, not from how many tasks you have. Concurrency is a property of the provider, not the workload.

survive rate limits with backoff

Even a bounded fan-out hits a 429 eventually, on a shared key or a burst. The fix is exponential backoff with jitter: on a RateLimitError, wait, then retry with a longer wait each time. Both SDKs raise a typed error on HTTP 429; see OpenAI's rate-limit guide (2026) for the equivalent class and the recommended backoff curve.

import random
from anthropic import RateLimitError

async def ask_resilient(prompt: str, retries: int = 5) -> str:
    delay = 1.0
    for attempt in range(retries):
        try:
            async with sem:
                return await ask(prompt)
        except RateLimitError:
            if attempt == retries - 1:
                raise
            await asyncio.sleep(delay + random.uniform(0, 0.5))
            delay *= 2                # 1s, 2s, 4s, 8s ...
    raise RuntimeError("unreachable")

The jitter matters: without it, every retry in a fan-out wakes up at the same moment and stampedes the API again. For OpenAI, swap the import for from openai import AsyncOpenAI, RateLimitError and the body is unchanged.

voting aggregation

Sectioning splits the work. Voting duplicates it. Run the same prompt n times and reduce, which trades tokens for reliability on noisy single-sample tasks.

from collections import Counter

async def vote(prompt: str, n: int = 5) -> str:
    answers = await asyncio.gather(*(ask_resilient(prompt) for _ in range(n)))
    normalized = [a.strip().lower() for a in answers]
    return Counter(normalized).most_common(1)[0][0]

Use voting for classification and guardrail decisions, not for open-ended generation where there is no single right answer to count.

When NOT to parallelize

Parallelization is a latency and reliability tool, not a default. Skip it when:

• The calls are dependent. If call B needs call A's output, this is prompt chaining, not parallelization. See agentic-workflow-patterns-python for the chaining pattern.
• The number of subtasks is dynamic. If a model has to decide how to split the work at runtime, you want the orchestrator-workers pattern instead; walk through it in orchestrator-workers-python.
• Fan-out is one call. Concurrency overhead is not free. A single call gains nothing from gather.
• Cost is the constraint, not latency. Voting five times is five times the tokens. Parallelism hides the wall-clock cost, not the bill.

Troubleshooting

RateLimitError / HTTP 429 under load. You fanned out past your requests-per-minute limit. Lower the Semaphore value and keep the backoff from Step 3. The semaphore prevents most 429s; the retry catches the rest.

One failed call kills the whole batch. By default asyncio.gather cancels everything when any coroutine raises. Pass return_exceptions=True to collect partial results instead:

results = await asyncio.gather(
    *(ask_resilient(p) for p in prompts),
    return_exceptions=True,
)
ok = [r for r in results if not isinstance(r, Exception)]

RuntimeError: asyncio.run() cannot be called from a running event loop. You are inside a notebook or an existing loop. Use await directly in Jupyter, or nest_asyncio.apply(), rather than nesting asyncio.run.

Verify your parallelization works

Time it. If the fan-out is real, wall time tracks your slowest call, not the sum.

import time

async def main():
    start = time.perf_counter()
    await review("def f(x): return eval(x)")
    print(f"wall time: {time.perf_counter() - start:.2f}s")

asyncio.run(main())

Three sequential calls print near nine seconds. The gather version prints near three. If yours prints nine, your calls are still running in series, usually because an await is sitting in a loop instead of inside a gather.

Limitations and open questions

• The Semaphore cap is a static guess. Provider limits shift by tier and by model; a token-bucket limiter that reads the retry-after header is more precise for heavy production fan-out.
• Voting cost scales linearly with n. There is no free reliability; measure whether three samples already capture most of the gain before paying for seven.
• asyncio parallelizes I/O-bound waits, not CPU work. If your aggregation step is heavy, move it off the event loop.

Parallelization is the cheapest latency win in the agentic-workflow toolkit once the calls are genuinely independent. Get the independence check right first; the gather is the easy part.