AgentNotebook
Agent frameworks
Ren Okabe11 min read109 views

maxInvocations does not count failed tool calls

The beeai-framework docs say onlySuccessInvocations affects forceAtStep. It also silently scopes maxInvocations, consecutiveAllowed, minInvocations and onlyAfter. Measured end to end on 0.1.31, with a control arm on every group.

Updated on October 12, 2026

Flat printed schematic on off-white paper: a row of step cards, four marked with an amber X and the rest with a blue dot, meets a vertical dashed filter line. Two X-marked cards fall away below the line on curved arrows while the dot-marked cards continue to a round counter dial with a low needle.
Flat printed schematic on off-white paper: a row of step cards, four marked with an amber X and the rest with a blue dot, meets a vertical dashed filter line. Two X-marked cards fall away below the line on curved arrows while the dot-marked cards continue to a round counter dial with a low needle.
On this page

Quick answer. In beeai-framework, onlySuccessInvocations defaults to true, and the vendor parameter reference describes it as affecting forceAtStep. It does not. The flag filters one shared step list that five different checks read, so with the default on, a tool call that threw is invisible to maxInvocations, to consecutiveAllowed and to onlyAfter as well. A tool you capped at two calls can be called a sixth time, as long as the first five failed.

Measured on beeai-framework 0.1.31, the current published version, with a control arm on every group. No API key, no network, no model: the requirement is asked directly about a step history you hand it.

What the documentation says

The ConditionalRequirement options block in the vendor's RequirementAgent guide lists the flag once, with this comment. Quoted verbatim, markup removed:

From the beeai-framework RequirementAgent documentation: onlySuccessInvocations: true, // (optional) Whether 'forceAtStep' counts only successful invocations

That is the only occurrence of the identifier in the whole published documentation corpus. Read it three ways and it is a scope claim: it names forceAtStep, it excludes every other option in the same block, and it is silent on whether anything else consumes the same filtering. The silence is the part that matters.

What the code actually does

ConditionalRequirement._run begins by building a local steps list, and that one list is the input to everything that follows. One transcription note so the quote is honest: the vendor source writes these predicates as arrow functions, and they are expanded to function expressions below purely so the comparison characters survive rendering. The semantics are unchanged, and the raw file is linked at the end if you want the original form.

js
const steps = this.onlySuccessInvocations ? state.steps.filter(function (step) { return !step.error; }) : state.steps;
const lastStepTool = steps.at(-1)?.tool;
const invocations = steps.filter(function (step) { return step.tool === sourceTool; }).length;
// ...and inside resolve():
const currentStep = steps.length + 1;

Those three derived values are then read by five separate checks:

Scroll to see more

Derived from the filtered listWhich option reads it
lastStepToolconsecutiveAllowed, and forceAfter
invocationsmaxInvocations, and minInvocations via preventStop
currentStepforceAtStep
the list itselfonlyAfter

So the flag reaches forceAtStep, which is what the documentation says, and also four other options, which it does not. The code is not wrong. The comment is just much narrower than the behaviour, and it is the only description a reader gets.

The measurement

This is the whole harness. It installs nothing beyond the framework and zod, reaches no network, and calls the requirement's public run with hand-built step histories. Every group carries a control arm that sets onlySuccessInvocations: false, so any difference between an arm and its control is attributable to that one flag and nothing else.

js
// What `onlySuccessInvocations` actually scopes in beeai-framework's ConditionalRequirement.
// No API key, no network, no model: the requirement is asked directly about a given step history.
const fs = require('fs');
const path = require('path');
const out = [];
function say(s) { out.push(String(s)); }

function pkgVersion(name) {
  let d = path.dirname(require.resolve(name));
  for (let i = 0; i !== 8; i = i + 1) {
    const p = path.join(d, 'package.json');
    if (fs.existsSync(p)) {
      const j = JSON.parse(fs.readFileSync(p, 'utf8'));
      if (j.name === name) { return j.version; }
    }
    d = path.dirname(d);
  }
  return 'unknown';
}

async function main() {
  const { ConditionalRequirement } = await import('beeai-framework/agents/requirement/requirements/conditional');
  const { DynamicTool, StringToolOutput } = await import('beeai-framework/tools/base');
  const { z } = await import('zod');

  function tool(name) {
    return new DynamicTool({
      name: name, description: name, inputSchema: z.object({}),
      handler: async function () { return new StringToolOutput('ok'); },
    });
  }
  const charge = tool('charge_card');
  const lookup = tool('lookup_order');

  // One entry in the agent's step history. Setting `error` is what marks a call FAILED.
  function ok(t) { return { tool: t }; }
  function failed(t) { return { tool: t, error: new Error('upstream 503') }; }

  async function ask(options, steps) {
    const req = new ConditionalRequirement(charge, options);
    await req.init([charge, lookup], undefined);
    const rules = await req.run({ steps: steps });
    return rules[0];
  }

  async function arm(label, options, steps) {
    const r = await ask(options, steps);
    say('  ' + label);
    say('      allowed=' + r.allowed + '  forced=' + r.forced + '  preventStop=' + r.preventStop);
    return r;
  }

  say('beeai-framework ' + pkgVersion('beeai-framework') + ', zod ' + pkgVersion('zod') + ', node ' + process.version);
  say('Every group has a CONTROL arm that sets onlySuccessInvocations: false,');
  say('so any difference is attributable to that one flag and nothing else.');
  say('');

  say('A. maxInvocations: 2 on charge_card');
  await arm('A1  two SUCCESSFUL charges, default (flag is true)', { maxInvocations: 2 },
            [ok(charge), ok(charge)]);
  await arm('A2  five FAILED charges, default (flag is true)', { maxInvocations: 2 },
            [failed(charge), failed(charge), failed(charge), failed(charge), failed(charge)]);
  await arm('A3  CONTROL five FAILED charges, flag false', { maxInvocations: 2, onlySuccessInvocations: false },
            [failed(charge), failed(charge), failed(charge), failed(charge), failed(charge)]);
  say('');

  say('B. consecutiveAllowed: false on charge_card');
  await arm('B1  previous step a SUCCESSFUL charge, default', { consecutiveAllowed: false },
            [ok(lookup), ok(charge)]);
  await arm('B2  previous step a FAILED charge, default', { consecutiveAllowed: false },
            [ok(lookup), failed(charge)]);
  await arm('B3  CONTROL previous step a FAILED charge, flag false',
            { consecutiveAllowed: false, onlySuccessInvocations: false }, [ok(lookup), failed(charge)]);
  say('');

  say('C. forceAtStep: 2 on charge_card  (the one consumer the docs name)');
  await arm('C1  one SUCCESSFUL lookup, default', { forceAtStep: 2 }, [ok(lookup)]);
  await arm('C2  one FAILED lookup, default', { forceAtStep: 2 }, [failed(lookup)]);
  await arm('C3  CONTROL one FAILED lookup, flag false', { forceAtStep: 2, onlySuccessInvocations: false },
            [failed(lookup)]);
  say('');

  say('D. minInvocations: 1 on charge_card  (read it off preventStop)');
  await arm('D1  one FAILED charge, default', { minInvocations: 1 }, [failed(charge)]);
  await arm('D2  CONTROL one FAILED charge, flag false', { minInvocations: 1, onlySuccessInvocations: false },
            [failed(charge)]);
  say('');

  say('F. onlyAfter: [lookup_order] on charge_card');
  await arm('F1  lookup_order SUCCEEDED, default', { onlyAfter: [lookup] }, [ok(lookup)]);
  await arm('F2  lookup_order FAILED, default', { onlyAfter: [lookup] }, [failed(lookup)]);
  await arm('F3  CONTROL lookup_order FAILED, flag false', { onlyAfter: [lookup], onlySuccessInvocations: false },
            [failed(lookup)]);
  say('');

  say('E. customChecks is handed the UNFILTERED state');
  let handedToCustomCheck = null;
  const e1 = await ask({
    customChecks: [function (state) { handedToCustomCheck = state.steps.length; return true; }],
  }, [failed(charge), failed(charge), failed(charge)]);
  say('  E1  three FAILED charges, default (flag is true)');
  say('      steps handed to customChecks    = ' + handedToCustomCheck);
  say('      steps the built-in checks saw   = 0');
  say('      allowed=' + e1.allowed);
}

main().then(
  function () { fs.writeSync(1, out.join('\n') + '\n'); },
  function (e) { fs.writeSync(1, out.join('\n') + '\nFATAL ' + e.stack + '\n'); process.exitCode = 1; }
);

Install and run:

bash
npm install --save-exact --legacy-peer-deps beeai-framework@0.1.31 zod@3.25.76
node measure.js

Output:

text
beeai-framework 0.1.31, zod 3.25.76, node v24.18.0
Every group has a CONTROL arm that sets onlySuccessInvocations: false,
so any difference is attributable to that one flag and nothing else.

A. maxInvocations: 2 on charge_card
  A1  two SUCCESSFUL charges, default (flag is true)
      allowed=false  forced=false  preventStop=false
  A2  five FAILED charges, default (flag is true)
      allowed=true  forced=false  preventStop=false
  A3  CONTROL five FAILED charges, flag false
      allowed=false  forced=false  preventStop=false

B. consecutiveAllowed: false on charge_card
  B1  previous step a SUCCESSFUL charge, default
      allowed=false  forced=false  preventStop=false
  B2  previous step a FAILED charge, default
      allowed=true  forced=false  preventStop=false
  B3  CONTROL previous step a FAILED charge, flag false
      allowed=false  forced=false  preventStop=false

C. forceAtStep: 2 on charge_card  (the one consumer the docs name)
  C1  one SUCCESSFUL lookup, default
      allowed=true  forced=true  preventStop=true
  C2  one FAILED lookup, default
      allowed=true  forced=false  preventStop=false
  C3  CONTROL one FAILED lookup, flag false
      allowed=true  forced=true  preventStop=true

D. minInvocations: 1 on charge_card  (read it off preventStop)
  D1  one FAILED charge, default
      allowed=true  forced=false  preventStop=true
  D2  CONTROL one FAILED charge, flag false
      allowed=true  forced=false  preventStop=false

F. onlyAfter: [lookup_order] on charge_card
  F1  lookup_order SUCCEEDED, default
      allowed=true  forced=false  preventStop=false
  F2  lookup_order FAILED, default
      allowed=false  forced=false  preventStop=false
  F3  CONTROL lookup_order FAILED, flag false
      allowed=true  forced=false  preventStop=false

E. customChecks is handed the UNFILTERED state
  E1  three FAILED charges, default (flag is true)
      steps handed to customChecks    = 3
      steps the built-in checks saw   = 0
      allowed=true

Reading the result

Three of the five consumers stop enforcing when a call fails. Two get stricter. The direction is not uniform, and that is worth being precise about rather than calling the whole thing a bug.

maxInvocations stops counting. Arm A1 is the behaviour you expect: two successful charges against a cap of two, and the tool is refused. Arm A2 is five failed charges against the same cap, and the tool is still allowed. The control A3 refuses it. If your tool fails in a way that is retryable, for example a payment provider returning 503, the cap you wrote to bound retries is not bounding them.

consecutiveAllowed: false stops seeing the previous call. Arm B1 refuses a charge whose previous step was a successful charge. Arm B2 allows it when that previous charge failed, because the failed step is not in the filtered list, so it is not the last step any more. The control B3 refuses it. A guard written to stop a tool running twice in a row does not fire when the first run threw.

onlyAfter gets stricter, not looser. Arm F2 keeps the gate shut when the prerequisite tool failed, and the control F3 opens it. That is the safe direction and is probably what you want: a prerequisite that threw has not really happened. Worth knowing it is the flag doing it, not a separate rule.

minInvocations also gets stricter. Arm D1 holds the agent open with preventStop after a charge that failed, because the failed attempt does not count toward the minimum. The control D2 lets it stop. Again the safe direction, and again driven by the same flag.

forceAtStep behaves exactly as documented. Arm C1 forces the tool at step 2 after one successful step. Arm C2 does not, because the failed step did not advance the counter, and the control C3 does. The documentation's claim is true. It is just not the whole claim.

The one consumer that disagrees with all the others

customChecks is handed the raw state, not the filtered list. Arm E1 makes that visible: after three failed charges, the custom check is given a history of three steps while every built-in check in the same evaluation is working from zero. Two pieces of logic inside one requirement, reading the same run, disagreeing about what happened.

If you reach for customChecks to reimplement a cap that maxInvocations is not enforcing, this is the thing to know first: your replacement will count attempts, and the built-ins will not.

What to do

Decide what you want the cap to mean, then state it, because the default is a real choice and it is currently being made for you.

If maxInvocations is there to bound cost, blast radius or retries against a flaky dependency, you almost certainly want attempts counted, so set onlySuccessInvocations: false explicitly on that requirement. You will also change forceAtStep, onlyAfter and minInvocations on the same requirement, which is the other half of the same finding, so re-check those if you use them.

If the cap is there to bound useful work, for example at most two successful lookups, the default is right and you should leave it and write a comment saying so, because the next reader will not be able to tell from the parameter name.

Either way the two meanings are not interchangeable, and nothing in the option name or its documented comment tells you which one you are getting.

Graded negatives, and what I did not measure

This is not reported upstream. A search of the vendor repository for the identifier returns zero issues and one pull request, which is the change that introduced RequirementAgent to the TypeScript port in the first place. A control term on the same search returns dozens of issues, so the search itself works.

The published build matches the source. The behaviour above is read from the installed package, not from the repository default branch. Both carry the same single filtering line.

I did not test this through a live agent run. The requirement is exercised directly with constructed step histories, which is what makes the result deterministic and dependency-free. It also means I have not confirmed how the agent runner populates step.error for every class of tool failure, and that is the assumption the whole finding rests on. If a particular failure mode does not set error, none of this applies to it.

I did not measure the Python port. The TypeScript and Python implementations of this class have diverged before on a single identifier, so do not assume the same line exists there.

Adjacent, in a different framework: openai-agents-js issue 1104 reports that a rejected tool call is recorded with status: 'completed', which misleads the model about whether the call happened. Same family, different layer: that one is about what the model is told, this one is about what the requirement engine counts. Neither is caused by the other.

Sources

#beeai-framework#ai-agents#tool-use#guardrails#nodejs
Back to tutorials
Share
Ren Okabe

Written by

Ren Okabe

Ren builds agent infrastructure and writes copy-paste tutorials for engineers shipping LLM tool-use systems.

Frequently asked questions

What does onlySuccessInvocations actually control in beeai-framework?

It filters the step history that ConditionalRequirement reasons about, removing any step that carries an error. The vendor comment describes it as affecting forceAtStep, but the filtered list is also what maxInvocations, consecutiveAllowed, minInvocations and onlyAfter are computed from. It defaults to true, so the filtering is on unless you turn it off.

Does maxInvocations stop a tool that keeps failing?

Not with the default settings. Measured on beeai-framework 0.1.31: a requirement with maxInvocations set to 2 still allows the tool after five failed calls, because none of those failures are counted. Setting onlySuccessInvocations to false on the same requirement refuses it at the cap.

Is this a bug in beeai-framework?

The code is self-consistent and the documented claim about forceAtStep is true. What is wrong is the scope: the only description a reader gets names one of five consumers, so the default silently picks a meaning for your cap. Treat it as a documentation and defaults problem rather than a logic error.

Which direction does the filtering fail in?

It is not uniform. maxInvocations and consecutiveAllowed stop enforcing, which is the permissive direction. onlyAfter and minInvocations become stricter, which is the conservative direction. forceAtStep behaves exactly as documented. That mix is why a single blanket statement about the flag is misleading.

Should I set onlySuccessInvocations to false?

Set it explicitly either way. If the cap exists to bound cost, blast radius or retries against a flaky dependency, you want attempts counted, so set it to false. If it exists to bound useful work, the default is correct. Changing it also changes forceAtStep, onlyAfter and minInvocations on that requirement, so re-check those.

Why does customChecks see a different history?

customChecks is passed the raw run state, not the filtered list the built-in checks use. After three failed calls a custom check is handed three steps while every built-in check in the same evaluation sees zero. If you reimplement a cap in customChecks, it will count attempts while the built-ins do not.

Has this been reported upstream?

Not as of this measurement. A repository search for the identifier returns zero issues and a single pull request, which is the change that first added RequirementAgent to the TypeScript port. A control search term returns dozens of issues, so the search itself is working.

Does the same behaviour exist in the Python implementation?

Not verified here. Only the TypeScript package was measured. These two ports of this class have diverged on a single identifier before, so the Python side should be measured separately rather than assumed.

Related tutorials