Passive Testing

Passive testing is how Mibo evaluates your Agent. Your system sends traces of real interactions, and Mibo checks them against your test cases. Mibo never calls your Agent. You get visibility into AI quality on real-world scenarios, including edge cases that synthetic tests miss.

How it works

1. Your AI system handles a real user interaction.
2. Your system sends a canonical trace (input, output, and any execution details) to Mibo’s trace ingestion endpoint.
3. Mibo schedules evaluation for the enabled test cases that apply to the trace.
4. Review the results in your quality dashboard.

Tip

For evaluation timing and replay limits, see Your API response codes and retries and OTLP passive test execution.

Setting it up

Create an Agent before you open its Traces page. If you have not set one up, follow Set up your Agent. Passive testing works the same way for every Agent type.

Until the Agent has an evaluated trace, its Traces page shows a setup checklist in the same order as the steps below:

1. Create an API key

   Follow Creating an API key. You use this key to authenticate trace requests. You can create the key before verifying your email, but verify your email before sending the first trace. Until then, Mibo rejects the request with 403 EMAIL_NOT_VERIFIED before storing or evaluating it. Return to Mibo and click I’ve verified my email in the Verify your email banner after opening the verification link.

   For Your API, scope the key to a single agent if you can — it removes the need to specify platformId on requests. For OTLP, set the agent’s OTLP Service Name to match your exporter’s service.name; key scope only authorizes the matched agent. See API Keys & Trace Routing.

2. Enable a test

   Open the Agent’s Tests page and click Create new, or open an existing test. In General Configuration, set Passive behavior to Always run or Run when relevant. Keep the test Enabled in the test list; Disabled tests are skipped. Save the test before sending traces. See Creating Test Cases.

3. Send traces and review results

   Send traces after you create an API key and save an enabled test. Evaluated traces appear on the Agent’s Traces page after the run completes, when a test applies. See Sending Traces for the decision matrix.

The two paths

OpenTelemetryStream traces from any OTel-instrumented stack.

Your APIPOST the canonical {spans:[...]} JSON directly.

Both arrive at the same endpoint, both produce the same canonical trace internally, and both feed identical assertion evaluation.

The ingestion endpoint

POST https://api.mibo-ai.com/public/traces

Request — Custom API example

curl -X POST "https://api.mibo-ai.com/public/traces" \
  -H "Content-Type: application/json" \
  -H "x-api-key: <YOUR_API_KEY>" \
  -H "x-request-id: chat-001" \
  -d '{
    "spans": [
      {
        "span_id": "root-001",
        "name": "Customer Support Agent",
        "attributes": {
          "gen_ai.response.text": "We are open Monday to Friday, 9am to 5pm."
        },
        "status": { "code": 1 }
      }
    ]
  }'

For the OTLP envelope shape, see OTLP ingestion.

Required

• x-api-key header: your workspace API key.
• Content-Type: application/json.
• A body matching either Your API ({ spans: [...] }) or OTLP ({ resourceSpans: [...] }). Any other shape returns 400.

Optional

• x-request-id header: the trace’s external identifier. See Trace identity for the rules for each ingestion path.
• platformId (top-level UUID): explicit Your API agent target.
• metadata.mibo.platform_id (Your API): an alternative explicit target for a multi-agent key. Other metadata fields do not affect assertion evaluation; put assertion data in span attributes.
• Content-Encoding: gzip: compressed body. Useful for large traces.

Agent resolution

Mibo resolves the target differently for each path:

• Your API: a key scoped to one agent selects it. Otherwise, include platformId at the top level or metadata.mibo.platform_id.
• OTLP: a single-Agent key selects that Agent. Other keys match service.name within their allowed Agents. A key for all agents in the workspace can fall back to its only active non-demo Agent when the service name is unconfigured. OTLP does not use mibo.platform_id.

If the target is missing, unmatched, or unauthorized, the request returns a routing error.

Tip

For Your API, the simplest setup is one API key per agent. OTLP also routes a single-Agent key directly to that Agent. Other keys follow OTLP routing.

What happens after a trace is received

When Mibo receives a new trace:

1. Mibo redacts secrets, and personal data if your workspace enables it, then stores the trace. See Data handling.
2. Mibo checks whether the Agent has enabled tests.
3. Mibo selects the applicable tests and schedules passive evaluation. OTLP waits for its debounce window so additional batches can be merged first.
4. Mibo evaluates the trace against the selected test cases. A Run when relevant test can be skipped when its scenario does not match the interaction.
5. Results appear in your dashboard. If no test applies, the trace is still stored but has no evaluated result.

The evaluation happens asynchronously when at least one test applies. The trace endpoint responds immediately with a 201 for a new trace, or 200 when updating an existing trace. Evaluation runs in the background.

If a trace appears without an evaluated result

A trace receipt and an evaluated result are separate milestones. If the trace appears on the Agent’s Traces page but no result appears:

1. Wait through the ingestion delay, then refresh the dashboard. For OTLP, the debounce period lasts about five seconds after the final batch arrives, with a 60-second cap from the first batch. Evaluation starts asynchronously after that period. If the result is still pending, keep waiting and refreshing until evaluation finishes before you change the test configuration or resend the trace.
2. Open the Agent’s Tests page.
3. Check that the test is Enabled, not Disabled.
4. Open the test and, under General Configuration, check that Passive behavior is Always run or Run when relevant.
5. Save the test. For Run when relevant, make sure the trace’s user interaction matches the test’s Scenario description. Otherwise Mibo skips the test.
6. Send a trace with a fresh identity after saving the change. For Your API, use a new x-request-id. For OTLP, use a new traceId and do not reuse an x-request-id header. Mibo does not reevaluate a previously accepted trace after this configuration change. Retrying its accepted payload with the same identity is not a new trace: Your API updates the existing trace, while OTLP merges or deduplicates the payload.

If the trace does not appear at all, check the ingestion response and follow the retry steps in Your API response codes and retries or OTLP troubleshooting.

If scenario routing is unavailable

The Agent’s Traces page can show Routing unavailable for a completed passive run. Open the trace to see No conditional checks were evaluated. Mibo still runs universal checks, including tests set to Always run, but it cannot evaluate Run when relevant tests until scenario routing recovers.

Use a new trace to check whether scenario routing has recovered:

1. For Your API, use a new x-request-id. For OTLP, use a new traceId and do not reuse an x-request-id header.
2. Check the new trace after the passive run completes.

If the new trace also shows Routing unavailable, wait and repeat these steps with another fresh identity.

For a predictable retry, do not resend an accepted trace with the same identity. Your API updates the existing trace. OTLP deduplicates a later batch without another passive evaluation when every span ID in the batch is already stored. A batch with a new span ID can schedule another passive evaluation, but use a new traceId for a deliberate retry. For the full identity and retry rules, see If a trace appears without an evaluated result.

If no input was available for scenario routing

The Agent’s Traces list can show No usable input for a trace. Open the trace to see No conditional checks were evaluated and This trace had no usable user input. Mibo cannot compare the trace with the Scenario description when it cannot extract input. Tests set to Always run still run, but Run when relevant tests do not.

Add the user’s message to the span that carries your AI system’s input before sending the next trace. Use gen_ai.input.messages for an OpenTelemetry message array or gen_ai.prompt for a plain-text fallback. Mibo uses the latest user message in gen_ai.input.messages; each message’s text belongs in a parts array. The Canonical Trace Reference lists both input attributes.

This illustrative span-attributes fragment shows the message structure. For a complete payload, see the Canonical Trace Reference.

{
  "gen_ai.input.messages": [
    {
      "role": "user",
      "parts": [{ "type": "text", "content": "I want to cancel my subscription" }]
    }
  ],
  "gen_ai.response.text": "I can help you cancel your subscription."
}

For a predictable recovery, wait for the current passive run to finish, then send a new trace with the updated instrumentation. For Your API, use a new x-request-id. For OTLP, use a new traceId and do not reuse an x-request-id header. A later OTLP batch with a new input-bearing span under the same traceId can schedule another evaluation, but use a fresh identity for a deliberate retry.

Recover from a passive semantic evaluation issue

A passive trace can be accepted and still fail to produce a usable semantic evaluator score. Open the evaluated result and read the skipped reason in the semantic assertion row. For Error, Reason, or Reasoning, click View test, expand Last Result, and select the semantic assertion. A temporary evaluator outage can mark the check as skipped. Skipped checks do not count toward the overall score, so the result can show Passed even though that check was not evaluated.

To retry after the current passive execution reaches a terminal result:

1. If the issue is an invalid threshold, open the test from the Agent’s Tests page. In the Form tab, set Strictness to a finite number from 0 through 1. In the JSON tab, set threshold to a finite number in the same range. Then click Save.
2. Wait for the current result to finish before sending another trace. Refresh the dashboard while it is pending.
3. Send a new trace after the evaluator is available. For Your API, use a new x-request-id. For OTLP, use a new traceId and do not reuse an x-request-id header.

For accepted-trace replay limits and incremental OTLP evaluation, see the passive trace retry rules.

Which test cases run on a real conversation

Picture your n8n or Flowise assistant handling real people all day. Some ask about pricing, some want a refund, some just say hello. You’ve written a test for each of those situations.

Should your “refund” test run on a conversation where someone only asked for your opening hours? No. It would turn red, even though your assistant did nothing wrong. That’s a false alarm, and a dashboard full of false alarms is one you stop trusting.

To prevent this, every test has one simple setting: Passive behavior. It tells Mibo when to run that test on real conversations. You pick it from a dropdown when you create or edit a test. There are two choices:

Choice	What it means	Good for
Always run	Mibo checks this on every conversation, no matter what the person said.	Rules that should hold for every single reply. “The answer is never empty.” “The assistant always replies in English.”
Run when relevant (default)	Mibo only runs this when the conversation actually matches the situation you described.	Most tests. “When someone asks to cancel, the assistant explains how.” This test won’t run on a “what are your hours?” chat.

How “Run when relevant” decides

For these tests, Mibo reads each real conversation and compares it to the Scenario you wrote. If it matches, the test runs. If it clearly doesn’t, Mibo doesn’t evaluate it for that conversation. It never counts as a failure.

So your Scenario does two jobs: it tells Mibo how to judge the reply, and it tells Mibo when this test applies. Write it the way you’d explain the situation to a coworker:

• Clear: “The user asks to cancel their subscription.”
• Too vague to match: “Handle the request.”

Tip

When Mibo isn’t sure a conversation matches, it skips the test instead of guessing. A skipped test is always better than a red one that isn’t a real problem.

Choosing it (no code needed)

1. Open a test in the editor, or create a new one.
2. In the form, find Passive behavior under General Configuration.
3. Pick Always run or Run when relevant.

If you don’t touch it, the test stays on Run when relevant, which is the safe default for most tests. That’s all there is to it.

Comfortable editing the raw test?

If you switch to the JSON tab, this same setting is the passive_class field. Its values are invariant (Always run) and conditional (Run when relevant). Leaving it out is the same as conditional.

{
  "scenario": "When the user asks to cancel their subscription, the agent explains how",
  "passive_class": "conditional",
  "user_message": "I want to cancel my subscription",
  "assertions": {
    "semantic": [
      {
        "criteria": "The agent explains how to cancel and does not refuse outright",
        "threshold": 0.8
      }
    ]
  }
}

Missing instrumentation, not silent pass/fail

If a token_limit, http_status, or node_call assertion can’t find the attribute it needs in your trace, Mibo flags it as missing instrumentation rather than passing or failing silently. The dashboard renders it distinctly so you know to add the right attribute to your spans.

The attributes Mibo reads are listed under each path’s deep-dive:

• OTLP — what Mibo reads from your spans
• Your API — attributes Mibo reads

Use cases

Production monitoring

Connect your system’s logging pipeline to Mibo. New trace submissions are evaluated against your test cases, giving you continuous quality visibility without any request to your system.

Post-mortem analysis

To investigate a production problem, submit a new trace for the interaction to evaluate it against applicable test cases. See Your quality dashboard for how to inspect the results.

Shadow testing

Send traces from a staging Agent to evaluate changes before you deploy them. Compare quality scores over time to catch regressions early.

Note

Passive testing requires an API key. See API Keys & Trace Routing for how to create and scope them.

Sending TracesDecision matrix: OTLP vs. Your API.

Assertion ReferenceAll assertion types and matchers.