Passive Testing

Active testing verifies your setup works. Passive testing evaluates traces from real user interactions against your test cases. You get visibility into AI quality, including edge cases and real-world scenarios that synthetic tests can never cover.

How it works

1. Your AI system handles a real user interaction.
2. Your system sends a canonical trace (input, output, and any execution details) to Mibo’s trace ingestion endpoint.
3. Mibo schedules passive evaluation for applicable test cases that are Active and eligible for passive evaluation.
4. Review the results in your quality dashboard. Passive evaluation does not send another request to your system.

Tip

For evaluation timing and replay limits, see Your API response codes and retries and OTLP passive test execution.

Setting it up

Create an Agent before you open its Traces page. If you have not connected your system, follow Add an Agent. Passive testing works with n8n, Flowise, and HTTP API Agents. The Agent type controls active-testing semantics. Passive ingestion is separate.

The Traces page combines test-run traces and live traffic. Use the Live traffic filter to focus on passive evaluations. Its empty state follows the same order as the steps below:

1. Create an API key

   Follow Creating an API key. You use this key to authenticate trace requests. You can create the key before verifying your email, but verify your email before sending the first trace. Until then, Mibo rejects the request with 403 EMAIL_NOT_VERIFIED before storing or evaluating it. Return to Mibo and click I’ve verified my email in the Verify your email banner after opening the verification link.

   For Your API, scope the key to a single agent if you can — it removes the need to specify platformId on requests. For OTLP, set the agent’s OTLP Service Name to match your exporter’s service.name; key scope only authorizes the matched agent. See API Keys & Trace Routing.

2. Keep a passive test active

   Open the Agent’s Tests page and click Create new, or open an existing test. In General Configuration, set Passive behavior to Always run or Run when relevant. These settings make a test eligible for passive evaluation. The Manual runs only setting does not evaluate real traces. Keep the test Active in the test list; Disabled tests are skipped. Save the test before sending traces. Multi-turn tests run manually only, so use a single-turn test for passive evaluation. See Creating Test Cases.

3. Send traces and review results

   Send traces after you create an API key and save an Active, passive-eligible test. The trace appears on the Agent’s Traces page after Mibo receives it. Choose Live traffic to find it among passive evaluations. Evaluated results appear after the passive run when a test applies. See Sending Traces for the decision matrix.

The two paths

OpenTelemetryStream traces from any OTel-instrumented stack.

Your APIPOST the canonical {spans:[...]} JSON directly.

Both arrive at the same endpoint, both produce the same canonical trace internally, and both feed identical assertion evaluation.

The ingestion endpoint

POST https://api.mibo-ai.com/public/traces

Request — Custom API example

curl -X POST "https://api.mibo-ai.com/public/traces" \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "x-request-id: chat-001" \
  -d '{
    "spans": [
      {
        "span_id": "root-001",
        "name": "Customer Support Agent",
        "attributes": {
          "gen_ai.response.text": "We are open Monday to Friday, 9am to 5pm."
        },
        "status": { "code": 1 }
      }
    ]
  }'

For the OTLP envelope shape, see OTLP ingestion.

Required

• x-api-key header: your project API key.
• Content-Type: application/json.
• A body matching either Your API ({ spans: [...] }) or OTLP ({ resourceSpans: [...] }). Any other shape returns 400.

Optional

• x-request-id header: the trace’s external identifier. See Trace identity for the rules for each ingestion path.
• platformId (top-level UUID): explicit Your API agent target.
• metadata.mibo.platform_id (Your API): an alternative explicit target for a multi-agent key. Other metadata fields do not affect assertion evaluation; put assertion data in span attributes.
• Content-Encoding: gzip: compressed body. Useful for large traces.

Agent resolution

Mibo resolves the target differently for each path:

• Your API: a key scoped to one agent selects it. Otherwise, include platformId at the top level or metadata.mibo.platform_id.
• OTLP: service.name must match the agent’s OTLP Service Name. mibo.platform_id is not used for OTLP routing. The API key must also be allowed to write to the matched agent.

If the target is missing, unmatched, or unauthorized, the request returns a routing error.

Tip

For Your API, the simplest setup is one API key per agent. OTLP still needs a matching service.name even when the key is limited to one agent.

What happens after a trace is received

When Mibo receives a new trace:

1. The trace data is stored.
2. Mibo checks whether the Agent has Active tests that are eligible for passive evaluation.
3. Mibo selects the applicable tests and schedules passive evaluation. OTLP waits for its debounce window so additional batches can be merged first.
4. The worker evaluates the trace against the selected test cases. A Run when relevant test can be skipped when its scenario does not match the interaction.
5. Results appear in your dashboard, just like active test results. If no test applies, the trace is still stored but has no evaluated result.

The evaluation happens asynchronously when at least one test applies. The trace endpoint responds immediately with a 201 for a new trace, or 200 when updating an existing trace. Evaluation runs in the background.

Note

On the Traces page, choose Live traffic to show passive evaluations. These traces have a Passive badge. Choose Test runs to show traces captured during active tests. They have an Active badge and show Not checked because passive testing does not run for them.

If a trace appears without an evaluated result

A trace receipt and an evaluated result are separate milestones. If the trace appears on the Agent’s Traces page but no result appears:

1. Wait through the ingestion delay, then refresh the dashboard. For OTLP, the debounce period lasts about five seconds after the final batch arrives, with a 60-second cap from the first batch. Evaluation starts asynchronously after that period. If the result is still pending, keep waiting and refreshing until evaluation finishes before you change the test configuration or resend the trace.
2. Open the Agent’s Tests page.
3. Check that the test status is Active, not Disabled.
4. Open the test and, under General Configuration, set Passive behavior to Always run or Run when relevant. Manual runs only never evaluates live traces.
5. Save the test. For Run when relevant, make sure the trace’s user interaction matches the test’s Scenario description. Otherwise Mibo skips the test.
6. Send a trace with a fresh identity after saving the change. For Your API, use a new x-request-id. For OTLP, use a new traceId and do not reuse an x-request-id header. Mibo does not reevaluate a previously accepted trace after this configuration change. Retrying its accepted payload with the same identity is not a new trace: Your API updates the existing trace, while OTLP merges or deduplicates the payload.

If the trace does not appear at all, check the ingestion response and follow the retry steps in Your API response codes and retries or OTLP troubleshooting.

If scenario routing is unavailable

The Agent’s Traces page can show Routing unavailable for a completed passive run. Open the trace to see No conditional checks were evaluated. Mibo still runs universal checks, including tests set to Always run, but it cannot evaluate Run when relevant tests until scenario routing recovers.

Use a new trace to check whether scenario routing has recovered:

1. For Your API, use a new x-request-id. For OTLP, use a new traceId and do not reuse an x-request-id header.
2. Check the new trace after the passive run completes.

If the new trace also shows Routing unavailable, wait and repeat these steps with another fresh identity.

For a predictable retry, do not resend an accepted trace with the same identity. Your API updates the existing trace. OTLP deduplicates a later batch without another passive evaluation when every span ID in the batch is already stored. A batch with a new span ID can schedule another passive evaluation, but use a new traceId for a deliberate retry. For the full identity and retry rules, see If a trace appears without an evaluated result.

If no input was available for scenario routing

The trace detail page can show No user input was available, so conditional tests were not routed for this trace. Mibo cannot compare the trace with the Scenario description when it cannot extract input. Tests set to Always run still run, but Run when relevant tests do not.

Add the user’s message to the span that carries your AI system’s input before sending the next trace. Use gen_ai.input.messages for an OpenTelemetry message array or gen_ai.prompt for a plain-text fallback. Mibo uses the latest user message in gen_ai.input.messages; each message’s text belongs in a parts array. The Canonical Trace Reference lists both input attributes.

This illustrative span-attributes fragment shows the message structure. For a complete payload, see the Canonical Trace Reference.

{
  "gen_ai.input.messages": [
    {
      "role": "user",
      "parts": [{ "type": "text", "content": "I want to cancel my subscription" }]
    }
  ],
  "gen_ai.response.text": "I can help you cancel your subscription."
}

For a predictable recovery, wait for the current passive run to finish, then send a new trace with the updated instrumentation. For Your API, use a new x-request-id. For OTLP, use a new traceId and do not reuse an x-request-id header. A later OTLP batch with a new input-bearing span under the same traceId can schedule another evaluation, but use a fresh identity for a deliberate retry.

Recover from a passive semantic evaluation issue

A passive trace can be accepted and still fail to produce a usable semantic evaluator score. Open the evaluated result and read the skipped reason in the semantic assertion row. For Error, Reason, or Reasoning, click Go to test, expand Last Result, and select the semantic assertion. A temporary evaluator outage can mark the check as skipped. Skipped checks do not count toward the overall score, so the result can show Passed even though that check was not evaluated.

To retry after the current passive execution reaches a terminal result:

1. If the issue is an invalid threshold, open the test from the Agent’s Tests page. In the Form tab, set Strictness to a finite number from 0 through 1. In the JSON tab, set threshold to a finite number in the same range. Then click Save.
2. Wait for the current result to finish before sending another trace. Refresh the dashboard while it is pending.
3. Send a new trace after the evaluator is available. For Your API, use a new x-request-id. For OTLP, use a new traceId and do not reuse an x-request-id header.

For accepted-trace replay limits and incremental OTLP evaluation, see the passive trace retry rules.

Which test cases run on a real conversation

Picture your n8n or Flowise assistant handling real people all day. Some ask about pricing, some want a refund, some just say hello. You’ve written a test for each of those situations.

Should your “refund” test run on a conversation where someone only asked for your opening hours? No. It would turn red, even though your assistant did nothing wrong. That’s a false alarm, and a dashboard full of false alarms is one you stop trusting.

To prevent this, every test has one simple setting: Passive behavior. It tells Mibo when to run that test on real conversations. You pick it from a dropdown when you create or edit a test. There are three choices:

Choice	What it means	Good for
Always run	Mibo checks this on every conversation, no matter what the person said.	Rules that should hold for every single reply. “The answer is never empty.” “The assistant always replies in English.”
Run when relevant (default)	Mibo only runs this when the conversation actually matches the situation you described.	Most tests. “When someone asks to cancel, the assistant explains how.” This test won’t run on a “what are your hours?” chat.
Manual runs only	Mibo never runs this on real conversations. It only runs when you press Run yourself.	Tricky inputs you’d only send on purpose. “Send a broken request and check the error message.”

How “Run when relevant” decides

For these tests, Mibo reads each real conversation and compares it to the Scenario you wrote. If it matches, the test runs. If it clearly doesn’t, Mibo skips it for that conversation (you’ll see it counted as skipped, never as a failure).

So your Scenario does two jobs: it tells Mibo how to judge the reply, and it tells Mibo when this test applies. Write it the way you’d explain the situation to a coworker:

• Clear: “The user asks to cancel their subscription.”
• Too vague to match: “Handle the request.”

Tip

When Mibo isn’t sure a conversation matches, it skips the test instead of guessing. A skipped test is always better than a red one that isn’t a real problem.

Choosing it (no code needed)

1. Open a test in the editor, or create a new one.
2. In the form, find Passive behavior under General Configuration.
3. Pick Always run, Run when relevant, or Manual runs only.

If you don’t touch it, the test stays on Run when relevant, which is the safe default for most tests. That’s all there is to it.

Comfortable editing the raw test?

If you switch to the JSON tab, this same setting is the passive_class field. Its values are invariant (Always run), conditional (Run when relevant), and active_only (Manual runs only). Leaving it out is the same as conditional.

{
  "scenario": "When the user asks to cancel their subscription, the agent explains how",
  "passive_class": "conditional",
  "user_message": "I want to cancel my subscription",
  "assertions": {
    "semantic": [
      {
        "criteria": "The agent explains how to cancel and does not refuse outright",
        "threshold": 0.8
      }
    ]
  }
}

This setting only affects real conversations. Manual runs ignore it because you choose the input on purpose.

Missing instrumentation, not silent pass/fail

If a token_limit, http_status, or node_call assertion can’t find the attribute it needs in your trace, Mibo flags it as missing instrumentation rather than passing or failing silently. The dashboard renders it distinctly so you know to add the right attribute to your spans.

The attributes Mibo reads are listed under each path’s deep-dive:

• OTLP — what Mibo reads from your spans
• Your API — attributes Mibo reads

Use cases

Production monitoring

Connect your system’s logging pipeline to Mibo. New trace submissions are evaluated against your test cases, giving you continuous quality visibility without another request to your system.

Post-mortem analysis

To investigate a production problem, submit a new trace for the interaction to evaluate it against applicable test cases. See Your quality dashboard for how to inspect the results.

Shadow testing

Evaluate production traffic against new or updated test cases before deploying changes. Catch regressions early by comparing quality scores over time.

Note

Passive testing requires an API key. See API Keys & Trace Routing for how to create and scope them.

Sending TracesDecision matrix: OTLP vs. Your API.

Assertion ReferenceAll assertion types and matchers.