Creating Test Cases

The Test Architect is Mibo’s AI assistant for creating tests. Instead of writing test cases manually, you describe what your AI system should do in plain language, and the AI generates the test scenarios for you. Mibo then evaluates the traces your Agent sends against these tests.

Note

AI features require a verified email. Before verifying, you can open the editor, create a test manually, and click Save. The chat and real-trace generation become available after you verify.

Opening the Test Architect

Navigate to your agent and click Create tests with AI. You’ll see a split-screen interface: a chat panel on the left and a test case editor on the right.

Tip

The Test Architect receives the selected Agent name as context. Include the important details about your system in your request. Then describe the specific scenarios you want to test.

Creating tests through conversation

Start by telling the AI about your system. For example:

“My agent helps users book restaurant reservations. It should ask for the date, time, and party size before confirming the booking.”

The AI will generate a set of test cases based on your description. You can keep the conversation going to refine or add more:

“Also test what happens when the user asks for a date that’s already fully booked.”

“Add a test for when the user doesn’t specify a time.”

Each time you describe a scenario, the Test Architect creates or updates the test cases in the editor panel.

You can also attach JSON files to the conversation, for example an n8n workflow export, a Flowise chatflow config, or a JSON document describing your API. The AI analyzes the structure to detect testable nodes and suggest test targets. You can also paste JSON directly into the chat instead of uploading a file.

Note

Only JSON files are supported (max 500 KB). Other formats like PDFs, Word documents, or plain text files are not accepted.

Attach the original workflow export or paste the complete JSON object when you want the Test Architect to identify workflow steps. It accepts a JSON object or a JSON code block. Pasted JSON, including a raw object or fenced JSON code block, must be 100,000 characters or less to be analyzed. Attached JSON files can be up to 500 KB. For large attached exports, Mibo reduces unrelated parameter detail to retain node, step, and module identities and their connections. If the reduced workflow is still over 50,000 characters, Mibo returns a general result without workflow targets. A top-level JSON array or malformed JSON is not analyzed.

The chat removes secrets from what you send before it reaches the AI model. See Data handling.

Generate tests from a real trace

A real trace records an execution of your Agent, including its input, output, and execution context. Use one to ground generated tests in behavior your Agent has actually produced.

1. In the Test Architect chat, click Use real trace. Before you send your first message, you can also click Attach a real trace to get started.
2. In Ground generation on a real trace, select a completed trace for this Agent, review its preview, and click Use this trace.
3. Mibo asks the Test Architect to generate test cases from the trace. The trace is attached to that message. The AI uses the trace’s real node names and argument keys instead of guessing them.

If the dialog says there are no completed traces for this Agent yet, send a trace first. See Sending Traces.

What a test case contains

Each test case has these parts:

• Name: a short description of what’s being tested.
• Scenario: a description of the situation and what the expected behavior is.
• Input: a representative input for the scenario.
• Assertions: the checks Mibo runs against each trace the test applies to.
• Passive behavior: controls when this test evaluates incoming traces. See Which test cases run on a real conversation.

Test input

The input describes what a user sends in this scenario. It documents the test and helps you and the Test Architect reason about it. Mibo does not send it to your Agent. When it evaluates a trace, Mibo reads the input recorded in that trace.

You can provide the input in two ways.

User message

A plain text message, like a real user would type. This is the most common option.

{
  "scenario": "User asks about store hours",
  "user_message": "What time do you close on Sundays?",
  "assertions": {
    "procedural": [],
    "semantic": [
      {
        "criteria": "The agent gives the Sunday closing time",
        "threshold": 0.8
      }
    ]
  }
}

Request body

A JSON object, for systems that receive structured data instead of a chat message. For example, a webhook that receives orders.

{
  "scenario": "Process an incoming order",
  "request_body": {
    "order_id": "ORD-5678",
    "customer": { "name": "Jane Doe", "email": "[email protected]" },
    "items": [
      { "product": "Widget A", "quantity": 2 }
    ]
  },
  "assertions": {
    "procedural": [
      {
        "target": "node_call",
        "condition": "MUST_CALL",
        "expected_name": "create_order"
      }
    ],
    "semantic": []
  }
}

Note

user_message and request_body are mutually exclusive. Use exactly one per test case.

Assertions

Tests use two kinds of checks: rule-based (deterministic, did the system call the right tool?) and AI-powered (scored by an evaluator, is the response good?). The Test Architect generates these automatically based on your descriptions.

Tip

Think about what you want to track on your dashboard — quality, consistency, hallucination rate. Design tests that will surface issues in these areas.

Assertion ReferenceComplete reference for all assertion types, matchers, expression functions, and examples.

Editing a test case by hand

Every test case the AI creates lands in the editor panel on the right. You can tweak any field there. To write a test case without the AI, open the Agent’s Tests page and click Create new. The editor opens with a starter definition that you can replace in the Form or JSON tab.

The form has three sections.

1. General configuration

Field	What to put here
Scenario description	One or two sentences describing the situation you’re testing. “User asks how to delete their account under GDPR.” AI evaluators read this to decide if the response is correct, and Run when relevant tests use it to decide whether a trace matches. Be specific.
Category (optional)	A label to group similar tests on the results dashboard — “Security”, “Tone”, “Refunds”. Whatever helps you slice the results.
Success metric (optional)	What this test is trying to prove — “cost-efficiency”, “latency”, “safety”. Shown in the report so you can spot weak areas at a glance.
Passive behavior (optional)	When this test evaluates incoming traces: Always run or Run when relevant (the default). See Which test cases run on a real conversation.

2. Test input

In Input source, choose a user message or Custom Payload (JSON):

• User message: a plain text message, like something a real user would type. “What time do you close on Sundays?” This is the common one.
• Custom Payload (JSON): a JSON object, for Agents that receive structured input, for example a webhook receiving an order.

For the exact shape of each option, see Test input above.

3. Assertions — the checks Mibo runs

Two kinds, added with the Add Procedural Check / Add Semantic Check buttons:

• Procedural — deterministic checks. “The create_booking tool was called with party_size: 4.” “The response is valid JSON.” “The HTTP status is 200.” These are yes/no.
• Semantic — AI-scored checks. “The agent confirms the booking, the date, and the time.” You write the criterion in plain English; an AI evaluator scores the response from 0 to 1 against it. Set Strictness with the slider from 0 to 1. Higher values require a higher score. The editor starts a new semantic check at 0.7, while JSON that omits threshold uses 0.8.

Each check shows a small form with its own fields — pick the target (node_call, json_match, semantic…), fill it in, save. The Assertion Reference covers every check type with copy-pasteable examples.

Tip

You don’t have to choose between the AI and the form — they edit the same test case. Let the AI draft, then jump to the form to polish the wording, tighten a threshold, or add one extra check by hand.

Managing test cases

Status

Each test case is either:

• Enabled: Mibo evaluates incoming traces against it. This is the default.
• Disabled: skipped during evaluation. Use this to temporarily exclude a test without deleting it.

Toggle a test case’s status from the test list. Use the Active and Disabled status filters to narrow the list.

Editing and organizing

After the AI generates your test cases, you can:

• Edit any test case to adjust the input, scenario, or assertions.
• Remove test cases you don’t need.
• Add more by continuing the conversation with the AI.
• Disable tests you want to keep but skip during evaluation.

Saving your tests

When you’re happy with the tests, click Save. Mibo saves them and takes you to the test list. Saved, enabled tests apply to the next traces your Agent sends.

Tips for effective tests

• Be specific. “Should respond with the store hours for the Manhattan location” is much better than “Should respond correctly.”
• Test one thing per case. Each test should verify a single behavior, not five things at once.
• Include edge cases. What happens with empty inputs? Very long messages? Unexpected formats?
• Ground tests in real traces. Generate tests from completed traces so assertions use your Agent’s real node names and arguments.
• Combine check types. Use a tool call check to verify the right function was called, and a semantic check to verify the response was well-worded.
• Set appropriate strictness. Use a finite threshold from 0 to 1 for semantic checks. Higher values require a higher evaluator score.