Creating Test Cases

The Test Architect is Mibo’s AI assistant for creating tests. Instead of writing test cases manually, you describe what your AI system should do in plain language, and the AI generates the test scenarios for you.

Note

AI features require a verified email. Before verifying, you can open the editor, create a test manually, and click Save. The chat, real-trace generation, and Save & run tests controls become available after you verify.

Opening the Test Architect

Navigate to your agent and click Create tests with AI. You’ll see a split-screen interface: a chat panel on the left and a test case editor on the right.

Tip

The Test Architect automatically receives the selected project name and Agent name as context. It does not receive their descriptions, so include the important details about your system in your request. Then describe the specific scenarios you want to test.

Creating tests through conversation

Start by telling the AI about your system. For example:

“My agent helps users book restaurant reservations. It should ask for the date, time, and party size before confirming the booking.”

The AI will generate a set of test cases based on your description. You can keep the conversation going to refine or add more:

“Also test what happens when the user asks for a date that’s already fully booked.”

“Add a test for when the user doesn’t specify a time.”

Each time you describe a scenario, the Test Architect creates or updates the test cases in the editor panel.

You can also attach JSON files to the conversation, for example an n8n workflow export, a Flowise chatflow config, or a JSON document describing your API. The AI analyzes the structure to detect testable nodes and suggest test targets. You can also paste JSON directly into the chat instead of uploading a file.

Note

Only JSON files are supported (max 500 KB). Other formats like PDFs, Word documents, or plain text files are not accepted.

Attach the original workflow export or paste the complete JSON object when you want the Test Architect to identify workflow steps. It accepts a JSON object or a JSON code block. Pasted JSON, including a raw object or fenced JSON code block, must be 100,000 characters or less to be analyzed. Attached JSON files can be up to 500 KB. For large attached exports, Mibo reduces unrelated parameter detail to retain node, step, and module identities and their connections. If the reduced workflow is still over 50,000 characters, Mibo returns a general result without workflow targets. A top-level JSON array or malformed JSON is not analyzed.

Generate tests from a real trace

A real trace records an execution of your Agent, including its input, output, and execution context. Use one to ground generated tests in behavior your Agent has actually produced.

1. In the Test Architect chat, click Use real trace. Before you send your first message, you can also click Attach a real trace to get started.
2. In Ground generation on a real trace, choose one of these options:
   • Pick existing trace: select a completed trace for this Agent, review its preview, and click Use this trace.
   • Dispatch new call: edit the JSON payload to replace every placeholder with a real value, then click Run trace now. Mibo sends the payload as-is to your Agent. This option counts against your execution quota.
3. After you select a trace, Mibo automatically asks the Test Architect to generate test cases from it. The trace is attached to that message. The AI uses the trace’s real node names and argument keys instead of guessing them.

Caution

If Run trace now shows Timed out waiting for the trace to arrive, check the Agent’s Traces page before trying again. The timeout ends the wait in this dialog but does not cancel the call, so the trace may still arrive. Retrying sends another real request and counts another execution. If the trace arrives, return to Use real trace → Pick existing trace and select it. If it does not arrive, follow Test your connection to verify and save the Agent connection before retrying.

If Pick existing trace is empty, click Dispatch a new call in the empty state to switch to Dispatch new call. If the dialog says the Agent has no request payload, click Configuration, follow the connection setup and save instructions, and return to this flow. When the call produces a completed trace, Mibo attaches it automatically. You can also send a trace through passive testing.

What a test case contains

Each test case has these parts:

• Name: a short description of what’s being tested.
• Scenario: a description of the situation and what the expected behavior is.
• Input: the message (or messages) that Mibo will send to your system.
• Assertions: the checks Mibo runs against the response.
• Passive behavior: controls when this test runs on real conversations. See Which test cases run on a real conversation.

Input types and how they’re sent

When you set up an agent, you configure a message template: a JSON structure that defines the request body Mibo sends to your system. This template uses {user_message} as a placeholder. Your message template is configured in the agent settings (see HTTP API setup for the universal HTTP path).

For example, an HTTP API agent might have this template:

{ "input": "{user_message}" }

And a Flowise agent uses:

{ "question": "{user_message}" }

When a test runs, Mibo takes this template, replaces {user_message} with the test’s input, and sends the result to your system. The rest of the template is sent as-is on every request. You configure the payload structure once for the agent, and each test case only provides the variable part.

You can provide the test input in three ways:

User message

A plain text message, like a real user would type. This is the most common option. When a test runs, Mibo replaces {user_message} in the agent’s message template with this text.

For example, with the template { "input": "{user_message}" } and this test case:

{
  "scenario": "User asks about store hours",
  "user_message": "What time do you close on Sundays?",
  "assertions": {
    "procedural": [],
    "semantic": []
  }
}

Mibo sends this request body to your system:

{ "input": "What time do you close on Sundays?" }

If your template has additional fields (like model or temperature), those are included as-is in every request.

Request body

A raw JSON object that replaces the entire message template. When a test uses request_body, the agent’s template is ignored completely and the request body is sent as-is.

Use this when your system expects a specific structure beyond a simple message, for example a webhook that receives structured data.

{
  "scenario": "Process an incoming order",
  "request_body": {
    "order_id": "ORD-5678",
    "customer": { "name": "Jane Doe", "email": "[email protected]" },
    "items": [
      { "product": "Widget A", "quantity": 2 }
    ]
  },
  "assertions": {
    "procedural": [],
    "semantic": []
  }
}

Multi-turn conversations (Turns)

To create or edit a multi-turn test, switch to the JSON tab. The form selector does not include a multi-turn option. If you open a multi-turn test in the form view, return to JSON to change its turns.

Each turn represents one message in the conversation, sent sequentially with shared session context.

This is essential for testing flows where your system needs to gather information across several exchanges, like booking a flight, filling out a form, or handling a multi-step support request.

{
  "scenario": "Complete booking flow",
  "turns": [
    { "user_message": "I want to book a flight to Paris" },
    { "user_message": "Next Monday, economy class" },
    { "user_message": "John Doe, passport ABC123" }
  ],
  "assertions": {
    "semantic": [
      {
        "criteria": "The agent confirms the booking with all details: destination, date, class, and passenger info",
        "threshold": 0.85
      }
    ]
  }
}

How multi-turn execution works:

1. Mibo generates a unique session ID for the test.
2. Each turn is sent sequentially to your system, using the same session ID to maintain conversation context.
3. Your system receives each message as if it were a real user continuing the conversation.
4. The top-level assertions are evaluated against the last turn’s response.

Per-turn assertions

You can optionally add assertions to individual turns to verify intermediate steps, not just the final response:

{
  "scenario": "Restaurant reservation with validation",
  "turns": [
    {
      "user_message": "Book a table for tomorrow at 8pm"
    },
    {
      "user_message": "4 people",
      "assertions": {
        "semantic": [
          {
            "criteria": "The agent should confirm the party size and ask for a name",
            "threshold": 0.8
          }
        ]
      }
    },
    {
      "user_message": "Under the name García"
    }
  ],
  "assertions": {
    "procedural": [
      {
        "target": "node_call",
        "condition": "MUST_CALL",
        "expected_name": "create_reservation",
        "expected_arguments": {
          "party_size": 4,
          "name": "García"
        }
      }
    ],
    "semantic": [
      {
        "criteria": "The agent confirms the full reservation: date, time, party size, and name",
        "threshold": 0.85
      }
    ]
  }
}

In this example, the second turn checks that the agent asks for a name after receiving the party size. The final assertions verify that the reservation tool was called correctly and the confirmation message is complete.

Note

Multi-turn tests run manually only. Mibo disables passive behavior for them because passive traces do not contain the full conversation sequence.

Note

user_message, request_body, and turns are mutually exclusive. Use exactly one per test case. You can’t combine a single input with turns.

Assertions

Tests use two kinds of checks: rule-based (deterministic, did the system call the right tool?) and AI-powered (scored by an evaluator, is the response good?). The Test Architect generates these automatically based on your descriptions.

Tip

Think about what you want to track on your dashboard — quality, consistency, hallucination rate. Design tests that will surface issues in these areas.

Assertion ReferenceComplete reference for all assertion types, matchers, expression functions, and examples.

Editing a test case by hand

Every test case the AI creates lands in the editor panel on the right. You can tweak any field there. To write a test case without the AI, open the Agent’s Tests page and click Create new. The editor opens with a starter definition that you can replace in the Form or JSON tab.

The form has three sections.

1. Scenario

Field	What to put here
Scenario description	One or two sentences describing the situation you’re testing. “User asks how to delete their account under GDPR.” This is what the AI evaluators read to decide if the response is correct, so be specific.
Category (optional)	A label to group similar tests on the results dashboard — “Security”, “Tone”, “Refunds”. Whatever helps you slice the results.
Success metric (optional)	What this test is trying to prove — “cost-efficiency”, “latency”, “safety”. Shown in the report so you can spot weak areas at a glance.
Passive behavior (optional)	When this test runs on real conversations: Always run, Run when relevant (the default), or Manual runs only. It has no effect when you press Run yourself. See Which test cases run on a real conversation.

2. Input — what Mibo sends to your system

In the form, choose one of these input types from the selector:

• User message — a plain text message, like something a real user would type. “What time do you close on Sundays?” This is the common one.
• Custom payload (JSON) — a raw JSON object that Mibo sends as-is, bypassing the message template you set for the agent. Use it when your system expects structured input, for example a webhook receiving an order.

For multi-turn input, follow the Multi-turn conversations instructions and JSON examples.

Note

You can only pick one input type per test case — they’re mutually exclusive.

For the exact shape of each option (and per-turn assertions in multi-turn tests), see Input types and how they’re sent above.

3. Assertions — the checks Mibo runs

Two kinds, added with the Add Procedural Check / Add Semantic Check buttons:

• Procedural — deterministic checks. “The create_booking tool was called with party_size: 4.” “The response is valid JSON.” “The HTTP status is 200.” These are yes/no.
• Semantic — AI-scored checks. “The agent confirms the booking, the date, and the time.” You write the criterion in plain English; an AI evaluator scores the response from 0 to 1 against it. Set Strictness with the slider from 0 to 1. Higher values require a higher score. The editor starts a new semantic check at 0.7, while JSON that omits threshold uses 0.8.

Each check shows a small form with its own fields — pick the target (node_call, json_match, semantic…), fill it in, save. The Assertion Reference covers every check type with copy-pasteable examples.

Tip

You don’t have to choose between the AI and the form — they edit the same test case. Let the AI draft, then jump to the form to polish the wording, tighten a threshold, or add one extra check by hand.

Managing test cases

Status

Each test case has a status:

• Active: included in test runs. This is the default.
• Disabled: skipped during test runs. Use this to temporarily exclude a test without deleting it.

You can toggle a test case’s status from the test list.

Editing and organizing

After the AI generates your test cases, you can:

• Edit any test case to adjust the input, scenario, or assertions.
• Remove test cases you don’t need.
• Add more by continuing the conversation with the AI.
• Disable tests you want to keep but skip during runs.

Saving your tests

When you’re happy with the tests, you have two options:

• Save: saves the tests and takes you to the test list.
• Save & run tests: saves the tests and immediately runs them against your system.

Recover after saving and running

Mibo saves your tests before it starts a run. If saving fails, the editor stays open and shows Could not save the test. Fix the problem and click Save & run tests again.

If the tests are saved but the run does not start, the page shows Test saved and keeps them available. Click Run saved test to retry the run without creating more copies.

Sometimes Mibo cannot tell whether the run request was accepted. The page shows Run status unknown. Click Check run status before retrying. If Mibo finds an accepted run, it opens the test when you saved one test. If you saved several, it returns you to the test list. If Mibo finds no run, click Run saved test. Checking first helps you avoid sending the same test twice.

Tips for effective tests

• Be specific. “Should respond with the store hours for the Manhattan location” is much better than “Should respond correctly.”
• Test one thing per case. Each test should verify a single behavior, not five things at once.
• Include edge cases. What happens with empty inputs? Very long messages? Unexpected formats?
• Use realistic inputs. Write test inputs that sound like actual user interactions, not robotic commands.
• Combine check types. Use a tool call check to verify the right function was called, and a semantic check to verify the response was well-worded.
• Set appropriate strictness. Use a finite threshold from 0 to 1 for semantic checks. Higher values require a higher evaluator score.
• Use multi-turn for conversations. If your system is designed for back-and-forth interaction, test the full flow, not just isolated messages.
• Add per-turn assertions sparingly. Only check intermediate steps when the order of operations matters. Keep most assertions on the final response.