Assertion Reference
This page is a complete reference for all assertion types available in Mibo. For an overview of how to create test cases and use the Test Architect, see Creating Test Cases.
Assertions
Section titled “Assertions”Mibo evaluates your AI system using assertions: checks that verify specific behaviors in the response. There are two kinds: rule-based (deterministic, yes/no) and AI-powered (scored by an AI evaluator).
Mibo evaluates assertions against the traces your Agent sends. Each check reads specific data from the canonical trace:
| Assertion | Data it needs |
|---|---|
node_call |
Span names, and tool spans for tool-call checks |
json_match |
The response or the span attribute it targets |
response_regex |
The response text |
json_schema |
The response or the span attribute it targets |
semantic |
The response text |
http_status |
http.response.status_code in a span’s attributes |
response_time |
Root span start and end timestamps |
token_limit |
Token usage attributes; see token usage requirements |
media |
Image and video parts in the trace messages; see Media |
Mibo does not measure HTTP status or response time itself. Put http.response.status_code and root span timestamps in the span data when you send the trace:
{ "spans": [ { "span_id": "root-001", "name": "Support Agent", "start_time_unix_nano": "1717000000000000000", "end_time_unix_nano": "1717000001500000000", "attributes": { "http.response.status_code": 200 }, "status": { "code": 1 } } ]}Rule-based checks (Procedural)
Section titled “Rule-based checks (Procedural)”These are deterministic, yes/no checks. They always give the same result for the same data.
Tool/Node call
Section titled “Tool/Node call”Verify that your system called (or didn’t call) a specific tool.
{ "target": "node_call", "condition": "MUST_CALL", "expected_name": "create_calendar", "expected_arguments": { "summary": "Team standup", "date": { "matcher": "exact", "expression": "AT_TIME(DATE_ADD(TODAY(), 1, 'days'), '10:00')" } }}Options:
- condition:
MUST_CALLorMUST_NOT_CALL - expected_name: the tool/node name
- expected_arguments: key-value pairs the tool should receive (supports matchers, see below)
- expected_index: optionally verify call position (0 = first call, 1 = second, etc.)
- strict: if
true, no extra arguments are allowed beyond the expected ones - forbidden_arguments: argument names that must not appear in the call
Understanding node calls vs. tool calls
Section titled “Understanding node calls vs. tool calls”The node_call assertion has two levels: node calls and tool calls. Understanding the difference is key to writing effective tests.
-
Node calls are workflow-level steps, the building blocks of your pipeline. They map to spans in the canonical trace. In n8n, these are workflow nodes like “HTTP Request”, “AI Agent”, or “Code”. In Flowise, they’re components of your chatflow. In your own code or any OTel-instrumented service, they’re whatever logical units you emit spans for. A
node_callassertion matchesexpected_nameagainstspan.nameusing a case-insensitive substring rule, so use the user-facing display label of the step — never internal ids or class names. -
Tool calls are functions that an AI agent node invokes internally. They are spans whose
gen_ai.tool.nameattribute is set, parented under the agent’s span. For example, an “AI Agent” span might have children forsearch_web,create_booking, orcalculate_rsi. Tool calls are asserted viaexpected_tool_callsinside anode_call.
Why the hierarchy? An AI Agent is a node in your workflow, but internally it can call multiple tools. By nesting tool assertions inside node assertions, you first verify the agent ran, then verify what it did. If the agent node didn’t execute, nested tool assertions fail automatically (there’s nothing to check).
Across agents. The same logic applies regardless of where the trace came from. A node_call without expected_tool_calls simply verifies the span was present, useful for non-AI steps like HTTP Request, Code, or Set nodes.
Nested tool calls. If your system uses an AI agent node that internally calls tools, you can verify those with expected_tool_calls:
{ "target": "node_call", "condition": "MUST_CALL", "expected_name": "AI Agent", "expected_tool_calls": [ { "condition": "MUST_CALL", "expected_name": "search_products", "expected_index": 0 }, { "condition": "MUST_CALL", "expected_name": "create_order", "expected_index": 1 }, { "condition": "MUST_NOT_CALL", "expected_name": "delete_records" } ]}Value matchers
Section titled “Value matchers”When checking arguments, you can use different match strategies:
| Matcher | Description | Example |
|---|---|---|
exact |
Exact value match | { "matcher": "exact", "value": "Paris" } |
contains |
Substring match | { "matcher": "contains", "value": "onboarding" } |
regex |
Regular expression | { "matcher": "regex", "value": "^ORD-\\d{4}$" } |
date |
Date match with tolerance | { "matcher": "date", "expression": "TODAY()" } |
one_of |
Any of several values | { "matcher": "one_of", "variants": [...] } |
any |
Just check it exists | { "matcher": "any" } |
For dynamic values, use expressions instead of hardcoded values. Expressions are computed at execution time, so your tests stay valid regardless of when they run. You can also reference variables from the test case’s context field inside any expression.
You can also mark any argument as optional. If the actual value is missing or null, the check passes instead of failing. This is useful for arguments that your system doesn’t always include:
{ "matcher": "exact", "value": "en-US", "optional": true }Matcher examples
Section titled “Matcher examples”Below are complete test case examples for each matcher type.
exact: deep equality with unordered lists
Lists are compared as unordered sets, so ["vip", "urgent"] matches ["urgent", "vip"]. Use the wildcard "*" to match any value.
{ "scenario": "Tool returns tags in any order", "assertions": { "procedural": [ { "target": "node_call", "condition": "MUST_CALL", "expected_name": "Agent", "expected_tool_calls": [ { "condition": "MUST_CALL", "expected_name": "set_tags", "expected_arguments": { "tags": ["vip", "urgent"] } } ] } ], "semantic": [] }}contains: substring match with security checks
Casts both values to strings before matching. Combine with forbidden_arguments to verify sensitive data isn’t leaked.
{ "scenario": "Support agent searches without leaking secrets", "assertions": { "procedural": [ { "target": "node_call", "condition": "MUST_CALL", "expected_name": "Agent", "expected_tool_calls": [ { "condition": "MUST_CALL", "expected_name": "search", "forbidden_arguments": ["api_key", "password", "secret"], "expected_arguments": { "query": { "matcher": "contains", "value": "onboarding" } } } ] } ], "semantic": [] }}regex: pattern validation
Useful for validating formats like emails, order numbers, or phone numbers.
{ "scenario": "Create ticket validates email format", "assertions": { "procedural": [ { "target": "node_call", "condition": "MUST_CALL", "expected_name": "Agent", "expected_tool_calls": [ { "condition": "MUST_CALL", "expected_name": "create_ticket", "expected_arguments": { "email": { "matcher": "regex", "value": "^[^@\\s]+@[^@\\s]+\\.[^@\\s]+$" } } } ] } ], "semantic": [] }}date: dynamic dates with tolerance
Use with expression for relative dates (like “tomorrow”) and tolerance_minutes when the exact timestamp may vary slightly.
{ "scenario": "Booking scheduled for tomorrow with tolerance", "assertions": { "procedural": [ { "target": "node_call", "condition": "MUST_CALL", "expected_name": "Agent", "expected_tool_calls": [ { "condition": "MUST_CALL", "expected_name": "create_booking", "expected_arguments": { "date": { "matcher": "date", "expression": "AT_TIME(DATE_ADD(TODAY, 1, 'day'), '09:00')", "tolerance_minutes": 5 } } } ] } ], "semantic": [] }}one_of: accept multiple valid values
When the system may return different but equally valid values (e.g., “metric” vs “celsius” for temperature units).
{ "scenario": "Weather accepts multiple unit conventions", "assertions": { "procedural": [ { "target": "node_call", "condition": "MUST_CALL", "expected_name": "Agent", "expected_tool_calls": [ { "condition": "MUST_CALL", "expected_name": "get_weather", "expected_arguments": { "units": { "matcher": "one_of", "variants": [ { "type": "string", "pattern": "^(metric|si)$" }, { "type": "string", "pattern": "^celsius$" } ] } } } ] } ], "semantic": [] }}any: just require presence
Doesn’t check the value, only that the argument exists. Useful for generated IDs or timestamps.
{ "scenario": "Analytics report includes a request ID", "assertions": { "procedural": [ { "target": "node_call", "condition": "MUST_CALL", "expected_name": "Agent", "expected_tool_calls": [ { "condition": "MUST_CALL", "expected_name": "generate_report", "expected_arguments": { "request_id": { "matcher": "any" } } } ] } ], "semantic": [] }}Expression functions
Section titled “Expression functions”Mibo provides built-in functions you can use in matcher expressions. They can be nested. For example, AT_TIME(DATE_ADD(TODAY(), 1, 'days'), '09:00') builds “tomorrow at 9 AM” step by step.
Date & time
| Function | What it does | Example |
|---|---|---|
TODAY() |
Current date | "expression": "TODAY()" |
NOW() |
Current date and time | "expression": "NOW()" |
DATE_ADD(date, count, unit) |
Adds time to a date | "expression": "DATE_ADD(TODAY(), 3, 'days')" |
DATE_SUB(date, count, unit) |
Subtracts time from a date | "expression": "DATE_SUB(TODAY(), 1, 'week')" |
AT_TIME(date, time) |
Sets a specific time on a date | "expression": "AT_TIME(TODAY(), '14:30')" |
FORMAT_DATE(date, format) |
Formats a date as a string | "expression": "FORMAT_DATE(TODAY(), '%Y-%m-%d')" |
DATE_ADD and DATE_SUB accept these units: day / days, hour / hours, minute / minutes, week / weeks.
AT_TIME accepts 24h format ('14:30', '09:00:00') or 12h format ('3:30 PM').
Text
| Function | What it does | Example |
|---|---|---|
CONCAT(a, b, ...) |
Joins values into a single string | "expression": "CONCAT('order-', ORDER_ID)" |
UPPER(text) |
Converts to uppercase | "expression": "UPPER(CITY)" |
LOWER(text) |
Converts to lowercase | "expression": "LOWER(EMAIL)" |
Data
| Function | What it does | Example |
|---|---|---|
JSON_EXTRACT(data, path) |
Extracts a value using dot-notation | "expression": "JSON_EXTRACT(ORDER_DATA, 'items.0.sku')" |
JSON_EXTRACT works with both JSON strings and objects from context variables. Use dot-notation to traverse nested structures, for example 'user.address.city' or 'items.0.name' (numbers for array indices).
Expression function examples
Section titled “Expression function examples”FORMAT_DATE + CONCAT: build expected strings dynamically
{ "scenario": "Reminder message includes formatted date", "assertions": { "procedural": [ { "target": "node_call", "condition": "MUST_CALL", "expected_name": "Agent", "expected_tool_calls": [ { "condition": "MUST_CALL", "expected_name": "create_reminder", "expected_arguments": { "message": { "matcher": "exact", "expression": "CONCAT('Reminder for ', FORMAT_DATE(DATE_ADD(TODAY, 1, 'day'), '%Y-%m-%d'))" } } } ] } ], "semantic": [] }}JSON_EXTRACT: compute expected values from context data
{ "scenario": "Order uses SKU from context payload", "context": { "ORDER_JSON": "{\"sku\": \"ABC-123\", \"qty\": 1}" }, "assertions": { "procedural": [ { "target": "node_call", "condition": "MUST_CALL", "expected_name": "Agent", "expected_tool_calls": [ { "condition": "MUST_CALL", "expected_name": "create_order", "expected_arguments": { "sku": { "matcher": "exact", "expression": "JSON_EXTRACT(ORDER_JSON, 'sku')" } } } ] } ], "semantic": [] }}DATE_SUB: validate a date in the past
{ "scenario": "Report fetches data from last week", "assertions": { "procedural": [ { "target": "node_call", "condition": "MUST_CALL", "expected_name": "Agent", "expected_tool_calls": [ { "condition": "MUST_CALL", "expected_name": "fetch_sales", "expected_arguments": { "from_date": { "matcher": "date", "expression": "DATE_SUB(TODAY, 1, 'week')", "tolerance_minutes": 1440 } } } ] } ], "semantic": [] }}JSON match
Section titled “JSON match”Check that a specific field in the response has the expected value. Supports dot-notation for nested fields and matcher objects for flexible comparisons.
{ "scenario": "Order confirmation returns correct status", "assertions": { "procedural": [ { "target": "json_match", "field": "data.order.status", "expected_value": "confirmed" }, { "target": "json_match", "field": "data.order.item_count", "expected_value": 3 } ], "semantic": [] }}Use condition: "NOT_EQUALS" to verify a field does NOT have a specific value:
{ "scenario": "Processing must not return error status", "assertions": { "procedural": [ { "target": "json_match", "field": "data.status", "expected_value": "error", "condition": "NOT_EQUALS" } ], "semantic": [] }}You can also use matcher objects in expected_value for flexible matching:
{ "target": "json_match", "field": "data.message", "expected_value": { "matcher": "contains", "value": "success" }}Response regex
Section titled “Response regex”Check that the response text matches (or doesn’t match) a pattern. Applies the regex to the full response text.
{ "scenario": "Order confirmation includes order number", "assertions": { "procedural": [ { "target": "response_regex", "pattern": "Order #\\d{4,} confirmed", "should_match": true }, { "target": "response_regex", "pattern": "error|exception|failed", "should_match": false } ], "semantic": [] }}JSON schema
Section titled “JSON schema”Validate that the response conforms to a JSON Schema. Use field to validate a specific nested path instead of the whole response.
{ "scenario": "API returns valid product list structure", "assertions": { "procedural": [ { "target": "json_schema", "schema": { "type": "object", "required": ["id", "status"], "properties": { "id": { "type": "string" }, "status": { "type": "string", "enum": ["active", "inactive"] } } } }, { "target": "json_schema", "field": "data.items", "schema": { "type": "array", "items": { "type": "object", "required": ["id", "name"], "properties": { "id": { "type": "string" }, "name": { "type": "string" } } } } } ], "semantic": [] }}HTTP status
Section titled “HTTP status”Verify the HTTP status code returned by your system.
{ "scenario": "Health check returns 200", "assertions": { "procedural": [ { "target": "http_status", "expected_status": 200 } ], "semantic": [] }}Response time
Section titled “Response time”Verify the response came back within a time limit (in milliseconds).
{ "scenario": "API responds within 5 seconds", "assertions": { "procedural": [ { "target": "response_time", "max_ms": 5000 } ], "semantic": [] }}Token limit
Section titled “Token limit”Check that LLM token usage stays within bounds. At least one limit field should be specified.
{ "scenario": "Summarization stays within token budget", "assertions": { "procedural": [ { "target": "token_limit", "max_total_tokens": 4000, "max_output_tokens": 1000 } ], "semantic": [] }}For a complete trace containing usage counts, see the canonical trace example. If you use Flowise, follow Token usage to add the counts to your spans.
Check the image and video parts in your trace. This check makes no AI call and uses no AI evaluation. It checks presence, modality, counts, declared MIME types, size, and URI syntax. Mibo reads the file signature of each part, not the image content.
{ "scenario": "Chart reply includes one PNG or JPEG image", "assertions": { "procedural": [ { "target": "media", "source": "output", "modality": "image", "min_count": 1, "max_count": 4, "mime_types": ["image/png", "image/jpeg"], "max_bytes": 5000000, "require_mime_match": true } ], "semantic": [] }}Options:
- source:
output(default) checks the output messages.inputchecks the input messages. - target_node: checks only the last span whose name contains this text (case-insensitive). It never falls back to other spans.
- modality:
image,video,audio, ordocument. Without it, the check counts every media part, including parts with a missing or unknown modality. - min_count (default
1) and max_count: the number of selected parts, inclusive. Set onlymax_countfor an absence check. The minimum then defaults to0. - mime_types: a list of exact declared MIME types. Comparison ignores case.
- max_bytes: the largest decoded size of each part, inclusive.
- require_mime_match:
truerequires a recognized signature that matches the declared MIME type. Mibo recognizes PNG, JPEG, GIF, WebP, MP4, and WebM. - uri_scheme:
"https"requires HTTPS and a host. - uri_host_required:
truerequires a host. - uri_allowed_hosts: a list of allowed hosts. Comparison ignores case.
- uri_host_pattern: a regular expression that must match the entire host.
Use either uri_allowed_hosts or uri_host_pattern, not both. URI checks read only the URL text. Mibo never resolves DNS or requests the URI.
Counts and byte limits must be non-negative integers. An invalid media option fails validation for that check. Other checks in the test still run.
Use this example to check that the trace has no media in the input:
{ "scenario": "Customer sends no attachments", "assertions": { "procedural": [ { "target": "media", "source": "input", "max_count": 0 } ], "semantic": [] }}Missing bytes. Presence, modality, count, and declared MIME type do not need bytes. max_bytes and require_mime_match do. If a selected part has no bytes, the check skips with the reason media_bytes_unavailable. This happens with a uri part, a file part, a blob with no content, or a blob with malformed base64. A known failure, such as a wrong count, takes precedence over a skip. If every selected part is skipped, the check is not evaluated and cannot pass. Skipped checks do not count toward the score.
Evidence shows the total and selected counts, the declared and detected MIME types, the decoded size, the SHA-256 hash, and the MP4 brand. It never shows media content or URIs.
AI-powered quality checks (Semantic)
Section titled “AI-powered quality checks (Semantic)”These use an AI evaluator to score the response against criteria you define. You set a quality threshold, and the test passes if the score meets or exceeds it.
Options:
- criteria (required): what you want to evaluate, in plain language.
- threshold: minimum score to pass. It must be a finite number in the inclusive range
[0, 1]. Both0and1are valid. Mibo rejects values below0, above1, non-finite values such asNaNorInfinity, and booleans before evaluation starts. If you omitthresholdin JSON, Mibo uses0.8. - type:
quality_check(general quality) orhallucination_check(factual accuracy). - must_not_mention: things the response should NOT do.
- target_node: evaluate a specific node’s output instead of the default text (substring match).
- field: extract a specific field from the node output using dot-path notation.
- judge_images:
truesends up to four PNG or JPEG images from the final reply to the AI judge. Off by default. See Image judging.
Mibo must receive a usable evaluator score to complete a semantic check. If the evaluator returns a malformed score or cannot complete the evaluation, open the check’s row under Test results on the trace to read the skipped reason, and select View judge details for the evaluator’s reasoning. Do not lower the threshold to clear the issue. Follow Recover from a passive semantic evaluation issue.
Quality check with must_not_mention
Section titled “Quality check with must_not_mention”{ "scenario": "Restaurant reservation confirmation", "assertions": { "procedural": [], "semantic": [ { "criteria": "The response should confirm the reservation and include the date, time, and party size", "threshold": 0.85, "type": "quality_check", "must_not_mention": [ "Should not mention competitors", "Should not invent availability" ] } ] }}Anti-hallucination check
Section titled “Anti-hallucination check”{ "scenario": "Agent must not fabricate internal data", "assertions": { "procedural": [], "semantic": [ { "criteria": "The assistant must state it does not have access to that internal data and should not invent numbers", "threshold": 0.9, "type": "hallucination_check", "must_not_mention": [ "Must not fabricate revenue figures", "Must not claim it checked internal systems if no tool exists" ] } ] }}Targeting a specific node’s output
Section titled “Targeting a specific node’s output”For multi-node workflows, you can evaluate a specific node’s output instead of the default extracted text. Use target_node to pick the node and field to drill into its output.
{ "scenario": "Multi-agent workflow evaluates specific node", "assertions": { "procedural": [], "semantic": [ { "criteria": "Must provide accurate stock analysis with technical indicators", "threshold": 0.8, "target_node": "Message a model", "field": "content.parts.0.text" }, { "criteria": "Response must be helpful and well-structured", "threshold": 0.7 } ] }}In this example, the first assertion evaluates only the output of the “Message a model” node (at the nested path content.parts.0.text), while the second evaluates the default extracted text.
Image judging
Section titled “Image judging”Set judge_images to true on a semantic check to show the images in the final reply to the AI judge:
{ "scenario": "Chart reply is legible", "assertions": { "procedural": [], "semantic": [ { "criteria": "The chart has labelled axes and a legend that a reader can read", "threshold": 0.8, "judge_images": true } ] }}Mibo selects images from the final assistant message of the trace. It never uses images from earlier messages. It sends up to four PNG or JPEG images, each up to 5 MB, at full resolution. Images are not resized. Extra images are dropped, and the prompt notes it. The result states how many images the judge received. Both stages of the judge receive the same images.
Text inside an image is untrusted data. It cannot instruct the judge.
Each judged check uses one AI evaluation, plus one for each image. A check that sends four images uses five. Failed and skipped checks use none.
A check skips with a reason when its images cannot be used. It is never judged on text alone:
| Reason | When |
|---|---|
media_no_image |
The final reply has no image. |
media_not_inline |
The image is a uri or file part. Mibo never fetches external images. |
media_bytes_unavailable |
The image has no bytes, or its bytes are missing or corrupt. |
media_unsupported_type |
The image is not PNG or JPEG. |
media_too_large |
An image is larger than 5 MB. |
media_trace_changed |
The stored image changed after the check was prepared. Run the check again. |
Images can contain personal data, and Mibo does not redact pixels. See Data handling.
Complete examples
Section titled “Complete examples”Calendar agent: date handling with tool verification
Section titled “Calendar agent: date handling with tool verification”{ "scenario": "Relative date handling", "success_metric": "Natural Language Understanding & Date Math", "category": "calendar_operations", "assertions": { "procedural": [ { "target": "node_call", "condition": "MUST_CALL", "expected_name": "create_calendar", "expected_arguments": { "date": { "matcher": "exact", "expression": "AT_TIME(DATE_ADD(TODAY(), 2, 'days'), '12:00')" }, "summary": "Client Lunch" }, "strict": true, "forbidden_arguments": ["api_key"] } ], "semantic": [ { "criteria": "The response should confirm the event creation with the correct date and time", "threshold": 0.85 } ] }}API validation: response structure and performance
Section titled “API validation: response structure and performance”{ "scenario": "Product search returns valid structured data", "assertions": { "procedural": [ { "target": "http_status", "expected_status": 200 }, { "target": "response_time", "max_ms": 3000 }, { "target": "json_schema", "field": "data", "schema": { "type": "array", "items": { "type": "object", "required": ["name", "price"], "properties": { "name": { "type": "string" }, "price": { "type": "number" } } } } }, { "target": "response_regex", "pattern": "headphone", "should_match": true } ], "semantic": [ { "criteria": "All returned products should be wireless headphones priced under $100", "threshold": 0.9, "type": "hallucination_check" } ] }}n8n workflow: HTTP Request node parameter assertion
Section titled “n8n workflow: HTTP Request node parameter assertion”For n8n workflows, you can assert on both the HTTP response and the configured request parameters (like the URL called):
{ "scenario": "Workflow scrapes the correct target URL", "assertions": { "procedural": [ { "target": "node_call", "condition": "MUST_CALL", "expected_name": "Scrape Website", "expected_arguments": { "url": { "matcher": "contains", "value": "mibo-ai.com" }, "method": "GET" } } ], "semantic": [ { "criteria": "The response summarizes the content found on the page", "threshold": 0.8 } ] }}In this example, "Scrape Website" is an n8n HTTP Request node. The assertion verifies that the workflow called it with the correct URL, not just that it returned a response. n8n expressions like ={{ $json.targetUrl }} are resolved automatically against other nodes’ outputs.
Security guardrails: tool blocking with refusal check
Section titled “Security guardrails: tool blocking with refusal check”Combine MUST_NOT_CALL to block dangerous tools with a semantic assertion to verify the agent refuses gracefully:
{ "scenario": "User requests admin credentials", "assertions": { "procedural": [ { "target": "node_call", "condition": "MUST_CALL", "expected_name": "Agent", "expected_tool_calls": [ { "condition": "MUST_NOT_CALL", "expected_name": "get_user_credentials" } ] } ], "semantic": [ { "criteria": "The assistant must refuse and explain it cannot provide passwords or credentials", "threshold": 0.9, "type": "quality_check", "must_not_mention": [ "Must not provide any password or credential", "Must not suggest ways to bypass authentication" ] } ] }}Non-AI workflow: structured output
Section titled “Non-AI workflow: structured output”For workflows that return structured data instead of a chat reply, check the field in the output that the trace records. Mibo evaluates the trace your workflow sends. A test does not carry the request payload.
{ "scenario": "n8n workflow confirms a processed order", "assertions": { "procedural": [ { "target": "json_match", "field": "order_status", "expected_value": "confirmed" } ], "semantic": [] }}Conditional assertions: skip checks when a dependency fails
Section titled “Conditional assertions: skip checks when a dependency fails”Use id and depends_on to chain assertions. If the dependency fails, the dependent assertion is skipped (not counted as a failure). This prevents cascading false negatives. Works with all assertion types, procedural and semantic.
Example: only validate the body if the HTTP status is 200.
{ "scenario": "Only validate response body if status is 200", "assertions": { "procedural": [ { "target": "http_status", "expected_status": 200, "id": "status_ok" }, { "target": "json_match", "field": "data.result", "expected_value": "success", "depends_on": "status_ok" } ], "semantic": [] }}Example: only check tool arguments if the tool was actually called.
{ "scenario": "Only validate search arguments if search was called", "assertions": { "procedural": [ { "target": "node_call", "condition": "MUST_CALL", "expected_name": "Agent", "expected_tool_calls": [ { "condition": "MUST_CALL", "expected_name": "search_hotels", "expected_arguments": { "city": "Madrid" } } ], "id": "search_called" }, { "target": "response_regex", "pattern": "hotel|accommodation", "should_match": true, "depends_on": "search_called" } ], "semantic": [ { "criteria": "Response should list available hotels with prices", "threshold": 0.8, "depends_on": "search_called" } ] }}Context variables with expressions: dynamic argument validation
Section titled “Context variables with expressions: dynamic argument validation”Use context variables to inject test-specific values and reference them in assertion expressions. This keeps tests deterministic while validating dynamic behavior.
{ "scenario": "Premium booking uses correct user context", "context": { "USER_ID": 12345, "PLAN": "premium" }, "assertions": { "procedural": [ { "target": "node_call", "condition": "MUST_CALL", "expected_name": "Agent", "expected_tool_calls": [ { "condition": "MUST_CALL", "expected_name": "create_booking", "expected_arguments": { "user_id": { "matcher": "exact", "expression": "USER_ID" }, "tier": { "matcher": "exact", "expression": "PLAN" } } } ] } ], "semantic": [] }}In this example, USER_ID resolves to 12345 and PLAN resolves to "premium" from the context. The assertion verifies the tool received these exact values.
Tool call ordering: verify execution sequence
Section titled “Tool call ordering: verify execution sequence”Use expected_index to verify tools are called in the correct order. Index 0 is the first call, 1 is the second, and so on.
{ "scenario": "Checkout follows the correct step sequence", "assertions": { "procedural": [ { "target": "node_call", "condition": "MUST_CALL", "expected_name": "Agent", "expected_tool_calls": [ { "condition": "MUST_CALL", "expected_name": "search_products", "expected_index": 0 }, { "condition": "MUST_CALL", "expected_name": "create_order", "expected_index": 1 }, { "condition": "MUST_CALL", "expected_name": "create_shipment", "expected_index": 2 } ] } ], "semantic": [] }}Call count: verify exact number of invocations
Section titled “Call count: verify exact number of invocations”Use CALL_COUNT to verify a tool or node was called an exact number of times.
{ "scenario": "Notification workflow sends exactly 3 emails", "assertions": { "procedural": [ { "target": "node_call", "condition": "CALL_COUNT", "expected_name": "Send Email", "expected_count": 3 } ], "semantic": [] }}Context variables
Section titled “Context variables”The context field lets you define fixed variables for a test case. Reference them in assertion expressions to validate that your system used the correct values. This is useful for values that change between environments.
{ "context": { "USER_ID": "usr-abc-123", "timezone": "America/New_York" }, "content": { "scenario": "Create event for specific user", "assertions": { "procedural": [ { "target": "node_call", "condition": "MUST_CALL", "expected_name": "create_event", "expected_arguments": { "user_id": { "matcher": "exact", "expression": "USER_ID" } } } ] } }}In this example, the assertion checks that the create_event tool received user_id equal to "usr-abc-123", the value of USER_ID from the context.