Skip to content

Assertion Reference

This page is a complete reference for all assertion types available in Mibo. For an overview of how to create test cases and use the Test Architect, see Creating Test Cases.

Mibo evaluates your AI system using assertions: checks that verify specific behaviors in the response. There are two kinds: rule-based (deterministic, yes/no) and AI-powered (scored by an AI evaluator).

Mibo evaluates assertions against the traces your Agent sends. Each check reads specific data from the canonical trace:

Assertion Data it needs
node_call Span names, and tool spans for tool-call checks
json_match The response or the span attribute it targets
response_regex The response text
json_schema The response or the span attribute it targets
semantic The response text
http_status http.response.status_code in a span’s attributes
response_time Root span start and end timestamps
token_limit Token usage attributes; see token usage requirements
media Image and video parts in the trace messages; see Media

Mibo does not measure HTTP status or response time itself. Put http.response.status_code and root span timestamps in the span data when you send the trace:

{
"spans": [
{
"span_id": "root-001",
"name": "Support Agent",
"start_time_unix_nano": "1717000000000000000",
"end_time_unix_nano": "1717000001500000000",
"attributes": {
"http.response.status_code": 200
},
"status": { "code": 1 }
}
]
}

These are deterministic, yes/no checks. They always give the same result for the same data.

Verify that your system called (or didn’t call) a specific tool.

{
"target": "node_call",
"condition": "MUST_CALL",
"expected_name": "create_calendar",
"expected_arguments": {
"summary": "Team standup",
"date": {
"matcher": "exact",
"expression": "AT_TIME(DATE_ADD(TODAY(), 1, 'days'), '10:00')"
}
}
}

Options:

  • condition: MUST_CALL or MUST_NOT_CALL
  • expected_name: the tool/node name
  • expected_arguments: key-value pairs the tool should receive (supports matchers, see below)
  • expected_index: optionally verify call position (0 = first call, 1 = second, etc.)
  • strict: if true, no extra arguments are allowed beyond the expected ones
  • forbidden_arguments: argument names that must not appear in the call

The node_call assertion has two levels: node calls and tool calls. Understanding the difference is key to writing effective tests.

  • Node calls are workflow-level steps, the building blocks of your pipeline. They map to spans in the canonical trace. In n8n, these are workflow nodes like “HTTP Request”, “AI Agent”, or “Code”. In Flowise, they’re components of your chatflow. In your own code or any OTel-instrumented service, they’re whatever logical units you emit spans for. A node_call assertion matches expected_name against span.name using a case-insensitive substring rule, so use the user-facing display label of the step — never internal ids or class names.

  • Tool calls are functions that an AI agent node invokes internally. They are spans whose gen_ai.tool.name attribute is set, parented under the agent’s span. For example, an “AI Agent” span might have children for search_web, create_booking, or calculate_rsi. Tool calls are asserted via expected_tool_calls inside a node_call.

Why the hierarchy? An AI Agent is a node in your workflow, but internally it can call multiple tools. By nesting tool assertions inside node assertions, you first verify the agent ran, then verify what it did. If the agent node didn’t execute, nested tool assertions fail automatically (there’s nothing to check).

Across agents. The same logic applies regardless of where the trace came from. A node_call without expected_tool_calls simply verifies the span was present, useful for non-AI steps like HTTP Request, Code, or Set nodes.

Nested tool calls. If your system uses an AI agent node that internally calls tools, you can verify those with expected_tool_calls:

{
"target": "node_call",
"condition": "MUST_CALL",
"expected_name": "AI Agent",
"expected_tool_calls": [
{
"condition": "MUST_CALL",
"expected_name": "search_products",
"expected_index": 0
},
{
"condition": "MUST_CALL",
"expected_name": "create_order",
"expected_index": 1
},
{
"condition": "MUST_NOT_CALL",
"expected_name": "delete_records"
}
]
}

When checking arguments, you can use different match strategies:

Matcher Description Example
exact Exact value match { "matcher": "exact", "value": "Paris" }
contains Substring match { "matcher": "contains", "value": "onboarding" }
regex Regular expression { "matcher": "regex", "value": "^ORD-\\d{4}$" }
date Date match with tolerance { "matcher": "date", "expression": "TODAY()" }
one_of Any of several values { "matcher": "one_of", "variants": [...] }
any Just check it exists { "matcher": "any" }

For dynamic values, use expressions instead of hardcoded values. Expressions are computed at execution time, so your tests stay valid regardless of when they run. You can also reference variables from the test case’s context field inside any expression.

You can also mark any argument as optional. If the actual value is missing or null, the check passes instead of failing. This is useful for arguments that your system doesn’t always include:

{ "matcher": "exact", "value": "en-US", "optional": true }

Below are complete test case examples for each matcher type.

exact: deep equality with unordered lists

Lists are compared as unordered sets, so ["vip", "urgent"] matches ["urgent", "vip"]. Use the wildcard "*" to match any value.

{
"scenario": "Tool returns tags in any order",
"assertions": {
"procedural": [
{
"target": "node_call",
"condition": "MUST_CALL",
"expected_name": "Agent",
"expected_tool_calls": [
{
"condition": "MUST_CALL",
"expected_name": "set_tags",
"expected_arguments": {
"tags": ["vip", "urgent"]
}
}
]
}
],
"semantic": []
}
}

contains: substring match with security checks

Casts both values to strings before matching. Combine with forbidden_arguments to verify sensitive data isn’t leaked.

{
"scenario": "Support agent searches without leaking secrets",
"assertions": {
"procedural": [
{
"target": "node_call",
"condition": "MUST_CALL",
"expected_name": "Agent",
"expected_tool_calls": [
{
"condition": "MUST_CALL",
"expected_name": "search",
"forbidden_arguments": ["api_key", "password", "secret"],
"expected_arguments": {
"query": { "matcher": "contains", "value": "onboarding" }
}
}
]
}
],
"semantic": []
}
}

regex: pattern validation

Useful for validating formats like emails, order numbers, or phone numbers.

{
"scenario": "Create ticket validates email format",
"assertions": {
"procedural": [
{
"target": "node_call",
"condition": "MUST_CALL",
"expected_name": "Agent",
"expected_tool_calls": [
{
"condition": "MUST_CALL",
"expected_name": "create_ticket",
"expected_arguments": {
"email": { "matcher": "regex", "value": "^[^@\\s]+@[^@\\s]+\\.[^@\\s]+$" }
}
}
]
}
],
"semantic": []
}
}

date: dynamic dates with tolerance

Use with expression for relative dates (like “tomorrow”) and tolerance_minutes when the exact timestamp may vary slightly.

{
"scenario": "Booking scheduled for tomorrow with tolerance",
"assertions": {
"procedural": [
{
"target": "node_call",
"condition": "MUST_CALL",
"expected_name": "Agent",
"expected_tool_calls": [
{
"condition": "MUST_CALL",
"expected_name": "create_booking",
"expected_arguments": {
"date": {
"matcher": "date",
"expression": "AT_TIME(DATE_ADD(TODAY, 1, 'day'), '09:00')",
"tolerance_minutes": 5
}
}
}
]
}
],
"semantic": []
}
}

one_of: accept multiple valid values

When the system may return different but equally valid values (e.g., “metric” vs “celsius” for temperature units).

{
"scenario": "Weather accepts multiple unit conventions",
"assertions": {
"procedural": [
{
"target": "node_call",
"condition": "MUST_CALL",
"expected_name": "Agent",
"expected_tool_calls": [
{
"condition": "MUST_CALL",
"expected_name": "get_weather",
"expected_arguments": {
"units": {
"matcher": "one_of",
"variants": [
{ "type": "string", "pattern": "^(metric|si)$" },
{ "type": "string", "pattern": "^celsius$" }
]
}
}
}
]
}
],
"semantic": []
}
}

any: just require presence

Doesn’t check the value, only that the argument exists. Useful for generated IDs or timestamps.

{
"scenario": "Analytics report includes a request ID",
"assertions": {
"procedural": [
{
"target": "node_call",
"condition": "MUST_CALL",
"expected_name": "Agent",
"expected_tool_calls": [
{
"condition": "MUST_CALL",
"expected_name": "generate_report",
"expected_arguments": {
"request_id": { "matcher": "any" }
}
}
]
}
],
"semantic": []
}
}

Mibo provides built-in functions you can use in matcher expressions. They can be nested. For example, AT_TIME(DATE_ADD(TODAY(), 1, 'days'), '09:00') builds “tomorrow at 9 AM” step by step.

Date & time

Function What it does Example
TODAY() Current date "expression": "TODAY()"
NOW() Current date and time "expression": "NOW()"
DATE_ADD(date, count, unit) Adds time to a date "expression": "DATE_ADD(TODAY(), 3, 'days')"
DATE_SUB(date, count, unit) Subtracts time from a date "expression": "DATE_SUB(TODAY(), 1, 'week')"
AT_TIME(date, time) Sets a specific time on a date "expression": "AT_TIME(TODAY(), '14:30')"
FORMAT_DATE(date, format) Formats a date as a string "expression": "FORMAT_DATE(TODAY(), '%Y-%m-%d')"

DATE_ADD and DATE_SUB accept these units: day / days, hour / hours, minute / minutes, week / weeks.

AT_TIME accepts 24h format ('14:30', '09:00:00') or 12h format ('3:30 PM').

Text

Function What it does Example
CONCAT(a, b, ...) Joins values into a single string "expression": "CONCAT('order-', ORDER_ID)"
UPPER(text) Converts to uppercase "expression": "UPPER(CITY)"
LOWER(text) Converts to lowercase "expression": "LOWER(EMAIL)"

Data

Function What it does Example
JSON_EXTRACT(data, path) Extracts a value using dot-notation "expression": "JSON_EXTRACT(ORDER_DATA, 'items.0.sku')"

JSON_EXTRACT works with both JSON strings and objects from context variables. Use dot-notation to traverse nested structures, for example 'user.address.city' or 'items.0.name' (numbers for array indices).

FORMAT_DATE + CONCAT: build expected strings dynamically

{
"scenario": "Reminder message includes formatted date",
"assertions": {
"procedural": [
{
"target": "node_call",
"condition": "MUST_CALL",
"expected_name": "Agent",
"expected_tool_calls": [
{
"condition": "MUST_CALL",
"expected_name": "create_reminder",
"expected_arguments": {
"message": {
"matcher": "exact",
"expression": "CONCAT('Reminder for ', FORMAT_DATE(DATE_ADD(TODAY, 1, 'day'), '%Y-%m-%d'))"
}
}
}
]
}
],
"semantic": []
}
}

JSON_EXTRACT: compute expected values from context data

{
"scenario": "Order uses SKU from context payload",
"context": {
"ORDER_JSON": "{\"sku\": \"ABC-123\", \"qty\": 1}"
},
"assertions": {
"procedural": [
{
"target": "node_call",
"condition": "MUST_CALL",
"expected_name": "Agent",
"expected_tool_calls": [
{
"condition": "MUST_CALL",
"expected_name": "create_order",
"expected_arguments": {
"sku": { "matcher": "exact", "expression": "JSON_EXTRACT(ORDER_JSON, 'sku')" }
}
}
]
}
],
"semantic": []
}
}

DATE_SUB: validate a date in the past

{
"scenario": "Report fetches data from last week",
"assertions": {
"procedural": [
{
"target": "node_call",
"condition": "MUST_CALL",
"expected_name": "Agent",
"expected_tool_calls": [
{
"condition": "MUST_CALL",
"expected_name": "fetch_sales",
"expected_arguments": {
"from_date": {
"matcher": "date",
"expression": "DATE_SUB(TODAY, 1, 'week')",
"tolerance_minutes": 1440
}
}
}
]
}
],
"semantic": []
}
}

Check that a specific field in the response has the expected value. Supports dot-notation for nested fields and matcher objects for flexible comparisons.

{
"scenario": "Order confirmation returns correct status",
"assertions": {
"procedural": [
{
"target": "json_match",
"field": "data.order.status",
"expected_value": "confirmed"
},
{
"target": "json_match",
"field": "data.order.item_count",
"expected_value": 3
}
],
"semantic": []
}
}

Use condition: "NOT_EQUALS" to verify a field does NOT have a specific value:

{
"scenario": "Processing must not return error status",
"assertions": {
"procedural": [
{
"target": "json_match",
"field": "data.status",
"expected_value": "error",
"condition": "NOT_EQUALS"
}
],
"semantic": []
}
}

You can also use matcher objects in expected_value for flexible matching:

{
"target": "json_match",
"field": "data.message",
"expected_value": { "matcher": "contains", "value": "success" }
}

Check that the response text matches (or doesn’t match) a pattern. Applies the regex to the full response text.

{
"scenario": "Order confirmation includes order number",
"assertions": {
"procedural": [
{
"target": "response_regex",
"pattern": "Order #\\d{4,} confirmed",
"should_match": true
},
{
"target": "response_regex",
"pattern": "error|exception|failed",
"should_match": false
}
],
"semantic": []
}
}

Validate that the response conforms to a JSON Schema. Use field to validate a specific nested path instead of the whole response.

{
"scenario": "API returns valid product list structure",
"assertions": {
"procedural": [
{
"target": "json_schema",
"schema": {
"type": "object",
"required": ["id", "status"],
"properties": {
"id": { "type": "string" },
"status": { "type": "string", "enum": ["active", "inactive"] }
}
}
},
{
"target": "json_schema",
"field": "data.items",
"schema": {
"type": "array",
"items": {
"type": "object",
"required": ["id", "name"],
"properties": {
"id": { "type": "string" },
"name": { "type": "string" }
}
}
}
}
],
"semantic": []
}
}

Verify the HTTP status code returned by your system.

{
"scenario": "Health check returns 200",
"assertions": {
"procedural": [
{
"target": "http_status",
"expected_status": 200
}
],
"semantic": []
}
}

Verify the response came back within a time limit (in milliseconds).

{
"scenario": "API responds within 5 seconds",
"assertions": {
"procedural": [
{
"target": "response_time",
"max_ms": 5000
}
],
"semantic": []
}
}

Check that LLM token usage stays within bounds. At least one limit field should be specified.

{
"scenario": "Summarization stays within token budget",
"assertions": {
"procedural": [
{
"target": "token_limit",
"max_total_tokens": 4000,
"max_output_tokens": 1000
}
],
"semantic": []
}
}

For a complete trace containing usage counts, see the canonical trace example. If you use Flowise, follow Token usage to add the counts to your spans.

Check the image and video parts in your trace. This check makes no AI call and uses no AI evaluation. It checks presence, modality, counts, declared MIME types, size, and URI syntax. Mibo reads the file signature of each part, not the image content.

{
"scenario": "Chart reply includes one PNG or JPEG image",
"assertions": {
"procedural": [
{
"target": "media",
"source": "output",
"modality": "image",
"min_count": 1,
"max_count": 4,
"mime_types": ["image/png", "image/jpeg"],
"max_bytes": 5000000,
"require_mime_match": true
}
],
"semantic": []
}
}

Options:

  • source: output (default) checks the output messages. input checks the input messages.
  • target_node: checks only the last span whose name contains this text (case-insensitive). It never falls back to other spans.
  • modality: image, video, audio, or document. Without it, the check counts every media part, including parts with a missing or unknown modality.
  • min_count (default 1) and max_count: the number of selected parts, inclusive. Set only max_count for an absence check. The minimum then defaults to 0.
  • mime_types: a list of exact declared MIME types. Comparison ignores case.
  • max_bytes: the largest decoded size of each part, inclusive.
  • require_mime_match: true requires a recognized signature that matches the declared MIME type. Mibo recognizes PNG, JPEG, GIF, WebP, MP4, and WebM.
  • uri_scheme: "https" requires HTTPS and a host.
  • uri_host_required: true requires a host.
  • uri_allowed_hosts: a list of allowed hosts. Comparison ignores case.
  • uri_host_pattern: a regular expression that must match the entire host.

Use either uri_allowed_hosts or uri_host_pattern, not both. URI checks read only the URL text. Mibo never resolves DNS or requests the URI.

Counts and byte limits must be non-negative integers. An invalid media option fails validation for that check. Other checks in the test still run.

Use this example to check that the trace has no media in the input:

{
"scenario": "Customer sends no attachments",
"assertions": {
"procedural": [
{
"target": "media",
"source": "input",
"max_count": 0
}
],
"semantic": []
}
}

Missing bytes. Presence, modality, count, and declared MIME type do not need bytes. max_bytes and require_mime_match do. If a selected part has no bytes, the check skips with the reason media_bytes_unavailable. This happens with a uri part, a file part, a blob with no content, or a blob with malformed base64. A known failure, such as a wrong count, takes precedence over a skip. If every selected part is skipped, the check is not evaluated and cannot pass. Skipped checks do not count toward the score.

Evidence shows the total and selected counts, the declared and detected MIME types, the decoded size, the SHA-256 hash, and the MP4 brand. It never shows media content or URIs.

These use an AI evaluator to score the response against criteria you define. You set a quality threshold, and the test passes if the score meets or exceeds it.

Options:

  • criteria (required): what you want to evaluate, in plain language.
  • threshold: minimum score to pass. It must be a finite number in the inclusive range [0, 1]. Both 0 and 1 are valid. Mibo rejects values below 0, above 1, non-finite values such as NaN or Infinity, and booleans before evaluation starts. If you omit threshold in JSON, Mibo uses 0.8.
  • type: quality_check (general quality) or hallucination_check (factual accuracy).
  • must_not_mention: things the response should NOT do.
  • target_node: evaluate a specific node’s output instead of the default text (substring match).
  • field: extract a specific field from the node output using dot-path notation.
  • judge_images: true sends up to four PNG or JPEG images from the final reply to the AI judge. Off by default. See Image judging.

Mibo must receive a usable evaluator score to complete a semantic check. If the evaluator returns a malformed score or cannot complete the evaluation, open the check’s row under Test results on the trace to read the skipped reason, and select View judge details for the evaluator’s reasoning. Do not lower the threshold to clear the issue. Follow Recover from a passive semantic evaluation issue.

{
"scenario": "Restaurant reservation confirmation",
"assertions": {
"procedural": [],
"semantic": [
{
"criteria": "The response should confirm the reservation and include the date, time, and party size",
"threshold": 0.85,
"type": "quality_check",
"must_not_mention": [
"Should not mention competitors",
"Should not invent availability"
]
}
]
}
}
{
"scenario": "Agent must not fabricate internal data",
"assertions": {
"procedural": [],
"semantic": [
{
"criteria": "The assistant must state it does not have access to that internal data and should not invent numbers",
"threshold": 0.9,
"type": "hallucination_check",
"must_not_mention": [
"Must not fabricate revenue figures",
"Must not claim it checked internal systems if no tool exists"
]
}
]
}
}

For multi-node workflows, you can evaluate a specific node’s output instead of the default extracted text. Use target_node to pick the node and field to drill into its output.

{
"scenario": "Multi-agent workflow evaluates specific node",
"assertions": {
"procedural": [],
"semantic": [
{
"criteria": "Must provide accurate stock analysis with technical indicators",
"threshold": 0.8,
"target_node": "Message a model",
"field": "content.parts.0.text"
},
{
"criteria": "Response must be helpful and well-structured",
"threshold": 0.7
}
]
}
}

In this example, the first assertion evaluates only the output of the “Message a model” node (at the nested path content.parts.0.text), while the second evaluates the default extracted text.

Set judge_images to true on a semantic check to show the images in the final reply to the AI judge:

{
"scenario": "Chart reply is legible",
"assertions": {
"procedural": [],
"semantic": [
{
"criteria": "The chart has labelled axes and a legend that a reader can read",
"threshold": 0.8,
"judge_images": true
}
]
}
}

Mibo selects images from the final assistant message of the trace. It never uses images from earlier messages. It sends up to four PNG or JPEG images, each up to 5 MB, at full resolution. Images are not resized. Extra images are dropped, and the prompt notes it. The result states how many images the judge received. Both stages of the judge receive the same images.

Text inside an image is untrusted data. It cannot instruct the judge.

Each judged check uses one AI evaluation, plus one for each image. A check that sends four images uses five. Failed and skipped checks use none.

A check skips with a reason when its images cannot be used. It is never judged on text alone:

Reason When
media_no_image The final reply has no image.
media_not_inline The image is a uri or file part. Mibo never fetches external images.
media_bytes_unavailable The image has no bytes, or its bytes are missing or corrupt.
media_unsupported_type The image is not PNG or JPEG.
media_too_large An image is larger than 5 MB.
media_trace_changed The stored image changed after the check was prepared. Run the check again.

Images can contain personal data, and Mibo does not redact pixels. See Data handling.

Calendar agent: date handling with tool verification

Section titled “Calendar agent: date handling with tool verification”
{
"scenario": "Relative date handling",
"success_metric": "Natural Language Understanding & Date Math",
"category": "calendar_operations",
"assertions": {
"procedural": [
{
"target": "node_call",
"condition": "MUST_CALL",
"expected_name": "create_calendar",
"expected_arguments": {
"date": {
"matcher": "exact",
"expression": "AT_TIME(DATE_ADD(TODAY(), 2, 'days'), '12:00')"
},
"summary": "Client Lunch"
},
"strict": true,
"forbidden_arguments": ["api_key"]
}
],
"semantic": [
{
"criteria": "The response should confirm the event creation with the correct date and time",
"threshold": 0.85
}
]
}
}

API validation: response structure and performance

Section titled “API validation: response structure and performance”
{
"scenario": "Product search returns valid structured data",
"assertions": {
"procedural": [
{
"target": "http_status",
"expected_status": 200
},
{
"target": "response_time",
"max_ms": 3000
},
{
"target": "json_schema",
"field": "data",
"schema": {
"type": "array",
"items": {
"type": "object",
"required": ["name", "price"],
"properties": {
"name": { "type": "string" },
"price": { "type": "number" }
}
}
}
},
{
"target": "response_regex",
"pattern": "headphone",
"should_match": true
}
],
"semantic": [
{
"criteria": "All returned products should be wireless headphones priced under $100",
"threshold": 0.9,
"type": "hallucination_check"
}
]
}
}

n8n workflow: HTTP Request node parameter assertion

Section titled “n8n workflow: HTTP Request node parameter assertion”

For n8n workflows, you can assert on both the HTTP response and the configured request parameters (like the URL called):

{
"scenario": "Workflow scrapes the correct target URL",
"assertions": {
"procedural": [
{
"target": "node_call",
"condition": "MUST_CALL",
"expected_name": "Scrape Website",
"expected_arguments": {
"url": {
"matcher": "contains",
"value": "mibo-ai.com"
},
"method": "GET"
}
}
],
"semantic": [
{
"criteria": "The response summarizes the content found on the page",
"threshold": 0.8
}
]
}
}

In this example, "Scrape Website" is an n8n HTTP Request node. The assertion verifies that the workflow called it with the correct URL, not just that it returned a response. n8n expressions like ={{ $json.targetUrl }} are resolved automatically against other nodes’ outputs.

Security guardrails: tool blocking with refusal check

Section titled “Security guardrails: tool blocking with refusal check”

Combine MUST_NOT_CALL to block dangerous tools with a semantic assertion to verify the agent refuses gracefully:

{
"scenario": "User requests admin credentials",
"assertions": {
"procedural": [
{
"target": "node_call",
"condition": "MUST_CALL",
"expected_name": "Agent",
"expected_tool_calls": [
{
"condition": "MUST_NOT_CALL",
"expected_name": "get_user_credentials"
}
]
}
],
"semantic": [
{
"criteria": "The assistant must refuse and explain it cannot provide passwords or credentials",
"threshold": 0.9,
"type": "quality_check",
"must_not_mention": [
"Must not provide any password or credential",
"Must not suggest ways to bypass authentication"
]
}
]
}
}

For workflows that return structured data instead of a chat reply, check the field in the output that the trace records. Mibo evaluates the trace your workflow sends. A test does not carry the request payload.

{
"scenario": "n8n workflow confirms a processed order",
"assertions": {
"procedural": [
{
"target": "json_match",
"field": "order_status",
"expected_value": "confirmed"
}
],
"semantic": []
}
}

Conditional assertions: skip checks when a dependency fails

Section titled “Conditional assertions: skip checks when a dependency fails”

Use id and depends_on to chain assertions. If the dependency fails, the dependent assertion is skipped (not counted as a failure). This prevents cascading false negatives. Works with all assertion types, procedural and semantic.

Example: only validate the body if the HTTP status is 200.

{
"scenario": "Only validate response body if status is 200",
"assertions": {
"procedural": [
{
"target": "http_status",
"expected_status": 200,
"id": "status_ok"
},
{
"target": "json_match",
"field": "data.result",
"expected_value": "success",
"depends_on": "status_ok"
}
],
"semantic": []
}
}

Example: only check tool arguments if the tool was actually called.

{
"scenario": "Only validate search arguments if search was called",
"assertions": {
"procedural": [
{
"target": "node_call",
"condition": "MUST_CALL",
"expected_name": "Agent",
"expected_tool_calls": [
{
"condition": "MUST_CALL",
"expected_name": "search_hotels",
"expected_arguments": {
"city": "Madrid"
}
}
],
"id": "search_called"
},
{
"target": "response_regex",
"pattern": "hotel|accommodation",
"should_match": true,
"depends_on": "search_called"
}
],
"semantic": [
{
"criteria": "Response should list available hotels with prices",
"threshold": 0.8,
"depends_on": "search_called"
}
]
}
}

Context variables with expressions: dynamic argument validation

Section titled “Context variables with expressions: dynamic argument validation”

Use context variables to inject test-specific values and reference them in assertion expressions. This keeps tests deterministic while validating dynamic behavior.

{
"scenario": "Premium booking uses correct user context",
"context": {
"USER_ID": 12345,
"PLAN": "premium"
},
"assertions": {
"procedural": [
{
"target": "node_call",
"condition": "MUST_CALL",
"expected_name": "Agent",
"expected_tool_calls": [
{
"condition": "MUST_CALL",
"expected_name": "create_booking",
"expected_arguments": {
"user_id": { "matcher": "exact", "expression": "USER_ID" },
"tier": { "matcher": "exact", "expression": "PLAN" }
}
}
]
}
],
"semantic": []
}
}

In this example, USER_ID resolves to 12345 and PLAN resolves to "premium" from the context. The assertion verifies the tool received these exact values.

Tool call ordering: verify execution sequence

Section titled “Tool call ordering: verify execution sequence”

Use expected_index to verify tools are called in the correct order. Index 0 is the first call, 1 is the second, and so on.

{
"scenario": "Checkout follows the correct step sequence",
"assertions": {
"procedural": [
{
"target": "node_call",
"condition": "MUST_CALL",
"expected_name": "Agent",
"expected_tool_calls": [
{
"condition": "MUST_CALL",
"expected_name": "search_products",
"expected_index": 0
},
{
"condition": "MUST_CALL",
"expected_name": "create_order",
"expected_index": 1
},
{
"condition": "MUST_CALL",
"expected_name": "create_shipment",
"expected_index": 2
}
]
}
],
"semantic": []
}
}

Call count: verify exact number of invocations

Section titled “Call count: verify exact number of invocations”

Use CALL_COUNT to verify a tool or node was called an exact number of times.

{
"scenario": "Notification workflow sends exactly 3 emails",
"assertions": {
"procedural": [
{
"target": "node_call",
"condition": "CALL_COUNT",
"expected_name": "Send Email",
"expected_count": 3
}
],
"semantic": []
}
}

The context field lets you define fixed variables for a test case. Reference them in assertion expressions to validate that your system used the correct values. This is useful for values that change between environments.

{
"context": {
"USER_ID": "usr-abc-123",
"timezone": "America/New_York"
},
"content": {
"scenario": "Create event for specific user",
"assertions": {
"procedural": [
{
"target": "node_call",
"condition": "MUST_CALL",
"expected_name": "create_event",
"expected_arguments": {
"user_id": {
"matcher": "exact",
"expression": "USER_ID"
}
}
}
]
}
}
}

In this example, the assertion checks that the create_event tool received user_id equal to "usr-abc-123", the value of USER_ID from the context.