Agent interface regression testing for MCP servers
Modelara tells you whether a change to the tools your AI agent uses breaks real workflows, changes how the agent behaves, increases cost, or gives the agent new authority, before you ship the change.
It connects to an MCP server, runs real tasks through a real model against it, records every turn and tool call locally, scores each run deterministically, and compares a candidate interface with an accepted baseline under an explicit, versioned policy. The result is one of four verdicts with stable CI exit codes and a reason for every rule that fired.
modelara init→ modelara test→ upgrade the server→ modelara test→ approve or explain
Everything runs on your machine. Prompts, tool arguments, tool results, model answers and credentials never leave it. Data leaves only through the model provider you configure, and through an optional, explicit sync of safe metadata.
Contents
- Quick start
- Core concepts
- Onboarding
- How a benchmark runs
- Verdicts and rules
- Baselines and approval
- Explaining failures
- Configuration
- Task suites
- CI integration
- Security and privacy
- Team service
- Command reference
- Requirements and limits
Quick start
Install the CLI on macOS or Linux (arm64 or x86_64). The script downloads the latest release and verifies its checksum:
curl -fsSL https://modelara.ai/install.sh | sh
To see what an upgrade changes in the interface without a model, tasks or API key, see Check an MCP upgrade without a model.
Then, in a project directory inside a Git repository:
export OPENAI_API_KEY=… # the model that drives the agent
export RESEND_API_KEY=… # whatever your MCP server needs (names only are stored)
modelara init # type the server's command line, e.g. npx -y resend-mcp@2.27.2
modelara test # first run: this benchmark becomes the baseline
After you change or upgrade the server, run modelara test again. It benchmarks the interface as it is now and compares it with your accepted baseline automatically:
MODELARA TEST
Testing: resend 2.27.2
Baseline: resend 2.26.0 (accepted 2026-10-29 19:40, first test)
✓ Correctness 25/25 → 25/25
✓ Behavior stable
✓ Efficiency total tokens +3%
✗ Security 3 new authority change(s) not approved
add-inbox-thread-labels (new, effect write, destructive: no)
remove-inbox-thread-label (new, effect write, destructive: no)
set-inbox-thread-labels (new, effect write, destructive: no)
RESULT: FAIL
Next:
modelara approve approve intended authority changes (exact capabilities)
modelara explain details
| Result | What happens | What you do |
|---|---|---|
| PASS | The new version becomes the baseline automatically | Nothing |
| FAIL new authority | Baseline unchanged | If the new write tools are intended: modelara approve |
| FAIL anything else | Baseline unchanged | modelara explain, then fix the cause |
| REVIEW | Baseline unchanged until a person accepts it | modelara explain; to accept: modelara approve |
| INVALID | Baseline unchanged | modelara explain: the comparison cannot be trusted |
Core concepts
Interface
The set of tools an MCP server exposes, normalized into a protocol-neutral capability description: names, schemas, and declared effects (read, write, destructive, unknown) from MCP annotations. Each interface has a content fingerprint, so any change to a tool definition is visible.
Task
A real job your agent must reliably complete, written as a user prompt plus how to recognize a correct answer. Tasks describe the desired outcome, never which tools to call.
Task suite
A versioned YAML file of tasks. Its fingerprint covers task IDs, prompts, expectations and limits, so a benchmark always records exactly which tasks it ran.
Environment
The data your tasks run against, identified by a fingerprint you supply or the hash of a file. Expected answers describe this data. Two benchmarks are comparable only if they ran against the same environment.
Benchmark
One recorded execution: every task, run several times (3 by default), through one model against one interface. Every turn, tool call and result is stored locally.
Baseline
The most recently accepted benchmark for this project, task suite, model settings and environment. modelara test compares every candidate with it.
Check
The decision: candidate versus baseline under a policy, producing PASS, REVIEW, FAIL or INVALID, a list of reasons with stable rule IDs, and a result fingerprint.
Lineage
Every benchmark and check records what it was made from (project, interface, suite, environment, execution policy, evaluator version), so any verdict can be traced to its inputs.
Onboarding: init and task bootstrap
modelara init asks a few questions and writes the smallest useful modelara.yaml. You type the server’s command line as you would in a shell (npx -y my-mcp@1.2.3, docker run -i --rm my-image, ./server --config "config files/dev.yaml"). It is split deterministically, without invoking a shell; quoted arguments are supported and a malformed quote is an error.
- Secrets are never written. You name the environment variables the server needs; their values stay in your shell. A value pasted by mistake (
TOKEN=…, ansk-orghp_token in an argument) is refused and never echoed. - The server sees only what it needs. The generated config sets
inherit_env: falseand passes through onlyPATH,HOMEand the variables you named. The model API key is stripped from the server’s environment. - An environment is declared. The config names the data your tasks run against, so later comparisons are valid.
Task bootstrap
Writing the first task suite is the hardest step for a new user. init offers to create one, and modelara tasks bootstrap does the same at any time:
- Connects to the server and suggests up to ten workflows built only from tools that are declared read-only, retry-safe and need no input. Write, destructive, unannotated and credential-related tools (API keys, tokens, OAuth grants, sessions) are never suggested. Each suggestion shows the tool it comes from.
- You choose which to try (
1,3,4, ranges,all,none). - Each selected workflow runs at least twice through your model as a read-only task, exactly as a benchmark would. A model that tries a write tool is blocked before the call reaches the server.
- Modelara proposes an expectation only from the tool’s own machine-readable output, and only when every run agrees.
- You confirm each one. Nothing becomes a release expectation without explicit confirmation. Scripts, CI and
init --yesnever confirm anything.
Tool output is read through one shared normalization, used everywhere results are analyzed, in this order:
| Tier | Read from | Rules |
|---|---|---|
| 1 | MCP structuredContent |
Authoritative whenever it decodes |
| 2 | JSON carried in text blocks | A block that is wholly one JSON object or array; prose blocks beside it are ignored, never parsed; JSON inside a sentence is never extracted |
| 3 | Strict key/value text | Every line must be Label: value, a short section header, or a leading Found <n> <noun>:. Any other line rejects the whole document. Values: integers, decimals, n / m pairs, booleans, short strings |
| — | Anything else (prose) | Not machine-readable. No expectation is inferred; you may type text a correct answer must contain, or skip |
Runs are compared on the canonical normalized value: key order and whitespace do not matter, array order does, and volatile fields are set aside. Identifiers, timestamps, URLs, email addresses and secret-like fields never become expected values. Embedded resources and binary content are never read as data. Observations are kept in memory and are not stored; only confirmed values are written to tasks.yaml.
How a benchmark runs
Before anything runs, a preflight checks every prerequisite and reports all problems at once: the configuration, the task suite, the model API key, the environment, and the MCP server (started, tools discovered). A suite that still contains the unedited starter task is refused, so a placeholder can never become a release gate.
MODELARA PREFLIGHT
✓ task suite loaded: 5 task(s) (my-agent-workflows)
✗ OPENAI_API_KEY is not set
✓ environment: external_asserted my-data-v1
✓ MCP server started successfully
✓ 136 tools discovered
Benchmark not started.
Fix:
export OPENAI_API_KEY=… (read from the environment; never stored or passed to the MCP server)
The agent loop
For each run, the model receives the task prompt and the interface’s tools, calls tools, receives their results, and ends with a structured final answer. The run stops at a final answer, the turn limit, the tool-call limit, the timeout, a provider error, or a lost MCP connection.
- Read-only enforcement. In a read-only task (the only mode currently supported), any tool not declared read-only is blocked before execution. The model is told the call was blocked, the call is recorded as
policy_blocked, and the run is scoredpolicy_violation. Where writes are prevented outside the interface (a read-only database role, a token without write scopes), an execution assertion lets read-only tasks use those tools; it is recorded as a dependency, never treated as intrinsic safety. - What the model sees of a tool result is exactly what is stored. Text blocks as returned; embedded text resources as their text; binary content only as a placeholder such as
[non-text resource: image/png], never as bytes or base64. - Round-robin repetitions. Every task runs once, then every task again, so a transient condition does not land on all runs of one task.
- Final-answer contract. Modelara defines the answer shape,
{"answer": "…", "facts": {…}}, and sends it as a strict JSON schema. The model learns which facts to report, never their expected values.
Deterministic evaluation
No model grades another. Each run gets exactly one outcome, in this precedence:
| Outcome | Meaning |
|---|---|
aborted timeout model_error mcp_error max_turns max_tool_calls |
The run ended without a final answer |
policy_violation |
At least one call was blocked by task policy |
forbidden_content |
A not_contains string appeared |
wrong_answer |
A reported fact differs from the expected value, or a contains string is missing |
missing_fact |
An expected fact was absent or null, or the answer was not structured |
success |
Every check passed |
Facts compare exactly after trimming and ignoring case. Numbers accept forms like "$1,234.50" with tiny float tolerance, booleans accept "true"/"false", an expected YYYY-MM-DD matches a timestamp on that date, and a list of values accepts any of them as equivalent forms.
Verdicts and rules
PASSexit 0
No rule hit.
REVIEWexit 3
Not clearly unsafe, but behavior changed or the evidence is inconclusive.
FAILexit 1
A regression gate was violated.
INVALIDexit 2
The comparison cannot be trusted (different suite, model, environment, incomplete runs).
Each model is judged on its own baseline/candidate pair; nothing is averaged across models. The overall verdict is the most severe, ordered INVALID > FAIL > REVIEW > PASS. Modelara reports every observed difference but gates only on differences with enough evidence; weaker ones are listed as observations (INFO) and never change the verdict.
Default policy (version 4)
| Rule | Hit when | Default |
|---|---|---|
comparability.* |
Different task suite, model, model parameters, runs per task or clock; a benchmark not completed; differing evaluator versions; incomplete runs | INVALID |
comparability.environment_mismatch |
Both benchmarks recorded an environment and they differ | INVALID |
comparability.environment_unknown |
Either benchmark has no recorded environment | INVALID (REVIEW when explicitly allowed) |
correctness.overall_regression |
A model’s success rate drops by more than 2 percentage points | FAIL if pooled Fisher p < 0.05, else REVIEW |
correctness.task_regression |
A task loses more than 1 success and the loss is corroborated: significant on its own, repeated strategy divergence, a significant overall regression, a structural change to a tool the task used, or new blocked behavior | FAIL |
correctness.task_loss_uncorroborated |
A task loses more than 1 success with nothing corroborating it | REVIEW |
correctness.task_borderline |
A task loses exactly 1 success | REVIEW |
correctness.new_failure_mode |
A failure class the baseline never showed | REVIEW |
drift.repeated_strategy_change |
The agent’s tool-selection strategy changed, against a stable baseline, in at least max(2, round(0.7 × runs)) candidate runs on a task that did not improve | REVIEW |
efficiency.regression |
Mean turns, tool calls or total tokens rise by more than 25% | REVIEW |
security.authority_expansion |
A new tool, or a stronger declared effect, that is not read-only and not approved | FAIL |
security.new_execution_dependency |
The candidate relies on externally enforced read-only execution the baseline did not | REVIEW |
security.new_blocked_behavior |
The agent now attempts a tool or effect that is blocked (any count if it may destroy state) | FAIL / REVIEW |
runs.insufficient |
Fewer than 3 runs per task | REVIEW |
Every reason carries structured evidence (the loss, its p-value, which corroborators applied, deviating and required runs, baseline variability), so a consumer can tell why a loss is FAIL or REVIEW without parsing text. Thresholds and actions are configurable; the effective policy is fingerprinted and stored with every check. Use 5 or more runs per task for release gating.
Approved authority expansion
New write tools fail by default. That is right for a team consuming an MCP server. A vendor intentionally adding capabilities approves them by exact name:
check:
security:
approved_capabilities: # exact names; no patterns, prefixes or wildcards
- create-inbox
- remove-inbox
Approval changes only the release classification. It does not grant Modelara permission to execute the tool. Read-only tasks still block it, an attempted call is still blocked behavior, and every other rule still applies. Any expansion not listed still fails. The list is part of the policy fingerprint.
Baselines and approval
modelara test selects the baseline itself: the most recently accepted benchmark of the same project with the same task suite, model settings (provider, API, model, parameters) and environment. The interface is deliberately not part of the match, since that is what changes. Changing the suite, model or environment starts a new baseline instead of making an invalid comparison.
| How a benchmark becomes the baseline | Recorded as |
|---|---|
The first modelara test |
initial, automatic |
| A candidate that passes | pass, automatic |
A REVIEW a person accepts with modelara approve |
review_accepted, with who accepted it |
| A FAIL whose new authority a person approves, and whose re-check then passes | approved, with who approved and which capabilities |
FAIL, INVALID and unaccepted REVIEW never become the baseline; the storage layer itself refuses them. modelara status shows the current baseline, why it is the baseline, and any pending result.
modelara approve acts on the latest test. For new authority it lists each pending capability with its effect and asks for approval by exact name (or --capability NAME, --all for every pending one). Approved names are added to check.security.approved_capabilities in modelara.yaml without disturbing the rest of the file, and the check re-runs. For REVIEW it shows the reasons and asks whether to accept the candidate (--accept-review in scripts). Commit the updated config so CI applies the same policy.
Explaining failures
modelara explain answers, for every failed run: what was asked, what was expected, what the model answered, which tools it called and how each call went, and why the evaluator classified the run as it did. Without an ID it explains the latest pending test, starting with the check’s reasons. It reads only the local database and makes no model call.
Task: read-auth-doc
Runs: 3
Failures: 3
Run #1
Status: wrong_answer
Expected:
session_timeout_minutes: 30
Observed:
session_timeout_minutes: 60
Tools used:
#1 get_file_contents {"owner":"…","path":"docs/authentication.md","repo":"…"}
→ returned successfully (100 bytes): successfully downloaded text file …
Classification:
wrong_answer: a reported fact differs from the expected value
Likely interpretation:
Tool execution succeeded.
The answer did not satisfy the evaluator expectation.
Tool errors, rejected arguments and blocked calls are shown with their messages and reasons. Where the stored evidence cannot determine a cause, explain says so rather than guessing. Output passes through the same redaction as stored traces.
Configuration (modelara.yaml)
Commands find the nearest modelara.yaml in the current directory or a parent, up to the repository root (outside a repository, only the current directory). --config always wins. Unknown keys are rejected, so a stray api_key: fails loudly instead of being ignored. Relative paths resolve against the config file’s directory.
version: 1
project:
name: my-agent
mcp:
transport: stdio # the supported transport
command: npx # executed directly, never through a shell
args: [-y, my-mcp@1.2.3]
inherit_env: false # the server sees only these variables:
env_passthrough: [PATH, HOME, MY_API_KEY]
timeout: 30s # per MCP request
environment:
name: my-data
kind: external_asserted # fixture | database_snapshot | service_snapshot | external_asserted
fingerprint: my-data-v1 # or fingerprint_file: ./snapshot.json (its sha256)
model:
provider: openai
model: gpt-4.1-mini
api: responses # or chat_completions for compatible gateways
temperature: 0
benchmark:
tasks: ./tasks.yaml
runs_per_task: 5
annotation_check: warn # warn | error | off
check: # optional overrides of the default policy
security:
approved_capabilities: []
| Section | Notable fields |
|---|---|
mcp |
env values may reference the parent environment as ${NAME}; dir; declared_overrides (operator-asserted annotations for servers that declare none); execution_assertions (writes prevented outside the interface) |
model |
api_key_env (default OPENAI_API_KEY), base_url, seed, max_output_tokens, request_timeout |
benchmark |
redact_keys (added to the built-in list), setup (a command run before each benchmark, e.g. to reset fixture data) |
check |
correctness, behavior, efficiency, security, runs, environment thresholds and actions (fail, review, ignore) |
storage |
path, default .modelara/modelara.db |
sync |
display_names: opt-in for the optional names channel (default off) |
Task suites
version: 1
suite:
id: my-agent-workflows
tasks:
- id: refund-bob-latest
prompt: Can Bob Martinez's most recent order be refunded?
expected:
facts: {order_id: ord_1008, eligible: false} # values the answer must report
contains: [final sale] # text that must appear (case-insensitive)
not_contains: [card number] # text that must not appear
limits: {max_turns: 8} # defaults: 12 turns, 20 calls, 3m
- Validation reports every problem at once, with what is missing and the supported keys.
modelara tasks initwrites a valid starter to edit. - Hard ceilings prevent runaway cost: 50 turns, 100 tool calls and 15 minutes per run.
- Derive expected values from the environment’s data, never from a model’s answer. Ask for a form the evaluator can compare, and list equivalent forms when one form cannot be enforced (
facts: {room: [C-104, Room C-104]}). - Optional analysis annotations (
required_evidence,reasoning) are never shown to the model;modelara analyzeuses them to locate interface problems.
CI integration
modelara test --ci # never prompts or approves
case $? in
0) echo "pass" ;;
3) echo "needs review" ;;
*) exit 1 ;; # 1 FAIL, 2 INVALID
esac
- Keep the
.modelaradatabase between CI runs so a baseline exists. Without one, nothing is compared and--ciexits 3 rather than appearing to pass. - Approvals live in
modelara.yaml; commit them before CI runs. - Pipelines that benchmark in a separate step can run
modelara test --candidate bench_…, or callmodelara check --baseline … --candidate …directly with--json,--quietor--review-exit-code 0.
Security and privacy
Stays on your machine
- Credentials and environment values
- Prompts and model responses
- Tool arguments and results
- Traces, expected and actual answers
- Business and fixture data
- Names of projects, suites, tasks and tools
explicit
safe sync
→
May be sent (opt-in, explicit)
-
Fingerprints and lineage
-
Counts, success rates, efficiency averages
-
Verdicts, rule IDs, structured evidence
-
Keyed pseudonyms of names (
task:…,tool:…) -
“Evidence exists locally” and its digest
-
Local-first. Inspection, model calls, tool calls, traces, evaluation, comparison and checks all run locally. Nothing is sent to Modelara as a side effect of any command.
-
Secrets. Configuration holds only the names of environment variables. The model API key is read from the environment and stripped from the MCP server’s environment. Before storage, values under keys like
password,secret,token,api_key,authorizationandcookie,key: valueandkey=valuepairs, andBearertokens are redacted. -
Execution safety. Tasks are read-only; tools not declared read-only are blocked before they reach the server. Release approval never changes this.
-
Safe by construction. The sync bundle format has no field that can hold a prompt, argument, result, answer, name or credential, so there is no content scanning to get wrong. Tests seed databases with tokens, passwords, internal hostnames and confidential names and assert none appear in bundle bytes. Keyed pseudonyms use a key that never leaves your machine, so names cannot be confirmed by guessing.
-
Storage. SQLite in WAL mode, pure Go (no CGO), forward-only migrations; a newer database is never opened by an older Modelara.
Optional team service
Modelara works fully offline. For teams, an optional service (currently offered to design partners) adds shared history and a web view. It never computes or overrides a verdict.
- Sync.
modelara sync exportwrites exactly what the service would receive, for offline review;modelara sync pushsends only what the service lacks, with a project-scoped machine token. Ingestion is atomic and idempotent. - Web UI. Check details, histories, per-model verdicts and task history, rendered only from synced safe evidence.
- Accounts. Invite-only email sign-in, revocable sessions, owner and member roles. Owners create, list and revoke machine tokens (shown once, stored only as hashes).
- Display names (opt-in). Readable project, task and tool names travel only on a separate, previewed, confirmed upload with a dedicated token, are visible only to project members, and can be deleted at any time. Safe sync never carries a name.
Command reference
Global flags: --config / -c (default: the nearest modelara.yaml), --log-level.
| Command | Purpose | Key flags |
|---|---|---|
init |
Create modelara.yaml; offer task bootstrap |
--yes --command --env --name --model --environment --force --no-tasks |
test |
Benchmark and check against the accepted baseline | --ci --verbose --candidate --allow-unknown-environment |
approve |
Approve intended authority changes, or accept a REVIEW | --capability --all --accept-review |
explain |
Why runs failed; the latest pending test by default | --benchmark --task --all |
status |
Current baseline and anything pending | |
tasks bootstrap |
Suggest, observe and confirm workflows | --runs |
tasks init |
Write a valid starter suite to edit | --file --force |
inspect |
List tools, declared effects and the interface fingerprint; call one read-only tool | --json --no-store --call --args |
benchmark |
Run the suite and record traces | --runs --model --suite --preflight |
check |
Verdict for explicit baseline/candidate benchmark pairs | --baseline --candidate --json --quiet --review-exit-code |
check --static |
Compare two interface versions without a model, tasks or tool calls | --baseline-command --candidate-command --baseline-interface --candidate-interface --json --sarif --sarif-output |
compare |
Detailed behavior comparison of two benchmarks | --baseline --candidate |
analyze |
Evidence-backed interface findings from one benchmark | --benchmark --json |
trace |
List runs or print one run’s full trace | --task --run --full --json |
history · show · diff · export |
Browse records and lineage; compare interfaces; export metadata without traces or credentials | |
sync · names |
Optional team service: safe evidence and opt-in display names | export inspect push |
propose |
Experimental automatic interface proposals; validate any change with test or check |
Requirements and limits
- Transport: MCP over stdio. Streamable HTTP is not yet supported.
- Model providers: OpenAI (Responses or Chat Completions, plus compatible gateways via
base_url). A scripted provider exists for demos and tests and is labelled as not a model measurement. - Task mode: read-only tasks only; write workflows are reserved and rejected.
- Tool content: MCP resource links are shown to the model as a placeholder and are not fetched; images, audio and binary resources are placeholders by design.
- Execution: runs are sequential.
- Task bootstrap: suggests only tools that need no input; tools that return only prose produce no automatic expectation.
- Platforms: release binaries for macOS and Linux (arm64 and x86_64); Windows builds are not available yet. The binary has no external runtime dependencies. A model API key is needed only for benchmarks and bootstrap;
inspect,check,compare,explain,historyandshowmake no model call.
Modelara · Agent interface regression CI for MCP servers. Exit codes: 0 PASS · 1 FAIL · 2 INVALID · 3 REVIEW.