← Modelara Docs/Modelara Technical Overview

Agent interface regression testing for MCP servers

Modelara tells you whether a change to the tools your AI agent uses breaks real workflows, changes how the agent behaves, increases cost, or gives the agent new authority, before you ship the change.

It connects to an MCP server, runs real tasks through a real model against it, records every turn and tool call locally, scores each run deterministically, and compares a candidate interface with an accepted baseline under an explicit, versioned policy. The result is one of four verdicts with stable CI exit codes and a reason for every rule that fired.

modelara init→ modelara test→ upgrade the server→ modelara test→ approve or explain

Everything runs on your machine. Prompts, tool arguments, tool results, model answers and credentials never leave it. Data leaves only through the model provider you configure, and through an optional, explicit sync of safe metadata.

Contents

  1. Quick start
  2. Core concepts
  3. Onboarding
  4. How a benchmark runs
  5. Verdicts and rules
  6. Baselines and approval
  7. Explaining failures
  8. Configuration
  9. Task suites
  10. CI integration
  11. Security and privacy
  12. Team service
  13. Command reference
  14. Requirements and limits

Quick start

Install the CLI on macOS or Linux (arm64 or x86_64). The script downloads the latest release and verifies its checksum:

curl -fsSL https://modelara.ai/install.sh | sh

To see what an upgrade changes in the interface without a model, tasks or API key, see Check an MCP upgrade without a model.

Then, in a project directory inside a Git repository:

export OPENAI_API_KEY=…            # the model that drives the agent
export RESEND_API_KEY=…            # whatever your MCP server needs (names only are stored)
modelara init                      # type the server's command line, e.g. npx -y resend-mcp@2.27.2
modelara test                      # first run: this benchmark becomes the baseline

After you change or upgrade the server, run modelara test again. It benchmarks the interface as it is now and compares it with your accepted baseline automatically:

MODELARA TEST

Testing:   resend 2.27.2
Baseline:  resend 2.26.0 (accepted 2026-10-29 19:40, first test)

  ✓ Correctness  25/25 → 25/25
  ✓ Behavior     stable
  ✓ Efficiency   total tokens +3%
  ✗ Security     3 new authority change(s) not approved
                   add-inbox-thread-labels    (new, effect write, destructive: no)
                   remove-inbox-thread-label  (new, effect write, destructive: no)
                   set-inbox-thread-labels    (new, effect write, destructive: no)

RESULT: FAIL

Next:
  modelara approve    approve intended authority changes (exact capabilities)
  modelara explain    details
Result What happens What you do
PASS The new version becomes the baseline automatically Nothing
FAIL new authority Baseline unchanged If the new write tools are intended: modelara approve
FAIL anything else Baseline unchanged modelara explain, then fix the cause
REVIEW Baseline unchanged until a person accepts it modelara explain; to accept: modelara approve
INVALID Baseline unchanged modelara explain: the comparison cannot be trusted

Core concepts

Interface

The set of tools an MCP server exposes, normalized into a protocol-neutral capability description: names, schemas, and declared effects (read, write, destructive, unknown) from MCP annotations. Each interface has a content fingerprint, so any change to a tool definition is visible.

Task

A real job your agent must reliably complete, written as a user prompt plus how to recognize a correct answer. Tasks describe the desired outcome, never which tools to call.

Task suite

A versioned YAML file of tasks. Its fingerprint covers task IDs, prompts, expectations and limits, so a benchmark always records exactly which tasks it ran.

Environment

The data your tasks run against, identified by a fingerprint you supply or the hash of a file. Expected answers describe this data. Two benchmarks are comparable only if they ran against the same environment.

Benchmark

One recorded execution: every task, run several times (3 by default), through one model against one interface. Every turn, tool call and result is stored locally.

Baseline

The most recently accepted benchmark for this project, task suite, model settings and environment. modelara test compares every candidate with it.

Check

The decision: candidate versus baseline under a policy, producing PASS, REVIEW, FAIL or INVALID, a list of reasons with stable rule IDs, and a result fingerprint.

Lineage

Every benchmark and check records what it was made from (project, interface, suite, environment, execution policy, evaluator version), so any verdict can be traced to its inputs.

Onboarding: init and task bootstrap

modelara init asks a few questions and writes the smallest useful modelara.yaml. You type the server’s command line as you would in a shell (npx -y my-mcp@1.2.3, docker run -i --rm my-image, ./server --config "config files/dev.yaml"). It is split deterministically, without invoking a shell; quoted arguments are supported and a malformed quote is an error.

Task bootstrap

Writing the first task suite is the hardest step for a new user. init offers to create one, and modelara tasks bootstrap does the same at any time:

  1. Connects to the server and suggests up to ten workflows built only from tools that are declared read-only, retry-safe and need no input. Write, destructive, unannotated and credential-related tools (API keys, tokens, OAuth grants, sessions) are never suggested. Each suggestion shows the tool it comes from.
  2. You choose which to try (1,3,4, ranges, all, none).
  3. Each selected workflow runs at least twice through your model as a read-only task, exactly as a benchmark would. A model that tries a write tool is blocked before the call reaches the server.
  4. Modelara proposes an expectation only from the tool’s own machine-readable output, and only when every run agrees.
  5. You confirm each one. Nothing becomes a release expectation without explicit confirmation. Scripts, CI and init --yes never confirm anything.

Tool output is read through one shared normalization, used everywhere results are analyzed, in this order:

Tier Read from Rules
1 MCP structuredContent Authoritative whenever it decodes
2 JSON carried in text blocks A block that is wholly one JSON object or array; prose blocks beside it are ignored, never parsed; JSON inside a sentence is never extracted
3 Strict key/value text Every line must be Label: value, a short section header, or a leading Found <n> <noun>:. Any other line rejects the whole document. Values: integers, decimals, n / m pairs, booleans, short strings
— Anything else (prose) Not machine-readable. No expectation is inferred; you may type text a correct answer must contain, or skip

Runs are compared on the canonical normalized value: key order and whitespace do not matter, array order does, and volatile fields are set aside. Identifiers, timestamps, URLs, email addresses and secret-like fields never become expected values. Embedded resources and binary content are never read as data. Observations are kept in memory and are not stored; only confirmed values are written to tasks.yaml.

How a benchmark runs

Before anything runs, a preflight checks every prerequisite and reports all problems at once: the configuration, the task suite, the model API key, the environment, and the MCP server (started, tools discovered). A suite that still contains the unedited starter task is refused, so a placeholder can never become a release gate.

MODELARA PREFLIGHT

✓ task suite loaded: 5 task(s) (my-agent-workflows)
✗ OPENAI_API_KEY is not set
✓ environment: external_asserted my-data-v1
✓ MCP server started successfully
✓ 136 tools discovered

Benchmark not started.

Fix:
  export OPENAI_API_KEY=…   (read from the environment; never stored or passed to the MCP server)

The agent loop

For each run, the model receives the task prompt and the interface’s tools, calls tools, receives their results, and ends with a structured final answer. The run stops at a final answer, the turn limit, the tool-call limit, the timeout, a provider error, or a lost MCP connection.

Deterministic evaluation

No model grades another. Each run gets exactly one outcome, in this precedence:

Outcome Meaning
aborted timeout model_error mcp_error max_turns max_tool_calls The run ended without a final answer
policy_violation At least one call was blocked by task policy
forbidden_content A not_contains string appeared
wrong_answer A reported fact differs from the expected value, or a contains string is missing
missing_fact An expected fact was absent or null, or the answer was not structured
success Every check passed

Facts compare exactly after trimming and ignoring case. Numbers accept forms like "$1,234.50" with tiny float tolerance, booleans accept "true"/"false", an expected YYYY-MM-DD matches a timestamp on that date, and a list of values accepts any of them as equivalent forms.

Verdicts and rules

PASSexit 0

No rule hit.

REVIEWexit 3

Not clearly unsafe, but behavior changed or the evidence is inconclusive.

FAILexit 1

A regression gate was violated.

INVALIDexit 2

The comparison cannot be trusted (different suite, model, environment, incomplete runs).

Each model is judged on its own baseline/candidate pair; nothing is averaged across models. The overall verdict is the most severe, ordered INVALID > FAIL > REVIEW > PASS. Modelara reports every observed difference but gates only on differences with enough evidence; weaker ones are listed as observations (INFO) and never change the verdict.

Default policy (version 4)

Rule Hit when Default
comparability.* Different task suite, model, model parameters, runs per task or clock; a benchmark not completed; differing evaluator versions; incomplete runs INVALID
comparability.environment_mismatch Both benchmarks recorded an environment and they differ INVALID
comparability.environment_unknown Either benchmark has no recorded environment INVALID (REVIEW when explicitly allowed)
correctness.overall_regression A model’s success rate drops by more than 2 percentage points FAIL if pooled Fisher p < 0.05, else REVIEW
correctness.task_regression A task loses more than 1 success and the loss is corroborated: significant on its own, repeated strategy divergence, a significant overall regression, a structural change to a tool the task used, or new blocked behavior FAIL
correctness.task_loss_uncorroborated A task loses more than 1 success with nothing corroborating it REVIEW
correctness.task_borderline A task loses exactly 1 success REVIEW
correctness.new_failure_mode A failure class the baseline never showed REVIEW
drift.repeated_strategy_change The agent’s tool-selection strategy changed, against a stable baseline, in at least max(2, round(0.7 × runs)) candidate runs on a task that did not improve REVIEW
efficiency.regression Mean turns, tool calls or total tokens rise by more than 25% REVIEW
security.authority_expansion A new tool, or a stronger declared effect, that is not read-only and not approved FAIL
security.new_execution_dependency The candidate relies on externally enforced read-only execution the baseline did not REVIEW
security.new_blocked_behavior The agent now attempts a tool or effect that is blocked (any count if it may destroy state) FAIL / REVIEW
runs.insufficient Fewer than 3 runs per task REVIEW

Every reason carries structured evidence (the loss, its p-value, which corroborators applied, deviating and required runs, baseline variability), so a consumer can tell why a loss is FAIL or REVIEW without parsing text. Thresholds and actions are configurable; the effective policy is fingerprinted and stored with every check. Use 5 or more runs per task for release gating.

Approved authority expansion

New write tools fail by default. That is right for a team consuming an MCP server. A vendor intentionally adding capabilities approves them by exact name:

check:
  security:
    approved_capabilities:     # exact names; no patterns, prefixes or wildcards
      - create-inbox
      - remove-inbox

Approval changes only the release classification. It does not grant Modelara permission to execute the tool. Read-only tasks still block it, an attempted call is still blocked behavior, and every other rule still applies. Any expansion not listed still fails. The list is part of the policy fingerprint.

Baselines and approval

modelara test selects the baseline itself: the most recently accepted benchmark of the same project with the same task suite, model settings (provider, API, model, parameters) and environment. The interface is deliberately not part of the match, since that is what changes. Changing the suite, model or environment starts a new baseline instead of making an invalid comparison.

How a benchmark becomes the baseline Recorded as
The first modelara test initial, automatic
A candidate that passes pass, automatic
A REVIEW a person accepts with modelara approve review_accepted, with who accepted it
A FAIL whose new authority a person approves, and whose re-check then passes approved, with who approved and which capabilities

FAIL, INVALID and unaccepted REVIEW never become the baseline; the storage layer itself refuses them. modelara status shows the current baseline, why it is the baseline, and any pending result.

modelara approve acts on the latest test. For new authority it lists each pending capability with its effect and asks for approval by exact name (or --capability NAME, --all for every pending one). Approved names are added to check.security.approved_capabilities in modelara.yaml without disturbing the rest of the file, and the check re-runs. For REVIEW it shows the reasons and asks whether to accept the candidate (--accept-review in scripts). Commit the updated config so CI applies the same policy.

Explaining failures

modelara explain answers, for every failed run: what was asked, what was expected, what the model answered, which tools it called and how each call went, and why the evaluator classified the run as it did. Without an ID it explains the latest pending test, starting with the check’s reasons. It reads only the local database and makes no model call.

Task: read-auth-doc
Runs: 3
Failures: 3

Run #1
Status: wrong_answer

Expected:
  session_timeout_minutes: 30
Observed:
  session_timeout_minutes: 60

Tools used:
  #1 get_file_contents {"owner":"…","path":"docs/authentication.md","repo":"…"}
     → returned successfully (100 bytes): successfully downloaded text file …

Classification:
  wrong_answer: a reported fact differs from the expected value

Likely interpretation:
  Tool execution succeeded.
  The answer did not satisfy the evaluator expectation.

Tool errors, rejected arguments and blocked calls are shown with their messages and reasons. Where the stored evidence cannot determine a cause, explain says so rather than guessing. Output passes through the same redaction as stored traces.

Configuration (modelara.yaml)

Commands find the nearest modelara.yaml in the current directory or a parent, up to the repository root (outside a repository, only the current directory). --config always wins. Unknown keys are rejected, so a stray api_key: fails loudly instead of being ignored. Relative paths resolve against the config file’s directory.

version: 1
project:
  name: my-agent
mcp:
  transport: stdio                # the supported transport
  command: npx                    # executed directly, never through a shell
  args: [-y, my-mcp@1.2.3]
  inherit_env: false              # the server sees only these variables:
  env_passthrough: [PATH, HOME, MY_API_KEY]
  timeout: 30s                    # per MCP request
environment:
  name: my-data
  kind: external_asserted         # fixture | database_snapshot | service_snapshot | external_asserted
  fingerprint: my-data-v1         # or fingerprint_file: ./snapshot.json (its sha256)
model:
  provider: openai
  model: gpt-4.1-mini
  api: responses                  # or chat_completions for compatible gateways
  temperature: 0
benchmark:
  tasks: ./tasks.yaml
  runs_per_task: 5
  annotation_check: warn          # warn | error | off
check:                            # optional overrides of the default policy
  security:
    approved_capabilities: []
Section Notable fields
mcp env values may reference the parent environment as ${NAME}; dir; declared_overrides (operator-asserted annotations for servers that declare none); execution_assertions (writes prevented outside the interface)
model api_key_env (default OPENAI_API_KEY), base_url, seed, max_output_tokens, request_timeout
benchmark redact_keys (added to the built-in list), setup (a command run before each benchmark, e.g. to reset fixture data)
check correctness, behavior, efficiency, security, runs, environment thresholds and actions (fail, review, ignore)
storage path, default .modelara/modelara.db
sync display_names: opt-in for the optional names channel (default off)

Task suites

version: 1
suite:
  id: my-agent-workflows
tasks:
  - id: refund-bob-latest
    prompt: Can Bob Martinez's most recent order be refunded?
    expected:
      facts: {order_id: ord_1008, eligible: false}   # values the answer must report
      contains: [final sale]                          # text that must appear (case-insensitive)
      not_contains: [card number]                     # text that must not appear
    limits: {max_turns: 8}                            # defaults: 12 turns, 20 calls, 3m

CI integration

modelara test --ci            # never prompts or approves
case $? in
  0) echo "pass" ;;
  3) echo "needs review" ;;
  *) exit 1 ;;                # 1 FAIL, 2 INVALID
esac

Security and privacy

Stays on your machine

explicit
safe sync
→

May be sent (opt-in, explicit)

Optional team service

Modelara works fully offline. For teams, an optional service (currently offered to design partners) adds shared history and a web view. It never computes or overrides a verdict.

Command reference

Global flags: --config / -c (default: the nearest modelara.yaml), --log-level.

Command Purpose Key flags
init Create modelara.yaml; offer task bootstrap --yes --command --env --name --model --environment --force --no-tasks
test Benchmark and check against the accepted baseline --ci --verbose --candidate --allow-unknown-environment
approve Approve intended authority changes, or accept a REVIEW --capability --all --accept-review
explain Why runs failed; the latest pending test by default --benchmark --task --all
status Current baseline and anything pending
tasks bootstrap Suggest, observe and confirm workflows --runs
tasks init Write a valid starter suite to edit --file --force
inspect List tools, declared effects and the interface fingerprint; call one read-only tool --json --no-store --call --args
benchmark Run the suite and record traces --runs --model --suite --preflight
check Verdict for explicit baseline/candidate benchmark pairs --baseline --candidate --json --quiet --review-exit-code
check --static Compare two interface versions without a model, tasks or tool calls --baseline-command --candidate-command --baseline-interface --candidate-interface --json --sarif --sarif-output
compare Detailed behavior comparison of two benchmarks --baseline --candidate
analyze Evidence-backed interface findings from one benchmark --benchmark --json
trace List runs or print one run’s full trace --task --run --full --json
history · show · diff · export Browse records and lineage; compare interfaces; export metadata without traces or credentials
sync · names Optional team service: safe evidence and opt-in display names export inspect push
propose Experimental automatic interface proposals; validate any change with test or check

Requirements and limits

Modelara · Agent interface regression CI for MCP servers. Exit codes: 0 PASS · 1 FAIL · 2 INVALID · 3 REVIEW.