Getting started with Modelara
Agent interface regression testing for MCP servers
Modelara tells you whether a change to the tools your AI agent uses breaks real workflows, changes agent behavior, increases cost, or introduces new authority before you ship the change.
The basic workflow is:
modelara init
↓
create and confirm workflows
↓
modelara test
↓
baseline created
↓
upgrade or change the MCP server
↓
modelara test
↓
PASS / REVIEW / FAIL / INVALID
Modelara runs locally. Your MCP credentials, benchmark traces and expected answers stay on your machine, and nothing is sent to Modelara unless you explicitly sync it. During model-based tests, prompts, tool results and the context the model needs are sent to the model provider you configure.
Installation
Install the latest Modelara release:
curl -fsSL https://modelara.ai/install.sh | sh
Make sure the install directory is on your PATH:
export PATH="$HOME/.local/bin:$PATH"
Verify the installation:
modelara --version
macOS
Modelara binaries are currently unsigned.
If macOS blocks the binary, remove the quarantine attribute:
xattr -d com.apple.quarantine ~/.local/bin/modelara
Then try:
modelara --version
Supported platforms
Release archives are published for macOS (Apple silicon and Intel) and Linux (arm64 and x86_64) on the releases page. Each release includes a checksums.txt; the install script verifies the archive against it before installing. Windows builds are not available yet.
Inspect Your First Server
Inspection and static checks need no model and no API key.
Create a configuration that says how to start your MCP server:
modelara init --yes --no-tasks --command "npx -y my-mcp-server@1.2.3"
Or run modelara init and answer its questions. Then inspect the server:
modelara inspect
Modelara starts the server, completes the MCP handshake, lists every tool and stops the server. It does not call any tool. For Filesystem MCP 2.2.0 the output starts with this summary:
MODELARA INSPECT
Source: secure-filesystem-server 2.2.0 (MCP 2025-11-25)
Fingerprint: sha256:28daa4cf473cea0b961f4339affca39603a55e75474254fae65e6ac2431c2020
Discovered 16 tools
Read-only: 10
Potential writes: 6
Destructive: 3
Unknown effects: 0
Retry-safe: 12
followed by one row per tool with its effect, whether it is destructive and retry-safe, and the size of its input and output schemas.
- Effects come from the hints the server declares about itself. A tool that declares nothing is reported as unknown and treated as possibly destructive.
- Fingerprint is a hash of the whole interface. The same interface always has the same fingerprint.
Useful flags:
| Flag | What it does |
|---|---|
--json |
Print the full interface, including schemas, as a modelara.inspect/v0 document |
--no-store |
Do not record this interface version in the local database |
--call <tool> --args '{…}' |
Call one tool once and show its result. Only tools declared read-only can be called |
Save a snapshot to compare against later:
modelara inspect --json --no-store > base.json
The snapshot records its own fingerprint. If the file is edited afterwards, Modelara refuses it.
Check an MCP Upgrade Without a Model
modelara check --static compares two versions of an interface: added, removed and changed tools, schema and description changes, and new authority. It starts each server only to list its tools.
Give each side as a command line or a snapshot, in any combination. Two versions of a published server:
modelara check --static \
--baseline-command 'npx -y my-mcp-server@1.0.0' \
--candidate-command 'npx -y my-mcp-server@1.1.0'
A saved snapshot against the version you are about to adopt:
modelara check --static \
--baseline-interface base.json \
--candidate-command 'npx -y my-mcp-server@1.1.0'
A command line runs the server directly on your machine with your permissions. For a server you have not reviewed, start it in a container instead, as in the reproducible example.
Example result for Filesystem MCP 2.1.0 → 2.2.0 (excerpt: informational findings omitted):
MODELARA STATIC CHECK
Baseline: secure-filesystem-server 2.1.0 (14 tools, sha256:bd1fd899684f…)
Candidate: secure-filesystem-server 2.2.0 (16 tools, sha256:28daa4cf473c…)
Interface changes
+ append_file write not approved
+ copy_file write not approved
~ directory_tree input schema changed review
input property added: depth
~ read_file input schema changed review
input property head changed: +maximum=9007199254740991, +minimum=0, type "number" → "integer"
input property tail changed: +maximum=9007199254740991, +minimum=0, type "number" → "integer"
~ read_text_file input schema changed review
input property head changed: +maximum=9007199254740991, +minimum=0, type "number" → "integer"
input property tail changed: +maximum=9007199254740991, +minimum=0, type "number" → "integer"
Summary
Added tools 2
Removed tools 0
Schema changes 3
Description changes 3
New authority 2 (0 approved, 2 not approved)
RESULT: FAIL
The exit code is 1.
Results and exit codes
| Result | Exit code | Meaning |
|---|---|---|
| PASS | 0 | No policy-significant change. Added read-only tools and description changes are informational |
| FAIL | 1 | A new or stronger write, destructive or unknown effect that is not approved |
| INVALID | 2 | The comparison cannot be trusted, for example a server did not start or a snapshot was edited |
| REVIEW | 3 | A schema changed or a tool was removed |
Use --review-exit-code 0 if a pipeline should continue on REVIEW, and --quiet to print only the result.
JSON and SARIF
| Flag | Output |
|---|---|
--json |
A modelara.check/v1 JSON document on stdout |
--sarif |
SARIF 2.1.0 on stdout |
--sarif-output check.sarif |
The text report on stdout, and SARIF 2.1.0 written to the file |
Choose one of --json, --sarif or --quiet. The JSON document contains the verdict, both interfaces with their fingerprints, a summary and one finding per change with its rule_id. SARIF results use these rule IDs:
| Rule | SARIF level | Result |
|---|---|---|
static.authority.new_non_read |
error | FAIL |
static.authority.stronger_effect |
error | FAIL |
static.interface.removed_tool |
warning | REVIEW |
static.interface.schema_changed |
warning | REVIEW |
static.interface.added_read_only |
note | informational |
static.authority.weaker_effect |
note | informational |
static.interface.description_changed |
note | informational |
static.interface.metadata_changed |
note | informational |
Approving intended authority
If a new write tool is intended, list its exact name in modelara.yaml. Static checks and modelara test both honor the list:
check:
security:
approved_capabilities:
- append_file
- copy_file
There are no wildcards or prefixes; anything not listed still fails. With both tools approved, the Filesystem MCP comparison reports New authority 2 (2 approved, 0 not approved) and RESULT: REVIEW (exit code 3), because the three schema changes remain.
What a static check does not tell you
It compares the declared interface only. It does not run an agent, so it cannot detect behavior changes behind an unchanged schema, and a PASS does not guarantee your agent will succeed. It does not verify that both sides are the same server. To test behavior, continue with the model-based workflow below.
Known issue in v0.1.9
--no-store cannot be combined with --static. Static checks never store results, so leave the flag out. In v0.1.9 the combination exits with code 2 and a misleading message that refers to benchmark flags:
Error: --static compares interfaces, not benchmarks: use --baseline-command or --baseline-interface (and the candidate equivalents) instead of --baseline/--candidate
--no-store is still valid with modelara inspect.
Reproduce the Filesystem MCP Example
The examples on this page and on the Modelara CLI page come from the public Filesystem MCP server (@modelcontextprotocol/server-filesystem), versions 2.1.0 and 2.2.0 on npm. Both versions are published with npm provenance from modelcontextprotocol/servers. The same comparison is published on Valyd.
These steps run each server inside a container, so no third-party code runs directly on your machine. You need Docker and Modelara CLI v0.1.9.
1. Download both versions into a Docker volume. This is the only step with network access. Install scripts are disabled and the container runs as an unprivileged user:
docker volume create modelara-example-fs
docker run --rm --user node --cap-drop ALL --security-opt no-new-privileges \
--memory 768m --pids-limit 256 -e npm_config_cache=/tmp/npm \
-v modelara-example-fs:/home/node node:24-bookworm-slim \
sh -c 'for v in 2.1.0 2.2.0; do mkdir -p ~/$v && cd ~/$v && npm init -y >/dev/null && npm install --ignore-scripts --no-audit --no-fund --save-exact @modelcontextprotocol/server-filesystem@$v; done'
2. Define how to start a server. Each server starts in a container with no network, a read-only file system and resource limits, and receives an empty directory (/tmp) as its only allowed directory:
RUN='docker run -i --rm --network none --read-only --user node --cap-drop ALL --security-opt no-new-privileges --memory 256m --pids-limit 64 -v modelara-example-fs:/home/node:ro node:24-bookworm-slim node'
FS=node_modules/@modelcontextprotocol/server-filesystem/dist/index.js
3. Compare the two versions:
modelara check --static \
--baseline-command "$RUN /home/node/2.1.0/$FS /tmp" \
--candidate-command "$RUN /home/node/2.2.0/$FS /tmp"
Expected: RESULT: FAIL, exit code 1, with fingerprints sha256:bd1fd899684f… (2.1.0, 14 tools) and sha256:28daa4cf473c… (2.2.0, 16 tools).
4. Inspect one version. inspect reads the server command from modelara.yaml:
modelara init --yes --no-tasks --name filesystem-example \
--command docker --args "${RUN#docker } /home/node/2.2.0/$FS /tmp"
modelara inspect
Expected: 16 tools and fingerprint sha256:28daa4cf473cea0b961f4339affca39603a55e75474254fae65e6ac2431c2020.
Why your results might differ
- Dependencies.
--save-exactpins the server package, not its dependencies. On 2026-11-09 they resolved to@modelcontextprotocol/server2.4.0 andzod4.6.7. Those libraries produce the JSON schemas, so a newer release of either can change schemas and therefore fingerprints. Check withnpm lsinside the volume. - Node.js image.
node:24-bookworm-slimis a moving tag. We verified with the image digestsha256:01d566e8d94d8c619591d78652babfccd3c6992a3dffa9ed5ab39e11cf6c70e9; usenode:24-bookworm-slim@sha256:…to pin it. - Modelara version. Output formats and rules can change between CLI releases. These results come from v0.1.9.
- Package integrity. npm verifies each download. The expected sha512 values are
sha512-BMihuyTNi975iQTQ…for 2.1.0 andsha512-mNLj/qhV+P4kj1qP…for 2.2.0.
To remove the example afterwards: docker volume rm modelara-example-fs.
Quick Start
The rest of this guide covers modelara test, which runs real tasks through a model against your server. It needs a model API key.
Create a directory for your Modelara project:
mkdir my-agent-tests
cd my-agent-tests
git init
Export the API key for the model Modelara will use:
export OPENAI_API_KEY="..."
Export any credentials required by your MCP server.
For example:
export MY_API_KEY="..."
Modelara stores only the environment variable name, not its secret value.
Now run:
modelara init
Modelara will ask for your MCP server command.
For example:
npx -y my-mcp-server@1.2.3
You can enter the complete command on one line.
Modelara executes the program directly. It does not invoke a shell.
Creating Your First Workflows
During modelara init, Modelara can inspect your MCP server and suggest starter workflows.
Modelara only suggests tools that are declared:
- read-only;
- retry-safe;
- non-destructive;
- usable without required input.
Write, destructive, unannotated, and credential-management tools are excluded from automatic suggestions.
Example:
Suggested workflows:
[1] List contacts
[2] Check usage
[3] List domains
[4] List segments
[5] List templates
Select workflows [1,2,3,4,5]:
Choose the workflows that represent behavior you want to protect.
How Bootstrap Works
For each selected workflow, Modelara runs the workflow multiple times.
Example:
Workflow: List contacts
Observed:
1 contacts
Observed the same result in all 2 runs.
Suggested expectation:
contacts_count: 1
Use this as a regression test? [Y/n]:
Nothing becomes a regression expectation until you explicitly approve it.
Modelara derives expectations from tool output using this order:
MCP structuredContent
↓
JSON returned as text
↓
strict Label: value text
↓
manual fallback
Modelara does not use an LLM to interpret arbitrary prose into expected values.
For example:
Contacts: 1
Domains: 0
can produce structured expectations.
But:
There are currently no domains configured.
does not automatically become an expectation.
This conservative behavior prevents Modelara from silently guessing what a tool response means.
Run Your First Test
After the workflows are confirmed:
modelara test
Before running a benchmark, Modelara checks:
- task suite;
- model API credentials;
- environment;
- MCP server startup;
- discovered tools.
Then it runs each workflow multiple times through the configured model.
Example:
MODELARA TEST
✓ Tested my-mcp 1.2.3
✓ 42 tools discovered
✓ 3 workflows tested
✓ 9 runs completed
BASELINE CREATED
Correctness 9/9
Your first successful benchmark becomes the baseline automatically.
Test an MCP Upgrade
Suppose you upgrade:
my-mcp 1.2.3
to:
my-mcp 1.3.0
Update the MCP command in modelara.yaml, then run:
modelara test
Modelara automatically finds the appropriate accepted baseline and compares the new benchmark against it.
You do not need to manage benchmark IDs during the normal workflow.
Understanding Results
Modelara produces one of four results.
PASS
The candidate passed the configured regression policy.
The successful benchmark automatically becomes the new baseline.
RESULT: PASS
No action is required.
REVIEW
Something changed, but Modelara does not have enough evidence to classify it as a definite regression.
Examples include:
- borderline correctness changes;
- significant behavior changes;
- efficiency regressions;
- insufficient evidence.
Inspect the result:
modelara explain
If the change is intentional and acceptable:
modelara approve
FAIL
A release gate was violated.
Examples include:
- significant correctness regression;
- unexpected new write authority;
- newly blocked behavior.
The current accepted baseline remains unchanged.
Inspect the failure:
modelara explain
INVALID
Modelara cannot trust the comparison.
Examples include incompatible environments or incomplete benchmark evidence.
The candidate does not become the baseline.
Run:
modelara explain
to see the reason.
Understanding New Authority
One important Modelara check is authority expansion.
Suppose an MCP upgrade introduces:
create-user
delete-user
update-account
These tools give an agent more authority than the previous interface.
Unexpected new authority fails by default.
Example:
Security
✗ 3 new authority changes
RESULT: FAIL
If the new capabilities are intentional, review them:
modelara approve
Approval is capability-specific.
Approving a capability changes the release classification only.
It does not give Modelara permission to execute write or destructive tools.
Explain a Failure
Run:
modelara explain
Modelara shows:
- what the workflow asked;
- what was expected;
- what the model answered;
- which tools were called;
- whether calls succeeded or were blocked;
- why the evaluator classified the run as it did.
modelara explain reads local benchmark evidence and does not make another model call.
Check Current Status
Run:
modelara status
This shows:
- current accepted baseline;
- how it became the baseline;
- latest candidate;
- pending REVIEW or FAIL results.
CI/CD
Once you have a baseline, Modelara can run in CI:
modelara test --ci
Exit codes:
0 PASS
1 FAIL
2 INVALID
3 REVIEW
Example shell logic:
modelara test --ci
case $? in
0)
echo "Modelara passed"
;;
3)
echo "Modelara requires review"
exit 1
;;
*)
echo "Modelara failed"
exit 1
;;
esac
CI must retain access to Modelara’s baseline data. Without an accepted baseline, Modelara will not pretend that a candidate passed.
Configuration
A basic modelara.yaml looks like:
version: 1
project:
name: my-agent
mcp:
transport: stdio
command: npx
args:
- -y
- my-mcp@1.2.3
inherit_env: false
env_passthrough:
- PATH
- HOME
- MY_API_KEY
environment:
name: my-environment
kind: external_asserted
fingerprint: my-environment-v1
model:
provider: openai
model: gpt-4.1-mini
benchmark:
tasks: ./tasks.yaml
Secrets should remain in environment variables.
Do not put API key values directly in this file.
Environment Fingerprints
Expected results describe the environment against which your workflows run.
If the underlying data intentionally changes, change the environment fingerprint.
For example:
environment:
name: staging
fingerprint: staging-v2
This prevents Modelara from comparing benchmarks that were run against materially different data and treating the difference as an MCP regression.
Security Model
Modelara is designed around conservative execution.
During normal workflow testing:
- read-only tasks are enforced;
- write tools are blocked;
- destructive tools are blocked;
- unknown-effect tools are not treated as safe;
- credentials remain environment variables;
- raw benchmark evidence stays local.
Modelara does not grant execution permission simply because a new capability was approved for release.
Privacy
Modelara’s benchmark evidence is stored locally.
This includes:
- prompts;
- model responses;
- tool arguments;
- tool results;
- expected and observed answers;
- traces;
- environment-specific business data.
Model-based tests send prompts, tool results and the context needed for inference to the model provider you configure. Inspection and static checks make no model call.
The optional team service receives data only when you run modelara sync push, and the sync format carries counts, verdicts, rule IDs and fingerprints, not prompts, tool data or credentials.
Useful Commands
modelara init
Set up a project and optionally bootstrap workflows.
modelara tasks bootstrap
Discover and confirm additional workflows.
modelara test
Run workflows and compare against the accepted baseline.
modelara explain
Explain the latest regression or review result.
modelara approve
Approve intentional authority changes or accept a REVIEW.
modelara status
Show the current baseline and pending result.
modelara inspect
Inspect the MCP server and discovered capabilities.
modelara check --static --baseline-command '…' --candidate-command '…'
Compare two versions of an interface without a model.
Current Limitations
Current Modelara releases focus on:
- MCP over stdio;
- read-only regression workflows;
- OpenAI-compatible model execution;
- macOS and Linux release binaries.
Some MCP tools may return only arbitrary natural-language prose. Modelara intentionally does not convert that prose into automatic regression expectations.
You can skip those workflows or define their expectations manually.
Recommended Workflow
For normal use:
modelara init
↓
modelara test
↓
change or upgrade MCP server
↓
modelara test
↓
PASS
├─ yes → new baseline automatically
└─ no → modelara explain
↓
intentional?
├─ yes → modelara approve
└─ no → fix the regression
The goal is simple:
Know whether an agent-facing interface change is safe before you ship or adopt it.