← Modelara Docs/Modelara Getting Started

Getting started with Modelara

Agent interface regression testing for MCP servers

Modelara tells you whether a change to the tools your AI agent uses breaks real workflows, changes agent behavior, increases cost, or introduces new authority before you ship the change.

The basic workflow is:

modelara init
      ↓
create and confirm workflows
      ↓
modelara test
      ↓
baseline created
      ↓
upgrade or change the MCP server
      ↓
modelara test
      ↓
PASS / REVIEW / FAIL / INVALID

Modelara runs locally. Your MCP credentials, benchmark traces and expected answers stay on your machine, and nothing is sent to Modelara unless you explicitly sync it. During model-based tests, prompts, tool results and the context the model needs are sent to the model provider you configure.


Installation

Install the latest Modelara release:

curl -fsSL https://modelara.ai/install.sh | sh

Make sure the install directory is on your PATH:

export PATH="$HOME/.local/bin:$PATH"

Verify the installation:

modelara --version

macOS

Modelara binaries are currently unsigned.

If macOS blocks the binary, remove the quarantine attribute:

xattr -d com.apple.quarantine ~/.local/bin/modelara

Then try:

modelara --version

Supported platforms

Release archives are published for macOS (Apple silicon and Intel) and Linux (arm64 and x86_64) on the releases page. Each release includes a checksums.txt; the install script verifies the archive against it before installing. Windows builds are not available yet.


Inspect Your First Server

Inspection and static checks need no model and no API key.

Create a configuration that says how to start your MCP server:

modelara init --yes --no-tasks --command "npx -y my-mcp-server@1.2.3"

Or run modelara init and answer its questions. Then inspect the server:

modelara inspect

Modelara starts the server, completes the MCP handshake, lists every tool and stops the server. It does not call any tool. For Filesystem MCP 2.2.0 the output starts with this summary:

MODELARA INSPECT

Source:       secure-filesystem-server 2.2.0 (MCP 2025-11-25)
Fingerprint:  sha256:28daa4cf473cea0b961f4339affca39603a55e75474254fae65e6ac2431c2020
Discovered 16 tools

Read-only:         10
Potential writes:  6
Destructive:       3
Unknown effects:   0
Retry-safe:        12

followed by one row per tool with its effect, whether it is destructive and retry-safe, and the size of its input and output schemas.

Useful flags:

Flag What it does
--json Print the full interface, including schemas, as a modelara.inspect/v0 document
--no-store Do not record this interface version in the local database
--call <tool> --args '{…}' Call one tool once and show its result. Only tools declared read-only can be called

Save a snapshot to compare against later:

modelara inspect --json --no-store > base.json

The snapshot records its own fingerprint. If the file is edited afterwards, Modelara refuses it.


Check an MCP Upgrade Without a Model

modelara check --static compares two versions of an interface: added, removed and changed tools, schema and description changes, and new authority. It starts each server only to list its tools.

Give each side as a command line or a snapshot, in any combination. Two versions of a published server:

modelara check --static \
  --baseline-command  'npx -y my-mcp-server@1.0.0' \
  --candidate-command 'npx -y my-mcp-server@1.1.0'

A saved snapshot against the version you are about to adopt:

modelara check --static \
  --baseline-interface base.json \
  --candidate-command 'npx -y my-mcp-server@1.1.0'

A command line runs the server directly on your machine with your permissions. For a server you have not reviewed, start it in a container instead, as in the reproducible example.

Example result for Filesystem MCP 2.1.0 → 2.2.0 (excerpt: informational findings omitted):

MODELARA STATIC CHECK

Baseline:  secure-filesystem-server 2.1.0  (14 tools, sha256:bd1fd899684f…)
Candidate: secure-filesystem-server 2.2.0  (16 tools, sha256:28daa4cf473c…)

Interface changes

  + append_file                 write                  not approved
  + copy_file                   write                  not approved
  ~ directory_tree              input schema changed   review
      input property added: depth
  ~ read_file                   input schema changed   review
      input property head changed: +maximum=9007199254740991, +minimum=0, type "number" → "integer"
      input property tail changed: +maximum=9007199254740991, +minimum=0, type "number" → "integer"
  ~ read_text_file              input schema changed   review
      input property head changed: +maximum=9007199254740991, +minimum=0, type "number" → "integer"
      input property tail changed: +maximum=9007199254740991, +minimum=0, type "number" → "integer"

Summary

  Added tools          2
  Removed tools        0
  Schema changes       3
  Description changes  3
  New authority        2 (0 approved, 2 not approved)

RESULT: FAIL

The exit code is 1.

Results and exit codes

Result Exit code Meaning
PASS 0 No policy-significant change. Added read-only tools and description changes are informational
FAIL 1 A new or stronger write, destructive or unknown effect that is not approved
INVALID 2 The comparison cannot be trusted, for example a server did not start or a snapshot was edited
REVIEW 3 A schema changed or a tool was removed

Use --review-exit-code 0 if a pipeline should continue on REVIEW, and --quiet to print only the result.

JSON and SARIF

Flag Output
--json A modelara.check/v1 JSON document on stdout
--sarif SARIF 2.1.0 on stdout
--sarif-output check.sarif The text report on stdout, and SARIF 2.1.0 written to the file

Choose one of --json, --sarif or --quiet. The JSON document contains the verdict, both interfaces with their fingerprints, a summary and one finding per change with its rule_id. SARIF results use these rule IDs:

Rule SARIF level Result
static.authority.new_non_read error FAIL
static.authority.stronger_effect error FAIL
static.interface.removed_tool warning REVIEW
static.interface.schema_changed warning REVIEW
static.interface.added_read_only note informational
static.authority.weaker_effect note informational
static.interface.description_changed note informational
static.interface.metadata_changed note informational

Approving intended authority

If a new write tool is intended, list its exact name in modelara.yaml. Static checks and modelara test both honor the list:

check:
  security:
    approved_capabilities:
      - append_file
      - copy_file

There are no wildcards or prefixes; anything not listed still fails. With both tools approved, the Filesystem MCP comparison reports New authority 2 (2 approved, 0 not approved) and RESULT: REVIEW (exit code 3), because the three schema changes remain.

What a static check does not tell you

It compares the declared interface only. It does not run an agent, so it cannot detect behavior changes behind an unchanged schema, and a PASS does not guarantee your agent will succeed. It does not verify that both sides are the same server. To test behavior, continue with the model-based workflow below.

Known issue in v0.1.9

--no-store cannot be combined with --static. Static checks never store results, so leave the flag out. In v0.1.9 the combination exits with code 2 and a misleading message that refers to benchmark flags:

Error: --static compares interfaces, not benchmarks: use --baseline-command or --baseline-interface (and the candidate equivalents) instead of --baseline/--candidate

--no-store is still valid with modelara inspect.


Reproduce the Filesystem MCP Example

The examples on this page and on the Modelara CLI page come from the public Filesystem MCP server (@modelcontextprotocol/server-filesystem), versions 2.1.0 and 2.2.0 on npm. Both versions are published with npm provenance from modelcontextprotocol/servers. The same comparison is published on Valyd.

These steps run each server inside a container, so no third-party code runs directly on your machine. You need Docker and Modelara CLI v0.1.9.

1. Download both versions into a Docker volume. This is the only step with network access. Install scripts are disabled and the container runs as an unprivileged user:

docker volume create modelara-example-fs

docker run --rm --user node --cap-drop ALL --security-opt no-new-privileges \
  --memory 768m --pids-limit 256 -e npm_config_cache=/tmp/npm \
  -v modelara-example-fs:/home/node node:24-bookworm-slim \
  sh -c 'for v in 2.1.0 2.2.0; do mkdir -p ~/$v && cd ~/$v && npm init -y >/dev/null && npm install --ignore-scripts --no-audit --no-fund --save-exact @modelcontextprotocol/server-filesystem@$v; done'

2. Define how to start a server. Each server starts in a container with no network, a read-only file system and resource limits, and receives an empty directory (/tmp) as its only allowed directory:

RUN='docker run -i --rm --network none --read-only --user node --cap-drop ALL --security-opt no-new-privileges --memory 256m --pids-limit 64 -v modelara-example-fs:/home/node:ro node:24-bookworm-slim node'
FS=node_modules/@modelcontextprotocol/server-filesystem/dist/index.js

3. Compare the two versions:

modelara check --static \
  --baseline-command  "$RUN /home/node/2.1.0/$FS /tmp" \
  --candidate-command "$RUN /home/node/2.2.0/$FS /tmp"

Expected: RESULT: FAIL, exit code 1, with fingerprints sha256:bd1fd899684f… (2.1.0, 14 tools) and sha256:28daa4cf473c… (2.2.0, 16 tools).

4. Inspect one version. inspect reads the server command from modelara.yaml:

modelara init --yes --no-tasks --name filesystem-example \
  --command docker --args "${RUN#docker } /home/node/2.2.0/$FS /tmp"
modelara inspect

Expected: 16 tools and fingerprint sha256:28daa4cf473cea0b961f4339affca39603a55e75474254fae65e6ac2431c2020.

Why your results might differ

To remove the example afterwards: docker volume rm modelara-example-fs.


Quick Start

The rest of this guide covers modelara test, which runs real tasks through a model against your server. It needs a model API key.

Create a directory for your Modelara project:

mkdir my-agent-tests
cd my-agent-tests
git init

Export the API key for the model Modelara will use:

export OPENAI_API_KEY="..."

Export any credentials required by your MCP server.

For example:

export MY_API_KEY="..."

Modelara stores only the environment variable name, not its secret value.

Now run:

modelara init

Modelara will ask for your MCP server command.

For example:

npx -y my-mcp-server@1.2.3

You can enter the complete command on one line.

Modelara executes the program directly. It does not invoke a shell.


Creating Your First Workflows

During modelara init, Modelara can inspect your MCP server and suggest starter workflows.

Modelara only suggests tools that are declared:

Write, destructive, unannotated, and credential-management tools are excluded from automatic suggestions.

Example:

Suggested workflows:

[1] List contacts
[2] Check usage
[3] List domains
[4] List segments
[5] List templates

Select workflows [1,2,3,4,5]:

Choose the workflows that represent behavior you want to protect.


How Bootstrap Works

For each selected workflow, Modelara runs the workflow multiple times.

Example:

Workflow: List contacts

Observed:
  1 contacts

Observed the same result in all 2 runs.

Suggested expectation:

  contacts_count: 1

Use this as a regression test? [Y/n]:

Nothing becomes a regression expectation until you explicitly approve it.

Modelara derives expectations from tool output using this order:

MCP structuredContent
        ↓
JSON returned as text
        ↓
strict Label: value text
        ↓
manual fallback

Modelara does not use an LLM to interpret arbitrary prose into expected values.

For example:

Contacts: 1
Domains: 0

can produce structured expectations.

But:

There are currently no domains configured.

does not automatically become an expectation.

This conservative behavior prevents Modelara from silently guessing what a tool response means.


Run Your First Test

After the workflows are confirmed:

modelara test

Before running a benchmark, Modelara checks:

Then it runs each workflow multiple times through the configured model.

Example:

MODELARA TEST

✓ Tested my-mcp 1.2.3
✓ 42 tools discovered
✓ 3 workflows tested
✓ 9 runs completed

BASELINE CREATED

  Correctness   9/9

Your first successful benchmark becomes the baseline automatically.


Test an MCP Upgrade

Suppose you upgrade:

my-mcp 1.2.3

to:

my-mcp 1.3.0

Update the MCP command in modelara.yaml, then run:

modelara test

Modelara automatically finds the appropriate accepted baseline and compares the new benchmark against it.

You do not need to manage benchmark IDs during the normal workflow.


Understanding Results

Modelara produces one of four results.

PASS

The candidate passed the configured regression policy.

The successful benchmark automatically becomes the new baseline.

RESULT: PASS

No action is required.


REVIEW

Something changed, but Modelara does not have enough evidence to classify it as a definite regression.

Examples include:

Inspect the result:

modelara explain

If the change is intentional and acceptable:

modelara approve

FAIL

A release gate was violated.

Examples include:

The current accepted baseline remains unchanged.

Inspect the failure:

modelara explain

INVALID

Modelara cannot trust the comparison.

Examples include incompatible environments or incomplete benchmark evidence.

The candidate does not become the baseline.

Run:

modelara explain

to see the reason.


Understanding New Authority

One important Modelara check is authority expansion.

Suppose an MCP upgrade introduces:

create-user
delete-user
update-account

These tools give an agent more authority than the previous interface.

Unexpected new authority fails by default.

Example:

Security
  ✗ 3 new authority changes

RESULT: FAIL

If the new capabilities are intentional, review them:

modelara approve

Approval is capability-specific.

Approving a capability changes the release classification only.

It does not give Modelara permission to execute write or destructive tools.


Explain a Failure

Run:

modelara explain

Modelara shows:

modelara explain reads local benchmark evidence and does not make another model call.


Check Current Status

Run:

modelara status

This shows:


CI/CD

Once you have a baseline, Modelara can run in CI:

modelara test --ci

Exit codes:

0  PASS
1  FAIL
2  INVALID
3  REVIEW

Example shell logic:

modelara test --ci

case $? in
  0)
    echo "Modelara passed"
    ;;
  3)
    echo "Modelara requires review"
    exit 1
    ;;
  *)
    echo "Modelara failed"
    exit 1
    ;;
esac

CI must retain access to Modelara’s baseline data. Without an accepted baseline, Modelara will not pretend that a candidate passed.


Configuration

A basic modelara.yaml looks like:

version: 1

project:
  name: my-agent

mcp:
  transport: stdio
  command: npx
  args:
    - -y
    - my-mcp@1.2.3
  inherit_env: false
  env_passthrough:
    - PATH
    - HOME
    - MY_API_KEY

environment:
  name: my-environment
  kind: external_asserted
  fingerprint: my-environment-v1

model:
  provider: openai
  model: gpt-4.1-mini

benchmark:
  tasks: ./tasks.yaml

Secrets should remain in environment variables.

Do not put API key values directly in this file.


Environment Fingerprints

Expected results describe the environment against which your workflows run.

If the underlying data intentionally changes, change the environment fingerprint.

For example:

environment:
  name: staging
  fingerprint: staging-v2

This prevents Modelara from comparing benchmarks that were run against materially different data and treating the difference as an MCP regression.


Security Model

Modelara is designed around conservative execution.

During normal workflow testing:

Modelara does not grant execution permission simply because a new capability was approved for release.


Privacy

Modelara’s benchmark evidence is stored locally.

This includes:

Model-based tests send prompts, tool results and the context needed for inference to the model provider you configure. Inspection and static checks make no model call.

The optional team service receives data only when you run modelara sync push, and the sync format carries counts, verdicts, rule IDs and fingerprints, not prompts, tool data or credentials.


Useful Commands

modelara init

Set up a project and optionally bootstrap workflows.

modelara tasks bootstrap

Discover and confirm additional workflows.

modelara test

Run workflows and compare against the accepted baseline.

modelara explain

Explain the latest regression or review result.

modelara approve

Approve intentional authority changes or accept a REVIEW.

modelara status

Show the current baseline and pending result.

modelara inspect

Inspect the MCP server and discovered capabilities.

modelara check --static --baseline-command '…' --candidate-command '…'

Compare two versions of an interface without a model.


Current Limitations

Current Modelara releases focus on:

Some MCP tools may return only arbitrary natural-language prose. Modelara intentionally does not convert that prose into automatic regression expectations.

You can skip those workflows or define their expectations manually.


For normal use:

modelara init
      ↓
modelara test
      ↓
change or upgrade MCP server
      ↓
modelara test
      ↓
PASS
 ├─ yes → new baseline automatically
 └─ no  → modelara explain
              ↓
        intentional?
        ├─ yes → modelara approve
        └─ no  → fix the regression

The goal is simple:

Know whether an agent-facing interface change is safe before you ship or adopt it.