Frequently asked questions
What Test Maze is, how your agent connects to it, what spec-based development enforces, how a verdict is decided, what we store and what it costs. Something missing? Ask us.
The basics
What is Test Maze?
Spec-driven delivery for AI coding agents. Test Maze holds your user stories and acceptance tests, keeps your coding agent inside them, verifies the work independently and gates the release. Your agent writes the code; Test Maze holds the spec and the verdict.
Why does an agent need an outside spec and verifier?
Because an agent that writes the requirements, the code, the tests and the result is grading its own homework against a spec that lives in a chat window. Test Maze keeps the story and its acceptance tests on record, reviewed by a person, and has a separate verifier grade the work, so a pass is something you can point at: this story, these cases, this run, this commit.
What are the four pillars?
Spec first (stories and acceptance criteria in one place, reviewed by people). Every change traced (no edit without a declared story, no commit without a Testmaze-Ref). Verified independently (a separate verifier runs the tests; fixed rules decide). Shipped on evidence (defects hold the release; the gate opens on results).
Is Test Maze a replacement for my test framework?
No. Keep Playwright, Jest, pytest or whatever you use. Test Maze records what those tests are meant to cover and what happened when your agent ran them, then grades the outcome.
Do I have to be an engineer to use it?
You need a coding agent and a project. You describe features and fixes in plain English; the agent calls the tools. Everything it does is visible in the app on the Agent Sessions page.
Connecting your agent
How do I connect my coding agent?
Create an MCP token in your workspace under Settings → MCP, run "npx -y @testmaze/mcp init <token>" in your project folder (it stores the token in .env.testmaze, which is git-ignored), then register the server with your agent — for Claude Code, "claude mcp add tm --scope project -- npx -y @testmaze/mcp".
Which coding agents are supported?
Any Model Context Protocol client. Setup instructions in the app cover Claude Code, Cursor, Cline, Gemini CLI and Codex CLI, and the same stdio server works anywhere else MCP is supported.
Do I call the tools myself?
No. You ask in plain English — "write test cases for the password reset flow", "verify this change" — and the agent picks from the 70 tools and fills in their inputs.
Can several people share a workspace?
Yes. Invite teammates from Settings → Team Management; each person gets a role, and every agent session in the workspace is visible to the team.
Spec-based development
What happens when I turn on spec-based development?
From Settings → Workspace → Spec enforcement, an admin switches the policy on for the workspace. From then on an agent cannot edit repository files until it has declared which user story it is implementing on that branch (spec.start_work), every commit must carry a Testmaze-Ref trailer naming the story or case, and the CI range check fails the build on an untraced commit. The hooks that enforce this in Claude Code and git are installed by npx @testmaze/mcp init; a warn mode lets everything through with a warning while you try it.
Can an agent approve its own story?
No. Reviewing a story or test case, ticking Definition of Ready and Done items, and shipping a release past a failing gate are human-only actions in the web app. There is no MCP tool for any of them, and the agent token cannot reach those routes. Agents propose; people approve.
What is the traceability matrix?
One row per user story: its feature, stage, review status, release, acceptance cases (passed, failed, not run, waived), the commits that name it, and open defects. It is the requirement → test → commit → verdict → release chain ISO 9001 / 12207-style processes ask for, with CSV export.
Verdicts and quality
How is the pass/fail verdict decided?
By fixed rules over the recorded results: pass when at least one case passed and none failed, fail when a case failed or a frozen regression-baseline case is missing or failing, otherwise in-progress. Waived acceptance criteria are excluded from the tally.
Is AI involved in the verdict?
No. The verdict path is deterministic — the same results always produce the same verdict. AI may only compose an optional hint about what to fix next.
What happens after a fail?
The verdict carries a next step: repair the code, repair the test, add coverage, waive a criterion, or ship. Your agent reads it and continues the loop without you having to translate. A failed case can also become a defect with a regression test and, if GitLab is connected, an issue.
Who runs the tests?
In Claude Code, a separate testmaze-verifier subagent installed by init: it runs every acceptance case against your running app in its own context and records results with recordedBy: verifier. Clients without subagents record results themselves, and the verdict reports whether it was independent.
When can a release ship?
When at least one test run is linked, no high-priority or high-severity case is failing or unexecuted, and no blocker defect is open. Agents cannot ship past a failing gate; an admin can, with a reason that is kept on the release.
What are frozen baselines and waivers?
Freezing a test run turns its cases into a regression baseline: if one of them later goes missing or fails, the verdict fails. A waiver marks an acceptance criterion as deliberately not covered yet, with a reason, so it does not silently drag the verdict down.
Data and security
Does Test Maze read my source code?
No. Test Maze never clones, scans or indexes your repository. What reaches it: the product summary your agent writes, stories, test cases and results, git identifiers (sha, branch, clean working tree), optional screenshots, any code snippet an agent chooses to paste into a code-quality check, and — only when spec-based development is on — commit subject lines and Testmaze-Ref trailers. Never diffs, never files.
What is stored about my test runs?
The cases, their results, who or what recorded them, and the git sha, branch and working-tree state of the commit under test. That provenance is what makes a verdict checkable later.
Where do my AI provider keys live?
If you bring your own GMI Cloud or OpenRouter key (any model OpenRouter serves), it is stored AES-256-GCM encrypted at rest and used only for your workspace’s calls. It is never shown back to the browser and never leaves the server.
Can I see everything my agent did?
Yes. The Agent Sessions page lists every tool call in the workspace with its inputs, results and timing, so nothing an agent does is invisible to the team.
Plans and billing
Is there a free plan?
Yes — Basic is free forever with nothing behind a paywall: the whole loop, spec-based development included. Pro is $19.99 per month per organisation (or $215.88 a year, about 10% off); seats are $5 a user beyond two. See the pricing page for details.
What are AI credits?
Credits pay for AI work that runs on Test Maze provider keys, at $2 per million tokens, topped up in packs of 5M, 10M or 25M tokens. Bring your own provider key and that usage bills to your provider instead.
How do I get help?
Write to us from the contact page and we reply by email. Pro workspaces get priority.
Looking for a specific tool? Every one of the 70 MCP tools has its own page under Tools.
Ready to give your agent a verifier?
Create a free workspace, generate an MCP token and connect your coding agent in under five minutes.