Evaluating Multi-Agent Systems in Production

Why does a demo that works once prove nothing?

A multi-agent system changes with every prompt, tool, model or context change. Without a fixed set of scenarios, nobody can tell an improvement from a regression.

A demo shows that the system can succeed. Evaluation shows how often it does, and what it costs when it does not.

What does evaluating an agent system actually mean?

The useful question is never “does it work?”. It is “is this version better than the previous one, on which scenarios, on which criteria, at what cost, with which errors?”. That is an experiment, not a review meeting.

FORGE, my evaluation lab, is built around that idea. Its central object is the evaluation run: one scenario version, one agent version, one configuration, the execution trace, the evaluations and the scores, frozen into a reproducible manifest. A comparison between two versions is then a comparison between sets of runs, with scores, cost, errors and a confidence level.

FORGE is a public beta, and its results depend on the scenarios you configure. It is the method I want to defend, not a claim that a tool solves evaluation for you.

Which evaluation layers do you need?

No single method is enough. Each layer catches a different kind of failure and has a limit that the next layer has to cover. I stack them in this order, from cheapest and most objective to most expensive and most nuanced.

How do you compare two versions of an agent system?

Pick a baseline and a candidate, run both on the same scenarios, repeat the runs because model outputs vary, and look at deltas per criterion rather than one global score. A single average hides the scenario where the candidate got worse.

• Freeze the scenarios and version them, so the test does not move while the agent does.
• Run the baseline and the candidate several times each, and compare with paired statistics, not a single run.
• Report deltas per criterion with a confidence interval, not only a composite score.
• List regressions per scenario, even when the average improves.
• End with a recommendation a person can act on: deploy, deploy with caution, do not deploy, or inconclusive.

How do you avoid fooling yourself with your own test set?

The biggest risk is not a bad agent, it is a good-looking one. Once a team tunes prompts against a public test set, the scores rise and the real behaviour may not. I separate three kinds of scenarios: public ones for everyday development, private ones whose content and expected result are hidden from the agent, and fresh ones added regularly to detect over-optimisation.

Canary strings help detect contamination, and structured error taxonomies (hallucination, contradiction, wrong tool, data leak, source error) turn a vague “it feels worse” into a countable, reviewable list.

Human calibration closes the loop. If the judges and the reviewers disagree, the first thing to fix is the judge, not the agent. Disagreement between judges is also the best way to prioritise what a human should read.

What does a RAG evaluation look like on a real platform?

On the DILA platform (public sector, services of the Prime Minister), the evaluation covered the whole RAG chain, not only the final text. That matters because a fluent answer can hide a retrieval failure. These figures were measured on the DILA platform, not by FORGE.

• Low faithfulness with high recall points at generation: the model had the facts and ignored them.
• Low recall points at retrieval: no prompt change will fix an answer whose evidence was never fetched.

Why score cost and latency like quality?

An agent that answers well but loops through tools for minutes is not better, it is more expensive. Cost, latency and tokens per step belong in the same scorecard as quality, with weights you set and can explain.

FORGE scores several dimensions, among them quality, coherence, reasoning, safety, robustness, cost, latency and user experience, and each composite score is published with its formula. The rule I hold to is simple: no score without an explanation. Every note must trace back to the criterion, the judge (model, version, prompt), the rule and the trace step concerned.

The DILA programme showed the same logic at platform level, where pipeline optimisation and reuse contributed to a 30% lower cost per AI request. That figure comes from the DILA platform, not from FORGE, but it shows why cost is a quality criterion and not an afterthought for finance.

What should a release gate decide?

A release gate turns evaluation into a decision. In practice it runs in continuous integration: if a candidate regresses on a criterion you declared critical, the pipeline stops, and the report shows which scenarios and which trace steps caused it.

The gate also needs a way out. A regression can be accepted, but only with a named owner and a reason, so the decision stays auditable. What the gate learns goes back into the scenarios: every incident found in production becomes a new test.

This is also what makes autonomy possible. When you can show what happens on a fixed set of scenarios, you can allow an agent to do more, and withdraw that permission when the evidence changes.