Skip to main content
The evals plugin checks that an agent still answers well. You save test cases in a dataset, pick scorers written in code, and start a run. Each case runs the agent once and gets a score from 0 to 1 per scorer. The run keeps the totals, so you can compare it with earlier runs.

Install

The plugin needs ai and at least one agent. It adds four collections in an Evals admin group:

Run a dataset

Create an eval-runs record for a dataset, in the admin or with the Local API. Creating it queues the frogbot-eval-run workflow on the default queue, so a jobs runner must be running for admin-started runs. Editing a run queues nothing, and the run’s own fields can’t be changed after it’s created. The run then:
  1. runs the dataset’s agent on each case, concurrency cases at a time (default 5), with the run’s model if one is set;
  2. applies the dataset’s scorers to each answer, in order, and saves one result per case;
  3. saves the totals and sets status to completed.
passed counts the cases where every scorer gave 1. scorerMeans holds each scorer’s mean over all cases, and score is the mean of those. A missing score counts as 0, so an errored case lowers every mean. A case’s agent error, a scorer that throws, and a score outside 0–1 are saved on the result and the run goes on. Only a failed result write fails the run; its retry skips the cases already saved. Cases don’t save a chat, so they don’t show in chat lists or training data. The model must be one of the agent’s model.options. Another model refuses the run when it’s created, before anything is queued. Leave it empty to use the agent’s default. To compare two models, start a run for each.

Scorers

A scorer receives the case and the answer and returns { score, reason? }, sync or async:

Checks

checks are scorers that give 0 or 1, with a reason when they fail:

Judge

judge({ rubric, model? }) asks a model to score the answer against your rubric. It makes one generateText call per case with the rubric, the question, the expected answer, the tool calls, and the answer. It reads back a score and a reason. It uses model, or ai.smallModel when model is unset. With neither, the config throws. The judge’s calls are normal AI calls: they’re metered in usage logs and checked against ai.access for the run’s user. A dataset of 200 cases with one judge costs 200 agent runs plus 200 judge calls per run.

Run from CI

aiEvalsPlugin() adds the evals:run command to the FrogBot CLI:
It creates a run of the dataset named, or with the ID, billing, runs it in this process with no jobs runner needed, and prints each scorer’s mean and the score. It exits 1 when any case errored, the run failed, or the score is below --min-score.

Access

The plugin sets no access on its collections; pass access to set your own, applied to all four. Without it, any signed-in user can read and write them. A run started by a user runs as that user: the agent’s access, its tools’ access and ai.access apply, as in chat. A run with no user, such as one from evals:run or a Local API call without req, runs trusted with overrideAccess: true.

Real tools

Cases run the agent’s real tools. A case such as “Email the customer their invoice” sends that email on every run. Run evals against test data and test accounts, or leave tools with side effects off the agent you evaluate.

Options