Install
ai and at least one agent. It adds four collections in an Evals admin group:
Run a dataset
Create aneval-runs record for a dataset, in the admin or with the Local API. Creating it queues the frogbot-eval-run workflow on the default queue, so a jobs runner must be running for admin-started runs. Editing a run queues nothing, and the run’s own fields can’t be changed after it’s created.
The run then:
- runs the dataset’s agent on each case,
concurrencycases at a time (default5), with the run’smodelif one is set; - applies the dataset’s scorers to each answer, in order, and saves one result per case;
- saves the totals and sets
statustocompleted.
passed counts the cases where every scorer gave 1. scorerMeans holds each scorer’s mean over all cases, and score is the mean of those. A missing score counts as 0, so an errored case lowers every mean.
A case’s agent error, a scorer that throws, and a score outside 0–1 are saved on the result and the run goes on. Only a failed result write fails the run; its retry skips the cases already saved. Cases don’t save a chat, so they don’t show in chat lists or training data.
The model must be one of the agent’s model.options. Another model refuses the run when it’s created, before anything is queued. Leave it empty to use the agent’s default. To compare two models, start a run for each.
Scorers
A scorer receives the case and the answer and returns{ score, reason? }, sync or async:
Checks
checks are scorers that give 0 or 1, with a reason when they fail:
Judge
judge({ rubric, model? }) asks a model to score the answer against your rubric. It makes one generateText call per case with the rubric, the question, the expected answer, the tool calls, and the answer. It reads back a score and a reason.
It uses model, or ai.smallModel when model is unset. With neither, the config throws.
The judge’s calls are normal AI calls: they’re metered in usage logs and checked against ai.access for the run’s user. A dataset of 200 cases with one judge costs 200 agent runs plus 200 judge calls per run.
Run from CI
aiEvalsPlugin() adds the evals:run command to the FrogBot CLI:
billing, runs it in this process with no jobs runner needed, and prints each scorer’s mean and the score. It exits 1 when any case errored, the run failed, or the score is below --min-score.
Access
The plugin sets no access on its collections; passaccess to set your own, applied to all four. Without it, any signed-in user can read and write them.
A run started by a user runs as that user: the agent’s access, its tools’ access and ai.access apply, as in chat. A run with no user, such as one from evals:run or a Local API call without req, runs trusted with overrideAccess: true.