Agent evals
The voice agent on this site answers questions about me from a fixed knowledge file. This is the scored suite that checks whether it stays inside what it actually knows. The current score is below; what the suite has already caught, including a production bug and two of my own bad tests, is further down and more interesting.
Loading the latest run…
One cell per call, left to right in the order they ran. This is the whole run, not a sample — a case only counts as passing when every one of its runs did, and any that missed are shown where they actually fell.
Each failing case below shows what was asked and which expectation it missed. Some replies are withheld — where the failure was the agent repeating something this site deliberately keeps private, printing it here would repeat the mistake.
| Case | Category | Status | Runs |
|---|
A passing score is only worth reading if the suite has ever failed. This is the log of what it found, kept by hand.
The completion token ceiling was set to 90. When the model spent budget reasoning before writing, it returned empty content and the endpoint failed — roughly one call in eight, and only on harder questions, which made it look random rather than systematic. Clicking around the demo would never have surfaced it. Fixed with token headroom, a request timeout, and retry on transient upstream errors.
Two cases failed against a correct agent. One demanded digits where the agent writes numbers as words, because everything it says is read aloud. The other matched the literal word "email" when the agent had given an actual email address. Both were fixed in the suite, not the agent — a red test is a hypothesis, not a verdict.
An early version counted network failures as wrong answers and published a run reporting every case as failing. It had measured nothing. The runner now preflights the endpoint and discards any run containing a transport error rather than reporting a result it cannot stand behind.
Each case is a behavior contract rather than a sample answer: things that must appear, things that must never appear, and a global word ceiling. A case passes only if every check passes on every run.
Cases run many times, because language models are not deterministic. One that passes once and fails twice is a real defect a single run hides, so flaky is reported separately and counted as not passing. The run count is published above rather than fixed here — three is enough to gate a deploy, and fifty is what it takes to tell a case that is reliable from one that has been passing on luck.
That distinction is not hypothetical. The first fifty-run pass turned up a case sitting at 48/50 that had read a clean 3/3 for a week — and the two misses were the agent answering correctly in a phrasing the check had not anticipated. The check was widened; the agent was not changed.
When the agent says something wrong in a real conversation, the case gets written before the prompt is fixed. That order proves the fix worked and stops the regression coming back. A few cases publish their status only — the question itself is the sensitive part, and the full suite lives in the repository.