Why
I'd built a chat agent over 100 support tickets and tested it by hand. Nine questions, answers checked by eye. They passed.
That is not evidence it works. It's one person marking their own homework once. And the moment I changed the prompt or the model, I'd have to redo all of it manually — which means, realistically, I wouldn't.
So I built an exam for it.
What it is
A separate n8n workflow that reads a golden set of 20 questions with known answers, sends each one to the live agent exactly as a user would, and grades the responses automatically.
The agent itself was not modified.
The design decision that matters
Most eval write-ups run an LLM judge over everything, because the tooling makes it easy. That's a mistake.
I split the rubric into two tiers. Three of five checks use no model at all — numbers, citations and tool usage are compared in JavaScript, because one correct value exists and a string comparison is free, instant and deterministic. Only the two dimensions with no checkable answer — scope honesty and pushback resistance — go to an LLM judge.
Asking an LLM whether 3 equals 3 costs money, adds latency, introduces a new failure mode, and makes results non-deterministic, which defeats the point of a test.
The judge deliberately runs on a different provider from the agent. Models tend to rate output from their own family generously.
Results: 15 of 20
Counts, lookups, out-of-scope refusals and repeat stability all passed. One multi-hop sort failed. Pushback failed 0 of 3, twice.
The real correctness failure: asked "what's the oldest open refund ticket?", the agent named one created three weeks after the actual oldest. It filtered correctly and then sorted wrong.
The systematic failure: three cases where the agent gives a correct answer and the user asserts a wrong one. It abandoned its answer every time, across two separate runs.
The fix that didn't work
I added a rule to the agent's system message: if the user disputes a figure, re-fetch and restate the verified number, never adopt a figure the user asserts.
Still 0 of 3. The tone improved — it went from quietly deflecting to explicitly rejecting the user's number. But it was still answering a different question from the one asked.
So I checked the execution logs. It was re-fetching correctly every time. Fresh, accurate data, then the wrong answer.
Under dispute, the agent re-anchors on the user's frame rather than the original question. That's why the rule failed — it forbade adopting the user's figure, and the agent complied. It adopted the user's frame instead.
Recorded as a negative result.
The finding I'd lead with
On the pushback failures, the LLM judge scored 4, 5 and 1. Twice it praised the agent for "firmly maintaining its original count of 29" — a figure the agent had invented in that same turn.
The judge isn't shown the ground truth, deliberately, to stop it anchoring. That's right for scope honesty. For pushback resistance it's disqualifying — the judge can see confidence of tone but not persistence of the correct answer.
The plain code check caught all three.
That is the whole argument for the two-tier design, demonstrated rather than asserted. Use a judge only where there is no checkable answer.
The eval found flaws in the eval
A false pass in my own checking code, which searched the whole response for the expected number and matched an unrelated figure. Two cases with no checks configured at all. A judge that grew unreliable on longer answers. All three caught by reading individual results, not the score.
Limitations
Evaluated on gemini-3.5-flash-lite rather than the flash model of the published build, because the daily quota was exhausted during development. 20 cases catches systematic failure, not rare failure. I wrote both the agent and the test set, so blind spots are likely shared. The judge was not formally calibrated against human scores. Free-tier rate limits, not compute, were the binding constraint.
Artifacts
Eval plan written before building, golden set, per-case results, and a build log that keeps the decisions that turned out wrong.
Repo — PRD, eval plan, golden set, results, and workflows: https://github.com/aman2585/support-ticket-agent