Chew Loong Nian’s diagnosis: a one-line rubric like “professional, natural and warm” is a ruler with no markings. Score the reply below and watch the 4.8 happen — then stress-test your own rubric against the three tests every rubric has to pass.
Built on Chew Loong Nian’s “How to Write an Agent Eval Rubric” (Towards AI) — framework and the 4.8 example via creator 煮caicai. The essay is member-only; this tool is built on its stated framework.
A real support reply from the essay. Score it 1–5 against the rubric “professional, natural and warm.”
That’s what the LLM judge gave this reply. Nothing wrong with it — and nothing useful either. It’s polite, warm, professional… and it solved nothing. A rubric with no markings can’t tell the difference between this and a reply that actually helps, so the judge defaults to a lenient, non-committal high score. Garbage rubric in, 4.8 out.
Every criterion in your rubric has to pass all three. One failure and the score is noise.
Does the criterion name concrete, checkable behavior in the reply — or vibe words like “natural” that no two readers would point to the same way?
Would it give a great reply and a useless-but-polite reply different scores — or a flat 4.8 for everything?
If a reply fails it, do you know exactly what to change — or just that the reply was “bad”?
Paste your rubric — one criterion per line. Answer the three questions honestly for each.