Evals and verification

An eval is a check you run on the assistant's output, repeatedly, so you know how often it is right instead of remembering the last time. Verification is the same idea inside one task: a test, a rubric, a second model acting as judge, a comparison against the source. It is how trust in an assistant gets earned rather than assumed.

Standing: Rising since 2025-04
Stars, week: +929 across 4 repositories
Stars, month: +6,742, through 2026-11-08
Listings: 5 chosen by hand

The standing

Where it stands

Among the 14 concepts. Bars: GitHub stars gained this week.

Established · 7

Rising · 7

How to use it today

In ChatGPT or Claude, as a person doing the work, not as a developer.

1Before automating a task, collect ten real inputs and the outputs you would accept; that is your first eval set.
2Ask a fresh chat to grade the output against a short rubric you wrote before you saw the output.
3For recurring tasks, keep a running tally of edits you had to make; a rising count means the context or the skill drifted.
4Use a second model, or a fresh chat, as a judge: paste the output and the brief and ask what is missing or wrong.
5In Claude Code, write the check (a test, a script, a comparison) before the task and let the loop run against it.

What changes day to day

You stop reading every output in full and start reading the verdicts. The time saved goes into writing better acceptance criteria, which turn out to be the hard part of the work.

Where it goes wrong

A model judging its own output in the same chat agrees with itself.

Rubrics written after seeing the output measure the output, not the task.

Listings that use it well

Chosen by hand after reading them, each with the reason. Install from the listing's own page.

Listing	Kind	Why it is here	Stars
ai-agent-book	Agent skills	agent-evaluation and eval-dataset-design: Pass@k, deterministic verifiers versus LLM as judge, rubrics and leakage control.	57k
llm-finetuning	Agent skills	eval-harness-first builds golden sets, per failure graders and judge calibration before any training run.	42k
AI Analyst	Agent skills	eval runs a held out gold suite, reliability repeats a question to measure drift, triangulation cross checks every number.	34
plugin-eval	Agent skills	Static lint plus LLM scoring for skills with a documented rubric and scoring formula.	42k
OpenJudge	Agent skills	A unified evaluation framework with LLM, function and agentic graders, prompt regression and RAG eval.	1.1k

Elsewhere

Repositories that embody the idea and are not in the catalog.

promptfoo/promptfoo ↗Declarative tests for prompts, agents and RAG with CI integration; now part of OpenAI, still MIT.
confident-ai/deepeval ↗Pytest-style unit tests for LLM outputs with research-backed metrics.
openai/evals ↗OpenAI's eval framework and open registry of benchmarks.

Standing

Our judgment, with the reason and the sources it rests on.

Rising since 2025-04. The vendors made evals easy in 2025 and 2026 (OpenAI's Evals API in April 2025, Anthropic's agent eval guidance in January 2026), but LangChain's late 2025 survey found only 52.4% of teams running offline evals and 37.3% online. Evals are recommended practice that about half of teams follow. Adopted faster each month; the vendors are building for it and new tools keep appearing.

Your AI Product Needs Evals2024-03-29
OpenAI Developers on X: Introducing the Evals API2025-04-08
Demystifying evals for AI agents2026-01-09
State of Agent Engineering2025-12