Tutto.Praxis · talking deck
Praxis: Advanced track · 45 minutes

What an eval is,
and when you
need one.

The theory before the workshop. What these things actually are, why the first one people build almost always gets abandoned, and how to pick the right kind for the work in front of you.

1
Tutto.Where we're going
Where we're going

Four questions, in order

Each one only makes sense once the previous one is answered.

What

  1. 1The problem evals solve
  2. 2Where the line sits

Which kind

  1. 3One right answer, or several
  2. 4The four graders

How

  1. 5The four moving parts
  2. 6What the number is for

What goes wrong

  1. 7Three ways it fails
  2. 8What it costs
2
Tutto.1 · What
The problem

You checked it once, and it looked fine

That is a demo. It tells you the tool can work. It does not tell you how often it does.

WHAT MOST PEOPLE DO Read one output. Decide it looks about right. Works until the day it quietly stops working. WHAT AN EVAL DOES Runs a fixed set. Scores it. Gives you a rate you can compare to last week's. Same set, same rules, every time you change something. An eval is a scorecard for the parts you cannot unit-test.
3
Tutto.2 · What
Where the line sits

Tests for the boring parts. Evals for the judgment.

Two tools, one line between them. Most confusion about evals is someone reaching for the wrong side of it.

DETERMINISTIC · TESTS Same input, same output, every time. Did the file parse? Did the total add up? Is the hash the same as yesterday? pass / fail PROBABILISTIC · EVALS Same input, a good answer that varies. Did it pull the right value from an ambiguous clause? Does the draft sound like us? a rate, out of a fixed set Same spirit, different half. Neither one measures the other.
4
Tutto.3 · Which kind
The choice that decides everything else

Does the question have one right answer, or several good ones?

Answer that and the kind of eval is already decided. Get it wrong and you will spend a week building the wrong instrument.

ONE RIGHT ANSWER → ANSWER KEY Extraction Pull the contract value out of this agreement. There is a correct number. It is in the document. Graded against a key someone wrote by hand. SEVERAL GOOD ONES → RUBRIC Judgment Draft a client update about this delay. Ten good versions exist. None is "the" answer. Graded against written criteria, not exact text. Between them you can measure both halves of the work. Neither one measures the other.
5
Tutto.4 · Which kind
How the scoring actually happens

Four ways to grade, cheapest first

Use the cheapest one that can express what you mean. Reaching for the expensive one first is the most common mistake.

1 · EXACT MATCH Does the string equal the key? Free. Brittle. Fails on a comma. 2 · NORMALISED Strip formatting from both sides, then match. Free. Correct. Where most work belongs. 3 · MODEL JUDGE Claude scores it against written criteria. Cheap. Needs checking. The only option for judgment. 4 · HUMAN A person reads it and decides. Expensive. Final. Calibrates the others. A model judge you have never checked against a human is a number with no one behind it.
6
Tutto.5 · How
Anatomy

Every eval is the same four parts

Whatever the kind, whatever the domain. If one of these is missing you do not have an eval, you have a spreadsheet.

1 · CASES A fixed set you own. Frozen, not live data. 2 · CRITERIA What counts as correct, written down in advance. 3 · RUNNER Runs the work, scores it, prints the result. 4 · A RATE 11 of 15 · 73% One number that moves. Part 2 is the hard one, and it is not a technical problem. Deciding what counts as correct is domain judgment. Whoever owns the standard has to write it, and that is rarely the person writing the code.
7
Tutto.6 · How
What you actually do with it

The number is not a grade. It is a direction.

Nobody needs to know they are at 73%. What they need is to know whether yesterday's change helped.

CHANGE THE RULE Not the output. Not the key. The written specification. RE-RUN THE SET Same cases. Same criteria. A minute, not a meeting. READ THE MOVE Up: the rule was the problem. Down: you learned something about your thinking. It does not replace reading the output. It tells you which output to read first.
8
Tutto.7 · What goes wrong
Why most first evals get abandoned

Three ways this goes wrong

People build one, get a bad number, conclude the tool cannot do the job, and stop. The number was usually measuring the wrong thing.

GRADING THE PUNCTUATION £659,400.00 vs 659400 The data agrees. The formatting does not. Normalise both sides. The most common one by far. MOVING THE KEY The output disagreed, so someone edited the key. Now the number only measures how willing you are to move it. HIDDEN REGRESSION You fixed one field and the average went up. Something else broke, and the average hid it. All three produce a number that looks like an answer and is not one.
9
Tutto.8 · What goes wrong
Before you promise this at work

Cheap to run. Expensive to start.

The cost falls the opposite way round to what most people assume, and getting it wrong is how these die in month two.

STARTING IT ~1 day A senior person writes the key by hand, checking every value against the source. RUNNING IT ~£0 Re-runs on demand, in about a minute, for as long as the work exists. Build it once, for the work you do repeatedly. Never build one for a job you will do twice.
10
Tutto.What's next
Where this goes

Two sessions, one for each half

You now have the map. The next two sessions are the two halves of it, built by hand.

NEXT · 75 MINUTES Measuring extraction Build an answer key against four contracts. One right answer per field. 340 graded values. LATER · LAB 07 Rubrics Score the judgment half against written criteria. Several good answers. A pass-rate that moves.
11
Tutto.Praxis
The whole thing, in one line

An eval does not prove
you are right. It tells you
which way you moved.

Write down what counts as correct. Freeze a small set of cases. Score them the cheapest way that expresses what you mean. Then change a rule and watch the number.

tutto.one/praxis · daniel@tutto.one

12
← → or space · N for notes · F11 fullscreen
SPEAKER NOTES ON · press N to hide