Tutto.

Praxis class plan · Advanced track

Measuring extraction: the golden dataset

The specification and the answer key are the same document. Write down what counts as correct at the moment you define the field, and the eval comes free.

Runs in 75 minutes. A 60-minute cut is on the last page.

Pairs with Lab 07, Evals. Rubrics score judgment. This scores extraction.

Learners leave with A schema, an extraction skill, an accuracy number, and a way to move it.

What this class teaches

One idea, taught by withholding it. The learner spends the first fifteen minutes writing a schema and believes they are doing preparation. Forty minutes later that schema turns out to be the only reason anything can be graded. The delay is the lesson.

Most people who try an eval for the first time abandon it. They build one, see 40% accuracy, conclude the tool cannot do the job, and stop. The number was never real. It was measuring punctuation. This class walks them into that failure on purpose, in a room, with someone there to name it.

The three things a learner should be able to say afterwards

An answer key turns "looks right" into a number. Eyeballing one output is a demo. Production needs a rate.
Most disagreements are formatting, not data. Normalise both sides before you score.
You fix the specification, never the output. A wrong answer is almost always an underspecified field.

The reveal. In Part 1 the learner fills in a column headed "match rule" and is told only that it records what counts as a correct answer. They are not told it is the grader. In Part 5 they discover that nothing can be scored without it. Hold that back. Saying it at the start converts a discovery into an abstraction, and the class stops working.

Who it is for

Anyone whose work involves pulling the same fields out of long documents: contracts, policies, medical reports, statements, invoices. It lands hardest with the Analyst archetype, and with anyone who has already been burned by a colleague's spreadsheet.

Prerequisite

A completed Plato build. Lab 07 is not required, but if the learner has done it, position the two directly: a rubric grades work with no single right answer, an answer key grades work with exactly one. Same family, opposite halves. If they have not done Lab 07, do not mention it.

Materials to build before the room

This class cannot be run from slides. It needs a corpus, and building it is the real cost. Budget a day, once. It then serves this class, the corporate programme, and the microsite demos.

The corpus

Four synthetic UK managed-services agreements, twenty to thirty pages each, internally consistent, written as a real supplier contract is written. Supplier agreements were chosen because every beachhead sector signs them: law, accounting, financial services, personal injury.

Agreement	Supplier	Initial term	Charge periods
northgate	Northgate Facilities Management Limited	36 months	36
caldera	Caldera Analytics Limited	48 months	48
wrenfield	Wrenfield Print & Mail Limited	24 months	24
holloway	Holloway Legal Support Services Limited	36 months	36

Terms differ on purpose. A learner who spots that all four run 36 months starts pattern-matching instead of reading.

The schema, handed out half-blank

Two linked tables. Table 1 holds one record per agreement. Table 2 holds the charge schedule, which repeats once per period and therefore cannot live in a flat row. Give the learner the first three rows filled in as worked examples and leave the rest empty.

Table 1: Agreement (one record per agreement). Bold rows are pre-filled for the learner.
Field	Description	Match rule
supplier_name	Full legal name of the supplier entity, including the suffix. Clause 1.1, "the Supplier".	Legal name with suffix. Not the trading name used in the body.
contract_value	Total charges payable across the initial term.	Initial term only. Excludes optional modules. Excludes VAT.
effective_date	Date the agreement becomes binding. Clause 2.1.	DD/MM/YYYY. Effective Date, not Service Commencement Date.
service_category		Facilities, IT Managed Services, Print & Mail, or Professional Services.
currency
service_start_date	Date live service begins, after onboarding. Clause 2.2.
initial_term_months	Length of the initial term in whole months.	Initial term only. Excludes any renewal extension.
renewal_term_months
notice_min_months	Minimum months of notice before expiry (the "not later than" bound). Blank if no renewal.
notice_max_months	Maximum months before expiry that notice may be served (the "not earlier than" bound).
security_amount
security_type		Performance Bond, Parent Company Guarantee, Cash Deposit, or blank.
sla_credit_basis	How service credits are calculated. Blank if there is no credit regime.

Table 2: Charge schedule (one record per period), linked to Table 1 by agreement.
Field	Description	Match rule
period_no	Sequential period number.	Starts at 1 at the Effective Date, including onboarding periods charged at zero.
payment_date	First day of the period.
payment_amount	Charge for that period.	Number only. Onboarding periods are 0, not blank.

The seven traps, and why each one is there

None of these are extraction failures. They are specification failures wearing an extraction costume. Every one of them is a plausible wrong answer that a careful human would also get wrong, which is exactly what makes them teach.

#	Trap	What the document does	What resolves it
1	Trading name against legal name	The body says "Northgate FM" forty times. The legal name appears once, in clause 1.1.	Match rule: legal name with suffix.
2	Three defensible contract values	A commercial summary headlines an annual fee of £240,000. The charge schedule totals £659,400. With optional modules it comes to £784,900.	Match rule: initial term only, excluding optional modules and VAT. Answer: 659400.
3	Two commencement dates	Effective Date is 01/02/2027. Service Commencement Date is 01/05/2027, after three months of onboarding. The second appears more often.	Two separate fields. The rule names which is which.
4	Initial term against renewal	36-month initial term with a 24-month renewal option in the same clause.	Match rule: initial term only.
5	One clause, two fields, inverted	"The Customer may serve a Renewal Notice not earlier than nine months and not later than six months before expiry."	min = 6, the later bound. max = 9. The clause states them in the opposite order to the field names.
6	Security that steps down	A £120,000 performance bond reducing to £80,000 at year two and £40,000 at year three.	Match rule: the amount at commencement. Answer: 120000.
7	Zero-value periods	The first three months are charged at £0 during onboarding.	Include them. Drop them and 33 of 36 rows carry the wrong period number.

Trap 7 is the one to save for the room. One omission, three rows missing, thirty-three rows silently misnumbered. Say the numbers out loud. It is the cheapest way to show that a small specification gap is not a small error.

The Northgate charge schedule, abridged

Period	Date	Amount	Note
1–3	2027-02-01 → 2027-04-01	0	Onboarding, no charge
4–15	2027-05-01 → 2028-04-01	19500	Year one rate
16–27	2028-05-01 → 2029-04-01	20000	Uplift
28–36	2029-05-01 → 2030-01-01	20600	Uplift

The answer key

One spreadsheet, two sheets, mirroring the schema. Someone builds it by hand and verifies every value against the document. There is no shortcut, and pretending otherwise is the single most common way this subject gets taught badly.

Sheet	Rows	Graded values
GoldenAnswers	4	52
ChargeSchedule	144	288
Total		340

Roughly 85% of the gradeable surface is the charge schedule. That is deliberate and it does two things. It puts the accuracy number where the real differences hide, and it makes eyeballing physically impossible. Nobody checks 340 values by eye, which is the whole reason the class exists.

The format collision, and why it stays in

The schema asks for dates as DD/MM/YYYY. The answer key stores them as YYYY-MM-DD, because it was built in a spreadsheet by a different person on a different day. A learner who follows the specification exactly will mismatch the key on every date field.

Leave it. It is not a trick, it is what actually happens when a specification and an answer key are produced independently. If a sharp learner challenges it, agree with them immediately and say so.

If challenged. "You are right, they disagree. They were built by different people at different times, and that is normal. The data agrees. The formatting does not. Which of those two is the eval supposed to be measuring?"

Keep the key out of reach

Answer keys live in a folder the learner does not open, and the completed schema lives there too. If someone opens the key during Part 4, the rest of the class is theatre.

Part 1 to Part 3: build the thing30 min

Three parts that look like ordinary work. The learner is building an extraction skill. They have no idea they are also building a grader.

Exercise 1 · 15 minutes

Finish the schema

Hand over the half-blank schema. Three columns per field: the name, the description of what it is and where to find it, and the match rule. Fill in the first two or three together, out loud, then let them work.

A schema is the formal definition of the data you want out. Three things for each field. What it is called. What it is, and where in the document it lives. And what counts as a correct answer.

That third one is the part people skip. "Contract value" sounds obvious until you open the document and find three numbers that all have a claim to it. Write down which one you mean.

Do not say that the match rule is the grader, that this is an eval, or that any of this will be scored later. If asked why the third column matters, answer with trap 2 and nothing more.
Watch for the learner who writes "the correct value" in the match rule column. Push once: correct according to what? That is the whole exercise.

Exercise 2 · 10 minutes

Extract one agreement

Point Claude Code at the schema and at the Northgate agreement. Ask for one row per field, following the match rule column. A usable prompt names the document, the schema, and the output form.

Using the schema as the field definitions, extract the Northgate agreement. Return one row per field and follow the match rule column.

Then stop. Read it together. Then refuse to resolve it.

Look it over. Does anything seem off to you?

Hold that. Do not fix it yet. We will come back to it with a number instead of an opinion.

This is the doubt you are planting. Somebody in the room will want to correct a field right now. Do not let them. The point of the class is that the fix comes from evidence, and there is no evidence yet.

Exercise 3 · 5 minutes

Package it as a skill

Turn the working prompt into a skill file so it can be re-run without retyping. Ask Claude Code to write it and to ask clarifying questions first, then review what it proposes before saving.

A skill is a text file that describes an action you want to run on demand. Nothing clever. The reason we are saving it now is that in ten minutes you are going to want to run this same extraction four times, and then run it again after you change it.

Part 4: run it wide10 min

The felt need. You cannot make anyone want a measurement until the output exceeds what their eye can hold. This part exists to produce that moment and nothing else.

Exercise 4 · 10 minutes

Four agreements, full schedules

Run the skill across all four agreements. Ask for one record per agreement, plus the complete charge schedule for each. Not a sample. All of it.

That is roughly three hundred and forty values on your screen.

Two minutes ago you were reading thirteen fields and asking yourself whether they looked right. Try that now. Tell me your error rate.

Let the silence sit. Nobody can answer. That is the answer. If someone starts scrolling and squinting, let them scroll for a while before you move on. The discomfort is the teaching.

Then name the gap plainly, and only then introduce the answer key.

Eyeballing one result is fine for a demo. In production you need to know how often you are right, and you need it as a number you can compare against last week's number.

So somebody sat down with these four agreements and wrote out the correct value for every field, by hand, checking each one against the document. That is the answer key. It took most of a day.

Say the cost out loud. "Most of a day" is the most useful sentence in this class for anyone deciding whether to do this at work. An eval is cheap to run and expensive to start. Trainers who skip this send people back to the office with a false estimate.

Part 5: grade it, and the reveal15 min

The centre of the class. The learner asks Claude Code to score its own output against the key, gets a number that is obviously wrong, and finds out why.

Exercise 5.1 · 5 minutes

Score it naively, and let it fail

Ask Claude Code to compare its output to the answer key, field by field, and report where it matches and where it misses. Do not warn anyone about formatting. Let the number come back low.

Expect something in the 40s or 50s. Every date fails on the format collision. Amounts fail on currency symbols and separators. "Ltd" fails against "Limited". The extraction is mostly right and the score says it is mostly wrong.

So the tool is wrong about half the time. Is it? Pick one of the failures and read both values out to me.

Somebody will read out something like £659,400.00 against 659400. Let the room land on it themselves.

Exercise 5.2 · 5 minutes

Normalise, then score again

Ask Claude Code to normalise both sides before comparing, then re-run the scoring. The number jumps. Grade the charge schedule period by period, because that is where the real differences hide.

A correct answer can look different from the answer key. Six hundred and fifty nine thousand four hundred with a pound sign and a comma, against the same number bare. The first of February written two ways. "Ltd" against "Limited". An empty cell against null.

Normalise the formatting on both sides before you score, or you will grade the punctuation instead of the data.

This is the line to deliver verbatim. It is the single most reusable sentence in the class and the reason most first evals get abandoned. Slow down for it.

Exercise 5.3 · 5 minutes

The reveal

Now go back to what remains failing after normalisation. Every one of those will trace to a field whose match rule was vague, or blank. Open the schema from Part 1 next to the results.

Go back to the schema you filled in an hour ago.

The third column. The one you thought was documentation. That is the only reason any of this could be scored at all. Without a written rule for what counts as correct, "wrong" is just disagreement.

You did not write a data definition this morning. You wrote the grader, and then you wrote a tool that gets graded by it. That is the whole method. The specification and the answer key are the same document.

Then stop talking. Do not restate it, do not summarise it, do not add an example. This is the beat the class was built around. Let them get there.

Part 6 and Part 7: see it, then move it16 min

An eval you cannot read is an eval you cannot act on. An eval you cannot move is a report card. These two parts turn the number into a control.

Exercise 6 · 8 minutes

Build a scorecard

Ask Claude Code to build a visual scorecard: one row per field, one column per agreement, matches and misses colour-coded, the running accuracy at the top, and a filter down to the misses only. This is the artefact you would put in front of a partner or a board.

A wall of pass and fail text is technically an eval and practically useless. You want to look at this and know within three seconds which field is your problem.

Exercise 7 · 8 minutes

Fix the specification, never the output

Take the top failing field. Diagnose which of three things it is, apply the matching fix in the schema and the skill, and re-run the whole thing. Watch the number move.

Symptom	Cause	Fix
Pulled the wrong number from the right area	Description too vague	Say where in the document to look
Defensible, but not the key's answer	Genuine ambiguity	Pin it down in the match rule
A value that is not in the document	No guardrail	Instruct it to return null when absent

Notice what we did not do. We did not correct the answer. Correcting the answer fixes one cell today and nothing tomorrow.

You change the rule and re-run. If the number goes up, the rule was the problem. If it goes down, you have learned something about your own thinking, which is worth more.

The most common derail in the whole class. Someone will edit the output spreadsheet, or worse, edit the answer key to match their output. Stop it immediately and name it: that is marking your own homework. The key does not move.

Part 8: the close4 min

Four minutes, and most of it is about what this does not give them. A class that oversells an eval produces someone who trusts a number they should not.

What you have is not proof the output is correct. It is proof the output agrees with a key that a person wrote, and that person can be wrong. The key is a judgment, written down and frozen.

What it gives you is direction. When you change a rule, you know within a minute whether the change helped or hurt, across a fixed set, with evidence. That is enough to stop guessing, and stopping guessing is most of the value.

It does not replace reading the output. It tells you which output to read first.

What this class deliberately does not teach

Name these. A trainer who leaves them out sends people away thinking evals are a solved problem.

Who builds the key. It was handed over here. In real work a senior person spends a day on it, and that day is the actual barrier to entry.
Regression. You fix one field and the headline number rises. Did another field break? With 288 schedule values dominating the score, a single agreement-level regression is invisible in the average.
Sampling. Four documents is a smoke test, not a sample. Nothing here tells you how many documents make a number trustworthy.
Abstention. Nothing teaches the tool to say "I could not find this". A confident wrong answer and a confident right answer score differently and look identical.
Fields with two right answers. This class is single-correct-answer only. The other half, where good is a matter of judgment, is Lab 07 and its rubrics.

The pair, stated once at the end. An answer key grades extraction, where one value is right. A rubric grades judgment, where several outputs could be good. Between them you can measure both halves of the work. Neither one measures the other.

Trainer run sheet

Part	Min	What the learner does	What must not be said
1	15	Fills in the half-blank schema	That the match rule column is the grader
2	10	Extracts one agreement, spots something off	Which field is wrong, or how to fix it
3	5	Packages the prompt as a skill	Why they will need to re-run it
4	10	Runs all four, hits 340 values	Anything. Let the silence do it
5	15	Scores naively, normalises, re-scores, gets the reveal	The normalisation warning, before they fail
6	8	Builds a scorecard
7	8	Fixes a rule, watches the number move	That editing the output is ever acceptable
8	4	Hears the limits	That an eval proves correctness

The 60-minute cut

Pre-fill more of the schema. Six rows instead of three. Part 1 drops to 8 minutes and the reveal is unaffected.
Demonstrate the scorecard. Show a prebuilt one rather than building it live. Part 6 drops to 3 minutes.
Cut nothing else. Parts 4, 5 and 7 are the class. Everything else is scaffolding for them.

Things that go wrong

Someone opens the answer key early. Keep it in a folder you have told them not to open, and check before Part 4.
The room wants to skip to the tool. Run Part 4 early if you have to. The 340-value wall answers the objection better than you can.
Someone fixes the answer key. Stop it and name it. The key does not move.
Someone asks what this costs to run. About twenty dollars a month.
A sharp learner spots the date collision before Part 5. Good. Agree, and ask them to hold it until the scoring. It becomes their moment instead of yours.