Write the Dataset Before You Write the Agent

I'm building an AI agent in public. Every phase, every mistake, every number, out in the open, so you can see what it actually takes to run one in production rather than in a demo.

Here is why that takes a series.

How Most Agents Are Built

Almost every AI application looks the same underneath. Something comes in. An LLM in the middle decides what to do, maybe calls a tool or two. Something goes out.

Three boxes. This is the part that gets built, and it demos beautifully.

You build this, you try it on a few examples, it works. You show someone. It works again. So it ships.

Then it starts being wrong in production, and here is the problem: nothing in that picture can tell you. There is no test set, so nothing catches a change that made things worse. There is no trace, so when someone reports a bad answer you cannot see what the agent actually did. There is no score, so "is it still working?" has no answer beyond someone's impression. You find out from the person who acted on a wrong answer.

The agent is not the hard part. Knowing whether the agent is still right is the hard part, and it is almost entirely missing from the demo.

What a Production One Actually Looks Like

Here is the full shape. Everything the agent needs around it before you can leave it running unattended.

The agent is one box in the middle. Everything above and below it is this series.

Reading that top to bottom: a hand-labelled dataset and scorers exist before the agent does. Running them produces a baseline, the first score, recorded once. Then the agent gets built. A CI gate reruns the dataset on every change and blocks anything that lowers the score. In production, traces record every step of every run, and live scoring watches real traffic rather than only the fixed test set. When the agent cannot decide, it escalates to a human, who can answer and resume the run. And when something does go wrong, rollback takes seconds.

That is the series. One piece per chapter, built on a real system, with the numbers published as they come.

What We Are Building

The agent watches a public register in Denmark. Municipalities publish the rules for what you may build on a piece of land: how tall, how many storeys, how much of the plot you can cover. Those rules change. A plan gets adopted with different numbers than it was proposed with, and nobody is notified.

If you are advising a client on a building project and the rules underneath it shift, you want to know that week, not at the planning meeting six months later. The register itself is also messy: numbers get typed into the wrong fields, and there is no built-in way to tell a real change from a correction.

It is called Tilsynsagent. If it had a launch poster, it would look something like this.

The 70% in the corner is real, and the rest of this article is about why it is printed on the poster.

Under the poster it is four steps.

Deterministic rules run first. The model only sees what rules cannot decide.

The code, the dataset, and every score are public as each phase lands.

TL;DR

This chapter is one step: the dataset comes before the agent, not after.

• A dataset written after the agent can only agree with it. You look at what it did, decide it looks fine, write that down as correct.
• The dataset is where the domain expert's judgment enters the system. Written requirements are always shorter than the judgment behind them, and labelling is where the missing part surfaces.
• The first eval run is a gap list, not a grade. Every failure is a place where what got built and what the business meant have drifted apart.
• Then record the baseline. The first score, before you fix anything. Without one you cannot tell whether a change helped.
• Mine came back at 70% on the metric I care about most. That is the number every later chapter gets measured against.

The Dataset Is Where the Business Gets a Vote

Here is the order I nearly built this in. Write the agent. Get it working. Then write some test cases.

That is backwards, and it hides well. If the agent already exists, you build your test set by running real inputs through it and deciding which outputs look right. Those become your expected answers. It feels careful, and it is not dishonest, but your answers came from the agent's own outputs. When the agent is confidently wrong in a plausible way, you nod and write that down as correct. I call this agreement by construction: the suite passes because it was built from the thing it is testing.

The difference is not effort or rigour. It is whether the dataset was ever in a position to say no.

That is the mechanical version. The version that actually matters is about who defines correct.

These projects run the same way everywhere. The business describes what it wants. Engineers turn that into rules, prompts, and code. That translation is where the damage happens, because a written requirement is always shorter than the judgment behind it. The expert knows a hundred things they never thought to say, because to them they are not rules, they are just obvious.

The dataset is where those unspoken things get captured, which is why the person labelling it should be the domain expert, not the engineer who wrote the rules.

Both sides did their job. The knowledge just never made the trip.

Here is a case from my set that shows exactly what that catches. One field changes: built percentage, the share of a plot you may cover, goes from 15 to 0.

The written rule is clear. A stated value changed to a different value, so file it. Any engineer would implement that, the agent does it, and the output looks entirely defensible.

Someone who knows Danish planning reads the same row differently: a municipality does not forbid all building on a plot. A built percentage of zero is not a policy anyone sets, it is almost always a data entry error. So the honest label is escalate, not file.

The rule is not wrong so much as incomplete. Nobody thought to say the obvious thing.

Run the two orderings against that case and the difference is stark. Label second and the agent outputs "file", you review it, it follows the spec, so you write "file" down and the expert's judgment never enters the system. Label first and the case fails on the very first run. That failure is not a bug report. It is the first time anyone noticed the requirement was missing a sentence.

That is what a first eval run is actually for. Not proving the agent works. Producing the list of every place where what got built and what the business meant have quietly drifted apart, while that list is still cheap to act on.

The Rules It Falls Through

The agent decides most cases with plain code, before any model is involved. Four rules, applied in order, first match wins. They are worth stating once because the rest of this series refers back to them.

First match wins, and the order is the design.

• R1: can the record be trusted? A physically impossible value, like a height of 0 metres or height and storeys swapped between versions. Escalate.
• R2: did a stated value change? A real limit moved from one number to another. File it.
• R3: was something added? A limit now exists where the field was previously blank. File it.
• R4: was something removed? A figure a reader would have quoted is now gone. Escalate, because the register cannot say why.
• Nothing matched? A new version exists but no watched field moved, so something changed somewhere this rule set does not look. Escalating is the honest answer; claiming "nothing changed" would be a guess.

R1 runs first deliberately. Nothing derived from a record you cannot trust should be filed as though it were fact.

The Dataset Itself

Mine is 34 real cases pulled from the register, each labelled by hand before the agent existed. Not generated, not sampled from any output, because there was no output to sample.

Here are four of them, exactly as they sit in the file. The cells with an arrow expand.

id	label	rule	origin	changed_fields	reason
ZL-013	escalate	R1	hand-labelled	{ bebygpct, anvendelsegenerel }▸Show full value	Built percentage drops from 15 to …▸Show full value
ZL-024	file	R3	rule-derived	{ bebygpct }▸Show full value	bebygpct now carries a value where…▸Show full value
ZL-030	escalate	R4	hand-labelled	{ maxbygnhjd }▸Show full value	A limit that was previously stated…▸Show full value
ZL-002	escalate	R1	hand-labelled	{ maxbygnhjd, maxetager }▸Show full value	Height and storeys are swapped bet…▸Show full value

Every column is there for a reason, and the reasons are worth walking through.

id is a stable handle. Once a case has a name, it can be cited in a commit message, a bug report, or an argument about whether something regressed. Cases that are only ever "the one with the zero" cannot be discussed precisely.

label is the answer: what the system should do. It is the only column the scoring actually compares against, and it is deliberately a small vocabulary, file or escalate, rather than free text. A label you cannot check mechanically is not a label.

rule records which rule the label is expected to come from. It is what makes a failure diagnosable: if a case labelled R1 comes back as a file, you know immediately which piece of logic to look at, rather than reading the whole engine.

origin records where the case came from. Most of mine are hand-labelled, and six are rule-derived, generated from the rule set rather than labelled by a person. Those six are honest cases, but they are a weaker kind of evidence, and the whole argument of this article collapses if the two get quietly mixed. The field is what keeps them separable.

changed_fields is the input, reduced to what actually moved between the two versions. Storing the diff rather than only the two full records means a case reads at a glance and reproduces exactly.

reason is prose, written by whoever labelled it, and it is the field that saves you six months later. A label alone tells you what was decided. The reason tells you why, which is the only thing that lets someone disagree with it on purpose rather than by accident.

The labelling also does not stop after the first batch. Once the agent runs, it produces real traces, and those go back to the same domain expert, who says what the right answer was. Each answer becomes a new case, so the dataset grows on real traffic rather than on anyone's imagination of it. In this project that promotion is a manual step on purpose. If it happened automatically, the dataset would drift back into being a record of what the agent did, which is the exact failure this article is about.

Two properties make a dataset worth having. It has to be able to say no: if every case in it is one the agent passes, it is not measuring anything. And labelling has to be uncomfortable, because doing it without a plausible answer in front of you is slow and hard, and that difficulty is the signal. If a case is easy to label, it was probably not testing much.

Thirty-four was enough. It is a floor to grow from, not a target to hit before starting.

The Scorers That Read the Output

A label says what the agent should decide. It does not say whether the answer it wrote is any good. That is a separate job, and it belongs to the scorers: small functions that read one output and return a pass or a fail.

There are four, and every one of them is plain code. No model grades another model anywhere in this system.

• citation_fidelity asks whether the link to the source document appears in the output, verbatim. This is the one that enforces the system's single rule, that every autonomous decision traces to the source that justified it.
• no_invented_numbers asks whether every numeral in the output traces back to a value that was actually in the record. A number the model made up is the failure mode that is hardest to spot by reading.
• states_both_values asks whether the output gives the before and the after when it reports a change. "The height limit changed" is not usable; "8.5 m became 10 m" is.
• did_not_read_the_document asks whether the model stayed inside its boundary. At this phase the agent works from the register only, and this catches an output that talks as though it had opened the PDF.

The reflex, when building these, is to hand the output to another LLM and ask how well it matches. Sometimes that is the only option. Often it is not, and reaching for it costs you the thing you were trying to measure: when the score drops, you cannot tell whether the agent got worse or the judge did. Where a check can be a plain fact, make it a fact. Citation fidelity is not a matter of degree. The link is in the output or it is not, and that is a substring test, not an opinion.

Then You Run Them, and Write Down the First Number

With cases labelled and scorers written, you run the suite and record what comes back. That number is your baseline: the first score, before you touch anything. Take it before you start tuning, because once you have changed things, "what it was like before" becomes something you remember rather than something you wrote down.

Mine:

Four metrics near the ceiling and one at 70%. The one at 70% is the one the system's core rule depends on.

The system has one rule above all others: every decision it makes on its own has to trace back to the source that justified it. Citation fidelity is the metric that enforces that rule, and it came back at 70%. The agent reaches its decision, then describes its source in prose instead of citing it, writing "as documented in the plan PDF" rather than including the link. Eight cases fail, seven of them on every run, so it is systematic rather than noise.

That is the number to beat. Everything I do to this system from here gets measured against it, and without it I would be guessing about whether a change helped. If you are not measuring, you are not improving, you are just editing.

Label first, and your dataset can disagree with you. Label second, and all it can do is agree. Mine disagreed with me eight times on the first run, which is exactly what I built it to do.

Coming Up in This Series

Next: the human review queue. The agent already escalates correctly, which sounds like a finished feature and is not. An escalation that pauses and can never be answered is an open loop wearing a control's clothing. Building the pause is the easy half. Closing it, resuming the paused run, and turning the human's answer into a new labelled case is where human-in-the-loop stops being a diagram and starts being a system.

---

The code, the dataset, the scorers, and the baseline are public, including the 70%: github.com/bhanuchaddha/Tilsynsagent.

I build and evaluate agent systems like this one. If you are trying to get an agent past the demo stage, my repo and my notes are all public, and my inbox is open: get in touch.

This series is written as each phase lands. If the concepts behind it are what you are after rather than the build log, the Agents in Production series covers them properly, including evaluation and the LLM-as-judge pitfalls this chapter only touches on.