How I built a training dataset to fine-tune an LLM judge for my AI agent

TL;DR

An LLM judge for an AI agent reads one recorded decision and grades it. I built 889 graded decisions from real runs of Aurora, my trading research agent, to fine-tune one. An AI checker changed 12 of its own 29 grades when I reran it, so I checked the labels with blind review instead, and 71 of them changed.

Training comes next: Qwen 3.5 9B with LoRA on one rented GPU, scored against hand-checked runs. The judge is step one of an agent that builds profitable trading strategies.

Training rows

889graded agent decisions, three critics

AI checker

12 of 29grades it changed on a rerun

Grades fixed

71by blind review and reconciliation

Tokens scored

0.23%223,685 of 96,128,264

Why my trading agent needs a judge

I'm building Aurora into an AI agent that builds profitable trading strategies. Profitable means beating the S&P 500 on downside-risk-adjusted return (the Sortino ratio) going forward.

Language models are trained to sound right, and a strategy that sounds right can still lose money. So the plan has two models. The first is a critic that grades the agent's work. The second is my own model, trained with reinforcement learning, and the critic decides which of its attempts count.

What an LLM judge for an agent grades

A request like "which AI stocks grew revenue four quarters straight?" has a right answer and a right path to it. The critic checks both, based only on whether the work was correct and honest. It never sees how a strategy performed, so a lucky result can't cover for bad work.

GOOD means the work was done right. REPAIR means it has a gap I can fix. BAD means something critical is wrong, such as an invented number, a false claim that something was verified, or an answer its own run contradicts.

1. Plan critic

   It grades the planner's plan before anything runs.

2. Executor critic

   It grades one executor step, which is a tool call or the final answer.

3. Whole-run critic

   It grades the finished run end to end.

Step 1: Turn real runs into self-contained packets

I used real recorded Aurora runs: what users actually asked, and every plan, tool call, result and answer the agent produced. Real runs carry the mistakes my agent actually makes.

I split them into training, validation and test sets by source family, so near-copies of one run never land on both sides. The test set is 300 source families, sealed until the end.

Each decision becomes one packet: the user's request, the agent's full instructions, every tool it called, what came back, and the answer being graded. The critic grades a packet in one call with no lookups, so everything a grade depends on has to be inside it.

My first packets left out the agent's instructions, and 127 grades cited rules the critic couldn't see, such as "retry at most once." Adding them made the data 2.7 times larger.

Step 2: Check the answer key before you train on it

My first grades came from Codex and Claude Code sessions, so I needed a way to check them. I built 29 cases where I already knew the right grade and tested AI checkers on them, for about $6.35 in API spend.

13 decision models, which are built to answer quick multiple-choice questions, mostly answered GOOD to everything. The best got 19 of 29. The mistakes hide across a long run: in the case above, the fees and the claim that the strategy never traded sit 771 lines apart.

DeepSeek V4.1 Flash did best, with 21 of 29. Then I ran it three times on the same cases, and it changed its grade on 12 of them. A checker that changes its mind that often can't tell you which grades are wrong.

How blind review checked the key

Step 1 of 5

Review blind

A fresh Claude Code subagent reads the raw run and never sees the saved grade.

What you should see

A grade, a reason and exact quotes with line numbers.

Step 2 of 5

Lock the verdicts

Every review is frozen by hash before anything is revealed.

What you should see

No review can be quietly edited to match the old grade.

Step 3 of 5

Reveal and compare

Only then does the saved grade come out.

What you should see

Agreements stand, and every disagreement moves on.

Step 4 of 5

Reconcile

A fresh reviewer checks every factual claim on both sides against the source and applies the written rules.

What you should see

A final grade with each claim marked verified or unsupported.

Step 5 of 5

Send rule questions to me

When the written rules don't cover a case, the question comes to me.

What you should see

I made 19 of these calls.

What the review changed

In the end, 71 training grades changed. Whole-run labels were mostly too lenient: 15 of the 16 wrong REPAIR labels should have been BAD. Plan and step labels leaned the other way, and 27 of them went up to GOOD.

On the 46 hand-checked validation runs, 8 saved grades were wrong (17%), all too lenient.

The blind reviewers were overruled in 43 of 117 reconciliations (37%), so one fresh review alone would have left errors in place too.

The final training set

Validation is separate: 46 hand-checked whole-run cases (13 GOOD, 13 REPAIR, 20 BAD) and 50 adjudicated plan cases (39 GOOD, 11 REPAIR, 0 BAD).

How I'm training the critic, in five steps

Step 1 of 5

Turn each example into tokens

A token is a chunk of a word, the piece a model reads in one bite. Each example glues three parts together: the grading rules, the full agent run with the agent's prompt, and the right answer. The answer lists the evidence first, then the reason, then the grade, so the model finds the proof before it judges.

What you should see

The example shown is 135,802 tokens, and its answer is about 280.

Step 2 of 5

Decide what gets graded

The model reads the whole run, but the loss, the training score it learns to lower, counts only the answer it writes.

What you should see

223,685 of 96,128,264 tokens are scored: 0.23%, fewer than 1 in 400.

Step 3 of 5

Make the runs fit

Qwen reads 262,144 tokens at once. I leave the 10 longer examples out instead of truncating them, because truncation can cut the evidence a grade depends on. Qwen can stretch to about 1,010,000 tokens with YaRN, which stretches how it tracks word positions, and I'll switch that on only when the finished critic grades a very long run.

What you should see

879 of 889 examples fit; the longest is 768,722 tokens.

Step 4 of 5

Train with LoRA

LoRA trains a small set of extra numbers on top of the model and leaves all 9 billion original numbers frozen. It is far cheaper, and the whole job fits on one GPU.

What you should see

I measured memory and speed on one H200 rented on RunPod at $5.29 an hour. The full run, all 879 examples twice, runs on one B200 at $7.99 an hour. I estimate $110 to $150, with a hard stop at $200.

Step 5 of 5

Race it against two baselines

My critic, base Qwen with no training, and DeepSeek V4.1 Flash all grade the same 46 hand-checked runs, which the critic never trains on. I save a copy after each round of training and keep the one that does best.

What you should see

Mine wins only if it catches more BAD runs than both without raising more false alarms on GOOD ones.

This part is planned and has not run yet. The base model is Qwen 3.5 9B. I picked it over Liquid's LFM 2.5, whose 128,000-token window can't fit 138 of my 286 whole-run examples, and which got cut off mid-answer on 12 of 30 plan grades in a small test.

What the race leaves open

BAD runs come first, because a bad run that slips through is the one that would let a cheating strategy earn a reward.

After the race, the sealed test set of 300 source families gets opened once, at the end, so I can't keep tweaking the critic until it passes.

Two gaps remain. The executor critic has no validation set yet, and plan validation has 0 BAD cases, so I can't measure how many BAD plans the critic catches. The fix is to label new validation cases from the reserved families with the same blind review.

What comes after the critic

Next I train my own model with reinforcement learning. It builds strategies over and over, and each one gets a Sortino score from real live or paper results, or from a walk-forward test on periods it never saw when it has none.

Before any score counts, the critic checks how the model got there. If it didn't do what the user asked, or its numbers aren't real, the score is zero.

Today, about 1 in 3 strategies my agent builds beats the S&P 500 going forward: 170 of 521 paper portfolios with at least 30 days of forward history (32.6%). That is the bar.

Watch the series

Questions

1. How many examples do you need to fine-tune an LLM judge?

   I can only speak to my own set: 889 training rows across three critics, plus 96 validation cases. Training hasn't run yet, so I don't know yet whether that is enough.

2. Can an AI model check my judge's labels?

   Measure it first on cases where you know the answer, and run it more than once. My best checker got 21 of 29 right and changed 12 of 29 grades on a rerun.

3. Should judge training data be synthetic or real?

   I used real recorded runs. The defects that matter most to me, like a claim the run's own tool results contradict, come from real tool output.

4. How will you evaluate the judge?

   On hand-checked validation cases, never the old labels. I'll measure how many BAD runs it catches, how many GOOD runs it wrongly flags, the full confusion matrix of grades, and whether every quote it cites exists in the case. Then the sealed test set, once.

How I measured

Continue exploring