Your AI trading bot will fail because it's optimizing the wrong thing. Here's how to fix it.

You built the agent. You gave it tools. You hooked up the memory. You ran it overnight on five years of historical data and it produced a 140% annual return.

You deploy it with $25,000 on Monday. By Friday you've lost 30%.

This is not a bug in your code. This is overfitting. Your agent didn't find a durable market edge. It memorized the historical data and learned to exploit noise that doesn't exist in live markets. The AI succeeded at the goal you gave it. The goal was wrong.

Evaluation is the part of agent development that nobody talks about because most people building AI demos have never run a system long enough to see it fail. This article covers the engineering that keeps it from happening: traces, LLM judges, and a feedback loop that makes agents actually improve over time.

Module 5, Lesson 1: Evaluation and Traces. How do you know your agent is doing what you actually want?

The Problem

The optimization trap: when the agent succeeds at the wrong thing.

Every machine learning practitioner knows about overfitting. You train a model on historical data. It learns the data perfectly, including all the noise and anomalies that are specific to that dataset. When you expose it to new data, it falls apart because what it learned wasn't a real pattern.

AI agents have the same problem, and it's harder to catch. When you tell an agent to "build a trading strategy with the highest possible backtest return," it will do exactly that. It will explore every combination of indicators, time windows, and position sizes until it finds something that maximizes the metric you asked for.

The result looks impressive. 126% return in 2024. You show it to your friends. You write about it on Twitter. You deploy it with $25,000 on Monday.

By Friday you've lost 30%. The 2022 bear market destroyed it completely — either because the agent never tested 2022, or because the evaluation criteria didn't penalize drawdown, so the agent ignored it.

The optimization trap is an evaluation failure, not a model failure. The model did what you asked. You asked for the wrong thing.

The fix is not a better model. It's a better evaluator: one that grades the agent on what actually matters, penalizes single-year outlier returns, requires multi-regime evidence, and gets stricter every round.

I know this because I ran the experiment. Five rounds of hill climbing on a live trading agent. $676 spent. The first round scored 71 and produced an Iron Condor with a 54% average annual return. By Round 5, the score had dropped to 27 and the agent was recommending long directional options with a -6.3% average and a 92% drawdown in 2022. The evaluator caused every step of the decline. The full story is in the war story section below. Everything between here and there — traces, LLM judges, the feedback loop — is the anatomy of how it happened and how to stop it.

Observability

The flight recorder: what a trace actually is.

When a traditional app crashes, you read the stack trace. When an agent fails on iteration 7 of a 12-step ReAct loop (Reasoning + Acting: think, call a tool, observe the result, repeat), you're guessing — unless you have a trace.

A trace is a structured log of every step in the agent's execution. Think of it as a flight recorder. Every input the model saw, every decision it made, every tool it called, every result it got back, every token it spent, every millisecond it waited. If something goes wrong at 3 AM, you can reconstruct exactly what happened.

Here's what a real NexusTrade agent trace looks like. Each iteration is a complete record of one loop through the ReAct cycle:

screen_stocks({
  filters: [
    "price_above_200sma",
    "vix_below_20"
  ],
  universe: "SPY,QQQ,IWM"
})

→ { regime: "bull",
    qualifying: ["SPY","QQQ"],
    vix: 16.2,
    spx_vs_200sma: "+2.4%" }

fetch_portfolios({})

→ [{ id: "p-8f3a...",
     name: "Public Portfolio",
     deployed: true,
     value: 25000,
     strategies: [] }]

backtest_portfolio({
  chat_portfolio_id: "cp-9d1b...",
  start_date: "2022-01-01",
  end_date: "2025-12-31",
  baseline_symbol: "SPY"
})

→ { annualReturn: "54.3%",
    maxDrawdown: "59%",
    sharpe: 1.12,
    beatBaseline: true }

analyze_portfolios({
  portfolio_ids: [
    "cp-9d1b...",
    "cp-2c4f...",
    "cp-7a8e..."
  ],
  baseline_asset: "SPY"
})

→ strongest: "cp-9d1b..."
  (Iron Condor SPY/QQQ)
  avg 54.3% across 4 regimes

edit_portfolio({
  operations: [{
    type: "addStrategies",
    portfolioId: "p-8f3a...",
    strategies: ["...yaml..."],
    strategyNames: [
      "Iron Condor SPY/QQQ"
    ]
  }]
})

"Deployed Always-On Iron Condors
on SPY/QQQ. Avg 54.3% annual
across 4 regimes (2022-2025).
Max drawdown 59%. Strategy is
live in Public Portfolio."

Status: completed
Iterations: 12 / 20
Total cost: $0.31

Each iteration is a complete record: what the model thought, what it called, what came back, and what it cost. 12 iterations, $0.31 total. The trace is what makes debugging possible.

Without traces, a failed agent run is a black box. You see the final answer (or the error), and you guess at what went wrong. With traces, you can pinpoint the exact iteration where the model made a bad assumption, called the wrong tool, or misread a result.

Evaluation

How to grade an agent: algorithmic metrics and LLM judges.

Not all evaluation is the same. Some things you can measure with code. Some things you need a second AI to grade.

For trading agents specifically, overfitting is the failure mode that pure algorithmic metrics miss. A high backtest return is an objective number. But whether that return is trustworthy — whether it comes from a durable edge or from memorized noise — requires judgment. That's where the LLM judge comes in.

Here's the actual system prompt for the NexusTrade Agent Run Evaluator:

The hard caps are the anti-overfitting mechanism. An agent that found 126% returns in 2024 but only tested one year cannot score above 4, because single-year outlier performance is exactly what overfitting looks like. The evaluator enforces multi-year evidence as a precondition for a high score.

One caveat embedded in the rubric above: the trace shows the agent calling a 59% max drawdown "acceptable" despite a 50% threshold. That's a deliberate judgment in the NexusTrade context — but it illustrates exactly where rubrics fail silently.

Here's what the evaluator looks like in the course exercises, running against a real Aurora agent run:

The evaluator reads the full agent trace and returns structured JSON: dimension scores, an overall grade, a deployment verdict, and a nextIteration note that seeds the next run.

And it returns structured JSON. Here's what Round 1 of the hill climbing experiment produced:

{
  "summary": "Deployed a robust Iron Condor across 4 regimes.
             Consistent positive years including 2022 bear.
             Return profile (54% avg) won't reach 100% goal.",
  "deployedStrategy": "Always-On Iron Condors (SPY/QQQ)",
  "deployedStrategyAvgReturn": "+54.34% avg (2022: +31.2%, 2024: +72.1%)",
  "deploymentVerdict": "iterate_first",
  "scores": {
    "deployedStrategyFitness": 7,
    "evidenceStrength": 7,
    "explorationCoverage": 6,
    "riskRealism": 5
  },
  "overallScore": 71,
  "verdict": "good",
  "failures": ["59% max drawdown exceeds safe threshold",
              "54% avg won't reach 100% annual goal"],
  "nextIteration": "Push for higher return while maintaining the 2022 floor."
}

The nextIteration field is what makes the loop work. It becomes the seed for the next agent run: here's what you did well, here's what was wrong, here's what to focus on. The evaluator writes the coach's notes.

You call it from anywhere via MCP:

run_agent_run_evaluator({
  agent_id: "69d49c51d06eee7b51cf5f68",
  model: "google/gemini-3-pro-preview"
})

// Returns structured JSON:
// scores, verdict, nextIteration, deploymentVerdict
// Inject nextIteration into the next run's context.

Going Deeper

Beyond hill climbing: four approaches to agent optimization.

Once you have traces and an evaluator, the natural next step is to close the loop: run the agent, score the output, use that score to improve the next run. The simplest version of this is hill climbing — run, grade, seed the next run with the feedback. Cheap, fast, and often enough for early iteration.

But hill climbing is a local search. It follows the gradient of whatever metric you give it. Point it at the wrong objective and it will confidently optimize you into a cliff. The research community has gone considerably further. Here's the landscape:

The common thread: all four require traces, all four require a scoring function, and all four fail if that scoring function points at the wrong objective. But they're not interchangeable. Here's why you don't just default to NSGA-II:

For most teams: start with hill climbing. Run it until it stalls. If you hit a genuine multi-objective tradeoff, consider NSGA-II — but only with automated infrastructure and a clear budget. The rubric is still load-bearing in all four cases. Here's what it looks like when the rubric fails.

Real-World Evidence

I ran this loop five times. The first round was still the best.

Production Architecture

The feedback loop: how evaluation connects to memory.

Evaluation doesn't mean much if the agent can't learn from it. The feedback loop is what turns a one-time grade into compounding improvement.

The full pipeline has five steps. Each one feeds the next:

agent.run({
  task: "Build options strategy",
  context: injected_memory,
  maxIterations: 20
})
→ agent_id: "69d49c..."
→ status: "completed"
→ iterations: 12

buildGradingInput({
  initialPrompt,
  plan,
  currentState.thought,
  currentState.finalAnswer,
  status,
  totalIterations,
  maxIterations
})

run_agent_run_evaluator({
  agent_id: "69d49c...",
})
→ overallScore: 71
→ verdict: "good"
→ nextIteration: "Push
   for higher return while
   maintaining 2022 floor."

AgentSummary.create({
  overallScore: 71,
  verdict: "good",
  proceduralLessons: [
    "Iron Condors survive 2022",
    "54% avg needs return boost"
  ],
  nextIteration: "Push for
    higher return..."
})

"Past run score: 71/100.
Iron Condors on SPY/QQQ
survived 2022 bear with
+31.2%. Max drawdown 59%
is too high. Next priority:
enhance return while
holding the 2022 floor."

↺ Loop restarts.

Run → Trace → Judge → Memory → Improve → repeat. Each loop produces a labeled training example: what the agent did, how it scored, and what to do differently next time.

What makes Step 5 concrete: the injected memory isn't an abstract "past context" flag. It's a literal block of text prepended to the next agent's system prompt — the AgentSummary record retrieved from MongoDB and formatted as natural language instructions. Here's what that block looks like at the start of Round 2:

## Previous Run Context
Score: 71/100 · Verdict: good

Lessons learned:
- Iron Condors on SPY/QQQ survived the 2022 bear market
- Multi-year average 54.3% — consistent across all regimes
- Max drawdown 59% exceeded the 50% threshold

Next iteration focus:
  Enhance return profile while holding the 2022 floor.
  Do not abandon the Iron Condor structure.

---
## Your Task
Build an options strategy for a $25,000 account...

The evaluator doesn't just grade — it writes the next run's starting conditions. That's the loop. And as the war story above shows, get the rubric wrong and the injected note corrupts every run that follows. A demo runs once. A system runs, gets graded, and improves — or degrades, depending on what you told it to optimize for.

Connect

Expose your agent to the world: the NexusTrade MCP server.

Everything in this article — traces, LLM judges, feedback loops — is built on Aurora's infrastructure. The same evaluator that graded the hill climbing experiment. The same backtester the agent used to test the Iron Condor across 2021-2024. Those tools are accessible via MCP, which means you can call them from Claude Desktop or Cursor without touching the NexusTrade UI.

The MCP server exposes the full tool surface: screener, portfolio system, backtester, indicator engine, and the evaluator itself. Here's the config:

{
  "mcpServers": {
    "nexustrade": {
      "url": "https://nexustrade.io/api/mcp",
      "headers": {
        "Authorization": "Bearer <your-api-key>"
      }
    }
  }
}

Then from Claude Desktop or Cursor, you can ask: "Use NexusTrade to screen for stocks with RSI below 40 trading above their 200-day moving average." Claude calls screen_stocks on the NexusTrade MCP server. The server returns live results from the same screener Aurora uses internally. No copy-paste. No API docs. One tool implementation, available from any MCP-compatible client.

You can also run the evaluator directly. run_agent_run_evaluator(agent_id) grades any completed agent run from any client. The evaluation loop becomes accessible from wherever you're working.

Check Your Understanding

Pop quiz.

The End

You've built the complete agent. This is where most people stop. It's where you start.

Five articles. Five modules. You started with a leaked source file and ended with a production evaluation loop — router, ReAct engine, long-term memory, rubric design, feedback loop. The only thing left is to run it.

The $676 hill climbing experiment is a better argument for evaluation than anything else in this article. The agent wasn't broken. The rubric was. Build the right rubric and the loop compounds toward something real. Build the wrong one and a perfectly obedient agent will follow it off a cliff, five rounds in a row, at $135 per run.

Before the capstone, here's the full series in five minutes. Every concept, every module, from the leaked source file to the production feedback loop.

The full course recap. Five modules in five minutes. Watch this before running the capstone.

Module 6 is where the capstone lives. The full pipeline, embedded directly in the lesson — no separate window, no instructions to follow in another tab. Aurora runs inline: screens the market, validates with news, builds a watchlist, then wires up a scheduled agent to manage it every week.

The capstone runs Aurora directly in the lesson page. Three steps: technical screen, watchlist, weekly monitor agent. The progress bar tracks where the agent is. You watch every tool call happen in real time.