← All Articles

AI agent system design · production case study

How to Design an AI Trading Bot: Junior to Senior

Aurora is the NexusTrade agent: a router, a durable ReAct loop, and a host that owns tools, approvals, and recovery. Use the real system as the junior-to-senior case study.

By Austin StarksJunior → mid-level → senior design pathAI trading bot case studyUpdated August 31, 2026
Open the final system map ↓
Interactive final design

The architecture of an AI agent

Follow the real boundary between request routing, ReAct decisions, and the executors that turn JSON into work.

Interactive AI trading agent architecture Router V5 classifies a request as Call Tool, Ask Clarity, Create Agent, or Continue Agent. Call Tool selects a versioned catalog prompt and produces a persisted response. Ask Clarity stores questions and waits. Create Agent and Continue Agent enter durable agent state before ReAct returns actions, a command, or a final answer. The stepper validates that decision and selects exactly one branch. Semi-automated mode can pause actions for approval. Traces observe every branch. ROUTING, PROMPT SELECTION, AND AGENT DECISION AGENT STATE, EXECUTOR, APPROVALS, AND EVALUATION ACTION AND COMMAND EXECUTION CREATE / CONTINUE AGENT TRACE SEMI-AUTOMATED ONLY ACTIONS[] LARGE RESULT ONLY COMMAND FINAL ANSWER ENTRY Trigger user · cron · API · event CLASSIFY + PLAN Router call tool · clarify · agent ROUTE Route result call tool · clarify create · continue VERSIONED Prompt registry prompt · model · schema ASSEMBLY ReAct prompt goal · plan · tools REACT ReAct model JSON decision DURABLE Agent state goal · plan · status HYDRATED Turn context recent turns · tool refs VALIDATE + DISPATCH Agent stepper actions · command · answer SEMI-AUTOMATED Pending approval approve · reject OBSERVE Traces version · cost · latency ACTIONS[] Action batch concurrent · ordered commit CAPABILITIES Tool registry schema · handler · scope COMMAND Command executor askUser · spawn · wait · stop LARGE OUTPUTS Private result blobs selected output · summary · ref COMMIT Observation result · error · answer
Guided path

Route the request

Router V5 reads the hydrated conversation and decides whether to run one catalog prompt, ask for clarification, create a durable agent, or continue an existing agent.

Design questions
  • Which requests actually need a durable agent?
  • What context must the router see before choosing a route?

The 60-second answer

An AI trading bot is a controller around a language model

Aurora's Router V5 chooses Call Tool, Ask Clarity, Create Agent, or Continue Agent. Call Tool runs one catalog prompt and commits. A durable agent returns schema-valid ReAct JSON. The stepper dispatches actions, commands, approval waits, or completion. Every observation is persisted before the next turn. Independent actions may run concurrently and still commit in order.

RouterCall Tool or durable ReActStepperAction, command, or answerPersist and repeat

Interview mode

Build the answer in 45 minutes

Scope one task and draw one model turn. Add context, authority, ownership, and financial reconciliation as the workload demands them. Test one failure at each boundary and explain the trade-off.

One interview clock

Move from useful to recoverable

The diagrams below are the answer. Use this clock to scope the product, draw the controller, introduce long-running work, and reserve time for failures and trade-offs.

ScopeAssistant or autonomous run, research or execution, trigger, and authority.
Prompt contractSystem rules, layered context, JSON schema, and a typed proposal.
Draw one turnAssemble context, decide in JSON, validate, call a tool, observe.
Design contextRoute versus durable run, turn context, retrieval, and token bounds.
Gate capabilitiesTool schemas, automation mode, approvals, and user scope.
Scale and evaluateQueues, durable waits, prompt evals, traces, cost, and recovery.
Incomplete answer

Draw an LLM between the user and an API

UserLLMToolAnswer

This skips prompt versioning, context selection, schemas, memory, permissions, invalid output, and the observation loop.

Interview-ready answer

Draw the complete agent turn

Versioned promptContextJSON decisionPolicyToolObservation

The model remains replaceable. Prompt, schema, memory, tool, and authority contracts define the agent product.

Interview trade-off = context → choice → reason → consequence → upgrade trigger

Junior interview · MVP

Design one correct AI trading-assistant loop

Design an assistant that answers a trading question, reads one user's portfolio, runs one paper backtest, and proposes a paper order for manual approval.

Junior interview scenario1 active user1 in-flight turn1 model request at a timeSeconds of latency acceptable
Functional requirements
  • Hold a back-and-forth conversation.
  • Read one authenticated portfolio.
  • Run one supported backtest command.
  • Return a typed recommendation or paper-order proposal.
Non-functional requirements
  • Reject malformed model output.
  • Keep portfolio access scoped to the current user.
  • Record the prompt, action, and result for debugging.
  • Require a person to approve every consequential action.
Constraints
  • One application process and one primary database.
  • No background scheduler or subagents.
  • Synchronous work is acceptable for the first demo.
  • Paper trading only.
Required failure handling
  • Reject invalid JSON and unknown tool names.
  • Deny access to a portfolio owned by another user.
  • Return a bounded error when the model or tool times out.
  • Never turn a recommendation into an unapproved order.

Junior · 1 of 4

Begin with an LLM conversation

A language model call starts with an array of messages. Append the assistant response and send the expanded array on the next request. That transcript creates the back-and-forth experience.

messages = [
  { role: "user", content: "Explain RSI" },
  { role: "assistant", content: "RSI measures recent momentum..." },
  { role: "user", content: "Now apply that to AAPL" }
]

response = model.generate({ messages })
User messageConversation historyLanguage modelAssistant message

This version can explain a trading concept. It has no current market data, portfolio state, backtesting engine, scheduler, or brokerage adapter. Controlled context and tools add those capabilities.

Junior · 2 of 4

Assemble the prompt in layers

Do not paste every available fact into one giant prompt. Build the model input from layers with different owners and lifetimes. Stable instructions come first; current goals, state, and retrieved evidence are added for this turn.

Assemble four layers: the versioned system contract, the task procedure, current turn state, and a small relevant evidence slice. Heavy tool results remain in durable storage and re-enter later turns as compact summaries or owner-scoped references.

REACT DECISION CONTRACT
Return JSON with "thought" and exactly one of:
  "actions"     independent product-tool calls
  "command"     agent-control work such as askUser or createSubagents
  "finalAnswer" the completed response

SIMPLIFIED BACKTEST TOOL PROCEDURE
Use "Backtest Portfolios" to launch a native backtest.
Pass the resolved portfolio ID and an explicit date window.
After launch, use the returned backtest ID with "Read Backtest".
Do not launch the same portfolio and window again.

Boundary 1 · The decision model chooses a tool

{
  "thought": "The portfolio exists and the requested window is explicit.",
  "actions": [
    {
      "tool": "Backtest Portfolios",
      "input": "Backtest PORTFOLIO_ID from 2021-01-01 through 2025-12-31."
    }
  ]
}

Boundary 2 · The tool prompt produces its schema

{
  "backtestConfigs": [
    {
      "portfolioId": "PORTFOLIO_ID",
      "startDate": "2021-01-01",
      "endDate": "2025-12-31"
    }
  ]
}

These are two separate model boundaries. The decision model returns an outer JSON decision, but each actions[].input value is plain text. The server appends that text as the user message for the selected tool prompt. The Backtest Portfolios prompt then returns the tool-specific backtestConfigs JSON shown above. Server code resolves the portfolio for the authenticated user, validates the dates and product constraints, launches the native backtest, and returns its backtest ID.

Putting escaped backtestConfigs inside actions[].input would only encode JSON as text for the second model to interpret. It would not provide typed arguments directly to the backtest implementation.

Host defenses

Reject blank inputs and unknown tools. Treat an object emitted as input as serialized text, never trusted arguments. Server code rejects invalid configurations, unauthorized portfolios, restricted jobs, and invalid dates before computation starts.

Junior · 3 of 4

Structured output turns a response into an action proposal

Give the decision model a catalog of supported tools. It selects a tool and writes a bounded text instruction for it. The server resolves the selected tool. Some tools handle a complete machine request directly; Backtest Portfolios invokes its own force-JSON prompt to produce the tool-specific schema.

{
  "thought": "I need one current quote and one portfolio read before sizing.",
  "actions": [
    { "tool": "Stock Screener", "input": "Read the current quote for SPY." },
    { "tool": "Read Portfolio", "input": "Read portfolio portfolio_123." }
  ]
}

That JSON is a proposal. The model has not executed a tool, granted itself authority, or written an observation. Host validation is the next step.

Junior · 4 of 4

The host validates the proposal before any tool runs

The host application owns parsing, validation, permissions, execution, and error handling. Reject unknown tools and invalid JSON. Persist a semi-automated proposal instead of running it.

const state = validateAgentState(modelJson)
const actions = getStateActions(state)

if (agent.automationMode === "semi-automated") {
  await persistPendingActionApproval(agent, state)
} else {
  const results = await executeActionBatch(agent, actions)
  await commitObservationsInActionOrder(agent, results)
}

This is the host-side contract, not model freedom. The stepper enforces exactly one decision branch and at most ten actions. Parallel actions are on by default and can be disabled with AGENT_MULTI_ACTIONS_ENABLED=false. Each named tool is resolved by the server before execution. Semi-automated runs persist the exact proposal and wait for approval.

Mid-level interview · product scale

Add routing, the ReAct executor, approvals, and durable runs

Evolve the assistant into a multi-user product with request routing, ReAct, typed tools, scheduled runs, asynchronous backtests, progress updates, and automation-mode approvals.

Illustrative interview load1,000 users50 concurrent turns at peakBacktests take 1 to 20 minutesSchedules burst on minute boundaries
Functional requirements
  • Classify a request before creating a durable agent.
  • Plan and execute multiple typed tool calls.
  • Pause for approval or clarification.
  • Schedule an agent and resume an asynchronous backtest.
  • Stream durable progress to the browser.
Non-functional requirements
  • Persist the request quickly, then let a worker claim it.
  • Commit agent state first. WebSockets are the fast path. The UI falls back to a 5-second status poll.
  • Recover a run after a web or worker restart.
  • Make product writes idempotent and define adapter-specific handling for brokerage uncertainty.
Constraints
  • Model, market-data, and brokerage APIs can time out.
  • Approvals may remain pending for hours.
  • Redis and WebSocket events may be missed.
  • Every resource lookup must enforce tenant ownership.
Required failure handling
  • Resume a run after the web process or agent worker restarts.
  • Release worker capacity while approval remains pending.
  • Adopt one durable result when a callback arrives twice.
  • Recover current state after a missed Redis or WebSocket event.

Mid-level · 1 of 5

Route the request before starting an agent

Many users and concurrent turns make a durable agent per message the wrong default. Classify the request first. NexusTrade gives Router V5 the hydrated conversation. The router chooses the smallest supported path and, when durable work is needed, authors the initial plan and title.

One request, four routing outcomes
User requestRouter
Call Toolone catalog prompt, then commitAsk claritypersist the question, then waitCreate agentpersist plan and initial stateContinue agentresearch follow-up, no new plan

Routing plays automatically when this figure enters view. Use the button to replay it.

type RouterDecision =
  | { action: "Call Tool"; toolName: CatalogPromptName; input: { text?: string } }
  | { action: "Create Agent"; input: { plan: string; title?: string } }
  | { action: "Ask Clarity"; input: { questions: string } }
  | { action: "Continue Agent"; input: {} }

The action set is closed. Catalog names go in toolName, never in action. Ask Clarity questions is one markdown string, not an array. The schema asks for a Create Agent title; the runtime still accepts a plan if the title is missing. Call Tool is one-shot catalog fulfillment. Create Agent and Continue Agent hand the durable loop to ReAct and the stepper. Ask Clarity and Call Tool also return suggested next messages.

Mid-level · 2 of 5

The executor turns ReAct JSON into work

Each ReAct decision is forced to JSON and contains exactly one of actions, command, or finalAnswer. The stepper validates that contract, checks whether approval is required, and sends the decision to the matching execution path.

Hydrate conversationStructured ReAct decisionAgent stepperDispatch branchCommit observations

Independent actions can execute in parallel

Suppose an agent needs the user's portfolios, watchlists, and scheduled agents. None of these reads depends on another, so the model can request all three in one decision. Every action sees the same conversation snapshot and remains valid even if either sibling fails.

One JSON decisionFetch portfoliosFetch watchlistsList scheduled agentsOrdered observations
{
  "thought": "These product reads are independent.",
  "actions": [
    { "tool": "Fetch User Portfolios", "input": "Load my portfolios" },
    { "tool": "Fetch User Watchlists", "input": "Load my watchlists" },
    { "tool": "List Scheduled Agents", "input": "Load my schedules" }
  ]
}
Why the parallel batch remains deterministic

Every action reads the same conversation snapshot and receives a stable batch ID plus action index. The runtime waits for every sibling with Promise.allSettled, sorts results by action index, and commits observations in that original order. A handled tool failure becomes a typed error observation; an unexpected rejection remains visible and cannot silently erase successful siblings. Recovery reuses the stable execution IDs so completed actions are not mistaken for new work.

Independent actions
One ReAct iteration
Decision Concurrent tool execution Commit
Model actions[3]
Action 0 Fetch User Portfolios
Action 1 Fetch User Watchlists
Action 2 List Scheduled Agents
Observations 0 → 1 error → 2

Run the batch to see concurrent settlement, a typed tool failure, and deterministic commit order.

1 model decision 3 tools overlap 1 ordered commit
Dependent actions Two ReAct iterations
Decision 1 Observe, decide again, execute Commit
Model request portfolio ID write backtest instruction
Tool 1 Fetch User Portfolios
Tool 2 Backtest Portfolios
Observations portfolioId = result.portfolioId backtestJobId committed
2 model decisions decision 2 receives portfolioId 2 persisted steps
claim = Agent.claimNextAgent(workerId)
renewClaimEvery(claim, 30_seconds)

messages = hydrateConversationTail(agent)
prompt = buildReActPrompt({
  entitlements,
  originalRequest,
  attachedPortfolio,
  currentPlan,
  currentIteration,
  recentCommittedActions,
  toolAllowlist
})

decision = model.respond(prompt, forceJSON = true)
state = validateExactlyOneOf(decision, ["actions", "command", "finalAnswer"])

if (state.actions) {
  snapshot = messages
  results = settleAllConcurrently(
    state.actions.map((action, actionIndex) =>
      execute(action, snapshot, stableExecutionId(actionIndex))
    )
  )
  persistObservations(results.sortByActionIndex())
}
if (state.command) executeAgentCommand(state.command)
if (state.finalAnswer) completeAgent(state.finalAnswer)

publishAgentUpdateAfterCommit()

Backtest Portfolios, optimizers, sandboxes, and ingestion tools can yield with durable computation IDs. The stepper persists those IDs, moves to waiting_for_computation, and releases the worker slot. Subagent joins use waiting_for_subagents; clarification uses awaiting_user_input.

Interview invariant

A process crash may lose the current in-flight operation. It must preserve every completed observation and prevent duplicate committed side effects.

Mid-level · 3 of 5

Gate every proposed action before the tool runs

The model proposes an action. It does not grant itself authority. Aurora uses an automation mode, not an LLM risk classifier. Automated runs execute after product checks. Semi-automated runs persist the exact proposal and wait for a person.

Automation mode, then product authority The host owns the gate. Tool code rechecks ownership, paid-job limits, and product constraints before any side effect.
Agent proposal tool name + text instruction + user scope
automationMode automated or semi-automated, set on the agent
Product authority user ownership, paid-job gates, argument validation
AUTOMATED Run the tool after product checks. No human pause.
SEMI-AUTOMATED Persist the batch on pending_plan_approval or pending_action_approval, then wait.
PRODUCT REJECT Wrong owner, unpaid job, or invalid config. Persist the error observation.
Product authority gate Revalidate user, portfolio, account, and tool constraints inside the handler. The model cannot skip this.
User approval Approval resumes the stored proposal. The model does not recreate it.
Rejected proposal No side effect runs. The next ReAct turn sees the error.
Authorized proposalPersist order intentDedicated executorReconcile broker state

Agent-created orders are staged proposals in product state. The model has no brokerage credentials. A dedicated execution path owns submission and reconciliation.

Mid-level · 4 of 5

The Router promotes the persisted chat Agent into a run

A NexusTrade conversation already has a persisted Agent document in chat status. Router V5 can answer or ask for clarification while that document remains the conversation container. When it chooses Create Agent, the server upgrades the same _id in place with the plan, title, execution settings, and a runnable status.

Persisted Agent: chatRouterCreate Agent: same _idpending_plan_approval or runningWorker claim
{
  "_id": "agent-id",
  "userId": "authenticated-user-id",
  "conversationId": "conversation-id",
  "title": "Portfolio Change Review",
  "initialPrompt": "Review my portfolio and summarize material changes.",
  "plan": "Inspect the portfolio, gather evidence, and summarize material changes.",
  "automationMode": "semi-automated",
  "origin": { "type": "manual" },
  "status": "pending_plan_approval",
  "maxIterations": 40,
  "currentIteration": 0,
  "runIterations": 0
}

These are representative fields after a semi-automated chat is promoted. _id remains the run identity. title and plan come from the Router. initialPrompt keeps the original request. Router and ReAct prompts are snapshotted in AgentPrompts. Chat Aurora uses Router V5. Strategy-triggered LaunchAgent still has a Planner V4 init path. Automated runs enter running instead of waiting for plan approval.

Mid-level · 5 of 5

Keep prior tool payloads out of the next decision

The current request determines which product data belongs in the next model call. This portfolio-creation request only needs its new specification, current plan, entitlements, and available tools. Full results from earlier research add no decision input.

A new portfolio request after a data-heavy conversation Each transformation below maps to a NexusTrade prompt or storage boundary.
Latest user request “Create a new equal-weight portfolio of AAPL, MSFT, NVDA, AMZN, and GOOGL. Rebalance it monthly.”
Unbounded conversation copy · over 400,000 characters
Latest portfolio-creation request
Full JSON for every previously fetched portfolio
Full backtest results, equity curves, trades, and history
Full watchlists and every saved symbol
Stock-price arrays and stock-screener table rows
Full article bodies and generated news summaries
Run Compute code, stdout, and step transcripts
1 · persist refs drop strategy trees + positions; keep references
2 · offload selected heavy results → private Tigris S3 object
3 · summarize counts + sample rows + owner-scoped object key
4 · cap model copy elide oldest message data above 400k chars
5 · assemble turn inject only current ReAct state
Bounded next-decision context
Cached system prompt and tool guidelines
Original portfolio-creation request
Subscription entitlements
Current plan and iteration
Recent committed tool actions and compact observations
Code-driven tool allowlist
Private Tigris result references available on demand
Dump everything into context Earlier research consumes the next decision’s budget

The model parses positions, historical trades, price arrays, and news that cannot change how this new portfolio is created.

NexusTrade context path The model receives the current goal and bounded state

Product and backtest data stay in their existing durable records. Selected oversized discovery, screening, and sandbox outputs move to private Tigris objects. The next turn receives compact observations and owner-scoped references instead of copying every payload again.

Mid-level · prompt operations

Version the prompt configuration and track schema compatibility

Once more than one person edits prompts, a save is not enough. Shared prompts need versioning. Promotion quality is a separate question from save.

A prompt is executable application configuration. Changing its system instructions, examples, model, JSON setting, or referenced schema can change agent behavior. NexusGenAI is the registry. A save snapshots the configuration and increments currentVersion. Restore is a version read, not a bakeoff. The referenced schema is a separate dependency, so compatibility belongs to the prompt version that uses it.

{
  "name": "Agent decision prompt",
  "currentVersion": 42,
  "model": "selected-model",
  "forceJSON": true,
  "schemaId": "agent-decision-schema-id",
  "prompt": "Return exactly one ReAct decision branch...",
  "examples": ["..."]
}
How you should evaluate before you promote This is the interview method and the bakeoff practice. It is not the save path. Save already created a version.
01 · frozen evidence Ground-truth cases Normal requests, edge cases, and deliberately broken outputs
02 · paired run Baseline vs candidate Same cases, model settings, tools, and evaluator version
03 · hard gates Deterministic checks JSON, schema, tool names, arguments, and allowlist rules
04 · calibrated judge Semantic evaluation Correct route, supported answer, task completion, and clarity
05 · unseen cases Validation and test Reject improvements that only memorize the training examples
06 · promotion decision Promote or keep Keep the candidate current, or restore the prior one
Invalid shape Fix the schema or tool contract
Wrong route or tool Fix instructions or examples
Unsafe proposal Fix policy and permission gates
Unsupported answer Fix retrieval or context assembly
Slow or expensive Fix model choice or context size

Select “Run evaluation method” to follow one bakeoff candidate.

Same casesSame evaluatorOne attributable changeHeld-out bakeoff

Calibrate a judge on reviewed good and broken examples before using its score. Compare versions on the same held-out cases, then change the component that owns the failure. Permission errors belong to product checks; stale evidence belongs to retrieval. Runtime code can restore a prior NexusGenAI version without changing the stepper.

Senior interview · fault-tolerant platform

Design for bursty work and ambiguous financial effects

Scale the product across stateless web instances and agent workers. Support parallel research, long computations, bounded costs, and multiple brokerage adapters while preserving recoverable state and user authority.

Illustrative interview load10,000 connected clients200 new turns per second during burstsThousands of market-open schedulesJobs range from seconds to an hour
Functional requirements
  • Partition runnable agents across worker processes.
  • Launch bounded subagents and join partial results.
  • Bound each run with maxIterations and a tree-level LLM cost ceiling (default $50 stop, $20 ops alert).
  • Reconcile every ambiguous financial submission.
Non-functional requirements
  • Zero loss of committed agent state.
  • Claimed execution writes are fenced by claim generation.
  • Bound iteration count, tree cost, and a 40-minute ownership generation.
  • Trace model, tool, status, and cost boundaries.
Constraints
  • Exactly-once execution is unavailable.
  • External providers impose independent rate limits.
  • Job runtimes and model costs have long tails.
  • Brokerage state remains external financial truth.
Required failure handling
  • Reject writes from an expired worker after ownership moves.
  • Recover a tool result committed outside the agent process.
  • Freeze retries when a broker submission has an unknown outcome.
  • Stop a tree at the cost ceiling and alert on queue backlog.

Senior · 1 of 2

Persist the workflow, then let workers claim one step

Agent {
  title, initialPrompt, lastUserRequest, plan, conversationId
  status, previousStatus, automationMode
  currentIteration, totalIterations, runIterations, maxIterations
  currentState: {
    thought, actionBatchId,
    actions[] | command | finalAnswer,
    commandExecution,
    pendingComputations[]
  }
  parentAgentId, childAgentIds[], subagentType
  claimedBy, claimedAt, claimToken, claimGeneration
  claimHeartbeatAt, claimExpiresAt, stateVersion
  origin, portfolioContext, costCeilingUsd, treeCostUsd
  updatedAt, lastProgressAt, completedAt, stoppedAt, error
}

Aurora stores the run as a durable agent document. A worker atomically claims a runnable agent, performs a bounded step, renews ownership while active, and releases the claim when the agent finishes or waits. Another worker can recover an abandoned run from the last committed observation.

100maximum active agent executions per worker process
500 msrunnable-agent polling interval
2 minrenewable lease with a 30 second heartbeat
40 minabsolute ceiling for one ownership generation
Durable execution and long-running workPersist identity before releasing capacity
Persisted agent RUNNING + context Atomic claim one worker owns step Bounded ReAct one decision branch Persist wait job ID or child IDs Yield free slot Backtest · optimizer · sandbox · subagents · approval completion writes an observation and returns status to RUNNING

Long work owns its compute lifecycle. The agent owns the goal, durable dependency, result, and next decision.

Senior · 2 of 2

Production scaling is queueing, isolation, and recovery

Pressure Design response Failure question
Many concurrent agents Cap each worker at 100 active executions, poll every 500 ms, and claim agents with renewable leases. Can two workers execute the same step?
Expensive prompts Use compact observations, a code-driven tool allowlist, and a tree-level cost ceiling. Can one user exhaust shared model capacity?
Slow tools Persist a job ID, enter a waiting state, release the worker, and wake from completion. What happens if completion arrives twice?
Parallel research Launch bounded subagents with narrower context and join their durable results. How do partial failures affect synthesis?
Rapid UI progress Commit canonical state first, then publish low-latency events to connected clients. Can a reconnect reconstruct missed updates?
External side effects Use provider-supported idempotency where available, dedicated adapters, and venue-specific reconciliation before retrying. Did a timeout occur before or after acceptance?

Observe every decision boundary

Each trace should identify the model, system prompt version, injected context, action schema, proposed arguments, permission result, tool observation, latency, token usage, cost, retries, and terminal state. Evaluators can grade task completion, unsupported claims, permission compliance, efficiency, and the final explanation.

Aurora also writes SystemAlert records for worker errors, stalled agents, queue backlog, and cost thresholds. Those alerts carry operational evidence for the responder; the user-facing agent trace carries the decision history. Both are required because a green HTTP route says nothing about a stuck claim or an unresolved order.

Agent traceDeterministic metricsLLM judge rubricFailure categoryPrompt, tool, or policy fix

A system design interview answer should close this loop. Reliability comes from replayable state and idempotent boundaries. Agent quality improves when traces produce specific changes to prompts, tools, routing, or policy.

Production walkthrough Production case study after the study ladder

Sequence 1: one recoverable ReAct decision

1Route + persistrequest, plan, origin, limits
2Claim + hydrateowner token, state, tools
3Decide in JSONactions, command, or answer
4Gate + executeapprove, run, or wait
5Commit + publishordered observations, client update

Each turn commits one recoverable state transition. Independent actions may run concurrently, but their observations retain stable identities and commit in action order before the next model decision.

Sequence 2: a backtest releases the agent worker
runningBacktest Portfolios returns IDRead Backtest sees processingpersist job + action identitywaiting_for_computation
worker released
wake adopts terminal resultrunning
next ReAct step

Play the lifecycle to see where durable state replaces a blocked worker.

Read Backtest yields when the returned backtest ID is still processing. The wait stores the computation type, backtest ID, message ID, and action execution identity. A wake check adopts the terminal result and transitions the agent back to running, even after a process restart.

Senior interview: walk the failure path

For each failure, identify the surviving record and explain how the system prevents a duplicate financial effect.

1 What happens when an agent worker crashes mid-step? Leases + fencing
Worker A owns generation 7Heartbeat stopsLease expiresWorker B claims generation 8

AnswerThe agent stores its owner, lease, generation, and last committed state. Worker B resumes from that state with generation 8. A late generation 7 write that goes through the execution fence is rejected. Tree-cost rollups and some HTTP handlers are unfenced by design.

2 What if a tool succeeds and the process dies before recording the observation? Idempotency
Persist action execution IDExecute toolCrash before commitAdopt prior result by ID

AnswerCreate the action identity before execution. At-least-once delivery is safe only when the tool can find or safely recreate its prior result and the agent can adopt the persisted observation.

3 How can a backtest run for minutes without holding an agent worker? Durable suspension
Start backtestPersist computation IDRelease worker slotCallback or pollResume agent

AnswerThe computation has its own job ID. The agent persists that ID, releases the worker, and lets the same waiting action adopt one terminal result.

4 What if an agent update is missed? Truth vs delivery
Persist agent state and messageWebSocket update is missed5-second status poll or reconnectLoad current agent and canonical conversation

AnswerAgent and chat records are canonical. WebSockets are the fast path. A status poll or reconnect reloads persisted state, and stateVersion prevents an older snapshot from replacing it.

5 A brokerage request times out. Is a retry safe? Ambiguous side effects
Adapter submits the approved orderBroker callTimeout before a final responseTreat outcome as unknown; resolve per adapter

AnswerA timeout leaves an unknown outcome, so a blind retry can duplicate a position. Alpaca and Public can carry stable client order identity. TradeStation returns its order ID in the response, so its unknown outcome requires venue-specific reconciliation.

Draw four trust boundaries

Language model Proposes JSON. Tool names, arguments, and claims are untrusted.
Schema
+ allowlist
Product services Authenticate users, scope resources, persist state, and run tools.
Risk
+ approval
Brokerage External financial truth. Submit through an adapter and reconcile.

Research ends at evidence or order intent. Crossing the second gate requires current product authority. A profitable backtest does not grant brokerage permission.

External content is data, never instructions

News, filings, web, screenersProvenance + content labelBounded contextSchema + product policy

This is a required design boundary for any research agent. Text retrieved from outside the product cannot change system prompts, tool allowlists, permissions, or approval state. Preserve its source, limit what enters context, and route every suggested action through the same schema and product-policy checks.

Estimate capacity from active work

Interview sizing shortcut active concurrency ≈ arrival rate × active step time

Five turns per second with four seconds of active work needs about 20 execution slots before headroom. A suspended backtest consumes no slot while it waits.

Watch the queue, not host count
  • oldest runnable agent age
  • active step duration
  • expired lease count
  • approval and computation backlog

Deployment topology

Browser, SDK, MCP, scheduleStateless web APIPrompt and model gatewayMongoDB agent stateAgent workersProduct tools + computeApproval + broker adapter

Redis handles low-latency delivery. MongoDB stores recoverable agent and product state. Compute workers and broker adapters keep long work and financial credentials outside model execution.

The senior-level answer: persist the workflow, assume at-least-once tools, release workers during waits, and keep financial authority outside the model.

Practical implementation paths

Choose the shell; keep authority in the product

Claude Code, Cursor, and Codex demonstrate the shell: instructions, tools, context, and approvals. Aurora is the NexusTrade implementation: portfolio context, durable research, staged orders, and financial authority outside the model. Any shell still depends on server-side validation.

The model proposes the next move. The surrounding system controls context, capabilities, permissions, persistence, recovery, cost, and financial side effects. That controller is the actual AI trading bot.

Build the strategy this article describes

Create a free account to backtest ideas against market history, inspect the risk, and deploy to paper or live markets when you're ready.

or

Free to browse. No credit card required.

Discussion

Sign in or create a free account to join the discussion.

No comments yet.