TL;DR
Published 2025-02-07. The article reports that o3-mini's buy rule matched the request (Pass), while Claude 3.5 Sonnet's buy rule used 1 plus the rate of change instead of 1 standard deviation (Fail).
- Published
- 2025-02-07
- Published model rows
- 2
- Evidence
- Historical task evaluation
Follow this evaluation
Published results
Claude 3.5 Sonnet and o3-mini building a Bitcoin trading strategy: source evidence dated 2025-02-07.
- Exact model version
- OpenAI o3-mini
- Buy rule matched request
- Pass
- Generated buy condition (as displayed)
- (3 Day BTC Rate of Change > SMA(3 Day BTC Rate of Change) + StdDev(3 Day BTC Rate of Change)) or (Positions Percent Change of (BTC) ≤ -15)
The displayed values belong to the identified model versions and this published task, with their original metric units.
What did this task require the model to return?
Read the tested version, task and rubric together. Check failures alongside successful answers; compare costs only within the reported unit.
2025-02-07: Claude 3.5 Sonnet and o3-mini building a Bitcoin trading strategyKeep the study date and exact model ID beside the result. Do not average different tasks into one winner.
Test a strategy with Aurora
Keep the study date and exact model ID beside the result. Do not average different tasks into one winner.
Test a strategy with AuroraKeep reporting dates and units beside the answer, and inspect its original citations.
Published results
See the complete published results
Scroll sideways to compare all columns. Select a heading to sort.
| OpenAI o3-mini | Pass | (3 Day BTC Rate of Change > SMA(3 Day BTC Rate of Change) + StdDev(3 Day BTC Rate of Change)) or (Positions Percent Change of (BTC) ≤ -15) |
| Claude 3.5 Sonnet | Fail | (1 Day BTC Rate of Change > ROC(BTC Price) + 1) or (Positions Percent Change of (BTC) < -15) |
No matching rows. Clear the filter to see all records.
1 strategy request, 2 models, one attempt each. Published 2025-02-07. Values retain their original units and grading scale.
Task and tested input
Turn a plain-English trading request into a syntactically valid, nested NexusTrade strategy JSON object with the requested conditions.
Published prompt excerpt
Create a trading strategy that buys 50% of my buying power in Bitcoin when the rate of change of Bitcoin is 1 standard deviation above its 3 day rate of change or my bitcoin position is down 15%. Sell 20% of my portfolio value if I am up 25% or I’m up 10% from the last time I sold it.
How the runs worked
1 strategy request, 2 models, one attempt each
NexusTrade AI chat portfolio-creation prompts that turn a plain-English request into nested strategy JSON
Step 1
Send the request to the NexusTrade chat with Claude 3.5 Sonnet selected, then with o3-mini.
Step 2
The portfolio, strategy and condition prompts run in sequence to produce the strategy object.
Step 3
Open the generated strategies and compare the buy condition with the request.
How to read the measurements
Grading and valid output
Author inspection of the generated buy rule. Claude 3.5 Sonnet produced a syntactically valid strategy on its first try, but its buy rule used 1 plus the 3-day rate of change instead of 1 standard deviation. The article states o3-mini created the strategies exactly.
Cost
The article publishes no per-run cost. It quotes list prices of $1.10 input and $4.40 output per million tokens for o3-mini and $15 output per million tokens for Claude 3.5 Sonnet. Its text gives Claude 3.5 Sonnet's input price as $3.75, while its pricing screenshot shows $3 input and $3.75 for prompt caching writes.
Time
The article publishes no per-model timing.
Why this model was selected
The article concluded that o3-mini, the cheaper model, performed significantly better at generating complex JSON objects than Claude 3.5 Sonnet.
See the published results
This image was published with the 2025-02-07 article. Read the source for the full test procedure and every result it reports.
Sample and scope
This February 2025 comparison is a single prompt with one attempt per model, judged by the author.
The screenshot captioned as o3-mini's zoomed buy strategy shows the same condition as Claude's, while the o3-mini chat screenshot shows the standard-deviation condition. The record relies on the chat screenshot and the article text.
The o3-mini chat screenshot is labeled with the app's 'Maximum Capability' tier rather than a model name, and both models have since been replaced.
No comments yet.