TL;DR
Published 2025-02-24. Claude 3.7 Sonnet's strategy returned 36.02% with a 1.03 Sharpe ratio and 17.55% max drawdown. o3-mini's strategy returned 31.87% with a 0.54 Sharpe ratio and 41.74% max drawdown.
- Published
- 2025-02-24
- Published model rows
- 2
- Evidence
- Historical task evaluation
Follow this evaluation
Published results
Claude 3.7 Sonnet and o3-mini building a vague leveraged-ETF strategy: source evidence dated 2025-02-24.
- Exact model version
- Claude 3.7 Sonnet
- Percent change
- 36.02%
- Sharpe ratio
- 1.03
- Sortino ratio
- 1.02
- Max drawdown
- 17.55%
- Average drawdown
- 1.83%
- Num trades
- 31.00
The displayed values belong to the identified model versions and this published task, with their original metric units.
What did this task require the model to return?
Read the tested version, task and rubric together. Check failures alongside successful answers; compare costs only within the reported unit.
2025-02-24: Claude 3.7 Sonnet and o3-mini building a vague leveraged-ETF strategyKeep the study date and exact model ID beside the result. Do not average different tasks into one winner.
Test a strategy with Aurora
Keep the study date and exact model ID beside the result. Do not average different tasks into one winner.
Test a strategy with AuroraKeep reporting dates and units beside the answer, and inspect its original citations.
Published results
1 ambiguous strategy request, 2 models, one generated strategy each, backtested once. Published 2025-02-24. Values retain their original units and grading scale.
See the complete published results
Scroll sideways to compare all columns. Select a heading to sort.
| Claude 3.7 Sonnet | 36.02% | 1.03 | 1.02 | 17.55% | 1.83% | 31.00 |
| GPT o3-mini | 31.87% | 0.54 | 0.60 | 41.74% | 10.09% | 6.00 |
No matching rows. Clear the filter to see all records.
1 ambiguous strategy request, 2 models, one generated strategy each, backtested once. Published 2025-02-24. Values retain their original units and grading scale.
Task and tested input
Turn an ambiguous plain-English request into a complete nested trading strategy JSON object, then backtest it.
Published prompt excerpt
Create a strategy using leveraged ETFs. I want to capture the upside of the broader market, while limiting my risk when the market (and my portfolio) goes up. No stop losses
How the runs worked
1 ambiguous strategy request, 2 models, one generated strategy each, backtested once
NexusTrade AI chat single-prompt Create Strategies JSON generation with the built-in backtester
Step 1
Send the request to the NexusTrade chat with o3-mini selected, then with Claude 3.7 Sonnet selected.
Step 2
A single Create Strategies prompt generates the full nested strategy JSON.
Step 3
Backtest each generated strategy and compare its statistics with holding SPY.
How to read the measurements
Grading and valid output
No score is assigned. The article compares percent change, Sharpe and Sortino ratios and drawdowns from each backtest, alongside the author's reading of the generated rules. Over the same window, holding SPY returned 20.58% with a 1.06 Sharpe ratio, 1.18 Sortino ratio and 9.42% max drawdown.
Cost
The article publishes no per-run cost. It lists Claude 3.7 Sonnet at $3 input and $15 output per million tokens and o3-mini at $1.10 input and $4.40 output per million tokens.
Time
The article publishes no per-model timing. It says o3-mini produced its strategy in less than a minute.
Why this model was selected
The article found Claude 3.7 Sonnet generated the better strategy and called it a strong choice, particularly for users who may not know exactly what they want, while noting it is more than 3x the price of o3-mini.
See the published results
This image was published with the 2025-02-24 article. Read the source for the full test procedure and every result it reports.
Sample and scope
This February 2025 comparison is a single prompt and a single generated strategy per model, so it does not separate model ability from sampling variance.
The backtest window is not stated in the article. Both screenshots show the same Hold SPY baseline (20.58%), which indicates a shared window.
Backtest returns depend on the period tested and are not evidence of future performance. Both models have since been replaced.
No comments yet.