TL;DR
o3-mini (openai/o3-mini) appears in 6 published studies. 2025-02-24, Claude 3.7 Sonnet and o3-mini building a vague leveraged-ETF strategy: 31.87% for percent change. 2025-02-05, Gemini 2.0 Flash, DeepSeek R1 and o3-mini writing financial SQL: 0.7/1 for q1: rddt vs spy return correlation. 2025-02-01, DeepSeek R1 and o3-mini writing a dividend stock screen in SQL: 1 for accuracy. 2025-04-10, Seven models generating financial SQL queries: 0.635 for average score. 2025-02-07, Claude 3.5 Sonnet and o3-mini building a Bitcoin trading strategy: Pass for buy rule matched request. 2025-04-15, Models generating SQL for financial questions, April 2025: 0.635 for average score. Each result belongs to its own task, date and grading method.
- Exact tested ID
- openai/o3-mini
- Published studies
- 6
- Provider
- openai
Inspect this exact model
Claude 3.7 Sonnet and o3-mini building a vague leveraged-ETF strategy
o3-mini: source evidence dated 2025-04-15.
- Percent change
- 31.87%
- Sharpe ratio
- 0.54
- Sortino ratio
- 0.60
- Max drawdown
- 41.74%
- Average drawdown
- 10.09%
- Num trades
- 6.00
The displayed values belong to the identified model versions and this published task, with their original metric units.
How did this version perform on the job I need?
The first measured row below belongs to this model, not its provider's other models. Open that study to see the other arms and tested input.
2025-02-24: Claude 3.7 Sonnet and o3-mini building a vague leveraged-ETF strategySelect a model for the specific tested task. A profile with one study does not establish performance on other jobs.
Test a strategy with Aurora
Select a model for the specific tested task. A profile with one study does not establish performance on other jobs.
Test a strategy with AuroraKeep reporting dates and units beside the answer, and inspect its original citations.
Claude 3.7 Sonnet and o3-mini building a vague leveraged-ETF strategy
See the complete claude 3.7 sonnet and o3-mini building a vague leveraged-etf strategy
o3-mini recorded 31.87% for percent change in the 2025-02-24 publication. 1 ambiguous strategy request, 2 models, one generated strategy each, backtested once.
| Published metric | Value |
|---|---|
| Percent change | 31.87% |
| Sharpe ratio | 0.54 |
| Sortino ratio | 0.60 |
| Max drawdown | 41.74% |
| Average drawdown | 10.09% |
| Num trades | 6.00 |
No matching rows. Clear the filter to see all records.
Claude 3.7 Sonnet and o3-mini building a vague leveraged-ETF strategy: read these measurements
View task, sample and measurement units
1 ambiguous strategy request, 2 models, one generated strategy each, backtested once
Turn an ambiguous plain-English request into a complete nested trading strategy JSON object, then backtest it.
Scoring
No score is assigned. The article compares percent change, Sharpe and Sortino ratios and drawdowns from each backtest, alongside the author's reading of the generated rules. Over the same window, holding SPY returned 20.58% with a 1.06 Sharpe ratio, 1.18 Sortino ratio and 9.42% max drawdown.
Cost units
The article publishes no per-run cost. It lists Claude 3.7 Sonnet at $3 input and $15 output per million tokens and o3-mini at $1.10 input and $4.40 output per million tokens.
Time units
The article publishes no per-model timing. It says o3-mini produced its strategy in less than a minute.
Gemini 2.0 Flash, DeepSeek R1 and o3-mini writing financial SQL
Open every published row
o3-mini recorded 0.7/1 for q1: rddt vs spy return correlation in the 2025-02-05 publication. 2 natural-language financial questions, 3 models, one attempt each.
| Published metric | Value |
|---|---|
| Q1: RDDT vs SPY return correlation | 0.7/1 |
| Q2: biotech revenue growth | 1/1 |
| Input price (per 1M tokens) | $1.1 |
| Output price (per 1M tokens) | $4.4 |
No matching rows. Clear the filter to see all records.
Gemini 2.0 Flash, DeepSeek R1 and o3-mini writing financial SQL: read these measurements
View task, sample and measurement units
2 natural-language financial questions, 3 models, one attempt each
Generate a BigQuery SQL query that answers a natural-language financial question over the NexusTrade stock dataset.
Scoring
Author-assigned per-question scores. On question 1, R1 misspelled adjustedClosingPrice and o3-mini used the ticker REDDIT, so both received 0.7 after a manual fix. On question 2, R1's query failed to execute (0/1), and o3-mini-high graded both the Gemini and o3-mini queries 9/10, which the author recorded as full credit.
Cost units
OpenRouter list prices per million tokens as shown in the article: Gemini Flash 2.0 $0.1 input and $0.4 output. DeepSeek R1 $0.75 input and $2.4 output. O3-mini $1.1 input and $4.4 output. No per-run cost is published.
Time units
The article publishes no per-model timing. It describes Gemini as responding in a couple of seconds, R1 as taking more than 30 seconds on question 1, and o3-mini as taking a few seconds.
DeepSeek R1 and o3-mini writing a dividend stock screen in SQL
Open every published row
o3-mini recorded 1 for accuracy in the 2025-02-01 publication. 1 natural-language screening request, 2 models, R1 asked twice, o3-mini once.
| Published metric | Value |
|---|---|
| Accuracy | 1 |
| Estimated cost | 6c |
| Latency | 2 minutes 24 seconds |
| Input price | $1.10/M |
| Output price | $4.4/M |
No matching rows. Clear the filter to see all records.
DeepSeek R1 and o3-mini writing a dividend stock screen in SQL: read these measurements
View task, sample and measurement units
1 natural-language screening request, 2 models, R1 asked twice, o3-mini once
Convert a plain-English dividend stock screen into an executable SQL query, run it and summarize the results.
Scoring
The author assigned 1 when manual inspection found the query conformed to the request and 0 otherwise. For R1 the article lists using total liabilities instead of debt, full-year instead of latest-quarter data and an average dividend yield as the defects.
Cost units
Estimated relative cost: with c as one R1 request, R1 used 5 retries plus 1 request (6c), and o3-mini used 1 retry, 1 evaluation and 1 summarization request at 2c each (6c). List prices shown were $0.75/M input and $2.4/M output for R1 on OpenRouter and $1.10/M input and $4.4/M output for o3-mini.
Time units
R1 gave up after 41 minutes according to the database logs. O3-mini responded 2 minutes 24 seconds after the 6:26 PM request, including one retry.
Seven models generating financial SQL queries
Open every published row
o3-mini recorded 0.635 for average score in the 2025-04-10 publication. 7 models on a shared SQL question set, the article does not state the number of questions.
| Published metric | Value |
|---|---|
| Average score | 0.635 |
| Median score | 1.000 |
| Std deviation | 0.464 |
| Success rate | 72.5% |
| Avg execution time (ms) | 8,854 |
| Perfect score (1.0) share | 57.5% |
No matching rows. Clear the filter to see all records.
Seven models generating financial SQL queries: read these measurements
View task, sample and measurement units
7 models on a shared SQL question set, the article does not state the number of questions
Generate SQL queries that answer natural-language financial questions over NexusTrade's stock data.
Scoring
Each query receives a 0 to 1 score. The table reports the distribution in buckets 0.0-0.2, 0.3-0.5, 0.6-0.7, 0.8-0.9 and 1.0. The grader is not described in this article.
Cost units
The article publishes no per-model cost.
Time units
Average execution time in milliseconds per model, from 1,296 ms (Gemini 2.0 Flash) to 26,892 ms (DeepSeek V3). The article does not define whether this includes model generation time.
Claude 3.5 Sonnet and o3-mini building a Bitcoin trading strategy
Open every published row
o3-mini recorded Pass for buy rule matched request in the 2025-02-07 publication. 1 strategy request, 2 models, one attempt each.
| Published metric | Value |
|---|---|
| Buy rule matched request | Pass |
| Generated buy condition (as displayed) | (3 Day BTC Rate of Change > SMA(3 Day BTC Rate of Change) + StdDev(3 Day BTC Rate of Change)) or (Positions Percent Change of (BTC) ≤ -15) |
No matching rows. Clear the filter to see all records.
Claude 3.5 Sonnet and o3-mini building a Bitcoin trading strategy: read these measurements
View task, sample and measurement units
1 strategy request, 2 models, one attempt each
Turn a plain-English trading request into a syntactically valid, nested NexusTrade strategy JSON object with the requested conditions.
Scoring
Author inspection of the generated buy rule. Claude 3.5 Sonnet produced a syntactically valid strategy on its first try, but its buy rule used 1 plus the 3-day rate of change instead of 1 standard deviation. The article states o3-mini created the strategies exactly.
Cost units
The article publishes no per-run cost. It quotes list prices of $1.10 input and $4.40 output per million tokens for o3-mini and $15 output per million tokens for Claude 3.5 Sonnet. Its text gives Claude 3.5 Sonnet's input price as $3.75, while its pricing screenshot shows $3 input and $3.75 for prompt caching writes.
Time units
The article publishes no per-model timing.
Models generating SQL for financial questions, April 2025
Open every published row
o3-mini recorded 0.635 for average score in the 2025-04-15 publication. 60 financial questions per the article text, 11 models.
| Published metric | Value |
|---|---|
| Average score | 0.635 |
| Median score | 1.000 |
| Std dev | 0.464 |
| Success rate | 72.5% |
| Avg execution | 8,854 ms |
| Input cost | $1.10/M |
| Output cost | $4.40/M |
No matching rows. Clear the filter to see all records.
Models generating SQL for financial questions, April 2025: read these measurements
View task, sample and measurement units
60 financial questions per the article text, 11 models
Generate SQL queries that correctly answer financial analysis questions.
Scoring
The article reports scores and success rates but refers readers to an earlier article for the methodology. The grader is not described in this article.
Cost units
List prices per million tokens, from $0.10/M input for GPT-4.1 Nano and Gemini 2.0 Flash to $3.00/M input and $15.00/M output for Claude 3.7 Sonnet and Grok-3-Beta. No measured run spend is published.
Time units
Average execution time in milliseconds, from 1,296 ms (Gemini 2.0 Flash) to 38,614 ms (GPT-4.1 Nano).
No comments yet.