TL;DR
o3 (openai/o3) appears in 2 published studies. 2025-07-11, Models generating financial SQL queries, July 2025: 0.900 for median score. 2025-08-11, Models turning plain-English trading rules into strategy configs: 0.900 for median score. Each result belongs to its own task, date and grading method.
- Exact tested ID
- openai/o3
- Published studies
- 2
- Provider
- openai
Inspect this exact model
Models generating financial SQL queries, July 2025
o3: source evidence dated 2025-08-11.
- Median score
- 0.900
- Avg score
- 0.635
- Success rate
- 77.53%
- Avg time
- 56,070 ms
- Input cost
- $2.00/M
- Output cost
- $8.00/M
The displayed values belong to the identified model versions and this published task, with their original metric units.
How did this version perform on the job I need?
The first measured row below belongs to this model, not its provider's other models. Open that study to see the other arms and tested input.
2025-07-11: Models generating financial SQL queries, July 2025Select a model for the specific tested task. A profile with one study does not establish performance on other jobs.
Test a strategy with Aurora
Select a model for the specific tested task. A profile with one study does not establish performance on other jobs.
Test a strategy with AuroraKeep reporting dates and units beside the answer, and inspect its original citations.
Models generating financial SQL queries, July 2025
See the complete models generating financial sql queries, july 2025
o3 recorded 0.900 for median score in the 2025-07-11 publication. About 88 financial questions per the article text, 11 models.
| Published metric | Value |
|---|---|
| Median score | 0.900 |
| Avg score | 0.635 |
| Success rate | 77.53% |
| Avg time | 56,070 ms |
| Input cost | $2.00/M |
| Output cost | $8.00/M |
No matching rows. Clear the filter to see all records.
Models generating financial SQL queries, July 2025: read these measurements
View task, sample and measurement units
About 88 financial questions per the article text, 11 models
Generate SQL queries that answer financial questions against a financial database.
Scoring
Three LLM judges assign a 0 to 1 quality score that is separate from whether the query executes. The article states the score measures query and answer assessment, not investment returns.
Cost units
List prices per million tokens, from $0.10/M input and $0.40/M output (Gemini 2.0 Flash) to $15.00/M input and $75.00/M output (Claude Opus 4). Total run spend was reported as over $200, with $171.88 on Requesty.
Time units
Average time in milliseconds, from 31,140 ms (Gemini 2.0 Flash) to 74,812 ms (Grok-4).
Models turning plain-English trading rules into strategy configs
Open every published row
o3 recorded 0.900 for median score in the 2025-08-11 publication. 14 models on one sample of strategy requests, this article does not state the count (the companion article of 2025-08-12 describes the same table as 12 financial questions).
| Published metric | Value |
|---|---|
| Median score | 0.900 |
| Avg score | 0.824 |
| Success rate | 100.0% |
| Avg time (ms) | 124,902 |
| Input cost | $2.00/M |
| Output cost | $8.00/M |
No matching rows. Clear the filter to see all records.
Models turning plain-English trading rules into strategy configs: read these measurements
View task, sample and measurement units
14 models on one sample of strategy requests, this article does not state the count (the companion article of 2025-08-12 describes the same table as 12 financial questions)
Convert a natural-language trading strategy request into a NexusTrade portfolio configuration of actions and conditions.
Scoring
LLM judges score each configuration from 0 to 1 using published guidelines (1.0 perfect match, 0.8 to 0.9 minor issues, 0.5 to 0.7 significant discrepancies, 0.2 to 0.4 major errors, 0.0 to 0.1 complete mismatch). Success rate is reported separately.
Cost units
The table lists each model's list price per million input and output tokens. It does not publish total run cost.
Time units
Average time per generation in milliseconds, as published in the table.
No comments yet.