TL;DR
Published 2025-07-11. Gemini 2.5 Pro has the highest published average score (0.757) and ties o4 Mini on median score (0.967). Grok-4 scores a 0.767 median and 0.666 average with the slowest average time (74,812 ms).
- Published
- 2025-07-11
- Published model rows
- 11
- Evidence
- Historical task evaluation
Follow this evaluation
Published results
Models generating financial SQL queries, July 2025: source evidence dated 2025-07-11.
- Exact model version
- Gemini 2.5 Pro
- Median score
- 0.967
- Avg score
- 0.757
- Success rate
- 86.52%
- Avg time
- 50,042 ms
- Input cost
- $1.25/M
- Output cost
- $10.00/M
The displayed values belong to the identified model versions and this published task, with their original metric units.
What did this task require the model to return?
Read the tested version, task and rubric together. Check failures alongside successful answers; compare costs only within the reported unit.
2025-07-11: Models generating financial SQL queries, July 2025Keep the study date and exact model ID beside the result. Do not average different tasks into one winner.
Test a strategy with Aurora
Keep the study date and exact model ID beside the result. Do not average different tasks into one winner.
Test a strategy with AuroraKeep reporting dates and units beside the answer, and inspect its original citations.
Published results
About 88 financial questions per the article text, 11 models. Published 2025-07-11. Values retain their original units and grading scale.
See the complete published results
Scroll sideways to compare all columns. Select a heading to sort.
| Gemini 2.5 Pro | 0.967 | 0.757 | 86.52% | 50,042 ms | $1.25/M | $10.00/M |
| o4 Mini | 0.967 | 0.675 | 79.78% | 48,481 ms | $1.10/M | $4.40/M |
| Gemini 2.5 Flash Preview | 0.900 | 0.703 | 87.64% | 33,344 ms | $0.15/M | $3.50/M |
| OpenAI o3 | 0.900 | 0.635 | 77.53% | 56,070 ms | $2.00/M | $8.00/M |
| Grok-4 | 0.767 | 0.666 | 77.53% | 74,812 ms | $3.00/M | $15.00/M |
| Claude Sonnet 4 | 0.733 | 0.652 | 80.90% | 44,833 ms | $3.00/M | $15.00/M |
| GPT-4.1 | 0.733 | 0.625 | 77.53% | 41,058 ms | $2.00/M | $8.00/M |
| Grok-3 Beta | 0.733 | 0.657 | 83.15% | 42,169 ms | $3.00/M | $15.00/M |
| Claude Opus 4 | 0.700 | 0.572 | 70.79% | 51,877 ms | $15.00/M | $75.00/M |
| Claude 3.7 Sonnet | 0.667 | 0.565 | 68.54% | 45,526 ms | $3.00/M | $15.00/M |
| Gemini 2.0 Flash | 0.567 | 0.525 | 74.16% | 31,140 ms | $0.10/M | $0.40/M |
No matching rows. Clear the filter to see all records.
About 88 financial questions per the article text, 11 models. Published 2025-07-11. Values retain their original units and grading scale.
Task and tested input
Generate SQL queries that answer financial questions against a financial database.
Published prompt excerpt
What are the top 10 stocks by revenue with a 2025 percent gain so far greater than its total 2024 percent gain? Sort by market cap descending
How the runs worked
About 88 financial questions per the article text, 11 models
EvaluateGPT: generated SQL executed against a financial database, then scored 0 to 1 by three LLM judges
Step 1
Submit each financial question to the model under test.
Step 2
Execute the generated SQL query against the database and fetch the results.
Step 3
Score the query and returned rows with three LLM judges (Claude Sonnet 4, Gemini 2.5 Pro, GPT-4.1) on a 0 to 1 scale, then aggregate mean, median and distribution per model.
How to read the measurements
Grading and valid output
Three LLM judges assign a 0 to 1 quality score that is separate from whether the query executes. The article states the score measures query and answer assessment, not investment returns.
Cost
List prices per million tokens, from $0.10/M input and $0.40/M output (Gemini 2.0 Flash) to $15.00/M input and $75.00/M output (Claude Opus 4). Total run spend was reported as over $200, with $171.88 on Requesty.
Time
Average time in milliseconds, from 31,140 ms (Gemini 2.0 Flash) to 74,812 ms (Grok-4).
Why this model was selected
The article named Gemini 2.5 Pro the best model and did not add Grok 4 to NexusTrade. It replaced Gemini Flash 2 with Gemini 2.5 Flash in the app, citing the median score rising from 0.567 to 0.900.
See the published results
This image was published with the 2025-07-11 article. Read the source for the full test procedure and every result it reports.
Sample and scope
This is a single July 2025 run with the model versions and prices of that week. Gemini 2.5 Flash appears as a preview build and Grok-3 as a beta.
Scores come from three LLM judges, one of which (Gemini 2.5 Pro) is also the top-ranked model. The article's own text says its ANOVA significance claim needs reconciliation and that the table supports only a descriptive ranking.
The article describes the question set as three examples "and 85+ more" and later as an 87-question design, while the published success rates are whole-number fractions of 89. The exact question count is not stated consistently.
No comments yet.