TL;DR
Published 2025-04-15. GPT-4.1 has the highest published average score (0.884) and success rate (93.3%) in this 11-model table, with Gemini 2.5 Pro at 0.880 and 92.5%. GPT-4.1 Nano has the lowest average score (0.382).
- Published
- 2025-04-15
- Published model rows
- 11
- Evidence
- Historical task evaluation
Follow this evaluation
Published results
Models generating SQL for financial questions, April 2025: source evidence dated 2025-04-15.
- Exact model version
- GPT-4.1
- Average score
- 0.884
- Median score
- 1.000
- Std dev
- 0.279
- Success rate
- 93.3%
- Avg execution
- 1,711 ms
- Input cost
- $2.00/M
- Output cost
- $8.00/M
The displayed values belong to the identified model versions and this published task, with their original metric units.
What did this task require the model to return?
Read the tested version, task and rubric together. Check failures alongside successful answers; compare costs only within the reported unit.
2025-04-15: Models generating SQL for financial questions, April 2025Keep the study date and exact model ID beside the result. Do not average different tasks into one winner.
Test a strategy with Aurora
Keep the study date and exact model ID beside the result. Do not average different tasks into one winner.
Test a strategy with AuroraKeep reporting dates and units beside the answer, and inspect its original citations.
Published results
60 financial questions per the article text, 11 models. Published 2025-04-15. Values retain their original units and grading scale.
See the complete published results
Scroll sideways to compare all columns. Select a heading to sort.
| GPT-4.1 | 0.884 | 1.000 | 0.279 | 93.3% | 1,711 ms | $2.00/M | $8.00/M |
| GPT-4.1 Mini | 0.787 | 1.000 | 0.382 | 83.3% | 1,526 ms | $0.40/M | $1.60/M |
| GPT-4.1 Nano | 0.382 | 0.050 | 0.454 | 53.3% | 38,614 ms | $0.10/M | $0.40/M |
| Claude 3.7 Sonnet | 0.660 | 1.000 | 0.455 | 75.0% | 2,003 ms | $3.00/M | $15.00/M |
| Gemini 2.5 Pro | 0.880 | 1.000 | 0.300 | 92.5% | 2,478 ms | $1.25/M | $10.00/M |
| Gemini 2.0 Flash | 0.717 | 1.000 | 0.392 | 92.5% | 1,296 ms | $0.10/M | $0.40/M |
| Llama 4 Maverick | 0.565 | 1.000 | 0.488 | 62.5% | 1,986 ms | $0.19/M | $0.85/M |
| DeepSeek V3 | 0.617 | 1.000 | 0.460 | 75.0% | 26,892 ms | $0.27/M | $1.10/M |
| Grok-3-Beta | 0.747 | 1.000 | 0.405 | 90.0% | 1,707 ms | $3.00/M | $15.00/M |
| Grok-3-Mini-Beta | 0.645 | 1.000 | 0.459 | 72.5% | 1,593 ms | $0.30/M | $0.50/M |
| OpenAI O3-Mini | 0.635 | 1.000 | 0.464 | 72.5% | 8,854 ms | $1.10/M | $4.40/M |
No matching rows. Clear the filter to see all records.
60 financial questions per the article text, 11 models. Published 2025-04-15. Values retain their original units and grading scale.
Task and tested input
Generate SQL queries that correctly answer financial analysis questions.
Instruction in brief (task summary)
Each model received financial questions and returned a SQL query to answer each one. The article's worked example is "Find companies with increasing profit margins over the last 3 years", shown as a NexusTrade usage example rather than an identified scored question.
How the runs worked
60 financial questions per the article text, 11 models
The author's one-shot SQL query generation test for financial analysis, with methodology deferred to an earlier linked article
Step 1
Ask each of 11 models the same 60 financial questions.
Step 2
Collect one generated SQL query per question (one-shot).
Step 3
Report average score, median score, standard deviation, success rate and average execution time per model.
How to read the measurements
Grading and valid output
The article reports scores and success rates but refers readers to an earlier article for the methodology. The grader is not described in this article.
Cost
List prices per million tokens, from $0.10/M input for GPT-4.1 Nano and Gemini 2.0 Flash to $3.00/M input and $15.00/M output for Claude 3.7 Sonnet and Grok-3-Beta. No measured run spend is published.
Time
Average execution time in milliseconds, from 1,296 ms (Gemini 2.0 Flash) to 38,614 ms (GPT-4.1 Nano).
Why this model was selected
The article called OpenAI the winner in pure performance by a very narrow margin (GPT-4.1) and Google the winner in cost effectiveness, and called GPT-4.1 Nano virtually unusable for this task.
See the published results
This image was published with the 2025-04-15 article. Read the source for the full test procedure and every result it reports.
Sample and scope
This April 2025 run reflects the model versions and prices of that week. Several models in the table (Grok-3-Beta, Gemini 2.5 Pro preview builds, o3-mini) have since been replaced.
The article defers the grading procedure to an earlier linked article and does not describe how scores and success were assigned in this run. Several success rates (92.5%, 62.5%, 72.5%) are not whole-number fractions of 60 questions, so the effective denominator is not stated.
The table comes from a single published run on one financial SQL task. The body text quotes GPT-4.1 at an "88.5% average score" while the table shows 0.884. The table value is used here.
The seven rows shared with the April 10, 2025 Grok 3 article carry identical values, so both articles publish the same run. This article adds the three GPT-4.1 versions and Gemini 2.5 Pro.
No comments yet.