TL;DR
Published 2025-04-10. Grok-3-Beta has the highest published average score (0.747) and perfect-score share (67.5%). Gemini 2.0 Flash has the highest success rate (92.5%) and fastest average execution time (1,296 ms), and Llama 4 Maverick has the lowest average score (0.565).
- Published
- 2025-04-10
- Published model rows
- 7
- Evidence
- Historical task evaluation
Follow this evaluation
Published results
Seven models generating financial SQL queries: source evidence dated 2025-04-10.
- Exact model version
- Grok-3-Beta
- Average score
- 0.747
- Median score
- 1.000
- Std deviation
- 0.405
- Success rate
- 90.0%
- Avg execution time (ms)
- 1,707
- Perfect score (1.0) share
- 67.5%
The displayed values belong to the identified model versions and this published task, with their original metric units.
What did this task require the model to return?
Read the tested version, task and rubric together. Check failures alongside successful answers; compare costs only within the reported unit.
2025-04-10: Seven models generating financial SQL queriesKeep the study date and exact model ID beside the result. Do not average different tasks into one winner.
Test a strategy with Aurora
Keep the study date and exact model ID beside the result. Do not average different tasks into one winner.
Test a strategy with AuroraKeep reporting dates and units beside the answer, and inspect its original citations.
Published results
7 models on a shared SQL question set, the article does not state the number of questions. Published 2025-04-10. Values retain their original units and grading scale.
See the complete published results
Scroll sideways to compare all columns. Select a heading to sort.
| Grok-3-Beta | 0.747 | 1.000 | 0.405 | 90.0% | 1,707 | 67.5% |
| Gemini 2.0 Flash | 0.717 | 1.000 | 0.392 | 92.5% | 1,296 | 55.0% |
| Claude 3.7 Sonnet | 0.660 | 1.000 | 0.455 | 75.0% | 2,003 | 57.5% |
| Grok-3-Mini-Beta | 0.645 | 1.000 | 0.459 | 72.5% | 1,593 | 60.0% |
| OpenAI O3-Mini | 0.635 | 1.000 | 0.464 | 72.5% | 8,854 | 57.5% |
| DeepSeek V3 | 0.617 | 1.000 | 0.460 | 75.0% | 26,892 | 52.5% |
| Llama 4 Maverick | 0.565 | 1.000 | 0.488 | 62.5% | 1,986 | 52.5% |
No matching rows. Clear the filter to see all records.
7 models on a shared SQL question set, the article does not state the number of questions. Published 2025-04-10. Values retain their original units and grading scale.
Task and tested input
Generate SQL queries that answer natural-language financial questions over NexusTrade's stock data.
Instruction in brief (task summary)
Generate a SQL query for each financial question in the evaluation set. The article does not publish the evaluation questions or prompt. It points readers to a separate SQL evaluation article for the testing method.
How the runs worked
7 models on a shared SQL question set, the article does not state the number of questions
SQL query generation evaluation for NexusTrade's financial data. The article links its separate SQL evaluation write-up for the testing method
Step 1
Run each of the seven models over the same set of SQL generation questions.
Step 2
Score each generated query from 0 to 1 and record whether it succeeded and how long it took to execute.
Step 3
Summarize average, median and standard deviation of scores, success rate, average execution time and the score distribution per model.
How to read the measurements
Grading and valid output
Each query receives a 0 to 1 score. The table reports the distribution in buckets 0.0-0.2, 0.3-0.5, 0.6-0.7, 0.8-0.9 and 1.0. The grader is not described in this article.
Cost
The article publishes no per-model cost.
Time
Average execution time in milliseconds per model, from 1,296 ms (Gemini 2.0 Flash) to 26,892 ms (DeepSeek V3). The article does not define whether this includes model generation time.
Why this model was selected
The article's walkthrough selected Grok 3 Mini Beta in the NexusTrade model settings, and states that this choice is distinct from the top-scoring Grok-3-Beta in the table.
See the published results
This image was published with the 2025-04-10 article. Read the source for the full test procedure and every result it reports.
Sample and scope
This April 2025 article publishes the results table without the question count, grader or prompt. Those are described only in a separately linked article.
Every model has a median score of 1.000, so the averages are driven by the share of low-scoring queries (0.0-0.2), which ranges from 22.5% to 42.5%.
The exact DeepSeek V3 snapshot is not named, and the Grok 3 beta models and other versions have since been replaced.
The April 15, 2025 GPT-4.1 article republishes these seven rows with identical values and adds four more models.
No comments yet.