TL;DR
Published 2024-07-19. GPT-4o mini passed, delivering accurate results on the first try, while GPT-3.5 Turbo failed to generate a valid query.
- Published
- 2024-07-19
- Published model rows
- 2
- Evidence
- Historical task evaluation
Follow this evaluation
Published results
GPT-4o mini and GPT-3.5 Turbo generating a valid SQL query: source evidence dated 2024-07-19.
- GPT-4o mini
- Pass
- GPT-3.5 Turbo
- Fail
The displayed values belong to the identified model versions and this published task, with their original metric units.
What did this task require the model to return?
Read the tested version, task and rubric together. Check failures alongside successful answers; compare costs only within the reported unit.
2024-07-19: GPT-4o mini and GPT-3.5 Turbo generating a valid SQL queryKeep the study date and exact model ID beside the result. Do not average different tasks into one winner.
Test a strategy with Aurora
Keep the study date and exact model ID beside the result. Do not average different tasks into one winner.
Test a strategy with AuroraKeep reporting dates and units beside the answer, and inspect its original citations.
Published results
See the complete published results
| Exact model version | Result |
|---|---|
| GPT-4o mini | Pass |
| GPT-3.5 Turbo | Fail |
No matching rows. Clear the filter to see all records.
1 SQL generation task, 2 models, prompt count and run count not published in this excerpt. Published 2024-07-19. Values retain their original units and grading scale.
Task and tested input
Generate a syntactically valid SQL query for a financial request and execute it against the database.
Instruction in brief (task summary)
Generate a syntactically valid SQL query that is executed against the database, then check whether the results look correct. The excerpt does not publish the exact request.
How the runs worked
1 SQL generation task, 2 models, prompt count and run count not published in this excerpt
NexusTrade AI trading platform, with generated SQL executed against the database
Step 1
Give each model the same SQL generation request in NexusTrade.
Step 2
Execute the generated query against the database.
Step 3
Examine whether the query ran and whether the results look correct.
How to read the measurements
Grading and valid output
Author judgment: a pass means a valid query with results that looked correct. A fail means no valid query.
Cost
The article publishes no per-model cost for this test.
Time
The article publishes no per-model timing.
Why this model was selected
The article reported GPT-4o mini succeeding where GPT-3.5 Turbo failed on both July 2024 tasks.
Sample and scope
July 2024 summary excerpt. The prompt, query and returned rows are not published on this page, and both models have since been replaced.
Pass or fail is the author's judgment of a single reported outcome per model, not a success rate over a question set.
The 60% cost and 128K context figures in the article are launch claims, not measurements from this test.
No comments yet.