GPT-oss and Gemini generating financial SQL queries

TL;DR

Published 2025-08-06. Gemini 2.5 Pro Preview and GPT-OSS-120B share the highest median score (0.900), but Gemini 2.5 Pro Preview has the higher average score (0.706 vs 0.614) and success rate (83.15% vs 70.79%). GPT-OSS-120B has the fastest average time (38,538 ms).

Published

2025-08-06

Published model rows

4

Evidence

Historical task evaluation

Follow this evaluation

Step 1 of 3

Published results

GPT-oss and Gemini generating financial SQL queries: source evidence dated 2025-08-06.

Exact model version

Gemini 2.5 Pro Preview

Median score

0.900

Avg score

0.706

Success rate

83.15%

Avg time

56,393 ms

Input cost

$1.25/M

Output cost

$10.00/M

Context window

1,048,576

What you should see

The displayed values belong to the identified model versions and this published task, with their original metric units.

Step 2 of 3

What did this task require the model to return?

Read the tested version, task and rubric together. Check failures alongside successful answers; compare costs only within the reported unit.

2025-08-06: GPT-oss and Gemini generating financial SQL queries →

What you should see

Keep the study date and exact model ID beside the result. Do not average different tasks into one winner.

Step 3 of 3

Test a strategy with Aurora

Keep the study date and exact model ID beside the result. Do not average different tasks into one winner.

Test a strategy with Aurora →

What you should see

Keep reporting dates and units beside the answer, and inspect its original citations.

Ask Aurora about this evidence →

Published results

Task and tested input

Generate BigQuery SQL queries that answer complex financial questions.

1. Instruction in brief (task summary)

   A question is submitted to the model under test, which generates a SQL query to answer it. The article does not publish the individual questions for this run.

How the runs worked

90 financial questions per the article text, 4 models

EvaluateGPT: generated BigQuery SQL executed against a financial database, then scored 0 to 1 by three LLM judges

1. Step 1

   Run the 90-question EvaluateGPT dataset on Gemini 2.5 Pro, Gemini 2.5 Flash, GPT-oss 120B and GPT-oss 20B.

2. Step 2

   Execute each generated SQL query against the database.

3. Step 3

   Score the query and results 0 to 1 with Claude Sonnet 4, Gemini 2.5 Pro and GPT-4.1, average the scores and compare mean, median and distribution per model.

How to read the measurements

1. Grading and valid output

   Three LLM judges score each query and its results from 0 to 1. The scores are averaged.

2. Cost

   List prices per million tokens, from $0.05/M input and $0.20/M output (GPT-OSS-20B) to $1.25/M input and $10.00/M output (Gemini 2.5 Pro Preview). No measured run spend is published.

3. Time

   Average time in milliseconds, from 38,538 ms (GPT-OSS-120B) to 56,393 ms (Gemini 2.5 Pro Preview).

Why this model was selected

The article said GPT-oss 120B overthrew Gemini 2.5 Flash in its price range for SQL generation and that Gemini 2.5 Pro still led, but the author stayed with the Gemini family for NexusTrade for now.

See the published results

This image was published with the 2025-08-06 article. Read the source for the full test procedure and every result it reports.

Sample and scope

This is a single run from the day GPT-oss was released (August 2025), with the provider pricing and Gemini 2.5 Pro preview build of that week.

Scores come from three LLM judges, one of which (Gemini 2.5 Pro) is also a tested model. The text says 90 questions while the published success rates are whole-number fractions of 89.

The article's separate JSON trading-strategy test (two prompts, author-inspected, GPT-oss variant not stated) is excluded here because it reports no comparable measurement.

Read the original evidence

Continue exploring