I tested every AI Model on a complex SQL Query Generation Task. Here’s where Grok 4 stands

I spent $200 to test every language model. Here’s the best

Grok 4 dominated the benchmarks at its unveiling yesterday. In every single major benchmark, Grok 4 outpeformed even the best and heaviest reasoning models in every single task.

It’s honestly not even close.

**Figure 1:** Grok 4’s performance on popular benchmarks including GPQA, AIME25, LCB, HMMT25, and USAMO25

These benchmarks are a decent proxy for general intelligence but they are not perfect. Popular benchmarks are easy to game and oftentimes unreliable for real-world tasks. So it got me thinking… how well does Grok 4 do on a real-world task?

I decided to test this objectively. Using a combination of Requesty and OpenRouter, platforms that allow aggregate LLMs into one API, I created a benchmark and spent over $200 testing each LLM on its SQL Query Performance.

**Figure 2**: My Requesty Analytics Dashboard showing that I spent $171.88 on Requesty alone today for this task

Here’s what I found.

A custom financial-query benchmark

To evaluate xAI’s new Grok 4 objectively, I’ve created a custom benchmark called EvaluateGPT.

GitHub - austin-starks/EvaluateGPT: Evaluate the effectiveness of a system prompt within seconds!

Evaluate the effectiveness of a system prompt within seconds! - austin-starks/EvaluateGPT

github.com

EvaluateGPT allows me to test how well each new language model can generate accurate SQL queries for tough financial questions.

**Figure 3:** An example query generated by the OpenAI o3 model for the question “What are the top 10 stocks by revenue with a 2025 percent gain so far greater than its total 2024 percent gain? Sort by market cap descending”

Think questions such as:

“What AI stocks have the highest market cap?”
“List companies with revenue growth of more than 20% year-over-year”
“What are the top 10 stocks by revenue with a 2025 percent gain so far greater than its total 2024 percent gain? Sort by market cap descending”

And 85+ more.

**Figure 4**: EvaluateGPT’s evaluation pipeline showing how three LLMs grade the accuracy of AI-generated SQL queries

As Figure 4 suggests, the process for evaluating a model is as follows:

A user submits a financial question to the model that we’re testing
The LLM generates the SQL Query that can answer this question
The SQL Query is executed against the database and fetch the results
The generated query and returned rows are assessed by three LLM judges: Claude Sonnet 4, Gemini 2.5 Pro and GPT-4.1. The linked EvaluateGPT framework uses a 0-to-1 quality score. A score is separate from whether the query executes. Illustrative check: SQL can run and return ten rows while sorting by revenue when the question asks for market-cap order. That result has valid syntax but does not answer the requested question. This is a teaching example, not an identified scored row from the July run.
The score is aggregated and we sort all of the models by their accuracy score

We then take the mean scores, median scores, and score distribution, an we create a table to compare each LLM to one another.

When I did this three months ago, I found that the Google models were among the most powerful for this particular task.

Despite all of the hype, Google BEATS OpenAI and remains the best AI company in the world.

O3, O4-mini and 4.1 are not enough to dethrone Google

medium.com

But three months is a long time in the AI space. I decided to re-run the entire experiment, and find out if Grok 4 was really as good as they want you to believe?

Spoiler alert: No, at least not for this task.

Grok 4, Claude 4, and Gemini 2.5 Pro. Which AI model is the best?

**Figure 5:** A Table Showing the Aggregate Scores of All of These LLMs Sorted by Their Median Score

Figure 5 is the model-score table; Figure 2 is the spend dashboard. In this July 2025 run, Gemini 2.5 Pro had the highest reported mean and median judge scores. These measure the model-generated query and answer assessment, rather than investment returns or the fraction of every possible financial question answered correctly.

The table shows the performance of different LLMs across cost, speed, accuracy, and score distribution. It also has notes on each of the models. At first glance, we can see that:

Gemini 2.5 Pro is the best model by far, scoring the highest median score and average score. Gemini 2.5 Flash is second best if you grade it according to its average score and input/output cost
OpenAI’s o4-mini is second place in terms of raw median performance and output cost (which is very small compared to the input). The O3 model is 4th place, but worse in nearly every way
Grok-4 trails behind, performing marginally better than Claude Sonnet 4 and GPT-4.1
Claude Opus 4 is arguably the worst model on this list, being obscenely expensive, having low median performance, low average performance, and slow execution speeds.

Comparing the reported model scores

To verify these results, I wanted to test for statistical significance. To do this, I used OpenAI’s O3 model to help me perform a Repeated Measures One-Way ANOVA.

The original article reported F(10, 960) = 4.77, p < 0.001 and partial η² near 0.048, alongside a Tukey HSD conclusion that Gemini 2.5 Pro beat every other model. That statistic needs reconciliation with the described 87-question, 11-model repeated-measures design. The original score matrix and analysis code are not supplied here, so I cannot verify the model, degrees of freedom or pairwise inference. Until those are checked, the score table supports the descriptive ranking, rather than the stated significance conclusion.

In the displayed sample, Gemini 2.5 Pro leads the reported scores. The exact inferential conclusion requires the original analysis described above; this ranking alone does not establish every pairwise difference.

Why Does This Matter?

For one, these results demonstrate that benchmarks don’t show the full story.

Even though Grok 4 demolished every other model on the benchmark, it did not translate for my real-world reasoning task. This has a massive impact for my real-world app.

Why is it important for LLMs to generate accurate SQL Queries?

For context, I’m building NexusTrade, a platform to help investors create automated investing strategies and perform financial research.

NexusTrade AI Chat - Talk with Aurora

Talk to an AI assistant to perform financial analysis and create algorithmic trading strategies.

nexustrade.io

One of the functionalities is configuring the model in the AI chat. Prior to this, I allowed 3 models: Gemini Flash 2, Gemini Flash 2.5, and Gemini 2.5 Pro.

**Figure 6:** The previous NexusTrade UI showed 3 models in the Advanced Chat Settings, all in the Gemini family. Gemini 2.5 Flash was much more expensive

The reported median judge score increased from 0.567 for Gemini Flash 2 to 0.900 for Gemini Flash 2.5. Those are quality scores in this one-shot SQL test; they are not 56.7% and 90.0% investing success rates.

However, to access this model, users had to pay 5x the “research tokens” within the app. This made this model completely inaccessible to free users. I decided that was unacceptable.

I removed Gemini Flash 2 from the model list and replaced it with Gemini Flash 2.5. As a result, my users will get a much better user experience at only a marginal increase in cost to me. This is insanely useful feedback. And it only cost me $200 to get it 😉.

**Figure 7:** The new simplified UI shows “General Purpose” which is Gemini 2.5 Flash and Highly Advanced which is Gemini 2.5 Pro. The simplified UI should greatly benefit users

How to take advantage of this?

If you’re an investor looking to make smarter investing decisions, NexusTrade is built with the most powerful LLMs to make it possible!

To start, NexusTrade’s financial analyses aren’t as simplistic as the benchmark. Each query goes through the following process.

**Figure 8**: How NexusTrade Fetches Accurate Financial Information Using Language Models

A user submits a financial question to the model that we’re testing
The LLM generates the SQL Query that can answer this question
The SQL Query is executed against the database and fetch the results
The Query and the results are fed into the same model and we generate a score and a score explanation
We repeat until the score is higher than our minimum score threshold

The production flow can revise a query after an execution or judge failure. A self-grading loop still needs inspection of the requested financial meaning, source dates and returned rows. The one-shot benchmark does not measure the accuracy or investing returns of that full loop.

NexusTrade Onboarding - Get Started in 3 Steps

Get started with automated trading in just 3 quick steps. Connect your broker, choose a strategy, and start trading…

nexustrade.io

Concluding Thoughts

With all of the hype around Grok 4, I would’ve expected an insane leap for this task.

Sadly, we didn’t get that.

These results show the Benchmark Fallacy in action. To summarize:

According to the data, Grok 4 is a mediocre model for SQL Query generation, particularly for its price
Google Gemini 2.5 Pro is cheaper (in terms of inputs and outputs), faster, and has a higher accuracy
Even OpenAI’s o4-mini is a better model in nearly every regard.

Grok 4 simply doesn’t live up to the hype. At least in this task.

This article shows the importance of creating your own custom benchmarks. While I wasn’t able to justify adding Grok 4 into my algorithmic trading platform NexusTrade, I used the results of these benchmarks to improve the platform, making it far better with only a marginal increase in cost to myself. Users will now enjoy a much better more accurate experience.

What has your experience been with Grok 4? I’d love to hear how it performs on other tasks, whether its SQL Query Generation, coding, or any other real-world use-cases!