TL;DR
Published 2025-01-21. On question 1, R1 reported 90 qualifying 7-day drops and o1 reported 150, and the article judged both correct. Both reported an average 365-day return of 20.29% after the drops. R1's list price was $0.55/M input and $2.19/M output against o1's $15.00/M and $60.00/M.
- Published
- 2025-01-21
- Published model rows
- 2
- Evidence
- Historical task evaluation
Follow this evaluation
Published results
DeepSeek R1 and OpenAI o1 answering SPY drawdown questions with SQL: source evidence dated 2025-01-21.
- Exact model version
- DeepSeek R1
- Q1 drop count
- 90
- Q1 verdict
- Correct
- Q2 avg 180d drawdown
- -16.69%
- Q2 avg 365d drawdown
- -19.33%
- Q3 avg 180d return
- +8.58%
- Q3 avg 365d return
- +20.29%
- Input price
- $0.55/M
The displayed values belong to the identified model versions and this published task, with their original metric units.
What did this task require the model to return?
Read the tested version, task and rubric together. Check failures alongside successful answers; compare costs only within the reported unit.
2025-01-21: DeepSeek R1 and OpenAI o1 answering SPY drawdown questions with SQLKeep the study date and exact model ID beside the result. Do not average different tasks into one winner.
Test a strategy with Aurora
Keep the study date and exact model ID beside the result. Do not average different tasks into one winner.
Test a strategy with AuroraKeep reporting dates and units beside the answer, and inspect its original citations.
Published results
3 chained SPY research questions in one conversation per model, 2 models, single run each. Published 2025-01-21. Values retain their original units and grading scale.
See the complete published results
Scroll sideways to compare all columns. Select a heading to sort.
| DeepSeek R1 | 90 | Correct | -16.69% | -19.33% | +8.58% | +20.29% | $0.55/M | $2.19/M |
| OpenAI o1 | 150 | Correct | -16.74% | -19.14% | 8.58% | 20.29% | $15.00/M | $60.00/M |
No matching rows. Clear the filter to see all records.
3 chained SPY research questions in one conversation per model, 2 models, single run each. Published 2025-01-21. Values retain their original units and grading scale.
Task and tested input
Answer multi-step historical market questions about SPY by generating SQL, executing it and summarizing the results.
Published prompt excerpt
Since Jan 1st 2000, how many times has SPY fallen 5% in a 7-day period? In other words, at time t, how many times has the percent return at time (t + 7 days) been -5% or more. Note, I'm asking 7 calendar days, not 7 trading days. In the results, include the data ranges of these drops and show the percent return. Also, format these results in a markdown table.
How the runs worked
3 chained SPY research questions in one conversation per model, 2 models, single run each
NexusTrade Aurora text-to-SQL research pipeline (query generated, executed, results returned to the model for summarization)
Step 1
Ask each model the 7-calendar-day SPY drop question in NexusTrade Aurora.
Step 2
Follow up in the same conversation with the average 180-day and 365-day max drawdown question.
Step 3
Follow up with the average 180-day and 365-day return question, then compare the answers and generated SQL side by side.
How to read the measurements
Grading and valid output
Author inspection only. Question 1 was judged 100% correct for both models, with o1 described as a little better for including a total-occurrences field and an untruncated table. Question 2 favored R1 for including a ratio. Question 3 was described as almost exactly alike.
Cost
List prices quoted in the article: R1 $0.55/M input and $2.19/M output tokens. O1 $15.00/M input and $60.00/M output tokens. No per-question spend is published.
Time
The article publishes no per-model timing.
Why this model was selected
The article switched the default model for all NexusTrade users to DeepSeek R1, citing comparable answers at about 2% of o1's price.
See the published results
This image was published with the 2025-01-21 article. Read the source for the full test procedure and every result it reports.
Sample and scope
January 2025 single-conversation comparison with no repeated runs. DeepSeek R1 and OpenAI o1 have since been replaced.
The article calls both question 1 answers 100% correct, but the screenshots report different counts (R1: 90 total instances over a period labeled January 2000 to August 2015. O1: 150 total occurrences), and the article does not reconcile them. Questions 2 and 3 were compared qualitatively, with no correctness grade.
Prices are list prices quoted in the article, not measured spend. The article also reports that R1 sometimes generated invalid SQL that the app retried and once timed out, and that a portfolio-generation prompt succeeded for R1. O1 was not shown on that prompt.
No comments yet.