TL;DR
Published 2024-06-25. GPT-4o returned 5 companies in 26 seconds. Claude 3.5 Sonnet returned no response because its query never terminated.
- Published
- 2024-06-25
- Published model rows
- 2
- Evidence
- Historical task evaluation
Follow this evaluation
Published results
Claude 3.5 Sonnet and GPT-4o turning a screen into a MongoDB query: source evidence dated 2024-06-25.
- Exact model version
- GPT-4o
- Response
- Returned 5 companies
- Time to response
- 26s
The displayed values belong to the identified model versions and this published task, with their original metric units.
What did this task require the model to return?
Read the tested version, task and rubric together. Check failures alongside successful answers; compare costs only within the reported unit.
2024-06-25: Claude 3.5 Sonnet and GPT-4o turning a screen into a MongoDB queryKeep the study date and exact model ID beside the result. Do not average different tasks into one winner.
Test a strategy with Aurora
Keep the study date and exact model ID beside the result. Do not average different tasks into one winner.
Test a strategy with AuroraKeep reporting dates and units beside the answer, and inspect its original citations.
Published results
See the complete published results
Scroll sideways to compare all columns. Select a heading to sort.
| GPT-4o | Returned 5 companies | 26s |
| Claude 3.5 Sonnet | No response | — |
No matching rows. Clear the filter to see all records.
1 screening request, 2 models, single run each. Published 2024-06-25. Values retain their original units and grading scale.
Task and tested input
Translate a plain-English stock screen into a MongoDB query, execute it and summarize the matching companies.
Published prompt excerpt
find the 5 companies with the highest cash flow increase from 2016 to 2020
How the runs worked
1 screening request, 2 models, single run each
NexusTrade Aurora beta stock screener, which translates plain English into a MongoDB query and executes it
Step 1
Send the screening request to GPT-4o-powered Aurora and record the response and its time.
Step 2
Send the same request to Claude 3.5 Sonnet-powered Aurora.
Step 3
Record that Claude's generated query never terminated and was killed manually in the cloud console.
How to read the measurements
Grading and valid output
Response or no response only. The article did not verify GPT-4o's returned figures.
Cost
The article publishes no per-model cost for this test.
Time
GPT-4o responded in 26 seconds. Claude 3.5 Sonnet never responded.
Why this model was selected
The article named GPT-4o the winner of this test.
See the published results
This image was published with the 2024-06-25 article. Read the source for the full test procedure and every result it reports.
Sample and scope
June 2024 single-request comparison on a feature the article labels beta. Both models have since been replaced.
GPT-4o's answer was not checked for accuracy. The article says some results, such as Onconova Therapeutics showing $0 free cash flow in 2016, look a little sketchy.
Claude's failure came from a generated query that never terminated and had to be killed manually, so the result reflects the screener's lack of a query timeout as well as the model.
No comments yet.