TL;DR
Published 2024-07-19. GPT-4o mini passed, generating a valid portfolio that passed a backtest configuration, while GPT-3.5 Turbo failed after running out of context.
- Published
- 2024-07-19
- Published model rows
- 2
- Evidence
- Historical task evaluation
Follow this evaluation
Published results
GPT-4o mini and GPT-3.5 Turbo generating a portfolio JSON object: source evidence dated 2024-07-19.
- GPT-4o mini
- Pass
- GPT-3.5 Turbo
- Fail
The displayed values belong to the identified model versions and this published task, with their original metric units.
What did this task require the model to return?
Read the tested version, task and rubric together. Check failures alongside successful answers; compare costs only within the reported unit.
2024-07-19: GPT-4o mini and GPT-3.5 Turbo generating a portfolio JSON objectKeep the study date and exact model ID beside the result. Do not average different tasks into one winner.
Test a strategy with Aurora
Keep the study date and exact model ID beside the result. Do not average different tasks into one winner.
Test a strategy with AuroraKeep reporting dates and units beside the answer, and inspect its original citations.
Published results
See the complete published results
| Exact model version | Result |
|---|---|
| GPT-4o mini | Pass |
| GPT-3.5 Turbo | Fail |
No matching rows. Clear the filter to see all records.
1 portfolio JSON task, 2 models, prompt count and run count not published in this excerpt. Published 2024-07-19. Values retain their original units and grading scale.
Task and tested input
Generate a syntactically valid portfolio JSON object for an algorithmic trading strategy that can be used downstream.
Instruction in brief (task summary)
Generate a syntactically valid portfolio JSON object that is used downstream to check whether it was generated appropriately. The excerpt does not publish the exact request.
How the runs worked
1 portfolio JSON task, 2 models, prompt count and run count not published in this excerpt
NexusTrade AI trading platform, with the generated portfolio JSON used downstream for a backtest
Step 1
Give each model the same portfolio generation request in NexusTrade.
Step 2
Use the generated JSON object downstream, including a backtest configuration.
Step 3
Record whether a valid object was produced.
How to read the measurements
Grading and valid output
Author judgment: a pass means a valid portfolio object that worked downstream. A fail means no valid object.
Cost
The article publishes no per-model cost for this test.
Time
The article publishes no per-model timing.
Why this model was selected
The article reported GPT-4o mini succeeding where GPT-3.5 Turbo failed on both July 2024 tasks.
Sample and scope
July 2024 summary excerpt. The request and generated JSON are not published on this page, and both models have since been replaced.
Pass or fail is the author's judgment of a single reported outcome per model, not a success rate over a request set.
GPT-3.5 Turbo's failure is attributed to running out of context, so the result reflects context length as well as reasoning.
No comments yet.