Selection bias in strategy research: keep the losers in the record

Follow the example

Choose a step to follow the example. Opening a request lets you review it; it does not submit a job or change your account.

Step 1 of 3

Write down the selection process

Record which assets, rule families, thresholds and periods were considered. The archive contains completed submissions, not every rejected idea or failed launch.

Available source runs

Recorded count: 222010 · Interpretation: The retained archive, not the live database

Optimizer-labeled source runs

Recorded count: 74518 · Interpretation: Recorded label across the source; excluded from this ranking cohort

Eligible daily stock runs

Recorded count: 102781 · Interpretation: After asset, cadence, duration, origin-label and metric filters

Incomplete or unsupported records excluded

Recorded count: 12233 · Interpretation: Typed normalization rejects missing or unsupported execution inputs

What you should see

A disclosed search universe and an unavailable failure denominator.

Step 2 of 3

Read count beside the score

Compare support, spread and losses across the same normalized population. Overlapping groups and historical winners do not isolate a causal effect.

What you should see

A selection-aware interpretation of aggregate evidence instead of a leaderboard treated as a future guarantee.

Step 3 of 3

Review the hypothesis before changing it

Ask Aurora to list the choices that informed your candidate and preview one frozen validation plan. Include restarts and rewrites rather than resetting the trial count.

What you should see

An auditable hypothesis and selection ledger, with no automatic research job.

Review this example in Aurora →

Audit the population behind the published corpus

The available archive contains 222,010 recorded runs, with its newest record dated 2026-09-06. The October reanalysis selects a daily stock cohort without options, excludes optimizer-labeled runs and keeps finite-Sharpe records lasting 365 through 2,000 days.

The selected records do not cover every attempted strategy, and the archive does not report how many jobs failed. Author origin is unknown. The archive does not identify which optimizer candidates were chosen before a later test, so it cannot establish optimizer superiority or clean unseen performance.

No matching rows. Clear the filter to see all records.

See how choosing the start date changes the story

The same five-dividend-grower rules beat the stored SPY return in the 2025 window and lagged it in the longer 2014 window. Publishing only the recent curve would hide that difference. These inspected periods are historical examples, not an untouched final test or proof of overfitting.

Each run starts with $10,000 and reinvests dividends. The strategy pays 0.1% stock transaction fees; the stored SPY baseline pays no candidate commissions. Historical point-in-time SP500 membership is unverified. Different assets, costs and exposure limit the comparison.

No matching rows. Clear the filter to see all records.

Audit the contest before copying a finalist

Open the actual completed study in Optimization and inspect every available candidate and fold, not just the first card. Save the original registry and the complete full-field portfolios. Separate candidates that completed and lost from jobs that failed operationally. If the tool output contains only a truncated leaderboard, retain the truncation and do not call it the full trial set.

For each finalist, record which validation window selected it and which outer windows were used to evaluate it. Then record any cross-fold summary used to choose the final assembled portfolio. That final choice needs its own untouched test because it used earlier outcomes.

The winner needs the size of the contest

A result selected because it was the strongest among many trials is harder to interpret than a predeclared single-rule test. The selection can happen through an optimizer, an agent conversation, manual parameter changes, asset screening or which chart gets published.

Record every evaluated candidate, including failed runs that produced usable evidence. A new prompt, random seed or renamed study does not erase the other attempts. Operationally failed jobs should be identified separately from strategies that completed and lost.

Separate the fold choice from the hindsight summary

NexusTrade’s walk-forward runner selects a fold candidate from validation statistics before promoting that candidate into the later outer test. A cross-fold retrospective summary may inspect all completed outer windows. Using that summary to pick a final book introduces another selection decision.

Do not report the retrospectively strongest candidate as though that exact choice had been made before the first test window. Freeze the assembled portfolio and evaluate it in a separate final holdout. If no candidate satisfies configured floors, show the fallback warning rather than certifying the best of the failures.

Publish the selection ledger

Save this ledger before publishing a winner. If earlier trials are unavailable, call the search history incomplete. A polished table of finalists cannot establish a complete search history.

candidate | first declared | trial count | selection window | test window | failed floors | included in report
[all candidates, including losses and rejected variants]
final choice | exact rules hash | information used | independent final holdout

Check who disappeared from the universe

A current list of successful companies can exclude delisted or failed names that were eligible at the historical start. An eligible-as-of universe needs a defined membership source and dates. A backtest over a current survivor list does not establish protection against survivorship bias.

Keep asset exclusions, missing-data policies and delisting handling explicit. Excluding a name because it lacked necessary observations is different from excluding it because its completed return was poor; both need an auditable reason.

Continue exploring