TL;DR
An LLM scoring old news may already know what happened next. That could make a sentiment backtest look better than live trading.
I scored 10 well-known headlines, each with its summary, using Claude Haiku 5.5. Each was scored 3 times as written and 3 times with the company masked. The masking gap averaged 0.15, about 50% larger than the 0.10 average rerun spread. Both peaked at 0.3.
Masking cannot separate hindsight from information loss at this sample size. Excluding ties, 6 of 8 masked means sat closer to zero, a one-sided sign test p of 0.14.
I label headlines as written. Masking deleted real information and left famous stories recognizable, so it is an unreliable fix. The test that settles it scores similar headlines from before and after the model's training cutoff and compares how well each set predicts returns. That test is planned.
Why hindsight could leak into a sentiment score
A model trained on text written after March 2023 has likely read about SVB's collapse. A backtest asks it to score SVB's Mar 8, 2023 headline as if that ending were unknown. In live trading, the ending is still in the future.
The model card I used for Claude Haiku 5.5 publishes no training cutoff, so I cannot say which of these ten stories it saw.
How the test hid the company
Claude Haiku 5.5 rewrote each headline and summary with the target company as Company A, other companies as Company B and onward, and dates and identifying names removed. The same model then scored both versions. The masked prompt named only the sector, such as “Company A, a large-cap financials company”.
Each version was scored 3 times on a 21-point scale from −1.0 to +1.0. The model saw the headline and its summary every time.
How I ran the test
The full scoring and masking prompts are below the results table.
- Model
- Claude Haiku 5.5 (openrouter::anthropic/claude-haiku-5.5)
- Routing
- NexusGenAI batch gateway, through OpenRouter
- Temperature
- Gateway default. The script set no temperature.
- Input
- Target company line, headline and summary
- Scale
- 21 values from −1.0 to +1.0 in steps of 0.1, enforced by the response schema
- Runs
- 3 per version, as written and masked
- Masker
- Claude Haiku 5.5, one rewrite per headline
SVB is the case that looks most like leakage
The SVB rewrite changed only names: SVB became Company A and General Atlantic became Company B. Every amount stayed. As written, the runs scored −0.8, −0.7, −0.9. Masked, all 3 scored −0.5.
That 0.3 gap is larger than the 0.2 spread of the as-written runs, and it points toward the ending: the stock fell ~60% the next session. Hindsight would produce exactly this. With one case and 3 runs, it is also within reach of chance.
Ten headlines, scored as written and masked
Read the explanation
I count a stock as doing the opposite of its text when the as-written mean and the stock's first full session after publication point in opposite directions. Hindsight would show most clearly in those cases.
Four qualify: Bear Stearns, First Republic, Credit Suisse and NVIDIA. First Republic's headline reports a rally already under way on Mar 16, so its first full session is Mar 17, when it fell ~33%. The Credit Suisse item was published at 00:43 UTC on Mar 16, 2023 (8:43 pm ET on Mar 15), so its first full session is Mar 16, when it rose ~19%. Bear Stearns has no recorded next-session move, so I use the next move the experiment recorded: emergency Fed-backed funding two days later.
All 12 as-written runs for those four stayed on the text's side of zero.
Hindsight can also leak through size. First Republic scored +0.5 as written and +0.67 masked: less bullish before a failure. NVIDIA scored −0.17 as written and −0.2 masked: less bearish before a 24% rise. Both shifts point toward the outcome, and both sit inside their own rerun spreads (0.3 and 0.1). Credit Suisse moved the other way, −0.8 as written against −0.6 masked, and Bear Stearns scored +0.2 both ways.
Across all ten, the masked mean sat closer to zero in 6 cases, farther in 2 and level in 2. Excluding the ties, 6 of 8 gives a one-sided sign test p of 0.14. Hindsight would push scores that way, and so does removing detail.
Masking also removed information
Tesla scored +0.5 on all three runs as written. The masked text said “a major stock index” instead of “S&P 500” and scored +0.2 on all three. In the third run, the as-written reasoning cited “forced buying from index funds”, and the masked reasoning described “a modest, temporary price bump”.
Masking left famous stories recognizable
The Boeing rewrite still read “killing 157 aboard Company A 737” and named the 737 MAX 8. The Cambridge Analytica rewrite kept “50 million Company A profiles harvested” for a voter-profiling system. The Bear Stearns summary kept the CEO's $17 billion liquidity quote word for word, under “Company A, a large-cap financials company”.
A reader who knows these stories can name the companies, and so can a model. The rewrites also came from Claude Haiku 5.5, the model under test.
Masking gap versus rerun noise
Shift toward zero: the as-written mean's distance from zero minus the masked mean's, 3 runs each. A positive bar means the masked score sat closer to zero. Rerun spread: for each headline, the larger of its two versions' highest minus lowest run. One step on the scale is 0.1.
- Mean masking gap
- 0.15
- Mean rerun spread
- 0.10
- Largest masking gap
- 0.3
- Largest rerun spread
- 0.3
Masking gap: for each headline, the distance between the masked mean and the as-written mean. Rerun spread: for each headline, the larger of its two versions' highest minus lowest run. Both are one number per headline.
The mean gap, 0.15, is about 50% larger than the mean rerun spread, 0.10. The largest gap and the largest spread are both 0.3. At 10 headlines, that difference is inconclusive.
What I decided, and the test that settles it
Masking cannot separate hindsight from information loss at this sample size.
I label headlines as written. Masking demonstrably destroys information: Tesla fell from +0.5 to +0.2 when “S&P 500” became “a major stock index”. It also left famous stories recognizable, so it is an unreliable fix for hindsight.
The test that settles it scores comparable headlines published before and after the model's training cutoff, then compares how well each set predicts returns. For news from July 2024 onward, I can also compare against Massive's contemporaneous sentiment labels. That test is planned and has yet to run.
Every headline, summary, rewrite and score
Show all 10 headlines, summaries, rewrites and runs
Scroll sideways to compare all columns. Select a heading to sort.
| Mar 12, 2008 | Bear Stearns (BSC) | CNBC | Bear Stearns CEO: No Liquidity Crisis for Firm | "We finished the year, and we reported that we had $17 billion of cash sitting at the bank's parent company as a liquidity cushion," the CEO said. "As the year has gone on, that liquidity cushion has been virtually unchanged." | Company A CEO: No Liquidity Crisis for Firm | "We finished the year, and we reported that we had $17 billion of cash sitting at the bank's parent company as a liquidity cushion," the CEO said. "As the year has gone on, that liquidity cushion has been virtually unchanged." | +0.2, +0.2, +0.2 | +0.2, +0.2, +0.2 | No next-session move recorded; emergency Fed-backed funding two days later | Yes | Needed emergency Fed-backed funding two days later and was sold for $2 a share that weekend. |
| Mar 17, 2018 | Facebook (FB) | The Guardian (headline); summary written from the article's reported facts | Revealed: 50 million Facebook profiles harvested for Cambridge Analytica in major data breach | A whistleblower says Cambridge Analytica took the Facebook data of millions of US voters without authorisation in early 2014 to build a voter-profiling system. Documents show Facebook learned of the harvesting by late 2015. | Revealed: 50 million Company A profiles harvested for Company B in major data breach | A whistleblower says Company B took the Company A data of millions of voters in one country without authorisation to build a voter-profiling system. Documents show Company A learned of the harvesting by a later point in time. | −0.7, −0.7, −0.7 | −0.5, −0.5, −0.5 | −7% next session | No | Fell ~7% the next session and kept falling for weeks. |
| Mar 10, 2019 | Boeing (BA) | Reuters | Ethiopian Airlines flight crashes, killing 157 aboard Boeing 737 | The plane was a new Boeing 737 MAX 8, the same model that crashed in Indonesia in October. It went down minutes after leaving Addis Ababa in clear weather. | Company B flight crashes, killing 157 aboard Company A 737 | The plane was a new Company A 737 MAX 8, the same model that crashed earlier in Asia. It went down minutes after leaving the capital city in clear weather. | −0.7, −0.8, −0.7 | −0.5, −0.5, −0.6 | Fell; 737 MAX grounded | No | Fell sharply the next sessions; the 737 MAX was grounded worldwide within days. |
| Apr 2, 2020 | Luckin Coffee (LK) | Paraphrase of the company's disclosure (no exact headline verified) | Luckin Coffee says internal probe found its COO fabricated about $310 million of 2019 sales | The company said investors should no longer rely on its previous financial statements for the nine months ended September 30, 2019. The investigation is at a preliminary stage. | Company A says internal probe found its COO fabricated about $310 million of prior-year sales | The company said investors should no longer rely on its previous financial statements for the nine months ended the third quarter of the prior year. The investigation is at a preliminary stage. | −1.0, −1.0, −1.0 | −0.9, −1.0, −0.9 | −80% same day | No | Fell ~80% the same day; later delisted. |
| Nov 16, 2020 | Tesla (TSLA) | S&P Dow Jones Indices press release | Tesla Set to Join S&P 500 | Tesla Inc. will be added to the S&P 500 effective prior to the open of trading on Monday, December 21, replacing a company to be named closer to the date. | Company A Set to Join Major Stock Index | Company A will be added to a major stock index effective prior to the open of trading on a Monday, replacing a company to be named closer to the date. | +0.5, +0.5, +0.5 | +0.2, +0.2, +0.2 | +8% next session | No | Rose ~8% the next session and ~70% into the inclusion date. |
| Feb 2, 2022 | Meta Platforms (Facebook) (FB) | Quartz (via Massive) | Facebook lost daily users for the first time ever | The metaverse is the future of Facebook and just in time—because its main app is plateauing. | Company A lost daily users for the first time ever | The metaverse is the future of Company A, and just in time, because its main app is plateauing. | −0.5, −0.5, −0.5 | −0.5, −0.5, −0.5 | −26% next session | No | Fell ~26% the next session. |
| Mar 8, 2023 | SVB Financial Group (SIVB) | Company press release | SVB Financial Group Announces Proposed Offerings of Common Stock and Mandatory Convertible Preferred Stock | SVB plans to sell $1.25 billion of common stock and $500 million of mandatory convertible preferred stock, and General Atlantic agreed to buy $500 million of common stock. SVB also sold substantially all of its available-for-sale securities, about $21 billion, for an expected after-tax loss of about $1.8 billion. | Company A Announces Proposed Offerings of Common Stock and Mandatory Convertible Preferred Stock | Company A plans to sell $1.25 billion of common stock and $500 million of mandatory convertible preferred stock, and Company B agreed to buy $500 million of common stock. Company A also sold substantially all of its available-for-sale securities, about $21 billion, for an expected after-tax loss of about $1.8 billion. | −0.8, −0.7, −0.9 | −0.5, −0.5, −0.5 | −60% next session | No | Fell ~60% the next session; the bank was seized two days later. |
| Mar 16, 2023 | First Republic Bank (FRC) | Benzinga (via Massive) | First Republic Rallies As Major US Banks Pledge Assistance To Calm Markets | First Republic Bank may get a substantial capital injection or perhaps a full acquisition from a consortium of large financial institutions, including JP Morgan Chase, Morgan Stanley, Citigroup, Bank of America and Wells Fargo. The deposit influx from the group of large banks is reportedly worth around $20 billion. | Company A Rallies As Major Banks Pledge Assistance To Calm Markets | Company A may get a substantial capital injection or perhaps a full acquisition from a consortium of large financial institutions, including Company B, Company C, Company D, Company E and Company F. The deposit influx from the group of large banks is reportedly worth around $20 billion. | +0.5, +0.5, +0.5 | +0.5, +0.7, +0.8 | Mar 17 (the headline reports a rally already under way on Mar 16): fell ~33% | Yes | Closed up ~10% that day, fell ~33% the next, and failed on May 1. |
| Mar 16, 2023, 00:43 UTC on Mar 16, 2023 (8:43 pm ET on Mar 15) | Credit Suisse (CS) | MarketWatch (via Massive) | Credit Suisse to borrow about $54 billion from Swiss central bank | Credit Suisse announced late Wednesday it was taking "decisive action" and borrowing up to 50 billion Swiss francs -- about $54 billion -- to ease investor fears. | Company A to borrow about $54 billion from a central bank | Company A announced late in the day it was taking "decisive action" and borrowing up to 50 billion local-currency units -- about $54 billion -- to ease investor fears. | −0.8, −0.7, −0.9 | −0.5, −0.7, −0.6 | Mar 16: rose ~19% | Yes | Rose ~19% the next session, then was forced into a takeover by UBS three days later. |
| May 24, 2023 | NVIDIA (NVDA) | GlobeNewswire (via Massive) | NVIDIA Announces Financial Results for First Quarter Fiscal 2024 | NVIDIA today reported revenue for the first quarter ended April 30, 2023, of $7.19 billion, down 13% from a year ago and up 19% from the previous quarter. | Company A Announces Financial Results for First Fiscal Quarter | Company A today reported revenue for its first fiscal quarter of $7.19 billion, down 13% from a year earlier and up 19% from the previous quarter. | −0.2, −0.2, −0.1 | −0.2, −0.2, −0.2 | +24% next session | Yes | Rose ~24% the next session on guidance the summary does not mention. |
No matching rows. Clear the filter to see all records.
Scores from Claude Haiku 5.5, 3 runs per version, from −1.0 to +1.0 in steps of 0.1. Each run saw the headline and the summary. Opposite of the text: the as-written mean and the first full session point in opposite directions. Outcomes are the approximate moves recorded with the experiment.
Scoring prompt
Show the full scoring prompt
You score how a news item affects the TARGET company's stock, from the point of view of an investor reading it at the moment it was published.
Use only the text given. Do not use anything you know about what happened later.
Return "value", one of -1.0, -0.9, ..., 0.9, 1.0:
- -1.0: existential (fraud, bankruptcy, failure of the core business)
- -0.5: clearly bad, material
- -0.2: mildly bad
- 0.0: neutral or not material
- 0.2: mildly good
- 0.5: clearly good, material
- 1.0: transformational (takeover at a large premium, a breakthrough that changes the business)
Write one or two sentences of "reasoning" first, then the value.
[user message]
TARGET company: <company (ticker), or “Company A, <sector>” when masked>
Headline: <headline>
Summary: <summary>Masking prompt
Show the full masking prompt
You anonymize financial news for a research study.
Rewrite the headline and summary so that no reader could identify any company or the date.
- Replace the TARGET company, its brands, products, subsidiaries, executives and ticker with "Company A".
- Replace every other company with "Company B", "Company C", and so on, consistently.
- Replace people's names with their role ("the CEO", "a whistleblower"), and remove or generalize anything that names a specific famous event, product line, regulator country or index that would identify the company (for example "a major stock index" instead of a named index, "a central bank" instead of a named one).
- Remove all dates, years and weekday names.
- Keep every fact that matters to an investor: amounts, percentages, direction of change, what happened.
Return JSON with "title" and "description".
[user message]
TARGET company: <company (ticker)>
Headline: <headline>
Summary: <summary>Limits
This is a small test: 10 headlines, 3 runs per version and one model, which also wrote the masked versions. The temperature was the gateway default.
The Luckin Coffee headline is a paraphrase of the company's disclosure, and the Cambridge Analytica summary was written from the Guardian article's reported facts. Outcomes are approximate moves.
No comments yet.