TL;DR
Our news classifier matched the saved company lists for 421 of 439 eligible GDELT headlines, or 95.90%. But it matched only 7 of the 20 headlines whose saved answers contained a tracked stock. The other 419 answers were empty, so saying "none" often looked correct.
Always saying "none" would score 95.44% overall and miss all 20 headlines with a tracked asset. A useful evaluation shows those counts beside accuracy. These references contain known errors, so the results measure agreement with the saved answer key.
Which headlines should return a stock?
Read the explanation
AAPL is Apple's stock-market code, called a ticker. For the real Apple earnings headline in this animation, the saved model output and reference both contain AAPL within the nine-security teaching test. For the local Hamilton traffic headline, both saved outputs are empty: []. These two examples explain what a match means.
The company example is the saved Zacks Apple headline. It is separate from the later GDELT evaluation. The real traffic example is "Rush-hour crash on key Hamilton route slows traffic". Its June 15, 2021 timestamp is when GDELT observed the record; the publisher's publication time is unknown in our saved sample.
In the measured GDELT test, we tracked 100 stocks plus SPY, a fund representing the broad US stock market. A correct answer had to match the complete saved ticker list. Seven matches out of 20 nonempty lists gives 35% agreement on that group. That metric asks whether the entire list matched; it is different from counting individual tickers recovered.
An empty answer is limited to the assets in the benchmark. News about another company, oil or a sector can matter financially even when this answer key contains no tracked ticker.
The arithmetic behind the impressive number
The broad probe contained 500 headlines. Of those, 439 had references marked clear, complete and inside our 100-stock-plus-SPY benchmark. The remaining 61 had unresolved identities, outside-benchmark targets or other excluded status combinations.
Within the 439 eligible headlines, 419 had empty reference arrays and 20 had one or more targets. The system correctly returned an empty array for 414 of the 419 empty cases. It matched the complete ticker arrays on seven of the 20 relevant cases.
That gives 421 correct arrays out of 439: (414 + 7) / 439 = 95.90%. The relevant-only result is 7 / 20 = 35%. Both calculations describe the same saved predictions.
A system that returned an empty array for every eligible headline would score 419 / 439 = 95.44% overall and recover zero relevant stories. Our measured system exceeded that overall agreement by about 0.46 percentage points.
The test denominator
439 eligible headlines = 419 empty reference arrays + 20 nonempty reference arrays.
The 20 relevant headlines contribute about 4.6% of the overall denominator.
The measured pipeline
414 correct empty arrays + 7 correct nonempty arrays = 421 correct arrays.
421 / 439 = 95.90% overall agreement. The relevant-only result is 7 / 20 = 35.00%.
The always-empty comparator
Return [] for all 439 eligible headlines. All 419 empty references match; all 20 relevant references are missed.
419 / 439 = 95.44% overall agreement, with zero relevant stories recovered.
Show the table · 3 rows, 4 columns
Scroll sideways to compare all columns. Select a heading to sort.
| Empty within the benchmark | 419 | 414 | 98.81% |
| One or more benchmark targets | 20 | 7 | 35.00% |
| Combined eligible population | 439 | 421 | 95.90% |
No matching rows. Clear the filter to see all records.
Exact arrays require the full predicted security set to equal the saved reference set. These labels include known errors discussed below.
What the relevant-news misses contained
Twelve of the 20 relevant references received an empty prediction. The other failed exact-array case named Berkshire Class B, and the system returned both Berkshire share classes. Finding the issuer still failed the exact-security contract.
Several empty predictions were clear coverage problems: Sam's Club to Walmart, Hummer to GM, and Cash App to Block. Others relied on entertainment-property relationships or macroeconomic interpretation. Their difficulty and relevance differ, which is why a cohort breakdown is useful.
Across all known associations, including known targets on partially annotated headlines, core target recall was 10 of 13 and contextual target recall was 5 of 20. Those denominators differ from the 20 complete relevant headlines. Keeping the populations explicit prevents a second misleading comparison.
High sentiment agreement can also cover a minority of targets
A separate Massive test of the same 100-stock-plus-SPY pipeline had 79.68% raw sentiment direction agreement on 443 clear reference pairs. With the confidence filter, 207 of 212 emitted scores agreed with their references, or 97.64%. Those 212 scores covered 41.41% of all 512 reference targets, leaving 300 without an emitted score.
For a strategy, the operational question includes how often a score exists when the rule needs it. A high-confidence score on a minority of articles creates different behavior from a score available on nearly every target.
The same filtered result recovered 31.82% of negative reference signals and 57.38% of positive reference signals. Those differences can matter to a rule designed to avoid bad news. A single combined agreement percentage hides the imbalance.
These sentiment figures were measured given reference target companies, independently of successful ticker extraction. An operational news pipeline must also recover the ticker before it can emit that ticker's score.
Start with 512 reference targets
The confidence rule emits a sentiment score for 212 of them.
The rule leaves 300 targets without an emitted score.
Inspect the 212 emitted scores
207 agree with their sentiment references; 5 do not.
Agreement is 207 / 212. Coverage is 212 / 512. The two percentages answer different questions.
Show the table · 6 rows, 3 columns
Scroll sideways to compare all columns. Select a heading to sort.
| All reference target pairs | 512 | Denominator for coverage |
| Clear sentiment reference pairs | 443 | Denominator for raw clear-direction agreement |
| Uncertain sentiment reference pairs | 69 | Counted as disagreement if a score is emitted |
| Scores emitted at the fixed cutoff | 212 | 212 / 512 = 41.41% coverage |
| Emitted scores agreeing with references | 207 | 207 / 212 = 97.64% agreement |
| Reference targets without an emitted score | 300 | 512 - 212 |
No matching rows. Clear the filter to see all records.
These are conditional sentiment results given reference target companies, across all known reference pairs. They do not assert successful ticker extraction on those pairs.
Review disagreement before treating it as model failure
The GDELT audit found a helicopter-crash headline labeled Berkshire Hathaway, with the reference explanation "Unrelated to this headline." Another reference linked Jerry Harris to Netflix, but the cached repeat returned an empty array.
One predicted SPY association counted as a false positive because the reference was empty: a headline about a falling dollar, weak US data and an impeachment inquiry. Under our intended economic-news policy, that case deserves review.
The target definition also limits the benchmark. An empty array among 100 stocks plus SPY can coexist with relevant news about another listed company, a commodity or a sector ETF. Empty within that universe cannot stand in for irrelevant to every tradable asset.
The right response is to preserve the original result, document disputed cases, and establish a reviewed reference policy for the next test. Quietly changing the labels after seeing predictions would make progress difficult to interpret.
The measurements I would want beside accuracy
Precision, recall and F-measures describe different aspects of classification; the definitions are collected in scikit-learn's model-evaluation documentation.
For this product, relevant-headline exact arrays show how often the full target list is right. Target recall shows how many supported associations survive. False nonempty outputs show how often unrelated stories acquire an asset. These answer different operational questions.
Sentiment direction agreement belongs beside emission coverage and separate negative, neutral and positive counts. The complete pipeline also needs a count of correct ticker-and-direction outputs, because a correct score attached to the wrong asset is unusable.
For repeatable comparisons, preserve the same article groups and reference definitions. Select thresholds on validation examples, then score the held-out examples. A new archive or publisher mix needs a separate source evaluation even when the earlier benchmark looks strong.
Turn missing coverage into explicit strategy behavior
A future news-driven backtest needs a declared rule for absent scores. It could leave a position unchanged, block an entry, or use a separately defined fallback. The choice belongs in the strategy, because the confidence filter changes when evidence is available.
Historical testing also needs a timestamp for when the signal became available to the system. Publisher time, archive capture, archive availability, ingestion and classification can occur at different times. The backtest should use an availability rule consistent with the actual collection process.
I paused the current model work with the experiments and error audit saved. The next useful result has to improve relevant-news recovery while controlling false assignments. The overall score already shows how easily an attractive number can survive weak coverage.
To inspect how I approach strategy research, open NexusTrade Explore.
No comments yet.