TL;DR
The small neural network, called a multilayer perceptron (MLP), won our controlled financial-sentiment comparison. At the 20,000-article rung, its average score error was 0.1170, compared with 0.1306 for XGBoost, a 10.4% reduction. Direction agreement rose from 60.3% to 65.5%.
Both models used the same frozen text embeddings, existing labels and held-out headlines. The benchmark covered nine securities, with one configuration and training seed per family. The MLP won this comparison; broader news coverage still requires separate testing.
Can a small model judge this Apple headline?
The task starts with a real headline: "Apple Posts Record Revenues, Beats on Earnings: ETFs to Buy" (Zacks, July 29, 2022). We ask whether the news is favorable, unfavorable or neutral for Apple. That judgment is sentiment. AAPL is Apple's ticker, the short code used to identify its stock.
The saved AI reference gives Apple +0.50 on a scale from -1 (negative) through 0 (neutral) to +1 (positive). Our small neural network predicts about +0.67. Its distance from the reference is about 0.17: |0.67 - 0.50|. Averaging that distance over many examples gives mean absolute error, or MAE.
The reference is an existing AI-generated judgment from headline and summary context. The trained scorer receives headline features. Matching that reference measures agreement with our saved labels; a stock-price forecast needs a separate test.
What I actually compared
I wanted a cheap way to turn financial news into ticker-specific sentiment for NexusTrade. A headline can mention several companies, and the same event can help one while hurting another. The task was to score a headline in relation to a particular company.
Both contenders received numerical features built with the same frozen Qwen embedding model. An embedding represents text as a vector of numbers. We combined the headline vector, a company vector, their element-by-element product and their dot product into 3,073 features.
XGBoost learned boosted decision trees over those features. The neural alternative was a small multilayer perceptron, or MLP, with hidden layers of 128 and 64 units. Each MLP had 401,793 parameters. We fitted separate models for ticker relevance and sentiment. The embedding model stayed frozen.
This comparison therefore tests two inexpensive scorers over an existing text representation. It is a different experiment from fine-tuning a transformer to read the original words.
The MLP made smaller sentiment errors at every data size
Same 1,267 reference company/headline pairs from the common 2,000-headline test; nine securities. Zero baseline. Sentiment scores span -1 to +1. Values are rounded to four decimals.
Show the table · 3 rows, 6 columns
Scroll sideways to compare all columns. Select a heading to sort.
| 1,000 | 800 | 0.1444 | 0.1419 | 53.9% | 57.0% |
| 5,000 | 4,000 | 0.1333 | 0.1257 | 57.7% | 61.2% |
| 20,000 | 16,000 | 0.1306 | 0.1170 | 60.3% | 65.5% |
No matching rows. Clear the filter to see all records.
Existing Massive/Haiku labels; nine securities; unchanged embeddings and article splits. MAE is in sentiment-score units. Lower MAE is better.
The MLP had lower sentiment error and higher direction agreement at all three rungs. Its advantage grew as we added training examples. At the largest rung, the absolute error reduction was 0.0136 on a sentiment scale spanning -1 to +1.
Mean absolute error measures the average distance between a predicted score and its reference. Direction agreement measures whether the prediction and reference fall in the same negative, neutral or positive category. We counted scores within plus or minus 0.05 as neutral.
How we kept the comparison useful
Read the explanation
The total dataset sizes were 1,000, 5,000 and 20,000 articles. Their grouped 80/10/10 partitions gave us 800, 4,000 and 16,000 articles for fitting weights. Each model was evaluated on a common set of 2,000 test headlines that stayed outside every training set. The split measures classification on held-out article groups; future-period strategy validation is a separate test.
Sentiment evaluation used all 1,267 reference company/headline pairs. Restricting it to tickers the extraction pipeline happened to find would have hidden some difficult examples.
A paired bootstrap over test articles gave a 95% error-reduction interval of 0.0094 to 0.0178 at the largest rung. That describes test-sample uncertainty for these fitted models. We used one training seed in this family comparison, so variation across new training seeds and news sources remains a separate question.
The labels came from our existing Massive/Haiku records, where the labeling process had headline and summary context. Our scorers received headline embeddings. That mismatch matters when an article discusses something the headline leaves out.
The MLP standardized each feature using training-only statistics. XGBoost used the same underlying pair features without that standardization. Both models reused the same candidate companies and identity corrections.
Why testing XGBoost was still sensible
XGBoost is an established implementation of tree boosting, described by Tianqi Chen and Carlos Guestrin in XGBoost: A Scalable Tree Boosting System.
The case for trees on many structured datasets is also supported by the 45-dataset benchmark in Why do tree-based models still outperform deep learning on tabular data?.
Our inputs were dense text embeddings. A result on ordinary structured columns cannot settle which scorer works best on this representation. The observed MLP advantage is a reason to run the comparison on the actual features, rather than choosing a model family by reputation.
XGBoost retained a storage advantage: the saved model pair occupied about 0.49 MB, compared with 3.27 MB for the MLP pair. The measured fitting loops at the largest rung took 13.69 seconds for XGBoost and 4.18 seconds for the MLP, using eight local CPU threads. Those timings exclude embedding generation, name extraction and feature preparation.
A better sentiment scorer still needs the right ticker
The scoring improvement left a large extraction problem. In this early nine-security pipeline, the frozen name extractor and dated resolver supplied 1,035 of 1,227 reference company associations, or 84.35%. A scorer could only assess the candidates it received.
At the largest rung, exact multiple-ticker arrays were correct for 35.0% of the XGBoost cases and 38.3% of the MLP cases. Both had 47.2% ticker recall within that cohort. More accurate sentiment on a known company cannot recover a company omitted upstream.
Sentiment itself also needed work. Of the MLP's 437 direction disagreements, 380 involved neutral versus nonneutral. Another 57 were opposite positive/negative directions. That separates a boundary problem from a more consequential polarity problem without erasing either.
One saved example was "ETFs in Focus Post Microsoft Q1 Earnings Beat, Soft View." For MSFT, the existing reference score was -0.30 and the MLP predicted +0.3706. The headline contains both favorable results and weaker guidance. This single disagreement shows why an average error needs actual examples beside it; the reference is an existing model-generated label.
Show the table · 3 rows, 4 columns
Scroll sideways to compare all columns. Select a heading to sort.
| Negative | 201 | 59 | 25 |
| Neutral | 61 | 142 | 185 |
| Positive | 32 | 75 | 487 |
No matching rows. Clear the filter to see all records.
The saved 20,000-article MLP confusion matrix covers all 1,267 reference pairs. Each row is the reference, and each column is the prediction. The fixed neutral band is plus or minus 0.05.
Longer training had already stopped helping
The largest MLP run selected epoch 3 for relevance and epoch 7 for sentiment using validation loss. Training continued beyond those checkpoints, and the later validation losses were worse. The saved curve gave us a concrete reason to keep the earlier weights.
The model-family comparison reused our cached features and labels, adding zero provider or cloud charges. That makes these small classifiers attractive to test locally. The expensive question is obtaining useful coverage and trustworthy supervision at archive scale.
Which would I choose for this task?
For this measured setup, I would carry the small MLP forward as the sentiment scorer and retain XGBoost as a baseline. The MLP earned that choice through lower score error and better direction agreement on the same examples.
A production decision also needs ticker recall, neutral handling, performance on the intended news source, and strategy results with realistic data availability. This experiment measured agreement with existing labels. The later 100-stock news pipeline is a separate system and should be judged with its own tests.
If you want to explore the research workflow I am building this for, open NexusTrade Explore.
No comments yet.