FinBERT outperforms word counts in financial news sentiment scoring
Transformer-based language models capture financial news sentiment far closer to human judgment than traditional lexicon methods, according to an ECB working paper published in September 2026. The researchers validated daily mood indices against 1,776 evaluations from 444 annotators.
Context breaks the word count trap
Analyzing 143,755 financial news articles from Factiva, the authors classified sentence-level sentiment with FinBERT and aggregated predictions into daily indicators.
A validation exercise with 444 incentivized participants across 588 articles produced 1,776 human ratings as an external benchmark.
Transformer-based models achieved in-sample macro-F1 scores between 0.67 and 0.69 and classification accuracy of 66 to 68 percent.
By contrast, traditional vocabulary-based benchmarks by Shapiro and Barbaglia reached a macro-F1 score of only 0.187 and an accuracy of 39 percent, as they collapsed most articles into the neutral category.
Polarity beyond compressed averages
Out-of-sample testing across chronological splits confirms the advantage.
The baseline FinBERT index achieved an R-squared of 0.632, whereas the Shapiro index reached 0.295 and the Barbaglia index 0.183. Downstream machine-learning models like Random Forest added little extra signal over the transformer scores.
Furthermore, the daily indices reveal systematic calendar variation: Tuesday posts the highest average tone at 0.744, while Friday falls to minus 0.025.
Counting words is officially obsolete
Bag-of-words metrics are no longer viable proxies for economic sentiment tracking.
The near-total failure of lexicons to separate positive from negative news exposes deep flaws in dictionary methods.
Central bank monitoring frameworks must abandon word counts and adopt context-aware transformer models.