Model comparison: Microsoft
Trained on 5327 days (Apr 2000 to Jun 2021) and tested on the following 1332 days (Jun 2021 to Sep 2026). Every pipeline scales (and optionally applies PCA) inside each time-series fold.
Verdict
0 of 7 models beat the best naive guess
That guess is to always predict "up", which scores 52.0% test accuracy. The best ROC AUC is 0.508 (Random Forest); 0.5 means no better than chance. Daily direction is close to a coin flip, and these results reflect that honestly.
Test-set metrics
Will tomorrow's close be higher? CV figures are the mean ± std over expanding-window folds of the training period. Test figures come from the same fitted pipeline that is backtested.
| Model | CV accuracy | CV AUC | Test accuracy | Precision | Recall | F1 | ROC AUC | vs naive | Tuned params |
|---|---|---|---|---|---|---|---|---|---|
| Random Forest | 51.4% ± 1.7% | 0.512 | 50.5% | 52.0% | 63.2% | 0.570 | 0.508 | below | max_depth=3 |
| Extra Trees | 51.2% ± 1.8% | 0.516 | 50.8% | 51.9% | 71.2% | 0.601 | 0.508 | below | defaults |
| Logistic Regression | 51.1% ± 1.7% | 0.523 | 50.4% | 51.9% | 61.8% | 0.564 | 0.507 | below | C=0.01 |
| SVM (RBF) | 51.3% ± 1.9% | 0.514 | 49.5% | 51.1% | 61.1% | 0.557 | 0.506 | below | C=0.3 |
| Gradient Boosting | 50.7% ± 0.8% | 0.513 | 50.4% | 52.0% | 56.9% | 0.544 | 0.506 | below | max_depth=3 |
| Linear SVC | 51.7% ± 1.7% | 0.517 | 49.1% | 50.9% | 57.2% | 0.539 | 0.501 | below | defaults |
| Bagging (KNN) | 49.2% ± 0.9% | 0.498 | 50.9% | 52.6% | 55.1% | 0.538 | 0.495 | below | defaults |
| Always up (baseline) | – | – | 52.0% | 52.0% | 100.0% | 0.684 | 0.500 |
Training period
How models scored during time-series cross-validation
Takeaway 6 of 7 models swing between beating and losing to a coin flip across folds, so their average CV score hides a lot of instability.
Test period
Unseen data: can the models rank up days above down days, and does any edge persist?
Takeaway Share of the test period each model spent above 50% (rolling 63 days): Logistic Regression 53%, SVM (RBF) 46%, Linear SVC 49%, Random Forest 51%, Extra Trees 57%, Bagging (KNN) 61%, Gradient Boosting 52%.
Walk-forward check
Would the result hold up if each model were refitted as new data arrived?
Takeaway Refitting 22 times changed test AUC by -0.004 on average (range -0.008 to +0.009). With 1,332 test days the 95% interval is about ±0.031 wide, so none of 7 models trained once and none walk-forward are reliably better than a coin flip.
Method The models above are fitted once on the training period. Walk-forward keeps the same tuned settings but refits every 63 trading days (22 times) on all earlier days, then predicts only the next block. The backtest still uses the model trained once.
Inside Logistic Regression
Pick another model from the selector or the table above
Takeaway Logistic Regression has a test ROC AUC of 0.507: pick a random up day and a random down day, and it ranks the up day higher 51% of the time (50% = guessing).
Method Permutation importance is measured on the last validation fold of the training period, never on the test set. It's always reported for the original named features, even for PCA pipelines.