Model comparison: Amazon
Trained on 5327 days (Apr 2000 to Jun 2021) and tested on the following 1332 days (Jun 2021 to Sep 2026). Every pipeline scales (and optionally applies PCA) inside each time-series fold.
Verdict
1 of 7 models beat the best naive guess
That guess is to always predict "up", which scores 50.9% test accuracy. The best ROC AUC is 0.517 (Random Forest); 0.5 means no better than chance. Daily direction is close to a coin flip, and these results reflect that honestly.
Test-set metrics
Will tomorrow's close be higher? CV figures are the mean ± std over expanding-window folds of the training period. Test figures come from the same fitted pipeline that is backtested.
| Model | CV accuracy | CV AUC | Test accuracy | Precision | Recall | F1 | ROC AUC | vs naive | Tuned params |
|---|---|---|---|---|---|---|---|---|---|
| Random Forest | 49.7% ± 2.0% | 0.507 | 50.7% | 50.9% | 87.9% | 0.645 | 0.517 | below | max_depth=3 |
| Logistic Regression | 50.8% ± 2.0% | 0.508 | 50.1% | 50.6% | 79.2% | 0.618 | 0.513 | below | C=1.0 |
| Linear SVC | 50.8% ± 1.8% | 0.508 | 49.5% | 50.3% | 79.4% | 0.616 | 0.512 | below | defaults |
| Extra Trees | 49.8% ± 1.9% | 0.499 | 50.8% | 50.9% | 95.0% | 0.663 | 0.511 | below | defaults |
| Bagging (KNN) | 49.9% ± 1.7% | 0.503 | 49.2% | 50.1% | 59.4% | 0.544 | 0.502 | below | defaults |
| Gradient Boosting | 51.1% ± 0.7% | 0.515 | 50.6% | 51.1% | 71.5% | 0.596 | 0.495 | below | max_depth=3 |
| SVM (RBF) | 50.6% ± 2.0% | 0.504 | 51.3% | 51.3% | 82.6% | 0.633 | 0.486 | above | C=3.0 |
| Always up (baseline) | – | – | 50.9% | 50.9% | 100.0% | 0.675 | 0.500 |
Training period
How models scored during time-series cross-validation
Takeaway 6 of 7 models swing between beating and losing to a coin flip across folds, so their average CV score hides a lot of instability.
Test period
Unseen data: can the models rank up days above down days, and does any edge persist?
Takeaway Share of the test period each model spent above 50% (rolling 63 days): Logistic Regression 50%, SVM (RBF) 64%, Linear SVC 48%, Random Forest 59%, Extra Trees 63%, Bagging (KNN) 43%, Gradient Boosting 54%.
Walk-forward check
Would the result hold up if each model were refitted as new data arrived?
Takeaway Refitting 22 times changed test AUC by -0.002 on average (range -0.008 to +0.013). With 1,332 test days the 95% interval is about ±0.031 wide, so none of 7 models trained once and none walk-forward are reliably better than a coin flip.
Method The models above are fitted once on the training period. Walk-forward keeps the same tuned settings but refits every 63 trading days (22 times) on all earlier days, then predicts only the next block. The backtest still uses the model trained once.
Inside Extra Trees
Pick another model from the selector or the table above
Takeaway Extra Trees has a test ROC AUC of 0.511: pick a random up day and a random down day, and it ranks the up day higher 51% of the time (50% = guessing).
Method Permutation importance is measured on the last validation fold of the training period, never on the test set. It's always reported for the original named features, even for PCA pipelines.