Why trees, specifically
Averaging only helps if the thing you're averaging has high variance and low bias. Deep decision trees are almost purpose-built for the role.
What a deep tree gets right
- Low bias. Grown without pruning, a tree can carve the space arbitrarily finely and fit almost any pattern.
- Non-linear and interactive by default. Every split conditions on the ones above it, so interactions are found without being specified.
- Scale-invariant. Splits are threshold comparisons, so monotone transformations of a feature change nothing. No standardization required, ever.
- Handles mixed types and tolerates missing values with surrogate splits.
What it gets badly wrong
Variance. The greedy top-down construction makes the whole structure hostage to the first split. Resample the training data and a slightly different split wins at the root, and every subsequent decision diverges from there.
Two trees trained on 95%-overlapping data can look nothing alike. As a single model that's a liability — the fitted structure isn't stable enough to interpret or trust. As an ensemble ingredient it's exactly the raw material you want.
Bootstrap, then aggregate
Draw n rows from your n training rows with replacement. Some appear several times, some not at all. Fit a tree. Repeat. To predict, average the probabilities across trees, or take a majority vote.
Why resampling produces different trees
Bootstrap samples share about 63% of the original rows on average, so any two trees see substantially different data. Because tree construction is greedy and unstable, that modest difference in input produces a large difference in structure — which is exactly what you're trying to buy.
What averaging does to the errors
Individual trees make errors in different places. Averaging cancels the parts that are independent and preserves the parts they agree on — and what they agree on is the signal.
Aggregation differs by task. For regression, average the predicted values. For classification, sklearn averages the predicted class probabilities rather than taking a hard majority vote, which is generally better calibrated and produces smoother boundaries.
The move that makes it a forest
At every single split, sample a random subset of the features and only consider those. Not once per tree — once per node. A feature that would have dominated is simply unavailable most of the time, so other features get their turn and the trees genuinely diverge.
The floor you cannot average away
Add more trees and the second term vanishes. The first term does not — it is set entirely by how correlated the trees are with each other. Drag ρ and watch the floor rise and fall.
Bagging alone might give you ρ ≈ 0.6. Feature subsampling might drop it to 0.2. That is not a marginal improvement in the limit — it is a threefold reduction in the irreducible variance of the whole ensemble, and no number of extra trees could have achieved it.
All of it
for b in 1…B: sample n rows from the training set WITH replacement # bagging grow a tree on that sample: at each node: pick m features at random from the p available # random subspace find the best split among ONLY those m split, and recurse grow deep — no pruning, stop only at min_samples_leaf record which rows were out-of-bag for this tree # free validation predict(x): regression → mean of the B tree predictions classification → mean of the B predicted probability vectors
Embarrassingly parallel
No tree depends on any other, so training scales linearly across cores with no coordination. Set n_jobs=-1 and forget about it. Boosting cannot do this — each tree needs the previous one's residuals.
Very few knobs
Defaults are genuinely good. A random forest fitted with no tuning at all is a serious baseline, which is not true of most competitive models.
Hard to break
No learning rate to diverge, no scaling to forget, no convergence to monitor. Adding trees never makes it worse. The failure modes are about the data, not the fit.
Validation you get for free
About 37% of rows are omitted from any given bootstrap sample. Those rows are out-of-bag for that tree — genuinely unseen. Predict each row using only the trees that didn't train on it, and you have a held-out estimate without holding anything out.
What to set, and what to leave alone
| Parameter | Effect | Guidance |
|---|---|---|
| n_estimators | Number of trees | More is never worse, only slower. Use 300–1000; stop when OOB plateaus. Not a tuning parameter in the usual sense |
| max_features | Features considered per split | The one that matters. sqrt(p) for classification, around p/3 for regression. Try a few values |
| max_depth | Tree depth cap | Leave unlimited by default. Cap it only if memory or inference latency demands it |
| min_samples_leaf | Smallest allowed leaf | 1 for classification, ~5 for regression. Raising it is the gentlest way to regularize |
| min_samples_split | Smallest node to split | Redundant with min_samples_leaf. Pick one |
| max_samples | Bootstrap sample size | Below 1.0 speeds up training on large data and slightly increases diversity |
| bootstrap | Resample rows at all | Keep it True — turning it off disables OOB and removes half the diversity |
| class_weight | Reweight rare classes | balanced_subsample for imbalanced problems. Or just move the threshold |
| criterion | Split quality measure | Gini vs entropy makes almost no practical difference. Don't spend time here |
The default importance is misleading
Forests will happily hand you a ranked list of features, and that list is one of the most-cited and least-scrutinized outputs in applied machine learning. The default method has known, systematic biases.
| Method | How | Problems |
|---|---|---|
| Impurity / Gini | Total impurity reduction attributed to each feature, summed over all splits | Biased toward high-cardinality and continuous features. Computed on training data, so it rewards overfitting. sklearn's default |
| Permutation | Shuffle one column, measure the drop in score | Much better — uses held-out data and measures what you actually care about. Slow, and misleading under correlated features |
| Drop-column | Refit without the feature | The most defensible definition. Costs a full refit per feature |
| SHAP (TreeSHAP) | Game-theoretic attribution, exact for trees | Per-prediction as well as global. The current best answer, and fast on tree models |
# do not ship the default. use permutation importance on held-out data. from sklearn.inspection import permutation_importance r = permutation_importance(rf, X_test, y_test, n_repeats=20, random_state=0) for i in r.importances_mean.argsort()[::-1]: print(f"{feature_names[i]:>24} {r.importances_mean[i]:+.4f} " f"± {r.importances_std[i]:.4f}") # the sanity check: add noise and see where it ranks X_test["random_noise"] = np.random.rand(len(X_test)) # anything scoring below the noise column is not doing real work
Strengths, and the things it genuinely cannot do
Where it shines
- Works immediately. Strong performance with default settings on most tabular problems.
- No preprocessing. No scaling, no normality assumptions, monotone transformations are irrelevant.
- Mixed feature types and non-linear interactions handled natively.
- Robust to outliers in the features, because splits only care about order.
- Free validation via OOB, and parallel training.
- Stable. Two runs with different seeds give similar models, unlike a single tree.
Where it fails
- Cannot extrapolate. A prediction is an average of training labels, so it can never exceed the range it saw. Fatal for trending time series.
- Axis-aligned splits. A diagonal boundary must be approximated by a staircase. Visible in the hero.
- Poor on very sparse high-dimensional data — text, one-hot encodings with thousands of columns. Linear models usually win there.
- Large models. Hundreds of deep trees is a lot of memory and non-trivial inference latency.
- Not interpretable, despite trees being the interpretable model. Five hundred trees is not a flowchart.
- Probabilities need calibration — vote proportions are pushed toward the middle and are not well-calibrated out of the box.
The same trick, aimed elsewhere
Extremely Randomized Trees
Randomize the split threshold too, rather than searching for the best one. More bias, less variance, and considerably faster since no threshold search is needed. Uses the whole dataset by default rather than bootstrapping. Often matches a forest for a fraction of the training time.
Isolation Forest
Anomaly detection with no labels. Split randomly and measure how few splits it takes to isolate a point — outliers separate quickly because they sit in sparse regions. Same machinery, entirely different objective.
Quantile Regression Forests
Keep every training value in the leaves instead of just the mean. You can then read off any conditional quantile, which gives you genuine prediction intervals rather than a point estimate.
Random Survival Forests
Split on a survival criterion to handle censored time-to-event data. Standard in medical and reliability work.
Proximity & embeddings
Two rows that repeatedly land in the same leaf are similar. That proximity matrix supports clustering, imputation and visualization — a supervised similarity metric derived from a fitted forest.
Causal / Generalized forests
Split to maximize heterogeneity in a treatment effect rather than in the outcome. Used to estimate who a treatment actually helps.
The comparison you actually need
| Random forest | Gradient boosting | |
|---|---|---|
| reduces | Variance | Bias |
| base trees | Deep, low bias, high variance | Shallow, high bias, low variance |
| trees are | Independent, fitted in parallel | Sequential — each fits the previous residuals |
| more trees | Never hurts; converges | Can overfit. Needs early stopping |
| tuning | Defaults are fine. One knob matters | Learning rate, depth, subsample, regularization — real work |
| peak accuracy | Very good | Usually better, when tuned |
| training speed | Fast, fully parallel | Slower, though LightGBM narrows the gap considerably |
| robustness to noisy labels | Better — averaging dilutes them | Worse — boosting chases the errors it can't fix |
| free validation | Yes, OOB | No |
A reasonable starting configuration
from sklearn.ensemble import RandomForestClassifier from sklearn.inspection import permutation_importance from sklearn.model_selection import train_test_split from sklearn.metrics import classification_report, roc_auc_score import numpy as np X_tr, X_te, y_tr, y_te = train_test_split( X, y, test_size=0.2, stratify=y, random_state=42) rf = RandomForestClassifier( n_estimators=500, max_features="sqrt", # the knob that matters min_samples_leaf=1, class_weight="balanced_subsample", oob_score=True, # free validation — turn it on n_jobs=-1, random_state=42, ).fit(X_tr, y_tr) print("OOB ", rf.oob_score_) # before you touch the test set proba = rf.predict_proba(X_te)[:, 1] print("AUC ", roc_auc_score(y_te, proba)) # check that more trees was enough — OOB should have plateaued for n in (50, 100, 250, 500): m = RandomForestClassifier(n_estimators=n, oob_score=True, n_jobs=-1, random_state=42).fit(X_tr, y_tr) print(n, round(m.oob_score_, 4)) # importance: permutation on HELD-OUT data, never the .feature_importances_ default r = permutation_importance(rf, X_te, y_te, n_repeats=20, n_jobs=-1) # calibrate if you need the probabilities to mean something from sklearn.calibration import CalibratedClassifierCV cal = CalibratedClassifierCV(rf, method="isotonic", cv="prefit")
Drills
| # | Do this | You'll know it when |
|---|---|---|
| 1 | Fit one deep tree five times on 90% bootstrap samples; plot all five boundaries | The instability that bagging exploits is visible |
| 2 | Implement bagging yourself in twenty lines, then add feature subsampling | You measure the accuracy gain from the second step alone |
| 3 | Plot OOB error against n_estimators from 1 to 500 | You can see the plateau and stop guessing tree counts |
| 4 | Sweep max_features from 1 to p and plot test error | The bias-diversity trade has a shape you've seen |
| 5 | Add a random noise column and compare impurity vs permutation importance | You stop trusting .feature_importances_ |
| 6 | Duplicate a strong feature and watch both importances halve | The correlated-credit problem is concrete |
| 7 | Train on a range and predict outside it | The extrapolation ceiling is something you've seen flatten |
| 8 | Compare against Extra Trees and LightGBM on the same data and time all three | You can defend a model choice on more than accuracy |
| 9 | Plot a reliability diagram for the raw forest probabilities | You know whether calibration is needed before you promise probabilities |
The one-paragraph summary
A random forest averages many deep, unpruned decision trees, each trained on a bootstrap sample of the rows and — critically — each split restricted to a random subset of the columns. Deep trees are chosen as the base learner because averaging reduces variance while leaving bias alone, so you want ingredients with low bias and are indifferent to their instability. The variance of the average is ρσ² + (1−ρ)σ²/M, which means adding trees eliminates only the second term and the floor is set by how correlated the trees are — this is precisely why feature subsampling at each node matters, since bagging alone leaves trees that all pick the same dominant feature at the root. Roughly 37% of rows are out-of-bag for any given tree, which yields a free held-out estimate during the same fit. Adding trees never hurts, defaults are strong, no scaling is required, and training parallelizes perfectly — but the model cannot extrapolate beyond its training range, splits only on axis-aligned thresholds, is not interpretable despite being built from interpretable parts, and its default impurity-based feature importances are biased toward high-cardinality features and should be replaced by permutation importance computed on held-out data.