Random Forest tutorial

Overview

What is Random Forest?

Random Forest is a ensemble learning technique. Combines diverse decision trees trained on sampled data. Its central idea is summarized by ŷ(x) = Aggregate({Tᵦ(x)}ᵦ₌₁ᴮ).

This guide connects theory to practice. You will trace the input, intermediate process, and output; decode the notation; run a dependency-light implementation; then repeat the workflow with scikit-learn and evaluate the result.

Preparation

Prerequisites and tools

You do not need an advanced software stack. Start with a recent Python environment and the fundamentals below, then install only the packages used by the practical example.

• Comfort with Python functions, NumPy arrays, and basic descriptive statistics.
• A clear distinction between training data, validation data, and untouched test data.
• Familiarity with features, targets, preprocessing, and task-appropriate evaluation metrics.

Process

How Random Forest works

1. 1

   Input

   Features and labels or target values.

2. 2

   Prepare and configure

   Check shapes, value ranges, ordering assumptions, missing values, and the parameters that control the algorithm’s behavior.

3. 3

   Algorithm process

   Bootstrap data, randomize features, and train many trees.

4. 4

   Output

   A majority vote or average prediction.

5. 5

   Validate

   Compare out-of-bag and held-out scores; a large gap can reveal distribution shift or leakage.

Core concept

Formula and intuition

The notation captures the main operation or complexity statement behind Random Forest. Read it together with the symbol key above and the step-by-step process in this guide.

Applications

When to use Random Forest

Robust structured-data prediction.

Implementation

Random Forest from scratch in Python

This dependency-light example emphasizes the algorithm’s mechanics so each important step remains visible.

import numpy as np

def stump(X, y):
    # a "stump" is a decision tree of depth 1: a single split with two leaves
    best = {"score": -1}
    for feature in range(X.shape[1]):
        for threshold in np.unique(X[:, feature]):
            left = y[X[:, feature] <= threshold]
            right = y[X[:, feature] > threshold]
            if len(left) == 0 or len(right) == 0:
                continue
            score = abs(left.mean() - right.mean())  # how well this split separates the classes
            if score > best["score"]:
                best = {"score": score, "feature": feature, "threshold": threshold,
                        "left_label": round(left.mean()), "right_label": round(right.mean())}
    return best

rng = np.random.default_rng(0)
n_samples, n_trees = 200, 9
X = rng.normal(0, 1, (n_samples, 4))
y = (X[:, 0] + X[:, 1] > 0).astype(int)

trees = []
for _ in range(n_trees):
    idx = rng.integers(0, n_samples, n_samples)  # bootstrap: sample rows with replacement
    trees.append(stump(X[idx], y[idx]))           # each tree trains on a slightly different sample

def predict(x):
    # combine every tree in the forest with a majority vote
    votes = [t["left_label"] if x[t["feature"]] <= t["threshold"] else t["right_label"] for t in trees]
    return round(np.mean(votes))

Practical tutorial

Build Random Forest with scikit-learn

RandomForestClassifier combines bootstrapping, randomized feature subsets, parallel training, and out-of-bag evaluation.

1. Prepare the data.Features and labels or target values. Validate its shape, type, range, and ordering before training or execution.
2. Configure the algorithm.Begin with explicit, conservative parameters and a fixed random seed whenever the library supports one.
3. Fit or execute.Bootstrap data, randomize features, and train many trees.
4. Inspect the result.A majority vote or average prediction. Then apply the evaluation checks in the next section.

from sklearn.datasets import load_wine
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split

X, y = load_wine(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, stratify=y, random_state=42
)
model = RandomForestClassifier(
    n_estimators=300, min_samples_leaf=2, oob_score=True,
    n_jobs=-1, random_state=42
)
model.fit(X_train, y_train)
print("out-of-bag score:", model.oob_score_)
print(classification_report(y_test, model.predict(X_test)))

API details and version-specific options: official scikit-learn reference →

Evaluation

How to evaluate the result

A successful run is not enough. Evaluate the output against the intended use, compare it with a simple baseline, and preserve a genuinely unseen test case whenever the task involves learned parameters.

• Compare out-of-bag and held-out scores; a large gap can reveal distribution shift or leakage.
• Use permutation importance on validation data for a less biased importance estimate.

Common mistakes

Pitfalls and how to avoid them

These failure modes are common in tutorials and production systems. Treat them as review questions, not just after-the-fact debugging advice.

• More trees reduce variance but do not correct biased labels or leakage.
• Large forests consume substantial memory and can slow inference.
• Default class weighting may underperform on imbalanced outcomes.

Project checklist

Before using Random Forest in a project

• ✓Define the prediction or discovery objective before selecting the algorithm.
• ✓Split data before fitting preprocessing and tune only inside cross-validation.
• ✓Record data versions, random seeds, features, hyperparameters, and evaluation metrics.
• ✓Compare against a simple baseline and inspect errors by meaningful subgroups.
• ✓Monitor input drift and real-world performance after deployment.

Further reading

Official documentation and next steps

This guide is an educational introduction, not a substitute for domain validation. For consequential applications, review the source documentation, test against representative data, and involve a subject-matter expert.

Continue learning

Related machine learning guides

• Linear RegressionFits a straight line that minimizes squared prediction errors.
• Logistic RegressionMaps a weighted feature score to a class probability.
• Support Vector MachineFinds a separating boundary with the widest possible margin.