Decision Tree tutorial

Overview

What is Decision Tree?

Decision Tree is a supervised learning technique. Splits the feature space into rule-based regions. Its central idea is summarized by s* = arg maxₛ ΔI(s).

This guide connects theory to practice. You will trace the input, intermediate process, and output; decode the notation; run a dependency-light implementation; then repeat the workflow with scikit-learn and evaluate the result.

Preparation

Prerequisites and tools

You do not need an advanced software stack. Start with a recent Python environment and the fundamentals below, then install only the packages used by the practical example.

• Comfort with Python functions, NumPy arrays, and basic descriptive statistics.
• A clear distinction between training data, validation data, and untouched test data.
• Familiarity with features, targets, preprocessing, and task-appropriate evaluation metrics.

Process

How Decision Tree works

1. 1

   Input

   Features and labels or target values.

2. 2

   Prepare and configure

   Check shapes, value ranges, ordering assumptions, missing values, and the parameters that control the algorithm’s behavior.

3. 3

   Algorithm process

   Choose splits that reduce impurity or error.

4. 4

   Output

   A leaf prediction reached through rules.

5. 5

   Validate

   Compare train and validation performance while increasing depth to identify overfitting.

Core concept

Formula and intuition

The notation captures the main operation or complexity statement behind Decision Tree. Read it together with the symbol key above and the step-by-step process in this guide.

Applications

When to use Decision Tree

Explainable classification and regression.

Implementation

Decision Tree from scratch in Python

This dependency-light example emphasizes the algorithm’s mechanics so each important step remains visible.

import numpy as np

def gini(y):
    # gini impurity: 0 when a node is pure (all one class), higher when mixed
    p = np.bincount(y) / len(y)
    return 1 - np.sum(p ** 2)

def best_split(X, y):
    # try every feature/threshold pair and keep the one that reduces impurity the most
    best = {"gain": -1}
    parent = gini(y)
    for feature in range(X.shape[1]):
        for threshold in np.unique(X[:, feature]):
            left = y[X[:, feature] <= threshold]
            right = y[X[:, feature] > threshold]
            if len(left) == 0 or len(right) == 0:
                continue
            weighted = (len(left) * gini(left) + len(right) * gini(right)) / len(y)
            if parent - weighted > best["gain"]:
                best = {"gain": parent - weighted, "feature": feature, "threshold": threshold}
    return best

def build_tree(X, y, depth=3):
    # recursively split until max depth is reached or a node is already pure
    if depth == 0 or len(np.unique(y)) == 1:
        return {"leaf": np.bincount(y).argmax()}  # majority class becomes the leaf prediction
    split = best_split(X, y)
    if split["gain"] <= 0:
        return {"leaf": np.bincount(y).argmax()}
    mask = X[:, split["feature"]] <= split["threshold"]
    return {"feature": split["feature"], "threshold": split["threshold"],
            "left": build_tree(X[mask], y[mask], depth - 1),
            "right": build_tree(X[~mask], y[~mask], depth - 1)}

rng = np.random.default_rng(0)
X = rng.normal(0, 1, (200, 4))
y = (X[:, 0] + X[:, 1] > 0).astype(int)
tree = build_tree(X, y, depth=3)

Practical tutorial

Build Decision Tree with scikit-learn

DecisionTreeClassifier exposes depth, leaf-size, pruning, and feature-importance controls while preserving readable rules.

1. Prepare the data.Features and labels or target values. Validate its shape, type, range, and ordering before training or execution.
2. Configure the algorithm.Begin with explicit, conservative parameters and a fixed random seed whenever the library supports one.
3. Fit or execute.Choose splits that reduce impurity or error.
4. Inspect the result.A leaf prediction reached through rules. Then apply the evaluation checks in the next section.

from sklearn.datasets import load_breast_cancer
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier, export_text

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42
)
model = DecisionTreeClassifier(
    max_depth=4, min_samples_leaf=8, random_state=42
)
model.fit(X_train, y_train)
print(classification_report(y_test, model.predict(X_test)))
print(export_text(model, max_depth=2))

API details and version-specific options: official scikit-learn reference →

Evaluation

How to evaluate the result

A successful run is not enough. Evaluate the output against the intended use, compare it with a simple baseline, and preserve a genuinely unseen test case whenever the task involves learned parameters.

• Compare train and validation performance while increasing depth to identify overfitting.
• Inspect leaf sample counts and class distributions, not only aggregate accuracy.

Common mistakes

Pitfalls and how to avoid them

These failure modes are common in tutorials and production systems. Treat them as review questions, not just after-the-fact debugging advice.

• Unrestricted trees memorize noise and produce unstable rules.
• Impurity-based feature importance can favor high-cardinality features.
• Small data changes can alter the selected split structure.

Project checklist

Before using Decision Tree in a project

• ✓Define the prediction or discovery objective before selecting the algorithm.
• ✓Split data before fitting preprocessing and tune only inside cross-validation.
• ✓Record data versions, random seeds, features, hyperparameters, and evaluation metrics.
• ✓Compare against a simple baseline and inspect errors by meaningful subgroups.
• ✓Monitor input drift and real-world performance after deployment.

Further reading

Official documentation and next steps

This guide is an educational introduction, not a substitute for domain validation. For consequential applications, review the source documentation, test against representative data, and involve a subject-matter expert.

Continue learning

Related machine learning guides

• Linear RegressionFits a straight line that minimizes squared prediction errors.
• K-Nearest NeighborsPredicts from the labels or values of nearby observations.
• Support Vector MachineFinds a separating boundary with the widest possible margin.