Linear Regression tutorial

Overview

What is Linear Regression?

Linear Regression is a supervised learning technique. Fits a straight line that minimizes squared prediction errors. Its central idea is summarized by ŷᵢ = β₀ + β₁ xᵢ.

This guide connects theory to practice. You will trace the input, intermediate process, and output; decode the notation; run a dependency-light implementation; then repeat the workflow with scikit-learn and evaluate the result.

Preparation

Prerequisites and tools

You do not need an advanced software stack. Start with a recent Python environment and the fundamentals below, then install only the packages used by the practical example.

• Comfort with Python functions, NumPy arrays, and basic descriptive statistics.
• A clear distinction between training data, validation data, and untouched test data.
• Familiarity with features, targets, preprocessing, and task-appropriate evaluation metrics.

Process

How Linear Regression works

1. 1

   Input

   Numeric features and a continuous target.

2. 2

   Prepare and configure

   Check shapes, value ranges, ordering assumptions, missing values, and the parameters that control the algorithm’s behavior.

3. 3

   Algorithm process

   Estimate slope and intercept by reducing residual error.

4. 4

   Output

   A continuous value on the fitted line.

5. 5

   Validate

   Report MAE in the target’s original units and R² as a scale-free goodness-of-fit summary.

Core concept

Formula and intuition

The notation captures the main operation or complexity statement behind Linear Regression. Read it together with the symbol key above and the step-by-step process in this guide.

Applications

When to use Linear Regression

Prices, revenue, demand, and trend estimation.

Implementation

Linear Regression from scratch in Python

This dependency-light example emphasizes the algorithm’s mechanics so each important step remains visible.

import numpy as np

# synthetic data: y is roughly 1.2 * x, plus random noise
rng = np.random.default_rng(0)
X = rng.uniform(0, 10, 100)
y = 1.2 * X + rng.normal(0, 2, 100)

# gradient descent on the squared-error loss: MSE = mean((y - (w*X+b))^2)
w, b = 0.0, 0.0
lr = 0.01
for _ in range(1000):
    y_pred = w * X + b
    dw = -2 * np.mean(X * (y - y_pred))  # d(MSE)/dw
    db = -2 * np.mean(y - y_pred)        # d(MSE)/db
    w -= lr * dw  # step downhill along the gradient
    b -= lr * db

print(f"slope={w:.2f} intercept={b:.2f}")

Practical tutorial

Build Linear Regression with scikit-learn

LinearRegression provides a dependable ordinary least-squares implementation with familiar fit and predict methods.

1. Prepare the data.Numeric features and a continuous target. Validate its shape, type, range, and ordering before training or execution.
2. Configure the algorithm.Begin with explicit, conservative parameters and a fixed random seed whenever the library supports one.
3. Fit or execute.Estimate slope and intercept by reducing residual error.
4. Inspect the result.A continuous value on the fitted line. Then apply the evaluation checks in the next section.

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, r2_score
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(7)
X = rng.uniform(0, 10, (240, 1))
y = 4.2 * X[:, 0] + 8 + rng.normal(0, 3, len(X))
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42
)

model = LinearRegression().fit(X_train, y_train)
prediction = model.predict(X_test)
print("slope:", model.coef_[0], "intercept:", model.intercept_)
print("MAE:", mean_absolute_error(y_test, prediction))
print("R²:", r2_score(y_test, prediction))

API details and version-specific options: official scikit-learn reference →

Evaluation

How to evaluate the result

A successful run is not enough. Evaluate the output against the intended use, compare it with a simple baseline, and preserve a genuinely unseen test case whenever the task involves learned parameters.

• Report MAE in the target’s original units and R² as a scale-free goodness-of-fit summary.
• Plot residuals against predictions; visible curves or funnels indicate violated linearity or constant-variance assumptions.

Common mistakes

Pitfalls and how to avoid them

These failure modes are common in tutorials and production systems. Treat them as review questions, not just after-the-fact debugging advice.

• Do not interpret correlation as causation.
• Outliers can move the fitted line substantially; inspect influential observations.
• Evaluate on held-out data instead of reporting training fit alone.

Project checklist

Before using Linear Regression in a project

• ✓Define the prediction or discovery objective before selecting the algorithm.
• ✓Split data before fitting preprocessing and tune only inside cross-validation.
• ✓Record data versions, random seeds, features, hyperparameters, and evaluation metrics.
• ✓Compare against a simple baseline and inspect errors by meaningful subgroups.
• ✓Monitor input drift and real-world performance after deployment.

Further reading

Official documentation and next steps

This guide is an educational introduction, not a substitute for domain validation. For consequential applications, review the source documentation, test against representative data, and involve a subject-matter expert.

Continue learning

Related machine learning guides

• Decision TreeSplits the feature space into rule-based regions.
• Random ForestCombines diverse decision trees trained on sampled data.
• K-Nearest NeighborsPredicts from the labels or values of nearby observations.