Logistic Regression tutorial

Overview

What is Logistic Regression?

Logistic Regression is a supervised learning technique. Maps a weighted feature score to a class probability. Its central idea is summarized by P(y = 1 | x) = σ(z), z = wᵀx + b.

This guide connects theory to practice. You will trace the input, intermediate process, and output; decode the notation; run a dependency-light implementation; then repeat the workflow with scikit-learn and evaluate the result.

Preparation

Prerequisites and tools

You do not need an advanced software stack. Start with a recent Python environment and the fundamentals below, then install only the packages used by the practical example.

• Comfort with Python functions, NumPy arrays, and basic descriptive statistics.
• A clear distinction between training data, validation data, and untouched test data.
• Familiarity with features, targets, preprocessing, and task-appropriate evaluation metrics.

Process

How Logistic Regression works

1. 1

   Input

   Numeric features and binary labels.

2. 2

   Prepare and configure

   Check shapes, value ranges, ordering assumptions, missing values, and the parameters that control the algorithm’s behavior.

3. 3

   Algorithm process

   Fit weights that maximize class likelihood.

4. 4

   Output

   A probability and a threshold-based class.

5. 5

   Validate

   Use ROC AUC or precision-recall AUC for ranking quality and inspect calibration when probabilities drive decisions.

Core concept

Formula and intuition

The notation captures the main operation or complexity statement behind Logistic Regression. Read it together with the symbol key above and the step-by-step process in this guide.

Applications

When to use Logistic Regression

Default risk, churn, fraud, and conversion.

Implementation

Logistic Regression from scratch in Python

This dependency-light example emphasizes the algorithm’s mechanics so each important step remains visible.

import numpy as np

# two features per sample; label is 1 when the features sum to a positive value
rng = np.random.default_rng(0)
X = rng.normal(0, 1, (200, 2))
y = (X[:, 0] + X[:, 1] > 0).astype(float)

# gradient descent on the cross-entropy loss using the sigmoid function
w = np.zeros(2)
b = 0.0
lr = 0.1
for _ in range(500):
    z = X @ w + b
    p = 1 / (1 + np.exp(-z))            # sigmoid: squashes the score into a 0-1 probability
    w -= lr * (X.T @ (p - y) / len(y))  # gradient of cross-entropy with respect to weights
    b -= lr * np.mean(p - y)

predictions = (1 / (1 + np.exp(-(X @ w + b))) > 0.5).astype(int)  # threshold at 0.5

Practical tutorial

Build Logistic Regression with scikit-learn

A scaling pipeline plus LogisticRegression is a strong, interpretable baseline for binary classification.

1. Prepare the data.Numeric features and binary labels. Validate its shape, type, range, and ordering before training or execution.
2. Configure the algorithm.Begin with explicit, conservative parameters and a fixed random seed whenever the library supports one.
3. Fit or execute.Fit weights that maximize class likelihood.
4. Inspect the result.A probability and a threshold-based class. Then apply the evaluation checks in the next section.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_classification(
    n_samples=600, n_features=8, n_informative=5, random_state=7
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42
)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
model.fit(X_train, y_train)
probability = model.predict_proba(X_test)[:, 1]
prediction = (probability >= 0.5).astype(int)
print("accuracy:", accuracy_score(y_test, prediction))
print("ROC AUC:", roc_auc_score(y_test, probability))

API details and version-specific options: official scikit-learn reference →

Evaluation

How to evaluate the result

A successful run is not enough. Evaluate the output against the intended use, compare it with a simple baseline, and preserve a genuinely unseen test case whenever the task involves learned parameters.

• Use ROC AUC or precision-recall AUC for ranking quality and inspect calibration when probabilities drive decisions.
• Choose the classification threshold from business costs rather than assuming 0.5 is always optimal.

Common mistakes

Pitfalls and how to avoid them

These failure modes are common in tutorials and production systems. Treat them as review questions, not just after-the-fact debugging advice.

• Scale features when regularization is active so coefficients receive comparable penalties.
• Accuracy can hide poor performance on an imbalanced minority class.
• Strong nonlinear relationships require feature engineering or a more flexible model.

Project checklist

Before using Logistic Regression in a project

• ✓Define the prediction or discovery objective before selecting the algorithm.
• ✓Split data before fitting preprocessing and tune only inside cross-validation.
• ✓Record data versions, random seeds, features, hyperparameters, and evaluation metrics.
• ✓Compare against a simple baseline and inspect errors by meaningful subgroups.
• ✓Monitor input drift and real-world performance after deployment.

Further reading

Official documentation and next steps

This guide is an educational introduction, not a substitute for domain validation. For consequential applications, review the source documentation, test against representative data, and involve a subject-matter expert.

Continue learning

Related machine learning guides

• Random ForestCombines diverse decision trees trained on sampled data.
• K-Nearest NeighborsPredicts from the labels or values of nearby observations.
• Support Vector MachineFinds a separating boundary with the widest possible margin.