Q-Learning tutorial

Overview

What is Q-Learning?

Q-Learning is a reinforcement learning technique. Learns the long-term value of actions by exploring rewards in an environment. Its central idea is summarized by Q(s,a) ← Q(s,a) + α[r + γ maxₐ′ Q(s′,a′) − Q(s,a)].

This guide connects theory to practice. You will trace the input, intermediate process, and output; decode the notation; run a dependency-light implementation; then repeat the workflow with Gymnasium and NumPy and evaluate the result.

Preparation

Prerequisites and tools

You do not need an advanced software stack. Start with a recent Python environment and the fundamentals below, then install only the packages used by the practical example.

• Comfort with Python functions, NumPy arrays, and basic descriptive statistics.
• A clear distinction between training data, validation data, and untouched test data.
• Familiarity with features, targets, preprocessing, and task-appropriate evaluation metrics.

Process

How Q-Learning works

1. 1

   Input

   States, available actions, rewards, and observed next states.

2. 2

   Prepare and configure

   Check shapes, value ranges, ordering assumptions, missing values, and the parameters that control the algorithm’s behavior.

3. 3

   Algorithm process

   Update action values from reward plus the best future value.

4. 4

   Output

   A policy that chooses the highest-value action in each state.

5. 5

   Validate

   Evaluate the greedy policy over many episodes without exploration and report mean return plus success rate.

Core concept

Formula and intuition

The notation captures the main operation or complexity statement behind Q-Learning. Read it together with the symbol key above and the step-by-step process in this guide.

Applications

When to use Q-Learning

Navigation, scheduling, games, and sequential decision-making.

Implementation

Q-Learning from scratch in Python

This dependency-light example emphasizes the algorithm’s mechanics so each important step remains visible.

import numpy as np

size = 5
Q = np.zeros((size, size, 4))  # up, right, down, left
rng = np.random.default_rng(0)
alpha, gamma, epsilon = .2, .95, .2

for _ in range(1000):
    state = [0, 0]
    while state != [4, 4]:
        action = rng.integers(4) if rng.random() < epsilon else Q[state[0], state[1]].argmax()
        move = [(-1,0), (0,1), (1,0), (0,-1)][action]
        nxt = [np.clip(state[0]+move[0], 0, 4), np.clip(state[1]+move[1], 0, 4)]
        reward = 10 if nxt == [4, 4] else -1
        target = reward + gamma * Q[nxt[0], nxt[1]].max()
        Q[state[0], state[1], action] += alpha * (target - Q[state[0], state[1], action])
        state = nxt

Practical tutorial

Build Q-Learning with Gymnasium and NumPy

Gymnasium supplies a standard environment API while NumPy holds the tabular action-value function.

1. Prepare the data.States, available actions, rewards, and observed next states. Validate its shape, type, range, and ordering before training or execution.
2. Configure the algorithm.Begin with explicit, conservative parameters and a fixed random seed whenever the library supports one.
3. Fit or execute.Update action values from reward plus the best future value.
4. Inspect the result.A policy that chooses the highest-value action in each state. Then apply the evaluation checks in the next section.

import gymnasium as gym
import numpy as np

env = gym.make("FrozenLake-v1", is_slippery=False)
Q = np.zeros((env.observation_space.n, env.action_space.n))
rng = np.random.default_rng(7)
alpha, gamma = 0.2, 0.95

for episode in range(5000):
    state, _ = env.reset(seed=episode)
    terminated = truncated = False
    epsilon = max(0.02, 1 - episode / 3500)
    while not (terminated or truncated):
        action = rng.integers(4) if rng.random() < epsilon else Q[state].argmax()
        nxt, reward, terminated, truncated, _ = env.step(action)
        target = reward + gamma * Q[nxt].max() * (not terminated)
        Q[state, action] += alpha * (target - Q[state, action])
        state = nxt

print("greedy policy:", Q.argmax(axis=1).reshape(4, 4))
env.close()

API details and version-specific options: official Gymnasium and NumPy reference →

Evaluation

How to evaluate the result

A successful run is not enough. Evaluate the output against the intended use, compare it with a simple baseline, and preserve a genuinely unseen test case whenever the task involves learned parameters.

• Evaluate the greedy policy over many episodes without exploration and report mean return plus success rate.
• Plot episodic return and Q-value change to verify that learning has stabilized.

Common mistakes

Pitfalls and how to avoid them

These failure modes are common in tutorials and production systems. Treat them as review questions, not just after-the-fact debugging advice.

• Do not bootstrap from terminal states.
• Insufficient exploration can lock the agent into a poor policy.
• A table becomes impractical when states or actions are continuous.

Project checklist

Before using Q-Learning in a project

• ✓Define the prediction or discovery objective before selecting the algorithm.
• ✓Split data before fitting preprocessing and tune only inside cross-validation.
• ✓Record data versions, random seeds, features, hyperparameters, and evaluation metrics.
• ✓Compare against a simple baseline and inspect errors by meaningful subgroups.
• ✓Monitor input drift and real-world performance after deployment.

Further reading

Official documentation and next steps

This guide is an educational introduction, not a substitute for domain validation. For consequential applications, review the source documentation, test against representative data, and involve a subject-matter expert.

Continue learning

Related machine learning guides

• Linear RegressionFits a straight line that minimizes squared prediction errors.
• Logistic RegressionMaps a weighted feature score to a class probability.
• Decision TreeSplits the feature space into rule-based regions.