Glossary · ML Atlas

A

A/B Testing: Comparing a new model against the current one on real users, by randomly sending some traffic to each and measuring which does better.(Technique)
A2C: Covered in Actor-Critic Methods
A3C: Covered in Actor-Critic Methods
accelerator: Covered in GPUs and AI Accelerators
Accuracy: The share of predictions a classifier gets right. Simple and intuitive, but misleading when one class is much rarer than the other.(Metric)
action: Covered in Markov Decision Process
Activation Function: The small nonlinear function each neuron applies to its weighted sum. Without it, a deep network would be no more powerful than a straight line.(Concept)
Active Learning: The model picks which unlabeled examples a human should label next, choosing the ones it is most unsure about, so you need far fewer labels.(Paradigm)
Actor-Critic Methods: RL methods with two learned parts: an actor that chooses actions (the policy) and a critic that estimates how good situations are, which makes learning more stable.(Family)
AdaBoost: The first practical boosting algorithm: trains many tiny models one after another, each focusing on the examples the previous ones got wrong, then lets them vote.(Model)
Adam and Other Optimizers: Improved versions of gradient descent. Momentum keeps the ball rolling; Adam also gives each weight its own step size based on how large its gradients have been.(Technique)
AdamW: Covered in Adam and Other Optimizers
adapters: Covered in LoRA (Low-Rank Adaptation)
adaptive boosting: Covered in AdaBoost
adaptive learning rate: Covered in Adam and Other Optimizers
ADAS: Covered in Driver Assistance and Self-Driving
advantage function: Covered in Actor-Critic Methods
adversarial attack: Covered in Adversarial Examples
Adversarial Examples: Inputs changed in ways humans barely notice that make a model confidently wrong, such as a few altered pixels that turn a panda into a gibbon.(Concept)
adversarial robustness: Covered in Adversarial Examples
adversarial testing: Covered in Red Teaming
adversarial training: Covered in Adversarial Examples
AE: Covered in Autoencoder
agent: Covered in Reinforcement Learning
agentic AI: Covered in AI Agents and Tool Use
agents: Covered in AI Agents and Tool Use
agglomerative clustering: Covered in Hierarchical Clustering
AI Act: Covered in EU AI Act and AI Regulation
AI Agents and Tool Use: A language model in a loop: it decides on a step, calls a tool (search, code, an API), reads the result and repeats until the task is done.(Technique)
AI Alignment: Making AI systems reliably pursue the goals and values their designers and users actually intend, including as they become more capable.(Concept)
AI ethics: Covered in Responsible AI, Safety and Security
AI feedback: Covered in Constitutional AI (RLAIF)
AI music: Covered in Music and Audio Generation
AI red teaming: Covered in Red Teaming
AI regulation: Covered in EU AI Act and AI Regulation
AI safety: Covered in Responsible AI, Safety and Security
AI security: Covered in Responsible AI, Safety and Security
Alexa: Covered in Voice Assistants
AlexNet: The deep convolutional network that won the 2012 ImageNet challenge by a huge margin and kicked off the deep learning boom.(Famous system)
algorithmic bias: Covered in Bias and Fairness
ALiBi: Covered in Positional Encoding
AlphaFold: DeepMind's system that predicts the 3D shape of proteins from their amino-acid sequence with near-experimental accuracy, recognized with the 2024 Nobel Prize in Chemistry.(Famous system)
AlphaFold 2: Covered in AlphaFold
AlphaGo: DeepMind's Go program that combined deep neural networks with tree search and, in March 2016, beat top professional Lee Sedol 4 games to 1.(Famous system)
AlphaGo Zero: Covered in AlphaZero
AlphaStar: DeepMind's StarCraft II agent, which reached Grandmaster level for all three races on the public ladder in 2019, under human-like viewing and speed limits.(Famous system)
AlphaZero: DeepMind's single algorithm that learned chess, shogi and Go from scratch through self-play, given only the rules, and beat the strongest existing programs in each.(Famous system)
ALS: Covered in Matrix Factorization
anchor boxes: Covered in Object Detection
ANN: Covered in Neural Network
annotation: Covered in Data Labeling
Anomaly Detection: Finding the rare data points that look unlike the rest, such as fraudulent payments, failing machines or intrusions, usually without labeled examples of the problem.(Technique)
API extraction: Covered in Model Extraction
Apple Face ID: Covered in Face ID and Face Recognition
approximate nearest neighbor: Covered in Vector Database
area under the curve: Covered in ROC Curve and AUC
ARIMA: Covered in Time Series Forecasting
artificial neural network: Covered in Neural Network
ASR: Covered in Speech Recognition
Atari: Covered in Deep Q-Network
Attention: Lets each element of a sequence look at every other element and decide how much each one matters, then mix their information accordingly.(Technique)
attention mechanism: Covered in Attention
AUC: Covered in ROC Curve and AUC
Audio Classification: Labeling sounds: is this a dog bark, glass breaking, a siren, a song, or the wake word 'Hey Siri'? Usually done by treating a spectrogram like an image.(Concept)
audio tokens: Covered in Music and Audio Generation
AudioSet: Covered in Audio Classification
augmentation: Covered in Data Augmentation
autocorrect: Covered in Keyboard Autocomplete
autodiff: Covered in Computational Graph and Chain Rule
Autoencoder: A network trained to copy its input through a narrow bottleneck, which forces it to learn a compact summary of the data.(Model)
autograd: Covered in Computational Graph and Chain Rule
automated valuation model: Covered in Home Price Estimators
automatic differentiation: Covered in Computational Graph and Chain Rule
automatic emergency braking: Covered in Driver Assistance and Self-Driving
automatic speech recognition: Covered in Speech Recognition
autonomous driving: Covered in Driver Assistance and Self-Driving
autoregression: Covered in Time Series Forecasting
autoregressive generation: Covered in Autoregressive Model
Autoregressive Model: A generative model that produces data one piece at a time, each piece predicted from all the pieces before it. GPT-style language models are the best-known example.(Family)
AVM: Covered in Home Price Estimators

B

backbone: Covered in Transfer Learning
backdoor attack: Covered in Data Poisoning
backgammon: Covered in TD-Gammon
backprop: Covered in Backpropagation
Backpropagation: The algorithm that computes how much each weight in a neural network contributed to the error, by applying the chain rule backward from the output.(Technique)
backward pass: Covered in Backpropagation
bagged trees: Covered in Bagging
Bagging: Trains many copies of a model on random resamples of the data and averages their predictions, which cancels out much of each model's jitter.(Technique)
base model: Covered in Pretraining
basis expansion: Covered in Polynomial Regression
Batch and Layer Normalization: Layers that rescale activations to a standard range inside the network, which makes deep networks train faster and more reliably.(Technique)
batch normalization: Covered in Batch and Layer Normalization
batch size: Covered in Epochs, Batches and Iterations
batching: Covered in Model Serving
BatchNorm: Covered in Batch and Layer Normalization
Baum-Welch: Covered in Hidden Markov Model
Bayes' rule: Covered in Probability Basics
Bayes' theorem: Covered in Naive Bayes
Bayesian classifier: Covered in Naive Bayes
Bayesian models: Covered in Probabilistic Models
Bayesian optimization: Covered in Hyperparameters
Bellman equation: Covered in Markov Decision Process
benchmark contamination: Covered in LLM Evaluation and Benchmarks
benchmarks: Covered in LLM Evaluation and Benchmarks
BERT: An encoder-only Transformer pretrained by filling in hidden words using context from both sides. Great at understanding text, not at generating it.(Model)
Bias and Fairness: Models trained on historical data can treat groups of people unequally. Fairness work measures those differences and tries to reduce them.(Concept)
bias and variance: Covered in Bias-Variance Trade-off
Bias-Variance Trade-off: Prediction error splits into bias (too simple to capture the pattern) and variance (too sensitive to the particular training sample). Reducing one often raises the other.(Concept)
Bidirectional Encoder Representations from Transformers: Covered in BERT
binary cross-entropy: Covered in Log Loss (Cross-Entropy)
bits per token: Covered in Perplexity
boosting: Covered in AdaBoost
bootstrap: Covered in Bagging
bootstrap aggregating: Covered in Bagging
bottleneck network: Covered in Autoencoder
bounding boxes: Covered in Object Detection
BPE: Covered in Tokenization
byte-pair encoding: Covered in Tokenization

C

CAI: Covered in Constitutional AI (RLAIF)
canary release: Covered in A/B Testing
Card Fraud Detection: Models that score every card payment for fraud risk in milliseconds, combining supervised classifiers such as gradient boosting with anomaly detection on spending patterns.(Famous system)
CART: Covered in Decision Tree
CASP: Covered in AlphaFold
catastrophic forgetting: Covered in Online Learning
CatBoost: Covered in Gradient Boosting
categorical cross-entropy: Covered in Log Loss (Cross-Entropy)
causal language model: Covered in GPT
causal language modeling: Covered in Autoregressive Model
CBOW: Covered in word2vec
CFG: Covered in Classifier-Free Guidance
chain rule: Covered in Computational Graph and Chain Rule
Chain-of-Thought: Getting a language model to write out intermediate reasoning steps before its final answer, which markedly improves results on math and logic problems.(Technique)
chat tuning: Covered in Instruction Tuning (SFT)
Chatbot Arena: Covered in LLM Evaluation and Benchmarks
ChatGPT: OpenAI's chat assistant, launched as a free research preview on 30 November 2022, which brought large language models to the general public.(Famous system)
checkpointing: Covered in Early Stopping
cheque reading: Covered in LeNet-5
Chinchilla: Covered in Scaling Laws
circuits: Covered in Mechanistic Interpretability
Classical Machine Learning: The pre-deep-learning toolkit: regression, trees, ensembles, nearest neighbors and clustering. Still the best choice for most tabular data.(Domain)
classification: Covered in Supervised Learning
classification tree: Covered in Decision Tree
Classifier-Free Guidance: A trick that makes diffusion models follow the prompt more closely by comparing a prompted and an unprompted prediction and pushing further in the difference.(Technique)
Claude: Anthropic's family of AI assistants: Transformer-based large language models trained with RLHF and Constitutional AI to be helpful, honest and harmless.(Famous system)
CLIP: A pair of encoders, one for images and one for text, trained so that a picture and a description of it land close together in the same embedding space.(Model)
cluster analysis: Covered in Clustering
Clustering: Grouping data points so that points in the same group are more similar to each other than to points in other groups, without any labels.(Family)
CNN: Covered in Convolutional Neural Network
coefficient of determination: Covered in Regression Metrics
collaborative filtering: Covered in Recommender Systems
Computational Graph and Chain Rule: Any calculation, including a neural network, can be drawn as a graph of simple operations. The chain rule then gives every input's effect on the output, one node at a time.(Concept)
compute-optimal training: Covered in Scaling Laws
Computer Vision: Teaching machines to understand images and video: what is in a picture, where it is, and which pixels belong to it.(Domain)
concept drift: Covered in Data Drift
conditional computation: Covered in Mixture of Experts
conditional diffusion: Covered in Classifier-Free Guidance
conditional distribution: Covered in Generative vs Discriminative Models
conditional probability: Covered in Probability Basics
confabulation: Covered in Hallucination
confusion matrix: Covered in Precision, Recall and F1
constitution: Covered in Constitutional AI (RLAIF)
Constitutional AI (RLAIF): Anthropic's method for training a harmless, helpful assistant using a written list of principles and AI-generated feedback, instead of relying only on human labels of harmful content.(Technique)
content-based filtering: Covered in Recommender Systems
context length: Covered in Context Window
context size: Covered in Context Window
Context Window: The maximum number of tokens an LLM can consider at once, including your prompt, any documents and its own answer. Anything outside it, the model cannot see.(Concept)
continual learning: Covered in Online Learning
continuous bag of words: Covered in word2vec
Contrastive Language-Image Pre-training: Covered in CLIP
Contrastive Learning: Learning embeddings by pulling matching pairs together and pushing non-matching pairs apart, without needing class labels.(Technique)
contrastive loss: Covered in Contrastive Learning
controlled experiment: Covered in A/B Testing
ConvNet: Covered in Convolutional Neural Network
convolution: Covered in Convolutional Neural Network
Convolutional Neural Network: A neural network that slides small learned filters across an image, so a pattern learned in one place is recognized anywhere. The backbone of computer vision for a decade.(Family)
core points: Covered in DBSCAN
cosine decay: Covered in Learning Rate
cosine similarity: Covered in Linear Algebra Basics
cost function: Covered in Loss Function
CoT: Covered in Chain-of-Thought
covariate shift: Covered in Data Drift
credit risk model: Covered in Credit Scoring
credit score: Covered in Credit Scoring
Credit Scoring: Predicting how likely a borrower is to repay, traditionally with interpretable scorecards based on logistic regression, increasingly with gradient boosting, under strict fairness and explainability rules.(Famous system)
critic: Covered in Actor-Critic Methods
cross-entropy: Covered in Perplexity
cross-entropy loss: Covered in Log Loss (Cross-Entropy)
Cross-Validation: Estimating how well a model will do on new data by training and testing it several times on different splits of the same dataset.(Technique)
crowdsourcing: Covered in Data Labeling
CUDA: Covered in GPUs and AI Accelerators
Curse of Dimensionality: As the number of features grows, data becomes sparse and distances lose meaning, so methods that rely on 'nearby' examples need exponentially more data.(Concept)
curve fitting: Covered in Polynomial Regression
CV: Covered in Cross-Validation

D

DALL-E: OpenAI's text-to-image models. The first (January 2021) generated images token by token with a Transformer; DALL-E 2 (April 2022) switched to diffusion guided by CLIP.(Famous system)
DALL-E 2: Covered in DALL-E
DALL·E: Covered in DALL-E
dangerous capability evaluations: Covered in Red Teaming
Data Augmentation: Creating extra training examples by changing existing ones in harmless ways, such as flipping, cropping or recoloring images.(Technique)
Data Drift: When the data a model sees in production gradually differs from its training data, so its accuracy quietly degrades.(Concept)
Data Labeling: Producing the correct answers (labels) that supervised models learn from, by people, by rules or by other models, and making sure those labels are consistent.(Technique)
data leakage: Covered in Train, Validation and Test Sets
data pipeline: Covered in Feature Engineering
Data Poisoning: Attacking a model by slipping manipulated examples into its training data, to degrade it or to plant a hidden backdoor that a secret trigger activates.(Concept)
Data, Features and Labels: The raw material of machine learning: examples described by features (the inputs) and, for supervised learning, a label (the answer to predict).(Concept)
dataset: Covered in Data, Features and Labels
DBSCAN: A clustering algorithm that grows clusters from dense regions of points, finds clusters of any shape and labels isolated points as noise.(Model)
DDPM: Covered in Diffusion Model
decision stumps: Covered in AdaBoost
Decision Tree: Predicts by asking a series of yes/no questions about the features, like a flowchart learned automatically from data.(Model)
decision tree methods: Covered in Tree-Based Models
decoder-only: Covered in Transformer
decoder-only Transformer: Covered in GPT
decoding: Covered in Sampling and Temperature
Deep Blue: IBM's chess computer that beat world champion Garry Kasparov in 1997. Included here as a contrast: it won by fast search and hand-tuned rules, not by learning.(Famous system)
deep learning: Covered in Neural Network
Deep Learning Architectures: How neural networks are wired for different data: convolutions for images, recurrence for sequences, attention and Transformers for nearly everything.(Domain)
Deep Q-Network: Q-learning with a deep neural network instead of a table, which learned to play dozens of Atari games from raw pixels and the score alone.(Model)
DeepSeek: Covered in DeepSeek-R1
DeepSeek-R1: An open-weight reasoning model from DeepSeek, released in January 2025 under the MIT license, trained with large-scale reinforcement learning to think step by step before answering.(Famous system)
demographic parity: Covered in Bias and Fairness
demonstrations: Covered in In-Context Learning
dendrogram: Covered in Hierarchical Clustering
denoising diffusion: Covered in Diffusion Model
dense network: Covered in Multilayer Perceptron
density-based clustering: Covered in DBSCAN
deployment: Covered in Model Serving
Differential Privacy: A mathematical guarantee that a model or statistic reveals almost nothing about any single person in the data, achieved by adding carefully calibrated noise.(Technique)
Diffusion Model: Learns to remove noise from images step by step. To generate, it starts from pure static and denoises it into a brand new image.(Model)
dimensionality: Covered in Curse of Dimensionality
Dimensionality Reduction: Methods that describe data with far fewer numbers while keeping what matters, to visualize it, speed up other models or remove noise.(Family)
Direct Preference Optimization: A simpler alternative to RLHF that trains a language model directly on pairs of preferred and rejected answers, with no separate reward model or reinforcement learning loop.(Technique)
discount factor: Covered in Markov Decision Process
Discover Weekly: Covered in Streaming and Video Recommendations
discriminative model: Covered in Generative vs Discriminative Models
discriminator: Covered in Generative Adversarial Network
disparate impact: Covered in Bias and Fairness
distillation: Covered in Knowledge Distillation
distillation attack: Covered in Model Extraction
distribution shift: Covered in Data Drift
document AI: Covered in Optical Character Recognition (OCR)
domain adaptation: Covered in Fine-Tuning
dot product: Covered in Linear Algebra Basics
double descent: Covered in Bias-Variance Trade-off
DP: Covered in Differential Privacy
DP-SGD: Covered in Differential Privacy
DPO: Covered in Direct Preference Optimization
DQN: Covered in Deep Q-Network
Dreamer: Covered in Model-Based Reinforcement Learning
Driver Assistance and Self-Driving: Cars that see the road with cameras, radar and lidar: from automatic emergency braking and lane keeping in ordinary cars to driverless robotaxis in some cities.(Famous system)
Dropout: During training, randomly switch off a fraction of neurons at every step, so the network cannot rely on any single neuron. Reduces overfitting.(Technique)

E

Early Stopping: Stop training when performance on a validation set stops improving, and keep the best version, before the model starts memorizing the training data.(Technique)
Edge AI (On-Device ML): Running models directly on phones, cars, cameras and sensors instead of in the cloud, for speed, privacy and offline use, which forces models to be small and efficient.(Concept)
elastic net: Covered in Ridge and Lasso Regression
Elman network: Covered in Recurrent Neural Network
EM algorithm: Covered in Gaussian Mixture Model
Email Spam Filters: The ML you use every day without noticing: classifiers that decide whether each email is spam, phishing or legitimate, evolving from naive Bayes to deep neural networks.(Famous system)
embedded ML: Covered in Edge AI (On-Device ML)
Embedding: A list of numbers (a vector) that represents a word, image or document, arranged so that similar things get similar vectors.(Concept)
embedding index: Covered in Vector Database
embedding visualization: Covered in t-SNE and UMAP
emergent abilities: Covered in Scaling Laws
encoder-decoder: Covered in Autoencoder
encoder-decoder RNN: Covered in Sequence-to-Sequence
encoder-only: Covered in Transformer
Ensemble Methods: Combine many models into one stronger model. Bagging averages independent models; boosting builds models that fix each other's mistakes.(Family)
ensembles: Covered in Ensemble Methods
environment: Covered in Reinforcement Learning
epoch: Covered in Epochs, Batches and Iterations
Epochs, Batches and Iterations: Training vocabulary: a batch is the group of examples used for one update, an iteration is one update, and an epoch is one full pass over the training data.(Concept)
epsilon: Covered in Differential Privacy
epsilon-greedy: Covered in Exploration vs Exploitation
equal opportunity: Covered in Bias and Fairness
error function: Covered in Loss Function
EU AI Act and AI Regulation: The European Union's law regulating AI by risk level: some uses are banned, high-risk systems face strict requirements, and general-purpose AI models have transparency duties.(Concept)
evals: Covered in LLM Evaluation and Benchmarks
example: Covered in Data, Features and Labels
expectation-maximization: Covered in Gaussian Mixture Model
expected value: Covered in Probability Basics
experience replay: Covered in Deep Q-Network
Experiment Tracking: Recording every training run's code version, data, hyperparameters, metrics and resulting model, so results can be compared, reproduced and safely deployed.(Technique)
Explainability: Methods that show why a model made a decision, such as which inputs mattered most, so people can check, debug and trust it.(Technique)
explained variance: Covered in Principal Component Analysis
exploding gradients: Covered in Vanishing and Exploding Gradients
Exploration vs Exploitation: The trade-off every learner faces: keep doing what has worked so far, or try something new that might be better.(Concept)
exponential smoothing: Covered in Time Series Forecasting
extended thinking: Covered in Reasoning Models

F

F1 score: Covered in Precision, Recall and F1
Face ID and Face Recognition: Unlocking a phone with your face: a depth camera and neural networks compare your face to a stored template, the best-known everyday use of face recognition.(Famous system)
face recognition: Covered in Face ID and Face Recognition
face verification: Covered in Face ID and Face Recognition
facial recognition: Covered in Face ID and Face Recognition
factual errors: Covered in Hallucination
fairness metrics: Covered in Bias and Fairness
FAISS: Covered in Vector Database
false positive: Covered in Precision, Recall and F1
false positive rate: Covered in ROC Curve and AUC
feature: Covered in Data, Features and Labels
Feature Engineering: Turning raw data into the input columns a model learns from: cleaning, combining, scaling and encoding information so patterns are easy to find.(Technique)
feature extraction: Covered in Dimensionality Reduction
feature importance: Covered in Explainability
feature scaling: Covered in Feature Engineering
features: Covered in Mechanistic Interpretability
FedAvg: Covered in Federated Learning
federated averaging: Covered in Federated Learning
Federated Learning: Training one shared model across many devices or organizations without collecting their raw data: each trains locally and only sends model updates.(Paradigm)
feedforward neural network: Covered in Multilayer Perceptron
few labels: Covered in Semi-Supervised Learning
few-shot: Covered in Few-Shot and Zero-Shot Learning
Few-Shot and Zero-Shot Learning: Handling new tasks or classes from a handful of examples (few-shot) or from a description alone (zero-shot), by reusing knowledge from pretraining.(Paradigm)
few-shot examples: Covered in In-Context Learning
few-shot prompting: Covered in In-Context Learning
FGSM: Covered in Adversarial Examples
FICO score: Covered in Credit Scoring
find dogs: Covered in Photo Library Search
Fine-Tuning: Continuing to train a pretrained model on your own examples to specialize it. LoRA does this cheaply by training small add-on matrices instead of all the weights.(Technique)
flips: Covered in Data Augmentation
flow models: Covered in Normalizing Flows
forecasting: Covered in Time Series Forecasting
forward pass: Covered in Computational Graph and Chain Rule
forward-backward algorithm: Covered in Hidden Markov Model
foundation model: Covered in Large Language Model
foundation model training: Covered in Pretraining
foundation models: Covered in Transfer Learning
Foundations: The shared ideas behind every ML model: data, a loss to minimize, an optimizer to minimize it, and checks that the model generalizes.(Domain)
fraud model: Covered in Card Fraud Detection
frequency: Covered in Spectrogram
full fine-tuning: Covered in Fine-Tuning
fully connected network: Covered in Multilayer Perceptron
function calling: Covered in AI Agents and Tool Use
Funk SVD: Covered in Matrix Factorization

G

game tree search: Covered in Deep Blue
GAN: Covered in Generative Adversarial Network
gated recurrent unit: Covered in Gated Recurrent Unit (GRU)
Gated Recurrent Unit (GRU): A simplified LSTM with two gates instead of three and no separate memory cell. Often just as accurate, with fewer parameters.(Model)
gating network: Covered in Mixture of Experts
Gaussian: Covered in Probability Basics
Gaussian kernel: Covered in Kernel Trick
Gaussian Mixture Model: Models data as a blend of several bell-shaped (Gaussian) clusters and assigns each point a probability of belonging to each one.(Model)
GBM: Covered in Gradient Boosting
Gboard: Covered in Keyboard Autocomplete
GCN: Covered in Graph Neural Network
GD: Covered in Gradient Descent
GELU: Covered in Activation Function
Gemini: Google DeepMind's family of AI models and assistant, designed from the start to work across text, images, audio, video and code.(Famous system)
general-purpose AI: Covered in EU AI Act and AI Regulation
generalization gap: Covered in Overfitting
generalized linear models: Covered in Linear Models
Generative Adversarial Network: Two networks compete: a generator makes fakes, a discriminator tries to spot them. Each improves until the fakes are hard to tell from real data.(Model)
Generative AI: Models that learn what data looks like well enough to create new examples: images, text, audio and video.(Domain)
generative classifiers: Covered in Probabilistic Models
generative model: Covered in Generative vs Discriminative Models
Generative Pre-trained Transformer: Covered in GPT
Generative vs Discriminative Models: Discriminative models learn to tell classes apart; generative models learn what the data itself looks like, so they can also create new examples.(Concept)
generator: Covered in Generative Adversarial Network
GLM: Covered in Linear Models
Glorot initialization: Covered in Weight Initialization
GloVe: Covered in word2vec
Glow: Covered in Normalizing Flows
Gmail spam filter: Covered in Email Spam Filters
GMM: Covered in Gaussian Mixture Model
GNMT: Covered in Google Translate
GNN: Covered in Graph Neural Network
Goodhart's law: Covered in Reward Hacking
Google Assistant: Covered in Voice Assistants
Google Neural Machine Translation: Covered in Google Translate
Google Photos: Covered in Photo Library Search
Google Translate: Google's translation service, which switched from phrase-based statistical translation to neural sequence-to-sequence models in 2016 and later to Transformers.(Famous system)
Google Veo: Covered in Veo
GPT: A decoder-only Transformer that generates text one token at a time, each time predicting the next token from everything before it.(Model)
GPT-2: OpenAI's 1.5-billion-parameter language model from 2019, famous for surprisingly fluent text and for being released in stages over misuse concerns.(Famous system)
GPT-3: OpenAI's 175-billion-parameter language model that showed a big enough model can do new tasks from just a few examples in its prompt.(Famous system)
GPT2: Covered in GPT-2
GPU: Covered in GPUs and AI Accelerators
GPUs and AI Accelerators: Specialized chips that do thousands of multiplications in parallel. Deep learning took off when graphics cards turned out to be perfect for neural networks.(Concept)
gradient boosted trees: Covered in Gradient Boosting
Gradient Boosting: Builds an ensemble of small trees one at a time, each trained to correct the remaining errors of all trees before it.(Model)
gradient clipping: Covered in Vanishing and Exploding Gradients
Gradient Descent: Finds good model parameters by repeatedly taking small steps downhill on the loss, in the direction that reduces error fastest.(Technique)
graph convolutional network: Covered in Graph Neural Network
Graph Neural Network: Neural networks for data shaped like a network, such as molecules, social graphs or road maps, where each node learns from its neighbors.(Family)
greedy decoding: Covered in Sampling and Temperature
grid search: Covered in Hyperparameters
ground truth: Covered in Data Labeling
grounding: Covered in Hallucination
GRU: Covered in Gated Recurrent Unit (GRU)
guidance scale: Covered in Classifier-Free Guidance

H

Hallucination: When a language model states something false or made up, fluently and confidently, as if it were fact.(Concept)
handwriting recognition: Covered in Optical Character Recognition (OCR)
HDBSCAN: Covered in DBSCAN
He initialization: Covered in Weight Initialization
Hidden Markov Model: A probabilistic model of sequences in which an unseen state changes step by step and each state emits an observation you can see.(Model)
Hierarchical Clustering: Builds a tree of clusters by repeatedly merging the two closest groups, so you can cut the tree at any level to get as many clusters as you like.(Model)
high-dimensional data: Covered in Curse of Dimensionality
high-risk AI: Covered in EU AI Act and AI Regulation
HMM: Covered in Hidden Markov Model
HNSW: Covered in Vector Database
holdout: Covered in Train, Validation and Test Sets
Holt-Winters: Covered in Time Series Forecasting
Home Price Estimators: Websites that estimate what any house is worth, such as Zillow's Zestimate, using regression models trained on past sales, home features and location.(Famous system)
house price prediction: Covered in Home Price Estimators
human in the loop: Covered in Active Learning
human pose estimation: Covered in Pose Estimation
hyperparameter tuning: Covered in Hyperparameters
Hyperparameters: Settings chosen before training, such as learning rate, tree depth or number of layers, as opposed to parameters, which the model learns from data.(Concept)

I

IBM Deep Blue: Covered in Deep Blue
ICL: Covered in In-Context Learning
iForest: Covered in Isolation Forest
image captioning: Covered in Vision-Language Model
Image Classification: Assigning one label to a whole image, such as cat, dog or car. The task that kicked off the deep learning era.(Concept)
image embeddings: Covered in Photo Library Search
image generation: Covered in Text-to-Image Generation
image patches: Covered in Vision Transformer
image recognition: Covered in Image Classification
image search: Covered in Photo Library Search
Image Segmentation: Labeling every pixel of an image, so the computer knows the exact outline of each object, not just a box around it.(Technique)
image-to-video: Covered in Text-to-Video
ImageNet: Covered in Image Classification
In-Context Learning: An LLM's ability to pick up a new task from examples written in the prompt, without any change to its weights.(Concept)
incremental learning: Covered in Online Learning
indirect prompt injection: Covered in Prompt Injection
induction heads: Covered in Mechanistic Interpretability
inference: Covered in Model Serving
inference-time scaling: Covered in Reasoning Models
InfoNCE: Covered in Contrastive Learning
inner alignment: Covered in AI Alignment
input variables: Covered in Data, Features and Labels
instance segmentation: Covered in Image Segmentation
Instance-Based Learning: Methods that keep the training examples themselves and predict by comparing a new input to the most similar stored ones, instead of learning a formula.(Family)
instruction following: Covered in Instruction Tuning (SFT)
Instruction Tuning (SFT): Fine-tuning a pretrained language model on examples of instructions paired with good responses, turning a text-continuer into an assistant that follows requests.(Technique)
int8: Covered in Quantization
inter-annotator agreement: Covered in Data Labeling
interpolation: Covered in Latent Space
interpretability: Covered in Explainability
intersection over union: Covered in Object Detection
invertible networks: Covered in Normalizing Flows
IoU: Covered in Object Detection
Isolation Forest: Spots anomalies by cutting the data with random splits: odd points get isolated after only a few cuts, normal points need many.(Model)
isolation trees: Covered in Isolation Forest
iteration: Covered in Epochs, Batches and Iterations

J

jailbreak: Covered in Prompt Injection
jailbreak testing: Covered in Red Teaming
joint distribution: Covered in Generative vs Discriminative Models
Jukebox: Covered in Music and Audio Generation
junk mail: Covered in Email Spam Filters

K

k-fold cross-validation: Covered in Cross-Validation
k-means: Covered in k-Means Clustering
k-Means Clustering: Splits data into k groups by alternating two steps: assign each point to its nearest center, then move each center to the middle of its points.(Model)
k-Nearest Neighbors: Predicts for a new point by finding the k most similar training examples and letting them vote. No real training step at all.(Model)
k-NN: Covered in k-Nearest Neighbors
Kaiming initialization: Covered in Weight Initialization
Kasparov: Covered in Deep Blue
kernel function: Covered in Kernel Trick
kernel machines: Covered in Kernel Methods
Kernel Methods: Methods that find curved boundaries by implicitly mapping data into a richer space where a simple straight-line method works, using the kernel trick.(Family)
Kernel Trick: Lets a linear method draw curved boundaries by computing similarities as if the data lived in a higher-dimensional space, without ever building that space.(Technique)
kernelized models: Covered in Kernel Methods
Keyboard Autocomplete: The word suggestions above your phone keyboard: small on-device language models that predict your next word, in Gboard's case trained with federated learning so your typing stays on your phone.(Famous system)
keypoint detection: Covered in Pose Estimation
keyword spotting: Covered in Audio Classification
king - man + woman: Covered in word2vec
kNN: Covered in k-Nearest Neighbors
Knowledge Distillation: Training a small, fast student model to imitate a large teacher model's outputs, keeping much of the quality at a fraction of the cost.(Technique)
KV cache: Covered in Context Window

L

L1: Covered in Regularization
L1 regression: Covered in Ridge and Lasso Regression
L2: Covered in Regularization
L2 regression: Covered in Ridge and Lasso Regression
label: Covered in Data, Features and Labels
label noise: Covered in Data Labeling
label propagation: Covered in Semi-Supervised Learning
labeling: Covered in Data Labeling
lane keeping: Covered in Driver Assistance and Self-Driving
Language and LLMs: How machines process language, from tokens and embeddings to the large language models behind ChatGPT, Gemini and Claude.(Domain)
language model: Covered in Large Language Model
language model evaluation: Covered in Perplexity
language modeling: Covered in Autoregressive Model
Large Language Model: A very large Transformer trained to predict the next token on vast amounts of text, then tuned to follow instructions. The engine behind chat assistants.(Family)
lasso: Covered in Ridge and Lasso Regression
latency: Covered in Model Serving
latent code: Covered in Latent Space
latent diffusion: Covered in Diffusion Model
latent factor model: Covered in Matrix Factorization
Latent Space: A compressed space of hidden coordinates where a model represents its data. Nearby points mean similar things, and moving through it changes the output smoothly.(Concept)
latent variable: Covered in Latent Space
latent vector: Covered in Latent Space
layer normalization: Covered in Batch and Layer Normalization
LayerNorm: Covered in Batch and Layer Normalization
lazy learning: Covered in Instance-Based Learning
Le Chat: Covered in Mistral AI Models
league training: Covered in AlphaStar
Learning Paradigms: The different ways a machine can learn: from labeled answers, from raw unlabeled data, from data that labels itself, or from rewards.(Domain)
Learning Rate: How big a step gradient descent takes on each update. Too small and training crawls; too large and it overshoots and diverges.(Concept)
learning rate schedule: Covered in Learning Rate
least squares regression: Covered in Linear Regression
leave-one-out: Covered in Cross-Validation
Lee Sedol: Covered in AlphaGo
LeNet: Covered in LeNet-5
LeNet-5: Yann LeCun's classic convolutional network for handwritten digits, which established the CNN blueprint and helped read millions of bank cheques a day in the 1990s.(Famous system)
let's think step by step: Covered in Chain-of-Thought
LightGBM: Covered in Gradient Boosting
LIME: Covered in Explainability
Linear Algebra Basics: Vectors, matrices and the dot product: the arithmetic that every model, from linear regression to transformers, runs on.(Concept)
Linear Models: Models that predict from a weighted sum of the input features. Simple, fast, interpretable, and the starting point for nearly everything else.(Family)
linear recurrence: Covered in State Space Models
Linear Regression: Predicts a number by fitting the straight line (or flat plane) that keeps the squared prediction errors as small as possible.(Model)
linkage: Covered in Hierarchical Clustering
Llama: Meta's family of language models whose weights are downloadable, which made it a foundation for a huge ecosystem of open models, fine-tunes and local AI.(Famous system)
LLM: Covered in Large Language Model
LLM Evaluation and Benchmarks: How we measure what language models can do: standardized benchmarks, human preference comparisons, model-graded tests and task-specific evals, each with its own blind spots.(Concept)
LLM security: Covered in Prompt Injection
LLM-as-a-judge: Covered in LLM Evaluation and Benchmarks
Lloyd's algorithm: Covered in k-Means Clustering
loan default prediction: Covered in Credit Scoring
Log Loss (Cross-Entropy): The standard loss and metric for probabilistic classifiers: it heavily punishes confident wrong answers and rewards well-calibrated probabilities.(Metric)
log-Mel: Covered in Spectrogram
Logistic Regression: Despite its name, a classifier: it turns a weighted sum of features into a probability between 0 and 1 using the S-shaped sigmoid curve.(Model)
logit model: Covered in Logistic Regression
long context: Covered in Context Window
Long Short-Term Memory: An RNN with gates that decide what to remember, what to forget and what to output, letting it keep information over much longer sequences.(Model)
LoRA (Low-Rank Adaptation): Fine-tunes a large model by freezing its weights and learning small low-rank 'patch' matrices instead, cutting trainable parameters by orders of magnitude.(Technique)
Loss Function: A formula that turns a model's mistakes into one number. Training means making that number as small as possible.(Concept)
lottery ticket hypothesis: Covered in Pruning
low-rank adaptation: Covered in LoRA (Low-Rank Adaptation)
LSTM: Covered in Long Short-Term Memory

M

machine learning operations: Covered in ML Engineering and MLOps
machine translation: Covered in Google Translate
MAE: Covered in Regression Metrics
magnitude pruning: Covered in Pruning
Mamba: Covered in State Space Models
manifold learning: Covered in Dimensionality Reduction
Markov Decision Process: The standard way to describe a reinforcement learning problem: states, actions, rewards, how actions change the state, and a discount for future rewards.(Concept)
Mask R-CNN: Covered in Image Segmentation
masked language model: Covered in BERT
masks: Covered in Image Segmentation
matrices: Covered in Linear Algebra Basics
Matrix Factorization: Fills in a huge, mostly empty user-by-item ratings table by describing every user and every item with a few hidden taste numbers.(Model)
matrix multiplication: Covered in Linear Algebra Basics
maximum margin: Covered in Support Vector Machine
MCTS: Covered in Monte Carlo Tree Search
MDP: Covered in Markov Decision Process
mean absolute error: Covered in Regression Metrics
Mechanistic Interpretability: Reverse-engineering what is happening inside a neural network: which internal features it represents and which circuits of neurons compute its behavior.(Technique)
Mel spectrogram: Covered in Spectrogram
Membership Inference and Data Extraction: Privacy attacks that reveal whether a specific record was in a model's training data, or that make the model regurgitate memorized training text.(Concept)
membership inference attack: Covered in Membership Inference and Data Extraction
memorization: Covered in Membership Inference and Data Extraction
memory-based learning: Covered in Instance-Based Learning
message passing: Covered in Graph Neural Network
meta-learning: Covered in Few-Shot and Zero-Shot Learning
MIA: Covered in Membership Inference and Data Extraction
Midjourney: A popular text-to-image service from an independent San Francisco lab, launched in open beta in July 2022 and first used entirely through a Discord bot.(Famous system)
Midjourney bot: Covered in Midjourney
mini-batch: Covered in Epochs, Batches and Iterations
mini-batch gradient descent: Covered in Gradient Descent
Mistral 7B: Covered in Mistral AI Models
Mistral AI Models: Language models from the Paris start-up Mistral AI, known for efficient open-weight releases such as Mistral 7B and the mixture-of-experts Mixtral 8x7B.(Famous system)
Mixtral: Covered in Mistral AI Models
Mixtral 8x7B: Covered in Mistral AI Models
mixture model: Covered in Gaussian Mixture Model
Mixture of Experts: A layer made of many expert sub-networks plus a router that sends each token to only a few of them, so the model can be huge while each token uses a small part.(Technique)
ML Engineering and MLOps: Everything it takes to get a model out of a notebook and keep it working in the real world: data pipelines, serving, hardware, monitoring and experiments.(Domain)
ML in production: Covered in ML Engineering and MLOps
MLflow: Covered in Experiment Tracking
MLM: Covered in BERT
MLP: Covered in Multilayer Perceptron
MMLU: Covered in LLM Evaluation and Benchmarks
MNIST: Covered in LeNet-5
mobile ML: Covered in Edge AI (On-Device ML)
mode collapse: Covered in StyleGAN
model complexity: Covered in Bias-Variance Trade-off
model compression: Covered in Knowledge Distillation
Model Extraction: Copying a model's behavior, or recovering parts of its parameters, just by sending it many queries and studying the answers.(Concept)
model monitoring: Covered in Data Drift
model registry: Covered in Experiment Tracking
Model Serving: Running a trained model as a service that answers requests quickly, reliably and affordably, which means balancing latency, throughput and cost.(Technique)
model stealing: Covered in Model Extraction
model theft: Covered in Model Extraction
Model-Based Reinforcement Learning: RL in which the agent learns or is given a model of how the world responds to its actions, and uses it to plan or to practice in imagination.(Family)
model-free vs model-based: Covered in Model-Based Reinforcement Learning
MoE: Covered in Mixture of Experts
momentum: Covered in Adam and Other Optimizers
Monte Carlo Tree Search: Plans by building a tree of possible futures, spending more simulations on promising moves, and choosing the move whose subtree looks best.(Technique)
MSE: Covered in Regression Metrics
multi-armed bandit: Covered in Exploration vs Exploitation
multi-head attention: Covered in Attention
Multilayer Perceptron: Layers of neurons where every neuron connects to every neuron in the next layer. The simplest deep network, and a building block inside Transformers.(Model)
Multimodal AI: Models that connect different kinds of data, such as images, text and audio, so they can describe pictures, answer questions about them or generate one from the other.(Domain)
multimodal LLM: Covered in Vision-Language Model
Music and Audio Generation: Models that generate music, sound effects or ambient audio from a text prompt, melody or other audio, usually by modeling compressed audio tokens or spectrogram latents.(Family)
MusicLM: Covered in Music and Audio Generation
MuZero: Covered in AlphaZero

N

Naive Bayes: A fast probabilistic classifier that applies Bayes' theorem while naively assuming every feature is independent. Famous as an early spam filter.(Model)
nearest neighbor: Covered in k-Nearest Neighbors
nearest-neighbor methods: Covered in Instance-Based Learning
negative log-likelihood: Covered in Log Loss (Cross-Entropy)
negative prompt: Covered in Classifier-Free Guidance
Netflix Prize: Covered in Streaming and Video Recommendations
Netflix recommendations: Covered in Streaming and Video Recommendations
neural audio codec: Covered in Music and Audio Generation
neural machine translation: Covered in Google Translate
neural net: Covered in Neural Network
Neural Network: A model made of layers of simple connected units (neurons). With enough layers and data it can learn almost any pattern, which is what deep learning means.(Family)
Neural Network Fundamentals: Artificial neurons, layers, activation functions and backpropagation: the building blocks shared by every deep learning model.(Domain)
next-token prediction: Covered in Autoregressive Model
next-word prediction: Covered in Keyboard Autocomplete
NLL: Covered in Log Loss (Cross-Entropy)
NMS: Covered in Object Detection
noise points: Covered in DBSCAN
non-max suppression: Covered in Object Detection
non-parametric methods: Covered in Instance-Based Learning
nonlinear dimensionality reduction: Covered in t-SNE and UMAP
nonlinearity: Covered in Activation Function
normal distribution: Covered in Probability Basics
normalization: Covered in Feature Engineering
Normalizing Flows: Generative models built from a chain of reversible transformations that turn simple noise into data and back, giving exact probabilities.(Model)
novelty detection: Covered in Anomaly Detection
novelty detection SVM: Covered in One-Class SVM
NPU: Covered in Edge AI (On-Device ML)
nucleus sampling: Covered in Sampling and Temperature

O

o1: Covered in Reasoning Models
Object Detection: Finding every object in an image and drawing a labeled box around each one, often in real time.(Technique)
objective function: Covered in Loss Function
OCR: Covered in Optical Character Recognition (OCR)
OCSVM: Covered in One-Class SVM
OLS: Covered in Linear Regression
on-device AI: Covered in Edge AI (On-Device ML)
on-device learning: Covered in Federated Learning
One-Class SVM: Learns a tight boundary around normal data only, then flags anything that falls outside it as an anomaly.(Model)
one-hot encoding: Covered in Feature Engineering
one-shot learning: Covered in Few-Shot and Zero-Shot Learning
online experiment: Covered in A/B Testing
Online Learning: Learning from data that arrives one example (or small batch) at a time, updating the model continuously instead of retraining from scratch.(Paradigm)
open-weight models: Covered in Llama
OpenAI Sora: Covered in Sora
OpenPose: Covered in Pose Estimation
Optical Character Recognition (OCR): Turning images of text, from scanned pages to street signs and handwriting, into machine-readable characters.(Concept)
optimizers: Covered in Adam and Other Optimizers
ordinary least squares: Covered in Linear Regression
outer alignment: Covered in AI Alignment
outlier detection: Covered in Anomaly Detection
Overfitting: When a model memorizes its training data, noise included, and then fails on new data. Its opposite, underfitting, is a model too simple to learn the pattern.(Concept)

P

parameter-efficient fine-tuning: Covered in LoRA (Low-Rank Adaptation)
patience: Covered in Early Stopping
payment fraud detection: Covered in Card Fraud Detection
PCA: Covered in Principal Component Analysis
PEFT: Covered in LoRA (Low-Rank Adaptation)
Perceptron: The first learning artificial neuron: a weighted sum of inputs followed by a hard yes/no threshold, with a simple rule to learn the weights.(Model)
Perplexity: How surprised a language model is by real text. Lower is better: a perplexity of 20 means the model is about as unsure as if it were choosing among 20 equally likely words each time.(Metric)
phishing detection: Covered in Email Spam Filters
Photo Library Search: Typing 'dog', 'beach' or 'receipt' into your photo app and finding matching pictures you never tagged, powered by image classifiers and image embeddings.(Famous system)
photo tagging: Covered in Photo Library Search
PixelCNN: Covered in Autoregressive Model
planning: Covered in Model-Based Reinforcement Learning
playing against itself: Covered in Self-Play
poisoning attack: Covered in Data Poisoning
policy: Covered in Markov Decision Process
Policy Gradient: Directly adjusts the agent's strategy to make actions that led to high reward more likely. PPO, a popular version, is also used in RLHF.(Technique)
polynomial features: Covered in Polynomial Regression
polynomial kernel: Covered in Kernel Trick
Polynomial Regression: Linear regression on powers of the input (x, x², x³...), so a 'linear' model can fit curves. The classic demo of underfitting and overfitting.(Model)
population-based training: Covered in Self-Play
Pose Estimation: Finding the positions of body joints (shoulders, elbows, knees...) in an image or video, producing a stick-figure skeleton for each person.(Concept)
position embeddings: Covered in Positional Encoding
Positional Encoding: Extra information added to each token so a Transformer knows word order, since attention by itself treats a sentence as an unordered bag of words.(Technique)
positive and negative pairs: Covered in Contrastive Learning
post-training: Covered in Instruction Tuning (SFT)
PPO: Covered in Policy Gradient
pre-training: Covered in Pretraining
precision: Covered in Precision, Recall and F1
Precision, Recall and F1: Precision asks how many of the model's alarms were real; recall asks how many real cases it caught. F1 combines both.(Metric)
predictive text: Covered in Keyboard Autocomplete
preference optimization: Covered in Direct Preference Optimization
preference tuning: Covered in Direct Preference Optimization
pretrained models: Covered in Transfer Learning
Pretraining: The first and most expensive stage of building an LLM: training on trillions of tokens of text to predict the next token, which produces a general-purpose base model.(Technique)
Principal Component Analysis: Finds the few directions along which data varies the most, so many columns can be summarized by a handful of new ones with little information lost.(Technique)
principal components: Covered in Principal Component Analysis
privacy attack: Covered in Membership Inference and Data Extraction
privacy budget: Covered in Differential Privacy
probabilistic graphical models: Covered in Probabilistic Models
Probabilistic Models: Models that describe data with probability distributions, so they can say how likely each answer is and reason about uncertainty.(Family)
probability: Covered in Probability Basics
Probability Basics: The language of uncertainty that most of ML is written in: distributions, conditional probability, Bayes' rule and expected values.(Concept)
probability distribution: Covered in Probability Basics
probing: Covered in Mechanistic Interpretability
projection: Covered in Dimensionality Reduction
prompt engineering: Covered in Prompting
Prompt Injection: Smuggling instructions into the text a language model reads, such as a web page or email, so it follows the attacker instead of the user.(Concept)
prompt to image: Covered in Text-to-Image Generation
Prompting: Steering a language model with the text you give it: clear instructions, examples of what you want, and room to reason step by step.(Technique)
protein structure prediction: Covered in AlphaFold
proximal policy optimization: Covered in Policy Gradient
Pruning: Removing weights, neurons or whole layers that contribute little, to make a trained network smaller and faster with little loss in accuracy.(Technique)
pseudo-labeling: Covered in Semi-Supervised Learning

Q

Q-Learning: Learns how good each action is in each state (its Q-value) from experience, by repeatedly nudging estimates toward 'reward now plus the best value next'.(Model)
Q-table: Covered in Q-Learning
Q-values: Covered in Q-Learning
QKV: Covered in Attention
QLoRA: Covered in LoRA (Low-Rank Adaptation)
Quantization: Storing a model's numbers with fewer bits, for example 8 or 4 instead of 16 or 32, so it becomes smaller, faster and cheaper with only a small loss in quality.(Technique)
query strategy: Covered in Active Learning

R

R squared: Covered in Regression Metrics
R-CNN: Covered in Object Detection
R1: Covered in DeepSeek-R1
R²: Covered in Regression Metrics
RAG: Covered in Retrieval-Augmented Generation
random crops: Covered in Data Augmentation
random decision forest: Covered in Random Forest
Random Forest: Many decision trees, each trained on a random sample of the data and features, vote together. Robust, accurate and hard to break.(Model)
random forests: Covered in Random Forest
random search: Covered in Hyperparameters
RBF kernel: Covered in Kernel Trick
ReAct: Covered in AI Agents and Tool Use
RealNVP: Covered in Normalizing Flows
reasoning model: Covered in DeepSeek-R1
Reasoning Models: LLMs trained with reinforcement learning to think through a long internal chain of reasoning before answering, trading extra inference time for accuracy on hard problems.(Family)
recall: Covered in Precision, Recall and F1
receiver operating characteristic: Covered in ROC Curve and AUC
recommendations: Covered in Recommender Systems
Recommender Systems: Systems that predict what you will like, from what you and people similar to you have watched, bought or listened to before.(Family)
Recurrent Neural Network: A neural network that reads a sequence one step at a time and carries a hidden memory from each step to the next.(Family)
Red Teaming: Deliberately attacking an AI system, as an adversary would, to find harmful behaviors, jailbreaks and security holes before real users or attackers do.(Technique)
regression: Covered in Supervised Learning
Regression Metrics: Ways to score predictions of numbers: average error size (MAE), average squared error (MSE, RMSE), and the share of variation explained (R²).(Metric)
regression tree: Covered in Decision Tree
Regularization: Any technique that discourages a model from becoming too complex, so it generalizes better to new data.(Technique)
Regulation (EU) 2024/1689: Covered in EU AI Act and AI Regulation
REINFORCE: Covered in Policy Gradient
Reinforcement Learning: Learning by trial and error: an agent acts in an environment, receives rewards, and gradually learns a strategy that collects as much reward as possible.(Domain)
reinforcement learning from AI feedback: Covered in Constitutional AI (RLAIF)
Reinforcement Learning from Human Feedback: Covered in RLHF
ReLU: Covered in Activation Function
representation: Covered in Embedding
representation space: Covered in Latent Space
reproducibility: Covered in Experiment Tracking
reset gate: Covered in Gated Recurrent Unit (GRU)
Residual Connection: A shortcut that adds a layer's input to its output, so each layer only learns a correction. It lets very deep networks train.(Technique)
residual network: Covered in ResNet
residual stream: Covered in Residual Connection
ResNet: A very deep CNN that uses shortcut connections, letting each block learn only a correction to its input. It made networks with 100+ layers trainable.(Model)
ResNet-50: Covered in ResNet
Responsible AI, Safety and Security: Making AI systems fair, understandable, secure against attacks and aligned with what people actually want.(Domain)
retrieval augmentation: Covered in Retrieval-Augmented Generation
Retrieval-Augmented Generation: Before answering, look up relevant passages in a document collection and paste them into the prompt, so the model answers from real sources instead of memory.(Technique)
reverse-mode automatic differentiation: Covered in Backpropagation
reward: Covered in Reinforcement Learning
Reward Hacking: When an agent finds a way to score high on the reward you wrote down without doing what you actually wanted.(Concept)
reward misspecification: Covered in Reward Hacking
reward model: Covered in RLHF
RF: Covered in Random Forest
Ridge and Lasso Regression: Linear regression with a penalty on large weights: ridge shrinks all weights a little, lasso pushes useless ones to exactly zero.(Model)
ridge regression: Covered in Ridge and Lasso Regression
RL: Covered in Reinforcement Learning
RLAIF: Covered in Constitutional AI (RLAIF)
RLHF: Tunes a language model toward answers people prefer: humans rank model outputs, a reward model learns those preferences, and the LLM is optimized to score well.(Technique)
RMSE: Covered in Regression Metrics
RMSNorm: Covered in Batch and Layer Normalization
RMSProp: Covered in Adam and Other Optimizers
RNN: Covered in Recurrent Neural Network
robotaxi: Covered in Driver Assistance and Self-Driving
ROC: Covered in ROC Curve and AUC
ROC Curve and AUC: A curve showing a classifier's true positive rate against its false positive rate at every threshold; the area under it (AUC) summarizes ranking quality in one number.(Metric)
ROC-AUC: Covered in ROC Curve and AUC
RoPE: Covered in Positional Encoding
rotary position embedding: Covered in Positional Encoding
router: Covered in Mixture of Experts
run tracking: Covered in Experiment Tracking

S

S4: Covered in State Space Models
SA-1B: Covered in Segment Anything
SAC: Covered in Actor-Critic Methods
SAE levels: Covered in Driver Assistance and Self-Driving
safety evals: Covered in Red Teaming
saliency maps: Covered in Explainability
SAM: Covered in Segment Anything
sample: Covered in Data, Features and Labels
Sampling and Temperature: How a language model picks each next token from its predicted probabilities. Temperature, top-k and top-p trade predictability for creativity.(Technique)
scalable oversight: Covered in AI Alignment
scaled dot-product attention: Covered in Attention
Scaling Laws: The observation that language model loss improves smoothly and predictably as you increase model size, training data and compute, following power laws.(Concept)
scene text: Covered in Optical Character Recognition (OCR)
score-based model: Covered in Diffusion Model
scorecard: Covered in Credit Scoring
scratchpad: Covered in Chain-of-Thought
seasonality: Covered in Time Series Forecasting
Segment Anything: Meta's promptable segmentation model: click on anything in an image and it returns that object's outline, even for things it was never specifically trained on.(Famous system)
self-attention: Covered in Attention
self-driving cars: Covered in Driver Assistance and Self-Driving
Self-Play: Training an agent by having it play against copies of itself, so its opponents automatically get stronger as it does.(Technique)
Self-Supervised Learning: Learning from unlabeled data by hiding part of each example and training the model to predict the hidden part. The basis of modern foundation models.(Paradigm)
self-training: Covered in Semi-Supervised Learning
semantic search: Covered in Vector Database
semantic segmentation: Covered in Image Segmentation
Semi-Supervised Learning: Learning from a few labeled examples plus many unlabeled ones, using the shape of the unlabeled data to spread the labels sensibly.(Paradigm)
seq2seq: Covered in Sequence-to-Sequence
Sequence-to-Sequence: An encoder reads a whole input sequence into a summary, and a decoder writes an output sequence from it. The architecture that launched neural machine translation.(Model)
SFT: Covered in Instruction Tuning (SFT)
SGD: Covered in Gradient Descent
SGD with momentum: Covered in Adam and Other Optimizers
shadow deployment: Covered in A/B Testing
shallow learning: Covered in Classical Machine Learning
SHAP: Covered in Explainability
short-time Fourier transform: Covered in Spectrogram
shortcut connection: Covered in Residual Connection
sigmoid: Covered in Activation Function
SimCLR: Covered in Contrastive Learning
sinusoidal positional encoding: Covered in Positional Encoding
Siri: Covered in Voice Assistants
skeleton tracking: Covered in Pose Estimation
skip connection: Covered in Residual Connection
skip-gram: Covered in word2vec
sleeper agent: Covered in Data Poisoning
smart compose: Covered in Keyboard Autocomplete
smart speaker: Covered in Voice Assistants
soft clustering: Covered in Gaussian Mixture Model
soft targets: Covered in Knowledge Distillation
softmax: Covered in Activation Function
Sora: OpenAI's text-to-video model, unveiled in February 2024, which generates video from text by denoising spacetime patches with a diffusion Transformer.(Famous system)
Sora 2: Covered in Sora
sound classification: Covered in Audio Classification
sound event detection: Covered in Audio Classification
spacetime patches: Covered in Text-to-Video
spam filtering: Covered in Email Spam Filters
sparse autoencoders: Covered in Mechanistic Interpretability
sparse model: Covered in Mixture of Experts
sparsity: Covered in Pruning
speaker diarization: Covered in Speaker Identification and Verification
Speaker Identification and Verification: Recognizing who is speaking from their voice: verifying a claimed identity, picking a speaker from a known group, or splitting a recording by speaker.(Concept)
speaker recognition: Covered in Speaker Identification and Verification
speaker verification: Covered in Speaker Identification and Verification
specification gaming: Covered in Reward Hacking
Spectrogram: A picture of sound: time runs left to right, pitch runs bottom to top, and brightness shows how loud each frequency is at each moment.(Concept)
Speech and Audio: Machine learning for sound: turning speech into text and back, recognizing sounds, and generating music and voices.(Domain)
Speech Recognition: Turning spoken audio into written text. Modern systems read a spectrogram and write out the words with a neural network.(Technique)
speech synthesis: Covered in Text-to-Speech
speech-to-text: Covered in Speech Recognition
Spotify recommendations: Covered in Streaming and Video Recommendations
SSL: Covered in Self-Supervised Learning
SSM: Covered in State Space Models
Stable Diffusion: An open-weights text-to-image model based on latent diffusion. Its public release in 2022 let anyone generate images on their own computer.(Famous system)
staged release: Covered in GPT-2
StarCraft II: Covered in AlphaStar
state: Covered in Markov Decision Process
State Space Models: Sequence models that carry a compact state forward like an RNN but can be trained in parallel, scaling linearly with sequence length instead of quadratically like attention.(Family)
step size: Covered in Learning Rate
step-by-step reasoning: Covered in Chain-of-Thought
STFT: Covered in Spectrogram
stochastic gradient descent: Covered in Gradient Descent
stratified k-fold: Covered in Cross-Validation
Streaming and Video Recommendations: The systems that pick what Netflix, Spotify and YouTube show you next, from matrix factorization made famous by the Netflix Prize to today's deep learning recommenders.(Famous system)
streaming learning: Covered in Online Learning
structured pruning: Covered in Pruning
style-based generator: Covered in StyleGAN
StyleGAN: NVIDIA's GAN that produced photorealistic faces of people who do not exist, famous through the website This Person Does Not Exist.(Famous system)
StyleGAN2: Covered in StyleGAN
subword tokenization: Covered in Tokenization
superposition: Covered in Mechanistic Interpretability
supervised fine-tuning: Covered in Instruction Tuning (SFT)
Supervised Learning: Learning from examples that come with the correct answer, so the model can predict answers for new, unlabeled examples.(Paradigm)
support vector data description: Covered in One-Class SVM
Support Vector Machine: Separates two classes with the widest possible margin. The kernel trick lets it draw curved boundaries by implicitly working in a higher-dimensional space.(Model)
support vectors: Covered in Support Vector Machine
SVD recommender: Covered in Matrix Factorization
SVM: Covered in Support Vector Machine
system prompt: Covered in Prompting

T

t-SNE: Covered in t-SNE and UMAP
t-SNE and UMAP: Methods that squash high-dimensional data into a 2D picture while keeping similar points close together, great for spotting clusters.(Technique)
tanh: Covered in Activation Function
target: Covered in Data, Features and Labels
target network: Covered in Deep Q-Network
TD learning: Covered in Q-Learning
TD-Gammon: A neural network that taught itself backgammon by playing against itself with temporal-difference learning and reached a level close to the best human players.(Famous system)
teacher-student: Covered in Knowledge Distillation
temperature: Covered in Sampling and Temperature
temporal difference: Covered in Q-Learning
tensors: Covered in Linear Algebra Basics
Tesauro: Covered in TD-Gammon
test-time compute: Covered in Reasoning Models
text recognition: Covered in Optical Character Recognition (OCR)
text-to-audio: Covered in Music and Audio Generation
Text-to-Image Generation: Creating a new image from a written description, by combining a text encoder that understands the prompt with a generative image model.(Technique)
text-to-music: Covered in Music and Audio Generation
Text-to-Speech: Generating natural-sounding speech from written text, including rhythm, intonation and, in some systems, a specific person's voice.(Technique)
Text-to-Video: Models that generate video clips from a text description, typically diffusion models that denoise a whole clip in a compressed space-time latent.(Family)
thinking models: Covered in Reasoning Models
This Person Does Not Exist: Covered in StyleGAN
throughput: Covered in Model Serving
Tikhonov regularization: Covered in Ridge and Lasso Regression
Time Series Forecasting: Predicting future values of a sequence measured over time, such as sales or electricity demand, from its own past patterns: trend, seasonality and noise.(Family)
TinyML: Covered in Edge AI (On-Device ML)
Tokenization: Splitting text into small pieces called tokens (whole words, word fragments or characters) and mapping each to a number the model can process.(Technique)
tokenizer: Covered in Tokenization
tokens: Covered in Tokenization
tool use: Covered in AI Agents and Tool Use
top-k: Covered in Sampling and Temperature
top-p: Covered in Sampling and Temperature
TPU: Covered in GPUs and AI Accelerators
traditional ML: Covered in Classical Machine Learning
train-test split: Covered in Train, Validation and Test Sets
Train, Validation and Test Sets: Splitting data so the model learns on one part, is tuned on a second and is honestly graded on a third it has never seen.(Concept)
training data attack: Covered in Data Poisoning
training data extraction: Covered in Membership Inference and Data Extraction
training step: Covered in Epochs, Batches and Iterations
transaction scoring: Covered in Card Fraud Detection
transcription: Covered in Speech Recognition
Transfer Learning: Reusing a model trained on one big task as the starting point for another, so the new task needs far less data and compute.(Paradigm)
Transformer: A neural network built from stacked self-attention and feed-forward layers. It processes all tokens in parallel and underlies today's language, vision and multimodal models.(Family)
transformer architecture: Covered in Transformer
tree ensembles: Covered in Tree-Based Models
tree models: Covered in Tree-Based Models
Tree-Based Models: Models built from decision trees, a series of yes/no questions about the features, from a single readable tree to forests and boosted ensembles of hundreds.(Family)
trojan: Covered in Data Poisoning
true positive: Covered in Precision, Recall and F1
true positive rate: Covered in ROC Curve and AUC
TrueDepth: Covered in Face ID and Face Recognition
trustworthy AI: Covered in Responsible AI, Safety and Security
TTS: Covered in Text-to-Speech

U

U-Net: An encoder-decoder network shaped like a U, with skip connections that carry fine detail across. Built for medical image segmentation, later the denoiser inside diffusion models.(Model)
UCB: Covered in Exploration vs Exploitation
UCT: Covered in Monte Carlo Tree Search
UMAP: Covered in t-SNE and UMAP
uncertainty sampling: Covered in Active Learning
underfitting: Covered in Overfitting
Unsupervised Learning: Finding structure in data that has no labels: groups, compressed representations and unusual points.(Paradigm)
update gate: Covered in Gated Recurrent Unit (GRU)
upper confidence bound: Covered in Exploration vs Exploitation

V

VAE: Covered in Variational Autoencoder
validation monitoring: Covered in Early Stopping
validation set: Covered in Train, Validation and Test Sets
value alignment: Covered in AI Alignment
value function: Covered in Markov Decision Process
Vanishing and Exploding Gradients: In deep or recurrent networks, gradients are multiplied through many layers and can shrink to nothing or blow up, so early layers stop learning or training becomes unstable.(Concept)
vanishing gradient problem: Covered in Vanishing and Exploding Gradients
Variational Autoencoder: An autoencoder whose compressed space is smooth and continuous, so you can sample or walk through it to generate new, realistic data.(Model)
Vector Database: A store for embeddings that can quickly find the vectors most similar to a query among millions, the retrieval engine behind semantic search and RAG.(Technique)
vector embedding: Covered in Embedding
vector search: Covered in Retrieval-Augmented Generation
vectors: Covered in Linear Algebra Basics
Veo: Google DeepMind's text-to-video model family, announced at Google I/O in May 2024 and offered in Google's AI products and cloud platform.(Famous system)
Veo 3: Covered in Veo
video diffusion: Covered in Text-to-Video
video generation: Covered in Text-to-Video
Visa Advanced Authorization: Covered in Card Fraud Detection
Vision Transformer: Applies the Transformer to images by cutting each image into small patches and treating the patches like words in a sentence.(Model)
Vision-Language Model: A language model that can also see: it takes images (and text) as input and answers questions, describes scenes or reads documents.(Family)
visual question answering: Covered in Vision-Language Model
ViT: Covered in Vision Transformer
Viterbi algorithm: Covered in Hidden Markov Model
VLM: Covered in Vision-Language Model
vocoder: Covered in Text-to-Speech
Voice Assistants: Spoken assistants such as Siri, Alexa and Google Assistant that chain wake-word detection, speech recognition, language understanding and text-to-speech.(Famous system)
voice biometrics: Covered in Speaker Identification and Verification
voice cloning: Covered in Text-to-Speech
voiceprint: Covered in Speaker Identification and Verification
VQA: Covered in Vision-Language Model

W

wake word: Covered in Voice Assistants
wake word detection: Covered in Audio Classification
Ward's method: Covered in Hierarchical Clustering
warmup: Covered in Learning Rate
WaveNet: Covered in Autoregressive Model
weight decay: Covered in Regularization
Weight Initialization: Choosing the random starting values of a network's weights so signals neither shrink nor blow up as they pass through many layers.(Technique)
weight pruning: Covered in Pruning
Weights & Biases: Covered in Experiment Tracking
Whisper: OpenAI's open-source speech recognition model, trained on 680,000 hours of audio from the web, that transcribes and translates speech in many languages.(Famous system)
word embedding: Covered in Embedding
word2vec: A fast, shallow neural method from Google that learns a vector for every word from which words appear near it, famous for 'king − man + woman ≈ queen'.(Model)
world model: Covered in Model-Based Reinforcement Learning

X

x-vectors: Covered in Speaker Identification and Verification
XAI: Covered in Explainability
Xavier initialization: Covered in Weight Initialization
XGBoost: Covered in Gradient Boosting

Y

YOLO: A family of fast object detectors that find all objects in an image with a single pass of a neural network, fast enough for live video.(Famous system)
You Only Look Once: Covered in YOLO
YouTube recommendations: Covered in Streaming and Video Recommendations

Z

zero-shot: Covered in Few-Shot and Zero-Shot Learning
zero-shot classification: Covered in CLIP
Zestimate: Covered in Home Price Estimators
Zillow: Covered in Home Price Estimators

#

/imagine: Covered in Midjourney
4-bit: Covered in Quantization