Epochs, Batches and Iterations

Training a neural network with gradient descent rarely uses all the data for each update. Three words describe how the data is chopped up:

• Batch (mini-batch): a small group of examples, often 32 to a few thousand. The gradient is averaged over the batch.
• Iteration (step): one weight update, computed from one batch.
• Epoch: one complete pass through the training set. With 50,000 examples and batches of 500, an epoch is 100 iterations.

Typical classic training runs for tens of epochs, reshuffling the data each time and checking validation loss after each one (see early stopping).

Batch size trade-offs:

• Small batches: noisier gradients (which can even help generalization), less memory, but less efficient use of the GPU.
• Large batches: smoother gradients and faster per example, but more memory and usually a larger learning rate.

Large language models are different: their datasets are so big that they see most text only once or a few times, so their training progress is measured in tokens rather than epochs.