Skip to content
ML Atlas

Neural Network Fundamentals

Artificial neurons, layers, activation functions and backpropagation: the building blocks shared by every deep learning model.

Role · kind
Domain
Level
Intermediate

A neural network is a huge team of tiny workers passing notes. Each worker does something very simple, but together, after lots of practice, the team can recognize faces or write stories.

Where it fits

Neural Network FundamentalsAdam and Other OptimizersAdam and Other OptimizersBackpropagationBackpropagationBatch and Layer NormalizationBatch and Layer Normalizati…Computational Graph and Chain RuleComputational Graph and Cha…

Not shown: 8 more kinds and parts. See the full list below.

+All relationships as a list

See it on the full map

Overview

A neural network is a function built from many simple units. Each unit (a neuron) computes a weighted sum of its inputs and passes it through an activation function. Stack units into layers and layers into a network, and you get a multilayer perceptron, the ancestor of all deep learning.

Training uses the same recipe as the rest of ML: a loss function plus gradient descent. The new ingredient is backpropagation, an efficient way to compute how every one of millions of weights should change.

On top of that come tricks that make deep networks actually train well, such as dropout against overfitting and residual connections that let very deep networks learn. Specific wiring patterns (convolutions, attention) live in the architectures domain.

Everyday use cases

  • Case 01

    Understanding modern AI

    CNNs, Transformers and diffusion models are all neural networks. These fundamentals apply to every one of them.