Hyperparameters
Settings chosen before training, such as learning rate, tree depth or number of layers, as opposed to parameters, which the model learns from data.
Also hyperparameter tuning · grid search · random search · Bayesian optimization
Baking a cake: the oven temperature and baking time are hyperparameters you pick. How the batter turns into cake is what the oven 'learns'. Pick the wrong settings and even great batter fails.
Where it fits
+All relationships as a list
Is a kind of / part of
Siblings
- Accuracy
- Bias-Variance Trade-off
- Cross-Validation
- Curse of Dimensionality
- Data, Features and Labels
- Gradient Descent
- Linear Algebra Basics
- Log Loss (Cross-Entropy)
- Loss Function
- Overfitting
- Precision, Recall and F1
- Probability Basics
- Regression Metrics
- Regularization
- ROC Curve and AUC
- Train, Validation and Test Sets
Uses / built on
Used by
How it works
A model has two kinds of numbers:
- Parameters are learned during training: the weights of a neural network, the split thresholds of a decision tree.
- Hyperparameters are chosen by you before training: the learning rate, the number of layers, the depth of the tree, in k-nearest neighbors, the strength of regularization.
Hyperparameters control how the model learns, so they have a big effect on the result.
How to choose them
- Grid search: try every combination of a few values. Simple but wasteful.
- Random search: sample combinations at random. Usually finds good settings faster, because only a few hyperparameters really matter.
- Bayesian optimization and successive halving: spend more trials near promising regions and stop bad runs early.
Always judge each setting on a validation set or with cross-validation. If you pick the best of 100 settings by their test score, that score is no longer an honest estimate. Logging every run with experiment tracking keeps the search reproducible.
Everyday use cases
- Case 01
Tuning a gradient boosting model
Searching over learning rate, depth and number of trees for a fraud model.
- Case 02
Choosing k in k-means or kNN
Too small or too large a k gives clearly worse results; a quick sweep finds a good value.
Strengths, weaknesses and when to use it
Strengths
- Good tuning often beats switching to a fancier model.
Weaknesses
- Search is expensive, and tuning against the test set silently inflates reported performance.
When to use itAfter you have a working baseline. Tune the few hyperparameters that matter most (often the learning rate) first.
Compare with
Check yourself
- Q01
Which of these is a hyperparameter, not a learned parameter?
Show answer
B: The number of trees in a random forest
- Q02
Why tune on a validation set instead of the test set?
Show answer
B: Tuning on the test set makes the final score overly optimistic