Naive Bayes

Bayes’ theorem turns “how likely is this evidence if the email is spam?” into “how likely is the email spam given this evidence?”. Naive Bayes applies it with a bold simplification: it treats every word (feature) as independent evidence, given the class.

Training is just counting:

• How often each word appears in spam and in normal email.
• How common spam is overall.

To classify a new email, multiply the class’s overall probability by the probability of each word under that class, and compare spam against not-spam. (In practice you add logarithms instead of multiplying, and add a small constant to counts so unseen words do not zero everything out.)

The independence assumption is clearly false, since “free” and “money” tend to appear together, yet the ranking of classes is often right anyway. That made naive Bayes the engine of early 2000s spam filters, and it is still a quick, sturdy baseline before trying logistic regression or neural networks.

The math

$$P(c \mid x_1, \dots, x_n) \propto P(c) \prod_{i=1}^{n} P(x_i \mid c)$$