Bias Variance Tradeoff

Let’s use this dataset to understand the impact of number of neighbours ‘k’.

👉If ‘k’ is very large, say, k=n,

• model simply predicts the majority class of the entire dataset for every query point , i.e, under-fitting.

👉If ‘k’ is very small, say, k=1,

• model is highly sensitive to noise or outliers, as it looks at only 1 nearest neighbor, i.e, over-fitting.

Let’s plot Error vs ‘K’ neighbors:

• Figure 1: k=1, Over-fitting

• Figure 2: k=n, Under-fitting

• Figure 3: k=11, Lowest Error (Optimum)