Mastering K-Nearest Neighbors: A Step-by-Step Guide for STAT 435

Introduction to K-Nearest Neighbors (KNN)

K-nearest neighbors (KNN) is a simple yet powerful non-parametric method used for classification and regression. In this tutorial, we'll walk through the key concepts and R implementations you'll need for STAT 435 homework assignments. Whether you're predicting student exam scores or analyzing the Boston housing dataset, KNN offers an intuitive approach: it classifies a new data point based on the majority class among its k nearest neighbors.

Generating Synthetic Data for KNN

In many assignments, you'll generate data from known distributions. For example, you might create two classes (red and blue) from bivariate normal distributions with different means. Use rnorm() in R to generate 25 observations per class. The code below creates a training set:

set.seed(123)
red_train <- cbind(rnorm(25, mean=0), rnorm(25, mean=0))
blue_train <- cbind(rnorm(25, mean=1.5), rnorm(25, mean=1.5))
train <- rbind(red_train, blue_train)
train_labels <- c(rep("red", 25), rep("blue", 25))

Plotting the data with proper labels and colors is essential. Use plot() with col parameter to distinguish classes. For test sets, generate another 25 per class and combine them in a single plot using different symbols (circles for training, squares for testing) and a legend.

Fitting KNN Models and Evaluating Error

The knn() function from the class library fits the model. For k values from 1 to 20, compute training and test error rates. Plot 1/k on the x-axis and error on the y-axis. You'll typically see that training error decreases as k decreases (more flexible), while test error has a U-shape due to the bias-variance tradeoff. The optimal k often lies where test error is minimized.

Understanding the Bayes Error Rate

The Bayes error rate is the minimum possible error for a given problem. For two normal distributions with equal covariance, the Bayes decision boundary is linear. In our example, the optimal classifier would misclassify some points due to overlapping distributions. The Bayes error can be computed using the cumulative distribution function of the normal distribution. Compare your KNN results to this theoretical lower bound.

Non-Linear Decision Boundaries with KNN

In another scenario, data is generated from uniform distributions with a circular decision boundary. This mimics real-world problems where classes are not linearly separable. KNN adapts well to such non-linear boundaries, especially with small k. For k=1, the decision boundary is highly flexible and follows the training data closely, leading to overfitting. As k increases, the boundary becomes smoother, reducing variance but increasing bias.

Bias-Variance Tradeoff in KNN

The bias-variance decomposition explains why KNN's performance depends on k. Low k (flexible model) has low bias but high variance; high k (inflexible) has high bias but low variance. The test error is the sum of bias², variance, and irreducible error. Plotting these curves helps visualize the tradeoff. In practice, choose k that minimizes test error, often via cross-validation.

Practical Tips for STAT 435 Homework

• Always set a seed for reproducibility when generating data.
• Label axes and include legends in all plots.
• Explain your results in the context of bias-variance tradeoff and Bayes error.
• Use scale() if features have different units, though in these exercises features are on similar scales.

Connecting KNN to Real-World Trends

KNN is widely used in recommendation systems (e.g., Netflix suggesting movies based on similar users) and in AI applications like image recognition. In sports analytics, KNN can classify player performance levels based on statistics. In finance, it helps detect fraud by comparing transactions to known patterns. Understanding KNN gives you a foundation for more advanced machine learning methods.

Conclusion

This tutorial covered the essential steps for completing KNN assignments in STAT 435: generating data, fitting models, evaluating errors, and interpreting results. Remember to always justify your choices and relate findings to theoretical concepts. For further practice, explore the Boston housing dataset to apply KNN for regression (predicting median home values) and compare with linear models.