The Flaw of GridSearchCV

Before using GridSearchCV, note that KNeighborsClassifier has more hyperparameters than n_neighbors. Two important ones are weights and p.

Weights

By default, the classifier uses weights='uniform', meaning all k neighbors vote equally. Setting weights='distance' gives closer neighbors more influence, often improving predictions when nearby points are more relevant.

P

The p parameter controls the distance metric:

• p=1: Manhattan distance;
• p=2: Euclidean distance.

A p parameter can take any positive integer. There are many different distances, but they are harder to visualize than p=1 or p=2.

Previously, only n_neighbors was tuned. To search over all three hyperparameters, use:

param_grid = {
    'n_neighbors': [1, 3, 5, 7],
    'weights': ['distance', 'uniform'],
    'p': [1, 2]
}

GridSearchCV tries all the possible combinations to find the best, so it will try all of those:

A larger grid like:

param_grid = {
    'n_neighbors': [...],
    'weights': ['distance', 'uniform'],
    'p': [1, 2, 3, 4, 5]
}

creates 100 combinations. With 5-fold cross-validation, the model is trained 500 times. This is fine for small datasets, but for bigger ones it becomes too slow.

To reduce computation time, RandomizedSearchCV tests only a random subset of combinations, usually finding strong results much faster than a full grid search.