Overfitting. Regularization

As shown in the previous chapter, using PolynomialFeatures, you can get a pretty complex decision boundary. Second-degree polynomial features can even get you the boundaries in the picture below.

And it is only a degree of two. A higher degree may yield even more complex shapes. But there is a problem with it. The decision boundary built by Logistic Regression may become too complicated, causing the model to overfit.
Overfitting is when the model, instead of learning general patterns in data, builds a very complex decision boundary to handle every training instance. Still, it does not perform as well on data it has never seen, while performing well on unseen data is a primary task of the Machine Learning model.

The regularization tackles the problem of overfitting. In fact, ℓ2 regularization is used in the LogisticRegression class by default. But you need to configure how strongly the model should be regularized. It is controlled by a C parameter.

• greater C – lower regularization, more overfitting;
• lower C – stronger regularization, less overfitting (but possibly underfitting).

What values of C will result in a good model depends on the dataset, thus better to choose it using the GridSearchCV.

Note

If you build a Logistic Regression with regularization, you must scale the data.

The LogisticRegression class includes regularization by default, so you should either remove regularization(by setting penalty=None) or scale the data(e.g., using StandardScaler).

Note

If you use both PolynomialFeatures and StandardScaler for preprocessing, the StandardScaler should be applied after the PolynomialFeatures. Generally, you want to apply the StandardScaler when all other changes to features are done.