course content

Course Content

Linear Regression with Python

MetricsMetrics

When we build a model, we want to measure the model's performance. We need some score associated with the model that describes how well the model fits. This score is called a metric, and there are a lot of different metrics. We will talk about the most widely used ones.

In this chapter, we are going to use the following notation:

We already know one metric, SSR, which we minimized to find the best parameters. Using our notation, we can write down the formula of SSR as follows:

or equally:

This metric was good for comparing models with the same number of instances. However, it doesn't give us an understanding of how well the model performs. Here is why:
Suppose you have two models on the different training sets (shown in the image below).

You can see that the first model fits well but still has a higher SSR than the second model, which visually fits the data worse. It happened only because the first model has much more data points, so the sum is higher, but on average, the first model's residuals are lower. So taking the average of squared residuals as a metric would describe the model better. That is precisely what the Mean Squared Error(MSE) is.

MSE

or equally:

To calculate the MSE metric using python, you can use NumPy's functions:

Or you can use Scikit-learn's mean_squared_error() method:

Where y_true is an array of actual target values and y_pred is an array of predicted target values for the same features.

The problem is the error it shows is squared. For example, suppose the MSE of the model predicting houses is 49 dollarsĀ². We are interested in price, not price squared, as given by MSE, so we would like to take the root of MSE and get 7 dollars. Now we have a metric with the same unit as the predicted value. This metric is called Root Mean Squared Error(RMSE).

RMSE

To calculate the RMSE metric using python, you can use NumPy's functions:

Or you can use Scikit-learn's mean_squared_error() method with squared=False:

MAE

In SSR, we squared the residuals to get rid of the sign. The second approach would be taking the absolute values of residuals instead of squaring them. That is the idea behind Mean Absolute Error(MAE).

or equally

It is the same as the MSE, but instead of squaring residuals, we take their absolute values.

To calculate the MAE metric using python, you can use NumPy's functions:

Or you can use Scikit-learn's mean_absolute_error() method:

For choosing the parameters, we used the SSR metric. That is because it was good for mathematical calculations and allowed us to get the Normal Equation. But to further compare the models, you can use any other metric.

Note

For comparing models, SSR, MSE, and RMSE will always identically choose which model is better and which is worse. And MAE can sometimes prefer a different model than SSR/MSE/RMSE since those penalize high residuals much more. Usually, you want to choose one metric a priori and focus on minimizing it.

Now you can surely tell that the second model is better since all its metrics are lower. However, lower metrics do not always mean the model is better. The following chapter will tell you why.

Section 4.

Chapter 2