Unbiased Estimation

Estimates can vary with each new set of samples, making them random variables with their own distributions. To determine the quality of an estimate, we need criteria to assess which estimates are better or worse, and how well they match our expectations.

In practice, estimates are often evaluated based on three key properties: unbiasedness, consistency, and efficiency.

Unbiased Estimation

Unbiased estimation in statistics ensures that the expected value of the estimator equals the true value of the parameter being estimated. This means the estimate is not consistently too high or too low on average.

Mathematically, it's expressed as follows:

To better understand this property, let's take a look at estimating the mean and standard deviation for a Gaussian samples using the method of moments.

Mean estimation

Thus, it can be concluded that estimating the mean of the Gaussian population by calculating the sample mean is unbiased since the expected value of the obtained value is equal to the real value of the mean.

Variance estimation

Now let's estimate the population's variance by calculating variance via samples:

We obtained a biased estimate where the sample variance's expected value is less than the true variance. However, as the number of samples increases, the expected value gets closer to the true variance (because 1/m approaches zero).

Note

An estimate that remains biased but where this bias decreases to zero as the number of samples approaches infinity is termed asymptotically unbiased.

To create a simply unbiased estimate of the population's variance, we can use the following formula:

The estimation obtained using the above formula is called the adjusted sample variance.

Note

The sample variance is only biased if we use the sample mean in the calculation. If we know exactly the mathematical expectation and want to estimate the variance, then we can use this expectation instead of the sample mean, and such estimation will be non-biased.

It is also important to mention that the above formulas for estimating the mean and variance can be used for the Gaussian distribution and any other distribution with a mathematical expectation and variance. Such estimates will also be non-biased.