In statistics, the difference between an estimator's expected value and the true value of the parameter being estimated is called the bias. An estimator or decision rule having nonzero bias is said to be biased. The term bias has many uses: In its most widely used form, bias is prejudice of some sort in terms of point of view, usually manifesting itself in written works as in editorial bias. ... This article is about the field of statistics. ... In statistics, an estimator is a function of the observable sample data that is used to estimate an unknown population parameter; an estimate is the result from the actual application of the function to a particular set of data. ... In probability theory the expected value (or mathematical expectation) of a random variable is the sum of the probability of each possible outcome of the experiment multiplied by its payoff (value). Thus, it represents the average amount one expects as the outcome of the random trial when identical odds are...

Although the term bias sounds pejorative, it is not necessarily used in that way in statistics. Biased estimators may have desirable properties. Not only do they sometimes have a smaller mean squared error than any unbiased estimator, but in some cases the only unbiased estimators are not even within the convex hull of the parameter space, so their use is absurd. In statistics the mean squared error of an estimator T of an unobservable parameter θ is i. ... Convex hull: elastic band analogy In mathematics, the convex hull or convex envelope for a set of points X in a real vector space V is the minimal convex set containing X. // For planar objects, i. ...

Definition

Suppose we are trying to estimate the parameter using an estimator (that is, some function of the observed data). Then the bias of is defined to be In statistics, an estimator is a function of the observable sample data that is used to estimate an unknown population parameter; an estimate is the result from the actual application of the function to a particular set of data. ...

In words, this would be "the expected value of the estimator minus the true value ." This may be rewritten as

which would read "the expected value of the difference between the estimator and the true value" (the expected value of is precisely ).

Examples

Estimating variance

Suppose X₁, ..., X_n are independent and identically distributed normal random variables with expectation μ and variance σ². Let expectation in the context of probability theory and statistics, see expected value. ... This article is about mathematics. ...

be the "sample average", and let

be a "sample variance". We also know that the variance σ² is defined by:

where N is the population size and x_i represents the member of the whole population.

Then S² is a "biased estimator" of σ² because

In other words, the sample variance does not equal the population variance, unless multiplied by the normalization factor.

Common sense would suggest to apply the population formula to the sample as well. The reason that it is biased is that the sample mean is generally somewhat closer to the observations in the sample than the population mean is, to these observations. This is so because the sample mean is, by definition, in the middle of the sample, while the population mean may even lie outside the sample. So the deviations to the sample mean will often be smaller than the deviations to the population mean, and so, if the same formula is applied to both, then this variance estimate will on average be somewhat smaller in the sample than in the population.

Note that when a transformation is applied to an unbiased estimator, the result is not necessarily itself an unbiased estimate of its corresponding population statistic. That is, for a non-linear function f and an unbiased estimator U of a parameter p, f(U) is usually not an unbiased estimator of f(p). For example the square root of the unbiased estimator of the population variance is not an unbiased estimator of the population standard deviation. In mathematics, a square root (√) of a number x is a number r such that , or in words, a number r whose square (the result of multiplying the number by itself) is x. ... This article is about mathematics. ... In probability and statistics, the standard deviation of a probability distribution, random variable, or population or multiset of values is a measure of the spread of its values. ...

Bias, however, is not the only consideration when choosing a statistic. Bias refers to the central tendency of the sampling distribution of a statistic, but the variance of the sampling distribution can also be an important consideration. Specifically, statistics with smaller sampling variances will yield greater statistical power. For example, while S² above is more biased than the traditional sample calculation The power of a statistical test is the probability that the test will reject a false null hypothesis (that it will not make a Type II error). ...

S² has a lower estimation variability than S²_unbiased because the denominator dividing the sum of squares is larger in the calculation of S², resulting in a smaller scale of final values, and therefore lower estimation variability, than that of S²_unbiased. Practically, this demonstrates that for some applications (where the amount of bias can be equated between groups/conditions) it is possible that a biased estimator can prove to be a more powerful, and therefore useful, statistic. The use of n − 1 rather than n is sometimes called Bessel's correction. In statistics, Bessels correction, named after Friedrich Bessel, is the use of n âˆ’ 1 instead of n when estimating variance, where n is the number of observations in a sample. ...

Estimating a Poisson probability

A far more extreme case of a biased estimator being better than any unbiased estimator is well-known: Suppose X has a Poisson distribution with expectation λ. It is desired to estimate In probability theory and statistics, the Poisson distribution is a discrete probability distribution that expresses the probability of a number of events occurring in a fixed period of time if these events occur with a known average rate, and are independent of the time since the last event. ...

(For example, when incoming calls at a telephone switchboard are modeled as a Poisson process, and λ is the average number of calls per minute, then e^−2λ is the probability that no calls arrive in the next two minutes.)

The only function of the data constituting an unbiased estimator is

If the observed value of X is 100, then the estimate is 1, although the true value of the quantity being estimated is obviously very likely to be near 0, which is the opposite extreme. And if X is observed to be 101, then the estimate is even more absurd: it is −1, although the quantity being estimated obviously must be positive.

The (biased) maximum likelihood estimator Maximum likelihood estimation (MLE) is a popular statistical method used to make inferences about parameters of the underlying probability distribution from a given data set. ...

is far better than this unbiased estimator. Not only is its value always positive, but it is also more accurate in the sense that its mean squared error (MSE) In statistics the mean squared error of an estimator T of an unobservable parameter θ is i. ...

is smaller; compare the unbiased estimator's MSE of

1 − e ^{− 4λ}

The MSEs are a functions of the true value λ. The bias of the maximum-likelihood estimator is:

.

Maximum of a discrete uniform distribution

The bias of maximum-likelihood estimators can be substantial. Consider a case where n tickets numbered from 1 through to n are placed in a box and one is selected at random, giving a value X. If n is unknown, then the maximum-likelihood estimator of n is X, even though the expectation of X is only (n + 1)/2; we can only be certain that n is at least X and is probably more. In this case, the natural unbiased estimator is 2X − 1.

See also

• Omitted-variable bias
• Consistency (statistics)

Omitted-variable bias is the bias that appears in an estimate of a parameter if a regression run does not have the appropriate form and data for other parameters. ... In statistics, a consistent estimator is an estimator that converges in probability to the quantity being estimated as the sample size grows. ...

External links

• An Illuminating Counterexample

ǪŲŲ