In statistics, the generalized linear model (GLM) is a useful generalization of ordinary least squares regression. A GLM stipulates that the random part of the experiment (the distribution function) and the systematic portion of the experiment (the linear predictor) are related through what's deemed the link function. A graph of a bell curve in a normal distribution showing statistics used in educational assessment, comparing various grading methods. ... Line of best fit redirects here. ...

Overview

In a GLM, the data (Y) are assumed to be generated from a distribution function in the exponential family (a very large range of distributions; also see below). The data's expectation μ is predicted by In probability and statistics, an exponential family is any class of probability distributions having a certain form. ...

where Xβ is the linear predictor, a linear combination (X, known from the experiment) of unknown parameters β, and g is called the link function.

In this framework, typically the random component is also a function V of the mean:

.

It is convenient if the variance follows from the exponential family distribution, but it may simply be that the variance is a function of the predicted value. In probability and statistics, an exponential family is any class of probability distributions having a certain form. ...

The unknown parameters β are typically estimated with maximum likelihood, quasi maximum likelihood, or Bayesian techniques. Maximum likelihood estimation (MLE) is a popular statistical method used to make inferences about parameters of the underlying probability distribution of a given data set. ... Bayesian refers to probability and statistics -- either methods associated with the Reverend Thomas Bayes (ca. ...

Components of the model

The GLM consists of three elements.

1. A distribution function f, from the exponential family.

2. A linear predictor η = X β.

3. A link function g such that E(y) = μ = g^-1(η).

Exponential Family of Distributions

The exponential family of distributions are those probability distributions, parameterized by θ and τ, whose density functions can be expressed in the form

.

τ, called the dispersion parameter, typically is known. The functions a, b, c, d, and h are known. Many, although not all, common distributions are in this family.

If a is the identity function, then the distribution is said to be in canonical form. If in addition b is the identity, then θ is called the canonical parameter. Generally, in mathematics, a canonical form is a function that is written in the most standard, conventional, and logical way. ...

Linear Predictors

The linear predictor is a quantity relating to the expectation of the data (thus, "predictor") through the link function. The symbol η ("eta") is typically used to denote a linear predictor.

η is expressed as linear combinations (thus, "linear") of unknown parameters β. The coefficients of the linear combination are represented as the matrix X; its elements are either fully known by the experimenters or stipulated by them in the modeling process.

Thus η can be expressed as

.

Link functions

The link function provides the relationship between the linear predictor and the distribution function (through its mean). There are many commonly used link functions, and their choice can be somewhat arbitrary. However, it is important to match the domain of the link function to the range of the distribution function's mean. In probability theory the expected value (or mathematical expectation) of a random variable is the sum of the probability of each possible outcome of the experiment multiplied by its payoff (value). Thus, it represents the average amount one expects as the outcome of the random trial when identical odds are...

Following is a table of some common link functions and their inverses (sometimes referred to as the mean function) used for several distributions in the exponential family.

Link Functions

Distribution	Name	Link Function	Mean Function
Normal	Identity
Exponential	Inverse
Gamma
Poisson	Log
Binomial	Logit
Multinomial

Examples

Linear regression

The simplest example of a GLM is linear regression. Here the distribution function is the normal distribution with constant variance and the link function is the identity. Line of best fit redirects here. ...

Binomial data

When the response data (Y) are binary (taking on only values 0 and 1), the distribution function is generally chosen to be the binomial distribution and the interpretation of μ_i is then the probability of Y_i taking on the value one. There are several popular link functions for binomial functions; the most typical is the logistic function: In probability theory and statistics, the binomial distribution is the discrete probability distribution of the number of successes in a sequence of n independent yes/no experiments, each of which yields success with probability p. ...

.

GLMs with this setup are logistic regression models. It has been suggested that Logit be merged into this article or section. ...

In addition, any inverse cumulative density function (CDF) can be used for the link since the CDF's range is [0, 1], the range of the binomial mean. The normal CDF Φ is a popular choice and yields the probit model. Its link is In probability theory, the cumulative distribution function (abbreviated cdf) completely describes the probability distribution of a real-valued random variable, X. For every real number x, the cdf is given by where the right-hand side represents the probability that the random variable X takes on a value less than... The probit model is a popular specification of a binary regression model, using a probit link function. ...

.

The identity link is also sometimes used for binomial data (this is equivalent to using the uniform distribution instead of the normal as the CDF) but this can be problematic as the predicted probabilities can be greater than one or less than zero. In implementation it is possible to fix the nonsensical probabilities outside of [0,1] but interpreting the coefficients can be difficult in this model. The model's primary merit is that near p=0.5 it is approximately a linear transformation of the probit and logit — econometricians sometimes call this the Harvard model.

The variance function for binomial data is given by:

where the dispersion parameter τ is typically exactly one. When it is not, the model is often described as binomial with overdispersion or quasibinomial.

Count data

Another example of generalized linear models includes Poisson regression which models count data using the Poisson distribution. The link is typically the logarithm. In statistics, the Poisson regression model attributes to a response variable Y a Poisson distribution whose expected value depends on a predictor variable x (written in lower case because the model treats x as non-random, in the following way: (where log means natural logarithm). ... In probability theory and statistics, the Poisson distribution is a discrete probability distribution. ...

The variance function is proportional to the mean:

,

where the dispersion parameter τ is typically exactly one. When it is not, the model is often described as poisson with overdispersion or quasipoisson.

References

• P. McCullagh and J.A. Nelder. Generalized Linear Models. London: Chapman and Hall, 1989.

• A.J. Dobson. Introduction to Generalized Linear Models, Second Edition. London: Chapman and Hall/CRC, 2001.

External links

• John Nelder FRS
• Royal Society citation for Nelder