In mathematics, invariant theory refers to the study of invariant algebraic forms (equivalently, symmetric tensors) for the action of linear transformations. This was a major field of study in the latter part of the nineteenth century, when it appeared that progress in this particular field (out of any number of possible mathematical formulations of invariance with respect to symmetry) was the key algorithmic discipline. Despite some heroic efforts that promise was not fulfilled, one can say; but many spin-off advances were connected. Current theories relating to the symmetric group and symmetric functions, commutative algebra, moduli spaces and the representations of Lie groups are rooted in this area.

In greater detail, given a finite-dimensional vector space V we can consider the symmetric algebra S(V), and the action on it of GL(V). It is actually more accurate to consider the projective representation of GL(V), if we are going to speak of invariants: that's because a scalar multiple of the identity will act on a tensor of rank r in S(V) through the r_th power 'weight' of the scalar. The point is then to define the subalgebra of invariants I(V) for the (projective) action. We are, in classical language, looking at n_ary r_ics, where n is the dimension of V.

These days it might be more natural to look to decompose the degree r part of S(V) into irreducible representations of GL(V): the formulation just given is the same as saying we are concerned only with the occurrence of one-dimensional representations. The representation theory required came later, though, with Issai Schur.

To give the broader picture: what was actually studied in the classical phase of invariant theory related in fact to

where V* is the dual vector space to V. That is, the invariants as polynomials involved a contragredient set of coordinates, transforming in a dual fashion.

It is customary to say that the work of David Hilbert, proving abstractly that I(V) was finitely presented, put an end to classical invariant theory. That is far from being true: the classical epoch in the subject may have continued to the final publications of Alfred Young, more than 50 years later. Explicit calculations for particular purposes have been known in modern times (for example Shioda, with the binary octavics).

The modern formulation of geometric invariant theory is due to David Mumford, and emphasizes the construction of a quotient by the group action that should capture invariant information through its coordinate ring. It is a subtle theory, in that success is obtained by excluding some 'bad' orbits and identifying others with 'good' orbits. In a separate development the symbolic method of invariant theory, an apparently heuristic combinatorial notation, has been rehabilitated.

For the invariant theory of finite groups, see Molien series. See also: invariant.