NationMaster - Encyclopedia: Reliability (statistics)

In statistics, reliability is the consistency of a set of measurements or measuring instrument, often used to describe a test. This can either be whether the measurements of the same instrument give or are likely to give the same measurement (test-retest), or in the case of more subjective instruments, such as personality or trait inventories, whether two independent assessors give similar scores (inter-rater reliability). Reliability is inversely related to random error. This article is about the field of statistics. ... In education, certification, counseling, the military, and many other fields, a test or an exam (short for examination) is a tool or technique intended to measure students expression of knowledge, skills and/or abilities. ... Inter-rater reliability or Inter-rater agreement is the measurement of agreement between raters. ... In statistics, the concepts of error and residual are easily confused with each other. ...

Reliability does not imply validity. That is, a reliable measure is measuring something consistently, but not necessarily what it is supposed to be measuring. For example, while there are many reliable tests of specific abilities, not all of them would be valid for predicting, say, job performance. In terms of accuracy and precision, reliability is precision, while validity is accuracy. In psychometrics a valid measure is one which is measuring what it is supposed to measure. ... â€œAccuracyâ€ redirects here. ...

In experimental sciences, reliability is the extent to which the measurements of a test remain consistent over repeated tests of the same subject under identical conditions. An experiment is reliable if it yields consistent results of the same measure. It is unreliable if repeated measurements give different results. It can also be interpreted as the lack of random error in measurement.^[1] In the scientific method, an experiment (Latin: ex- periri, of (or from) trying) is a set of observations performed in the context of solving a particular problem or question, to retain or falsify a hypothesis or research concerning phenomena. ...

In engineering, reliability is the ability of a system or component to perform its required functions under stated conditions for a specified period of time. It is often reported in terms of a probability. Evaluations of reliability involve the use of many statistical tools. See Reliability engineering for further discussion. Engineering is the discipline and profession of applying scientific knowledge and utilizing natural laws and physical resources in order to design and implement materials, structures, machines, devices, systems, and processes that realize a desired objective and meet specified criteria. ... Reliability engineering is an engineering field, that deals with the study of reliability: the ability of a system or component to perform its required functions under stated conditions for a specified period of time. ...

1 Bathroom scale analogy
2 Estimation
3 Classical test theory
4 Item response theory
5 See also
6 References
7 External links

Bathroom scale analogy

An often-used example used to elucidate the difference between reliability and validity in the experimental sciences is a common bathroom scale. If someone that weighs 200 lbs. steps on the scale 10 times, and it reads "200" each time, then the measurement is reliable and valid. If the scale consistently reads "150", then it is not valid, but it is still reliable because the measurement is very consistent. If the scale varied a lot around 200 (190, 205, 192, 209, etc.), then the scale could be considered valid but not reliable. Digital kitchen scales. ...

Estimation

Reliability may be estimated through a variety of methods that fall into two types: Single-administration and multiple-administration. Multiple-administration methods require that two assessments are administered. In the test-retest method, reliability is estimated as the Pearson product-moment correlation coefficient between two administrations of the same measure. In the alternate forms method, reliability is estimated by the Pearson product-moment correlation coefficient of two different forms of a measure, usually administered together. Single-administration methods include split-half and internal consistency. The split-half method treats the two halves of a measure as alternate forms. This "halves reliability" estimate is then stepped up to the full test length using the Spearman-Brown prediction formula. The most common internal consistency measure is Cronbach's alpha, which is usually interpreted as the mean of all possible split-half coefficients.^[2] Cronbach's alpha is a generalization of an earlier form of estimating internal consistency, Kuder-Richardson Formula 20.^[2] Test-retest is a statistical method used to examine how reliable a test is: A test is performed twice, e. ... In statistics, the Pearson product-moment correlation coefficient (sometimes known as the PMCC) (r) is a measure of the correlation of two variables X and Y measured on the same object or organism, that is, a measure of the tendency of the variables to increase or decrease together. ... In statistics and research, internal consistency is a measure based on the correlations between different items on the same test (or the same subscale on a larger test). ... The Spearman-Brown prediction formula (also known as the Spearman-Brown prophecy formula) is a formula relating psychometric reliability to test length: where is the predicted reliability; N is the number of tests combined (see below); and is the reliability of the current test. The formula predicts the reliability of... Cronbachs (alpha) has an important use as a measure of the reliability of a psychometric instrument. ... In statistics, the Kuder-Richardson Formula 20 (KR-20) is a measure of reliability for measures with dichotomous choices. ...

Each of these estimation methods is sensitive to different sources of error and so might not be expected to be equal. Also, reliability is a property of the scores of a measure rather than the measure itself and are thus said to be sample dependent. Reliability estimates from one sample might differ from those of a second sample (beyond what might be expected due to sampling variations) if the second sample is drawn from a different population because the true reliability is different in this second population. (This is true of measures of all types--yardsticks might measure houses well yet have poor reliability when used to measure the lengths of insects.)

Reliability may be improved by clarity of expression (for written assessments), lengthening the measure,^[2] and other informal means. However, formal psychometric analysis, called the item analysis, is considered the most effective way to increase reliability. This analysis consists of computation of item difficulties and item discrimination indices, the latter index involving computation of correlations between the items and sum of the item scores of the entire test. If items that are too difficult, too easy, and/or have near-zero or negative discrimination are replaced with better items, the reliability of the measure will increase.

$R (t) = 1 - F (t)$ .

$R (t) = exp( - λ t)$ . (where $λ$ is the failure rate)

Classical test theory

In classical test theory, reliability is defined mathematically as the ratio of the variation of the true score and the variation of the observed score. Or, equivalently, one minus the ratio of the variation of the error score and the variation of the observed score: Classical test theory is a body of related psychometric theory that predict outcomes of psychological testing such as the difficulty of items or the ability of test-takers. ...

${rho}_{xx'}=frac{{sigma}^2_T}{{sigma}^2_X}=1-frac{{{sigma}^2_E}}{{{sigma}^2_X}}$

where $ρ x x'$ is the symbol for the reliability of the observed score, X; ${sigma}^2_X$ , ${sigma}^2_T$ , and ${sigma}^2_E$ are the variances on the measured, true and error scores respectively. Unfortunately, there is no way to directly observe or calculate the true score, so a variety of methods are used to estimate the reliability of a test.

Some examples of the methods to estimate reliability include test-retest reliability, internal consistency reliability, and parallel-test reliability. Each method comes at the problem of figuring out the source of error in the test somewhat differently.

Item response theory

It was well-known to classical test theorists that measurement precision is not uniform across the scale of measurement. Tests tend to distinguish better for test-takers with moderate trait levels and worse among high- and low-scoring test-takers. Item response theory extends the concept of reliability from a single index to a function called the information function. The IRT information function is the inverse of the conditional observed score standard error at any given test score. Higher levels of IRT information indicate higher precision and thus greater reliability. Item response theory (IRT) is a body of related psychometric theory that provides a foundation for scaling persons and items based on responses to assessment items. ...

References

^ Rudner, L.M., & Shafer, W.D. (2001). Reliability. ERIC Digest. College Park, MD: ERIC Clearinghouse on Assessment and Evaluation. [1]
^ ^a ^b ^c Cortina, J.M., (1993). What Is Coefficient Alpha? An Examination of Theory and Applications. Journal of Applied Psychology, 78(1), 98-104.

External links

Uncertainty models, uncertainty quantification, and uncertainty processing in engineering
The relationships between correlational and internal consistency concepts of test reliability
The problem of negative reliabilities

Categories: Statistics | Psychometrics | Market research | Educational psychology research methods

COMMENTARY

Share your thoughts, questions and commentary here

Want to know more?
Search encyclopedia, statistics and forums:

Feeds | Contact
The Wikipedia article included on this page is licensed under the GFDL.
Images may be subject to relevant owners' copyright.
All other elements are (c) copyright NationMaster.com 2003-5. All Rights Reserved.
Usage implies agreement with terms.

Contents

Bathroom scale analogy GA_googleFillSlot("encyclopedia_square");