In statistics, effect size is a measure of the strength of the relationship between two variables. In scientific experiments, it is often useful to know not only whether an experiment has a statistically significant effect, but also the size of any observed effects. In practical situations, effect sizes are helpful for making decisions. Effect size measures are the common currency of meta-analysis studies that summarize the findings from a specific area of research. This article is about the field of statistics. ... In statistics, a result is significant if it is unlikely to have occurred by chance, given that a presumed null hypothesis is true. ... A meta-analysis is a statistical practice of combining the results of a number of studies. ...

Summary

The concept of effect size appears in everyday language. For example, a weight loss program may boast that it leads to an average weight loss of 30 pounds. In this case, 30 pounds is an indicator of the claimed effect size. Another example is that a tutoring program may claim that it raises school performance by one letter grade. This grade increase is the claimed effect size of the program.

An effect size is best explained through an example: if you had no previous contact with humans, and one day visited England, how many people would you need to see before you realize that, on average, men are taller than women there? The answer relates to the effect size of the difference in average height between men and women. The larger the effect size, the easier it is to see that men are taller. If the height difference were small, then it would require knowing the heights of many men and women to notice that (on average) men are taller than women. This example is demonstrated further below. For other uses, see England (disambiguation). ...

In inferential statistics, an effect size helps to determine whether a statistically significant difference is a difference of practical concern. In other words, given a sufficiently large sample size, it is always possible to show that there is a difference between two means being compared out to some decimal position. The effects size helps us to know whether the difference observed is a difference that matters. Effect size, sample size, critical significance level (α), and power in statistical hypothesis testing are related: any one of these values can be determined, given the others. In meta-analysis, effect sizes are used as a common measure that can be calculated for different studies and then combined into overall analyses. It has been suggested that this article or section be merged with statistical inference. ... In statistics, a result is significant if it is unlikely to have occurred by chance, given that a presumed null hypothesis is true. ... The power of a statistical test is the probability that the test will reject a false null hypothesis (that it will not make a Type II error). ... One may be faced with the problem of making a definite decision with respect to an uncertain hypothesis which is known only through its observable consequences. ... A meta-analysis is a statistical practice of combining the results of a number of studies. ...

The term effect size is most commonly used to describe standardized measures of effect (e.g., r, Cohen's d, odds ratio, etc.). However, unstandardized measures (e.g., the raw difference between group means, unstandardized regression coefficients, etc.) can equally be effect size measures. Standardized effect size measures are typically used when the metrics of variables being studied do not have intrinsic meaning to the reader (e.g., a score on a personality test on an arbitrary scale), or when results from multiple studies are being combined when some or all of the studies use different scales. Some students mistook the recommendation of Wilkinson & APA Task Force on Statistical Inference (1999, p. 599)--Always present effect sizes for primary outcomes--as that reporting standardized measures of effect like Cohen's d is the default requirement. Actually, just following the sentence the authors added that -- If the units of measurement are meaningful on a practical level (e.g., number of cigarettes smoked per day), then we usually prefer an unstandardized measure (regression coefficient or mean difference) to a standardized measure (r or d).

Types

Pearson R correlation

Pearson's R correlation, introduced by Karl Pearson, is one of the most widely used effect sizes. It can be used when the data are continuous or binary; thus the Pearson R is arguably the most versatile effect size. This was the first important effect size to be developed in statistics. Pearson's R can vary in magnitude from -1 to 1, with -1 indicating a perfect negative relationship, 1 indicating a perfect positive relationship, and 0 indicating no relationship between two variables. Cohen (1988, 1992) gives the following guidelines for the social sciences: small effect size, R = 0.1; medium, R = 0.3; large, R = 0.5. (Note that correlation coefficients for the physical sciences are typically of a different order of magnitude.) Positive linear correlations between 1000 pairs of numbers. ... Karl Pearson FRS (March 27, 1857 – April 27, 1936) established the discipline of mathematical statistics. ...

Another often-used measure of the strength of the relationship between two variables is the coefficient of determination (the square of r, referred to as "r-squared"). This is a measure of the proportion of variance shared by the two variables, and varies from 0 to 1. An r² of 0.21 means that 21% of the total variance is shared by the two variables. In statistics, the coefficient of determination R2 is the proportion of variability in a data set that is accounted for by a statistical model. ...

Cohen's d

Cohen's d is the appropriate effect size measure to use in the context of a t-test on means. d is defined as the difference between two means divided by the pooled standard deviation for those means. Thus, in the case where both samples are the same size, A t-test is any statistical hypothesis test in which the test statistic has a Students t-distribution if the null hypothesis is true. ... Pooled standard deviation is a way to find a better estimate of the true standard deviation given several different samples taken in different circumstances where the mean may vary between samples but the true standard deviation (precision) is assumed to remain the same. ...

where mean_i and SD_i are the mean and standard deviation for group i, for i = 1, 2.

Different people offer different advice regarding how to interpret the resultant effect size, but the most accepted opinion is that of Cohen (1992) where 0.2 is indicative of a small effect, 0.5 a medium and 0.8 a large effect size. This article is about mathematical mean. ... In probability and statistics, the standard deviation of a probability distribution, random variable, or population or multiset of values is a measure of the spread of its values. ...

So, in the example above of visiting England and observing men and women's height, the data (Aaron,Kromrey,& Ferron, 1998, November; from a 2004 UK representative sample of 2436 men and 3311 women) is:

• Men: Mean Height = 1750 mm; Standard Deviation = 89.93 mm
• Women: Mean Height = 1612 mm; Standard Deviation = 69.05 mm

The effect size (using Cohen's d) would equal 1.72 (95% confidence intervals: 1.66 - 1.78). This is very large and you should have no problem in detecting that there is a consistent height difference, on average, between men and women. In this diagram, the bars represent observation means and the red lines represent the confidence intervals surrounding them. ...

One point worth noting, though, is that in some cases it may be wise to use just one of the standard deviations (e.g., pre-treatment standard deviation in a therapeutic trial). Either way, note that sample size does not play a part in the calculation - points noted by Hedges.

Another way of calculating effect size is to subtract one mean away from the other (ignore the sign) and then divide the answer by the mean standard deviation.

Hedges' ĝ

Hedges and Olkin (1985) noted that one could adjust effect size estimates by taking into account the sample size. The problem with Cohen's d is that the outcome is heavily influenced by the denominator in the equation. If one standard deviation is larger than the other then the denominator is weighted in that direction and the effect size is more conservative. However, surely it makes more sense to put stock in the larger sample size? Hedges' ĝ incorporates sample size by both computing a denominator which looks at the sample sizes of the respective standard deviations and also makes an adjustment to the overall effect size based on this sample size. The formula for Hedges' ĝ (as used by software such as the Effect Size Generator) is:

In the above 'height' example, Hedges' ĝ effect size equals 1.76 (95% confidence intervals: 1.70 - 1.82). Notice how the large sample size has increased the effect size from Cohen's d? If, instead, the available data were from only 90 men and 80 women Hedges' ĝ would provide a more conservative estimate of effect size: 1.70 (with larger 95% confidence intervals: 1.35 - 2.05). In this diagram, the bars represent observation means and the red lines represent the confidence intervals surrounding them. ... In this diagram, the bars represent observation means and the red lines represent the confidence intervals surrounding them. ...

Cohen's f²

Cohen's f² is the appropriate effect size measure to use in the context of an F-test for multiple correlation or multiple regression. The f² effect size measure for multiple regression is defined as: An F-test is any statistical test in which the test statistic has an F-distribution if the null hypothesis is true. ... // Introduction In statistics, regression analysis is a method for explanation of phenomena and prediction of future events. ... It has been suggested that this article or section be merged into linear regression. ...

where R² is the squared multiple correlation.

The f² effect size measure for hierarchical multiple regression is defined as: In statistics, the coefficient of determination R2 is the proportion of a sample variance of a response variable that is explained by the predictor variables when a linear regression is done. ...

where is the variance accounted for by a set of one or more independent variables A, and is the combined variance accounted for by A and another set of one or more independent variables B.

By convention, f² effect sizes of 0.02, 0.15, and 0.35 are considered small, medium, and large, respectively (Cohen, 1988).

φ, Cramer's φ, or Cramer's V

The best measure of association for the chi-square test is phi (or Cramer's phi or V). Phi is related to the point-biserial correlation coefficient and Cohen's d and estimates the extent of the relationship between two variables (2 x 2).^[1] Cramer's Phi may be used with variables having more than two levels. A chi-square test is any statistical hypothesis test in which the test statistic has a chi-square distribution when the null hypothesis is true, or any in which the probability distribution of the test statistic (assuming the null hypothesis is true) can be made to approximate a chi-square... The point biserial correlation coefficient is a correlation coefficient used when one variable is dichotomous and nominal. ...

Phi can be computed by finding the square root of the chi-square statistic divided by the sample size.

Similarly, Cramer's phi can be found through a slightly more complex formula that takes the number of rows or columns into account (k).

Odds ratio

The odds ratio is another useful effect size. It is appropriate when both variables are binary. For example, consider a study on spelling. In a control group, two students pass the class for every one who fails, so the odds of passing are two to one (or more briefly 2/1 = 2). In the treatment group, six students pass for every one who fails, so the odds of passing are six to one (or 6/1 = 6). The effect size can be computed by noting that the odds of passing in the treatment group are three times higher than in the control group (because 6 divided by 2 is 3). Therefore, the odds ratio is 3. However, odds ratio statistics are on a different scale to Cohen's d. So, this '3' is not comparable to a Cohen's d of '3'. The odds-ratio is a statistical measure, particularly important in Bayesian statistics and logistic regression. ...

References

1. ^ Aaron, B., Kromrey, J. D., & Ferron, J. M. (1998, November). Equating r-based and d-based effect-size indices: Problems with a commonly recommended formula. Paper presented at the annual meeting of the Florida Educational Research Association, Orlando, FL. (ERIC Document Reproduction Service No. ED433353)

• Aaron, B., Kromrey, J. D., & Ferron, J. M. (1998, November). Equating r-based and d-based effect-size indices: Problems with a commonly recommended formula. Paper presented at the annual meeting of the Florida Educational Research Association, Orlando, FL. (ERIC Document Reproduction Service No. ED433353)
• Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum
• Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155-159.
• Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. San Diego, CA: Academic Press.
• Lipsey, M.W., & Wilson, D.B. (2001). Practical meta-analysis. Sage: Thousand Oaks, CA.
• Wilkinson, L., & APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604.

External links

Software

• Free Effect Size Generator Software - PC & Mac Software
• Free Odds Ratio Generator Software - PC Software
• Free GPower Software - PC & Mac Software
• Free Effect Size Calculator for Multiple Regression - Web Based
• Free Effect Size Calculator for Hierarchical Multiple Regression - Web Based

Further Explanations

• Effect Size (ES)
• Measuring Effect Size
• Effect size for two independent groups
• Effect size for two dependent groups