NationMaster - Encyclopedia: Selection bias

Selection bias is the error of distorting a statistical analysis by pre- or post-selecting the samples. Typically this causes measures of statistical significance to appear much stronger than they are, but it is also possible to cause completely illusory artifacts. Selection bias can be the result of scientific fraud which manipulate data directly, but more often is either unconscious or due to biases in the instruments used for observation. For example, astronomical observations will typically find more blue galaxies than red ones simply because most instruments are more sensitive to blue light than red light. Statistics is the science and practice of developing knowledge through the use of empirical data expressed in quantitative form. ... In statistics, a result is significant if it is unlikely to have occurred by chance, given that a presumed null hypothesis is true, but is not improbable if the null hypothesis is false. ... Scientific misconduct is the violation of the standard codes of scholarly conduct and ethical behavior in professional scientific research. ...

There are many types of possible selection bias, including:

Spatial:

Selecting end-points of a series. For example, to maximise a claimed trend, you could start the time series at an unusually low year, and end on a high one.
Early termination of a trial at a time when its results support a desired conclusion.
A trial may be terminated early at an extreme value (often for ethical reasons), but the extreme value is likely to be reached by the variable with the largest variance, even if all variables have a similar mean. As a result of that early termination, therefore, the means of variables with larger variances are overestimated.
Partitioning data with knowledge of the contents of the partitions, and then analyzing them with tests designed for blindly chosen partitions (see stratified sampling, cluster sampling, Texas sharpshooter fallacy).
Analyzing the lengths of intervals by selecting intervals that occupy randomly chosen points in time or space, a process that favors longer intervals.

Data: Ethics is a general term for what is often described as the science (study) of morality. In philosophy, ethical behavior is that which is good or right. ... In probability theory and statistics, the variance of a random variable is a measure of its statistical dispersion, indicating how far from the expected value its values typically are. ... In statistics, mean has two related meanings: the average in ordinary English, which is more correctly called the arithmetic mean, to distinguish it from geometric mean or harmonic mean. ... Stratified sampling is a method of sampling from a population in statistics. ... Cluster sampling is used when natural groupings are evident in the population. ... The Texas sharpshooter fallacy is a logical fallacy where a cluster of statistically non-significant data is taken from its context, and therefore thought to have a common cause. ...

Rejection of "bad" data on arbitrary grounds, instead of according to previously stated or generally agreed criteria

Participants:

Pre-screening of trial participants, or advertising for volunteers within particular groups. For example to "prove" that smoking doesn't affect fitness, advertise for both at the local fitness centre, but advertise for smokers during the advanced aerobics class, and for non-smokers during the weight loss sessions.
Discounting trial subjects/tests that did not run to completion. For example, in a test of a dieting program, the researcher may simply reject everyone who drops out of the trial. But most of those who drop out are those for whom it wasn't working.
Self-selection bias, which is possible whenever the group of people being studied has any form of control over whether to participate. Participants' decision to participate may be correlated with traits that affect the study, making the participants a non-representative sample. For example, people with strong opinions or substantial knowledge may be more willing to spend time answering a survey than those who don't.

Studies:

Selection of which studies to include in a meta-analysis
Performing repeated experiments and reporting only the most favourable results. (Perhaps relabelling lab records of other experiments as "calibration tests", "instrumentation errors" or "preliminary surveys".)
Presenting the most significant result of a data dredge as if it were a single experiment. (Which is logically the same as the previous item, but curiously is seen as much less dishonest.)

Selection bias is closely related to: A meta-analysis is a statistical practice of combining the results of a number of studies. ... Data dredging is the term used to refer to the unscrupulous search for statistically significant relationships in large quantities of data. ...

sample bias, a selection bias produced by an accidental bias in the sampling technique, as against deliberate or unconscious manipulation.
publication bias or reporting bias, the distortion produced in community perception or meta-analyses by not publishing uninteresting (usually negative) results, or results which go against the experimenter's prejudices, a sponsor's interests, or community expectations.
confirmation bias, the distortion produced by experiments that are designed to seek confirmatory evidence instead of trying to disprove the hypothesis.

In statistics, a biased estimator is one that for some reason on average over_ or underestimates what is being estimated. ... Publication bias, also called the positive outcome bias, is typically the tendency for researchers to publish experimental results that have a positive result (found something), while consequently not publishing findings which have a negative result (found that something did not happen). ... A meta-analysis is a statistical practice of combining the results of a number of studies. ... In statistical inference, confirmation bias is a type of cognitive bias toward confirmation of the hypothesis under study. ...