selection bias

Robert Todd Carroll

selection bias

Selection bias comes in two flavors: (1) self-selection of individuals to participate in an activity or survey, or as a subject in an experimental study; (2) selection of samples or studies by researchers to support a particular hypothesis.

Edzard Ernst, M.D., who was trained in various non-conventional medical therapies, provides an example of selection bias that occurred while he was studying the therapeutic effect of mistletoe injections on cancer patients. He was told that the effect would be a lessening of suffering.

Whenever I gave mistletoe injections, the results seemed encouraging. But young doctors are easily impressed, and I was no exception. What I didn't appreciate then was a relatively simple phenomenon: the hospital where I worked was well known for its approach across Germany; patients went there because they wanted this type of treatment. They were desperate and had very high expectations - and expectations can often move mountains, particularly in relation to subjective experience and symptoms. We call this "selection bias". It can give the impression that a therapy causes a positive health outcome even when it has no positive action of its own.*

Selection bias partly explains why there are so many satisfied customers who go to psychics, tarot card readers, palmists, and faith healers. The subjects are highly motivated to be helped and to have the reader or healer succeed. Subjects are often extremely generous in their efforts to personally validate the words, images, or advice of the reader/healer. Some will even assent to claims they know are false, as one of Gary Schwartz's subjects did with a medium who got the subject to agree that her husband was dead when in fact he was still alive. (Schwartz did not see this event as an example of the power of selection bias to affect subjective validation of psychic claims. Instead, he saw it as a evidence for clairvoyance on the part of the psychic and the sitter whose husband died in a car crash shortly after the session.) In fact, Schwartz engaged in selection bias when he published several papers in the Journal of the Society for Psychical Research that support the hypothesis of survival of consciousness after death. In his book, The Afterlife Experiments, he describes numerous subjects in his experiments who are conspicuously not mentioned in the published papers on those experiments.

Skeptics and parapsychologists have accused each other of selection bias in determining which studies to include in the ganzfeld meta-analysis (Bem and Honorton 1994; C.E.M. Hansel 1989; Hyman 1989; Marks 2000; Radin 1997). Ray Hyman did the first meta-analysis of 42 experiments and found no evidence of ESP. Honorton did his own meta-analysis and found evidence of "anomalous information transfer." In 1994, Bem and Honorton published the results of a meta-analysis of 28 ganzfeld studies and once again found evidence for anomalous information transfer. Julie Milton and Richard Wiseman (1999) published their own meta-analysis of ganzfeld studies and concluded that "the ganzfeld technique does not at present offer a replicable method for producing ESP in the laboratory."

selection bias in prayer studies

Elisabeth Targ's study on distance healing using prayer is "widely acknowledged as the most scientifically rigorous attempt ever to discover if prayer can heal" (Bronson 2002), even though it only had 40 participants. The study has since been discredited for improprieties in mining the data. The study was originally designed to test whether prayer could have an effect on the death rate from AIDS, but was changed after the study was completed to see if there were any significant correlations between prayer and about two dozen illnesses associated with AIDS such as diarrhea and oral thrush. The statistician was able to find about half a dozen significant correlations. These were selected as if they were the main purpose of the study, after it was apparent that prayer had no effect on survival.

selection bias in polls and surveys

Norman Bradburn, director of the National Opinion Research Center at the University of Chicago, coined the acronym SLOP to describe polls that use selection bias to get their samples. SLOP is an acronym for self-selected listener opinion polls. He compares them to radio talk shows: they attract a slice of America that is not representative of the country as a whole. “As a result, SLOP surveys litter misinformation and confusion across serious policy and political debates, virtually wherever and whenever they are used” (Morin).

The inaccuracy of such polls should be obvious. Those who call in to give their opinion are self-selected rather than randomly selected. It appears that people who are willing to call in their opinion once will sometimes call in their opinion more than once. For example, in a USA Today call-in poll 81 percent of the more than 6,000 respondents said that “Donald Trump symbolizes what made the U.S.A. a great country.” However, 72 percent of the favorable calls came from two telephones in one insurance company office.
CBS tried the gimmick of call-in polling in “America on the Line,” which featured two surveys conducted immediately after President [George Herbert Walker] Bush’s State of the Union speech. There were 314,786 self-selected callers in one survey and 1,241 adults previously selected by a more scientific method in the other survey. The latter was to act as a check on the call-in survey. CBS’s Dan Rather commented on the similarity of results in the surveys, a sentiment that was echoed the next day in the Washington Post, which wrote, “by and large, the two polls produced the same or similar results.” The facts, however, do not support this judgment. “On two of the nine questions asked in both polls, the results differed by more than 20 percentage points. On another five, the differences were 10 percentage points or more” (Morin, cited in Carroll 2005: 147).

Some very famous surveys have been based upon selection bias. The work of Alfred C. Kinsey, for example, is the source for a homosexuality statistic widely reported in both the mass media and in scientific publications, though the statistic is based on biased samples. Kinsey’s famous studies on sexual behavior in the 1950s have been repeatedly cited as the basis for the claim that 10% of the population is gay. In fact, numerous studies have been done that put the percentage of adults who describe themselves as exclusively gay as much lower than the 10% figure. Some found the rate to be between one and two percent.* However, it should be noted that "survey research methodologies often result in underreporting of stigmatized behaviors."*

Kinsey gathered his data, in part, by distributing questionnaires to prisoners and to people who attended his lectures on sexuality, neither of which was likely to be a good cross section of Americans (Carroll 2005: 140). For his studies on male sexuality, "he interviewed only white men, and these respondents were disproportionately from lower socioeconomic classes."*

A study that considered attraction to the same sex, in measuring homosexuality, found "8.7, 7.9, and 8.5% of males and 11.1, 8.6, and 11.7% of females in the United States, the United Kingdom, and France, respectively, report some homosexual attraction but no homosexual behavior since age 15."*

In 1994, sociologist Edward Laumann headed a team of sociologists that studied U.S. sexual behavior. They interviewed a representative sample of the U.S. population between the ages of 18 and 59. Laumann found that over a five-year period, 4.1 percent of U.S. men and 2.2 percent of U.S. women had sex with someone of their own sex. If the time period is extended to include their entire lives, these totals increase to 7.1 percent of the men and 3.8 percent of the women.*

Paul and Kirk Cameron reported in 1998: "The 1994 University of Chicago "definitive" survey of adults estimated prevalence of homosexuality among males at 2.8% and among females at 1.4%. Corrected for the exclusion of those over the age of 59 years, the estimates should be 2.3% and 1.2%."* A study in Britain in 2000 found that about 2.6% of men and women reported having had a same-sex partner within the previous five years and 8.4% of the men and 9.7% of the women reported having had at least one sexual experience with a member of the same sex.*

One wonders, however, if anything approaching unbiased data is possible for determining what percentage of any human population is homosexual. Given the long history of religious prohibition of homosexuality and the widespread revulsion of homosexual behavior that has often led to torment and persecution, it is likely that researchers in this area will be motivated by something other than a generic search for the truth. Results will differ depending on how one defines 'gay,' 'lesbian,' and 'homosexual'. Methods of gathering data samples will vary widely and the participants in such studies may not be highly motivated to reveal much about their sex lives.

There is some irony in the fact that the Kinsey studies are cited as the source of the statistic that 10% of the population is gay. As Michael Shermer notes, Kinsey made it clear that he did not believe human males "represent two discrete populations, heterosexual and homosexual." Kinsey maintained that "it is a fundamental of taxonomy that nature rarely deals with discrete categories. Only the human mind invents categories and tries to force facts into separate pigeon-holes" (Shermer 2005: 246). Nature has a bias toward variation. The idea that people should fall into neat categories as 'gay' and 'straight', or even 'male' and 'female', is not consistent with the lessons of evolution. Any study that creates such false dichotomies will be misleading.

further reading

Medicine man: Mistletoe - viable alternative cancer treatment, or just a placebo with dangerous side-effects? by Edzard Ernst, December 21, 2004 The Guardian (an example of selection bias by Rudolph Steiner)

Bem, Daryl J. and Charles Honorton (1994). "Does Psi Exist?" Psychological Bulletin, Vol. 115, No. 1, 4-18.

Bronson, Po. (2002). "A Prayer Before Dying," Wired Dec.

Carroll, Robert Todd. (2005). Becoming a Critical Thinker: A Guide for the New Millennium, 2nd ed. Pearson.

Hansel, C.E.M. The Search for Psychic Power : ESP and Parapsychology Revisited (Buffalo, N.Y.: Prometheus Books, 1989).

Hyman, Ray (1989). The Elusive Quarry: a Scientific Appraisal of Psychical Research. Prometheus Books.

Marks, David. The Psychology of the Psychic (Buffalo, N.Y.: Prometheus Books, 2000).

Milton, J. and R. Wiseman (1999). "Does psi exist? Lack of replication of an anomalous process of information transfer." Psychological Bulletin, 125 (4), 387-391.

Morin, Richard. “Call in polls: Pseudo-Science debases journalism,” Sacramento Bee, Feb. 12, 1992, p. B11.

Radin, Dean (1997). The Conscious Universe - The Scientific Truth of Psychic Phenomena. HarperCollins. (This entire book is an example of selection bias in favor of one's pet beliefs.)

Shermer, Michael. (2005). The Science of Good and Evil: Why People Cheat, Gossip, Care, Share, and Follow the Golden Rule. Owl Books.

séance

Last updated 12/03/07

selective thinking