From Abracadabra to Zombies
Evaluating Personal Experience
by
Robert Todd Carroll
"...the
self is hardly a neutral observer of the world." --Daniel
Schacter, The Seven Sins of Memory
Experts in the psychology of
human error have long been aware that even highly trained experts are
easily misled when they rely on personal experience and informal decision
rules to infer the causes of complex events. --Barry Beyerstein
This article is
dedicated to the memory of Barry L. Beyerstein
There are sound reasons
for preferring the data from randomized, double-blind, controlled
experiments to the data provided by anecdotes when we are searching for
causes. Even well-educated, highly trained experts are subject to many
perceptual,
affective, and cognitive biases that lead us into error when evaluating
personal experiences. The informal procedures most of us use to decide whether
events are causally related are vastly inferior to formal rules such as
Mill's methods. The formal rules aren't infallible
and can be misapplied but they are
orders of magnitude more reliable than naive sense perception, unaided
memory, or so-called intuition. Formal methods of
causal analysis are necessary even if we do not have an emotional,
doctrinal, or monetary stake in the acceptance of a particular causal claim.
Many of our beliefs are driven by our biases and are generated for their
comfort-value rather than for their truth-value. Formal methods of causal
analysis are especially necessary when the causal claim is not particularly comforting or
attractive. All things being equal, the more
impersonal and detached we are in evaluating potential causal events, the less likely error becomes.
Thus, when personal experience conflicts with the data of well designed and
executed controlled studies, it behooves us to examine our personal
experience for bias. We should be especially wary of any
product promising beneficial effects that is marketed before
controlled studies have been done. When there is no compelling supportive
evidence from controlled studies, we should subject a causal claim to a
detailed critical evaluation before considering that there might be a causal
connection between two events. For example, what evidence there is from
controlled studies indicates that the claims of
applied kinesiology
(AK) are bogus. This fact has not diminished the attractiveness of AK
for many people. In fact, some chiropractors reject controlled experiments
on the ground that they don't work, i.e., the results conflict with their
beliefs about and their personal experience with AK. This response is irrational,
however. Instead of
rejecting formal, scientific testing, a rational person would inspect his
conflicting belief and make every effort to discover the source of his error.
The benefits of formal analysis and the pitfalls of anecdotal evidence are
illustrated in the following e-mail I received from "Mike":
Hello!
I can’t attest to the validity of
David Hawkins’s assertions or opinions, but I
will share my personal experience with applied kinesiology with you.
[Hawkins is an AK practitioner, among other things.]
Twenty-five years ago (age 30) I began suffering from serious food
allergies. I began reacting to common foods that I had eaten for most of
my life with increasing severity including a couple of potentially life
threatening anaphylactic shocks that resulted in my being rushed to nearby
hospital emergency wards and saved with injections of epinephrine.
I exhausted the best traditional medical resources available in Boston
without results. Out of desperation, I visited a practitioner of applied
kinesiology on a recommendation from a local nutritionist. It was the
strangest thing. This very professional female in a white lab coat had me
lie down on a medical-style table in what looked like a traditional
doctor’s office. She placed dozens of small amounts food substances
enclosed within what looked like small plastic pill boxes on my stomach
while I laid down face up. She asked me to extend my right arm and resist
her efforts to press my arm down toward the ground. In most instances, I
easily resisted her pressure. In some instances my arm collapsed, which
was astounding to me at the time both physically and mentally. She
meticulously created a list that included several food substances to
eliminate from my diet. I did so.
From that day forward I stopped having allergic food reactions. After a
few months I visited her again. She tested me for the items I showed
sensitivity to during our first visit. I tested as cured. Cautiously, I
reintroduced those food items back into my diet without any repercussions.
I have been fine ever since.
I don’t know how or why this worked. I do know that very sophisticated
lab work conducted in a highly regarded hospital in Boston by a renowned
physician was unable to determine what I was allergic to and how to pursue
relief.
I am a conservative executive, a practicing Catholic, and have
undergraduate and graduate degrees in business.
Feel free to use my comments although I am not on a crusade for the
credibility of applied kinesiology. If you do so, please withhold my last
name.
Mike says he doesn't know how or why AK worked. However,
from this anecdotal evidence alone we can't justifiably say that AK worked.
Mike also implies that AK worked better than the best science available at
the time. After all, AK found what caused his allergies and scientific
medicine did not. The problem here is that at this point it is just an
assumption that Mike suffered from food allergies identified by the AK
lady.
It is understandable that a person suffering from some
problem would infer that since the problem went away after the
treatment that the treatment was effective. The only evidence we have that
the treatment worked (i.e., was effective) is that the problem went away
after the visit to the AK lady and after Mike stopped ingesting the foods on
the list she made for him. The fact that scientific medical tests did not
indicate that Mike's symptoms were due to an allergy is not evidence
in favor of the view that AK is superior to conventional scientific methods.
Had Mike not visited the AK lady, his symptoms might have ceased without any
intervention at all. We have no way of knowing what would have happened to
Mike had he not gone to the AK lady.
Mike's anecdote illustrates why scientists
prefer controlled studies to the details of personal experience, even though
those details are important for medical diagnosis. The first problem with
the anecdote is that we
have to rely on Mike's memory for details. The
events happened twenty-five years ago. A lot of things can conspire to
corrupt even a recent memory. The accuracy of the details may not be
reliable. Memory can be selective and important details may not be recalled.
Things that Mike experienced in the past twenty-five years might get mixed
up with the old memory. Time sequences can get confused. A properly designed
and executed controlled study can eliminate these problems.
Then there is the problem of interpreting the data. Mike seems convinced
that he suffered from food allergies. It would be relatively easy to test
AK's effectiveness at diagnosing food allergies. All we have to do is put the food or
a known non-allergen (for the client) in a pill box and lay it on the
client's stomach and
subject him to the arm pressure test. As long as neither client nor AK
practitioner knows what is in the box we should be able to do a fair test,
expecting distinctly different reactions to the pressure on the arm routine
depending on what's in the box. We could do a significant number of such
tests with various clients, randomly assigning pill boxes and their
contents, which would give us enough data to determine
whether there is likely anything to the AK claims.
To prevent complaints from the AK person that the test
wasn't fair, we'd let her do a few unblinded trials to make sure she thinks
she can do what she claims. John Renish suggests that, to throw a monkey
wrench into the mix, he would "put a compressive strain sensor between the
practitioner's hand and the patient's for both the known and double-blinded
materials: the record of force would demonstrate the variation due to the
practitioner's ideomotor or unconscious effort, or possibly the
practitioner's knowingly fraudulent actions."
There
are a "number of social, psychological, and cognitive factors that can
convince honest, intelligent, and well-educated people that
scientifically-discredited [or untested] treatments have merit."--Barry Beyerstein
On the
other hand, with the anecdote we have some problems. We can probably trust
Mike's memory about the anaphylactic shocks and the treatment with
injections of epinephrine. But we have no way of knowing whether the AK
practitioner's pill box routine reliably identified food allergies. The list
she gave Mike could have been determined by her pill box routine, but she
could have just made it up based on general knowledge of foods that might be
problematic. The amount of pressure she exerted might have been
unconsciously determined
by her beliefs about the allergenic potential of what she knew was in the
pill box for each test. Even more problematic is the fact that there is no compelling scientific evidence to
support the claim that this AK practitioner's method of identifying food
allergens is reliable.
Furthermore, we can't be sure that
the absence of further symptoms was due to the absence of the foods removed
from Mike's diet. One thing happened after the other, but that is
not enough to support a causal
connection. For all we know, some X-factor caused the anaphylactic
shocks, some event or substance not controlled for and completely
independent of any of the foods Mike believes caused his symptoms. The best
medical people in Boston couldn't identify the cause of Mike's symptoms, he
tells us. It is possible but unlikely that such competent medical people
couldn't identify a simple food allergy. The belief that food caused his
medical problems seems based on little more than the claims of the AK
practitioner and the fact that his symptoms did not recur after visiting her
and he stopped eating the foods on her list.
I find it puzzling,
however, that even though he received no therapy, he was told by the AK lady
several months after his first visit with her that he was "cured" of his allergies and could now return to eating the foods
that nearly killed him. He says he returned to eating these foods and hasn't
had any more problems. This evidence is consistent with the claim that an
X-factor, not the foods he avoided for a few months, caused his symptoms.
A properly controlled study would
produce unambiguous results. This anecdote, however, can be interpreted in
several ways. Perhaps Mike can now eat the killer foods because he never had
an allergic reaction to them in the first place. Perhaps his symptoms
weren't caused by an allergic reaction at all. Something else that we know
nothing about may have actually caused his symptoms. Whatever it was, it was
temporary. The symptoms might have been produced by voodoo or they may have
been a punishment from God. Maybe stress or an infection caused the
symptoms. Maybe aliens were experimenting on Mike, unbeknownst to him.
I am no expert on food allergies but from what I
do know, it seems unlikely a person could both go thirty years without a reaction
to certain foods that suddenly nearly kill him and then a few months later find that the
killers have returned to their former gentle selves.
This
anecdote illustrates the power of experience to deceive us into
thinking we have identified a causal agent when, in fact, we have very
little evidence in support of our belief. The astute reader will have
noticed that I made no mention of the placebo
effect to explain Mike's relief. In my Skeptic's Dictionary entry
on homeopathy I review Wendy
Kaminer's defense of the rationality of going to a homeopath even if
there is no compelling scientific evidence that homeopathic remedies are
effective. They may have some benefit due to the placebo effect, she argues,
and so it would be irrational not to go to the homeopath. I won't
repeat my arguments here, but I don't think Kaminer's position is defensible.
Even if AK has some sort of placebo effect, that is not good enough
reason to recommend it. It is possible, I suppose, that the AK lady was so
charming and sweet that she completely disarmed Mike and relieved him of all
stress, thereby relaxing him to such an extent that his own body was able to
produce natural substances that brought him relief or shut down the
production of the harmful chemicals that were causing his symptoms. However,
even if Mike's recovery was due to a placebo effect, we have no way of
knowing this without doing systematic formal testing of AK. With the amount
of evidence we have here, the placebo effect is on par with voodoo or divine
vengeance in terms of its supportive evidence.
It might seem that the evidence in Mike's anecdote that
AK is an effective diagnostic and therapeutic procedure is on par with the
evidence that the prescription a physician gives you for a diagnosed bladder
infection was effective. It might appear that the only evidence you have is
that the infection went away after you took the medicine. However,
presumably the physician chose the drug to prescribe based on the evidence
from scientific studies. The AK lady has no scientific studies to back up
her method of diagnosis. What she probably has is some experience using AK
where the method seems to work. She has probably seen demonstrations of AK,
has received some positive feedback from her customers, and so "knows from
experience" that it works.
The
evidence from satisfied customers is anecdotal and consists of nothing more
than that some customers claiming that after the treatment
they felt better. Again, these customers have no way of knowing that it was
the AK that brought about the desired outcome. All they know is that one
thing happened after another. The demonstrations of AK that the practitioner
has probably witnessed or produced herself were most likely not controlled
demonstrations. The method of testing AK used by practitioners would
probably involve
knowing what substance is being tested and then behaving in an expected
manner. If the substance is believed to be bad for you, then they press hard
or produce little resistance (depending on what role they are playing). If
the substance is believed to be good for you, then they don't press hard or
don't offer much resistance. This may all take place at the unconscious
level, however. So, we need not accuse the AK folks of fraud.
Ideomotor action combined with a strong
desire to succeed in providing support for their belief system is sufficient
to explain how this kind of self-deception
works. These factors also illustrate the importance of doing randomized,
double-blind, controlled studies rather than relying on personal experience
and anecdotes to establish causal connections.
While there haven't been any well designed controlled
studies indicating the effectiveness of AK, there have been numerous studies
that seem to provide support for the effectiveness of
acupuncture. Evaluating treatment by an
acupuncturist is a bit more tricky than evaluating AK treatment. I will try
to illustrate some of the issues involved in evaluating personal experiences
with acupuncture by using an e-mail sent to me from "Fabio." (English is not
Fabio's first language but I have kept my editing to a minimum. I believe
that Fabio expresses himself clearly enough to be understood and that is all
that is required for our purposes here. The reader should understand that it
is the ideas that we are interested in here, not the grammar.)
Regarding your article on acupuncture: I agree with you that there are
many people who claim to do acupuncture and what they do is just
junk. And I agree that there might be several studies that prove
acupuncture is not working, at least not in the way it should, or that it
is even harmful.
On the other hand, take this personal experience, which I witnessed
with my own eyes, from a person I know: A little girl of six
years has had big problems due to the fact that her left leg was 6 cm
[2.4 inches] shorter than the right one since birth. After taking her to hospital, the
doctors advised the parents [that they wanted] to operate [on] their
daughter and remove 6 cm from the longer leg. Accidentally, they met an
acupuncturist, who I know personally, and he offered ... to try to treat
the girl before the operation [and] to try [to] avoid it [the operation].
And funny thing, it worked, since it wasn't the leg that was 6 cm shorter,
but the hip, which was tilted relative to the backbone. No difficult
operation, almost no pain, almost no cost, and no more problems in the
years since the treatment up to now.
I don't say this to claim that acupuncture is THE great thing and can
replace conventional medical treatment. I am aware that there are a lot of
people around who are just frauds or even harm people. But this little
first-hand experience and some others I had myself on my own body,
convinced me that acupuncture definitely can work and can do something to
your body. Of course it will not work if not applied correctly. And it's
no wonder-healing. Just like conventional medicine is dependent on being
applied correctly and cannot perform wonders.
Of course you don't need to believe my story and can say that I am just
telling nonsense out of wishful thinking or whatever. My only advice, if
you want to be a "real" skeptic: go out, try a lot of people [and] a lot
of treatments and see what it can do... if it does not work, you just
waste money, if it helps you with something it is fine and if it harms
you, at least you know that it works, just not in the way you liked it to
have... no, sorry I was just kidding.
Anyhow, from my opinion it is very very important to try out as much as
one can do, and not only read studies and books. In my opinion, this is
what a skeptic in the sense of being a scientist should do. What can
happen? If it does not work, it simply does not work, and if something
works, it's just a starting point for thinking, more investigations and
maybe findings. And it can definitely be exciting.
If you want to post my mail as a reader comment on your page, feel free
to do so.
Fabio advises me and other skeptics to
personally experience acupuncture if we are to understand it. Experience can
provide us with lessons that books and scientific studies cannot.
Furthermore, Fabio and many others are satisfied customers in the sense that
they have experienced the effectiveness of acupuncture.
First, I must remind Fabio and the reader that in my
article on acupuncture I make a point to distinguish the claim that sticking
needles into various parts of a human body induces physiological changes in
that body and the claim that acupuncture works by unblocking chi in
meridians. The concepts of chi and meridians are metaphysical concepts that
evolved long before physiology and long before the discovery of nerve
endings, endorphins, etc. The fact that millions of people have sworn by
acupuncture or lucky charms or prayers or faith healers, etc. is irrelevant
to whether these practices are causal agents in the way their adherents
think they are. As I make clear in my article, appealing to unblocking chi
in meridians is superfluous metaphysical gibberish but the claim that
sticking needles in people has measurable physiological effects is an
empirical claim that has been tested and is correct.
We can test the claim, for example, that sticking needles
in a person's ear lobes effectively treats addiction. But we are
deceiving ourselves if we think we have good evidence for such a claim by
experiencing auricular acupuncture before we quit drinking, smoking, or
using heroin. I understand why a person who went to an acupuncturist to help
with some addictive behavior would think
that the acupuncture was the most significant causal factor in her treatment
if she did actually overcome the addiction. But this subjective certainty of a causal
connection is of little relevance to the accuracy of the general claim:
acupuncture is an effective causal agent in the treatment of addiction. To provide compelling evidence for such a claim we should
require controlled studies, not anecdotes. In our individual experience we
are unable to control for dozens of other factors that might be significant
and we are likely to give too much credit to the one thing that actually had
little or no effect.
Fabio might reply by asking: What difference does it
make? You quit smoking! That's what you wanted. So what if the
acupuncture actually had nothing to do with it? The treatment worked!
In fact, you don't know that the treatment worked, i.e.,
that the acupuncture was effective. You can't reasonably advise others to
try it simply on the grounds that after you did the treatment you quit
smoking. On the other hand, if dozens of your smoking friends and colleagues
all tried it and all quit smoking for several years, we would have to admit
that the odds of some X-factor being involved have diminished. Though the
placebo effect might account for these successes, such a mass of anecdotal
evidence would warrant a nice grant from your government to do a proper
scientific study of the issue. Any lingering doubts we might have about the
causal efficacy of acupuncture for treating nicotine addiction could be
erased by a few well designed and executed controlled studies.
I was intrigued by the story about the little girl
because I was told by my parents that when I was very young one of my legs
was shorter than the other. I received neither surgery nor acupuncture, but
a shoe with a lift. Eventually, I didn't need the shoe. As far as I can
tell, my legs are the same length now. Maybe my hip was out of whack and the
shoe with the lift slowly adjusted it in relation to my leg. I've never
heard of acupuncture making legs grow or hips relocate, so that is a new one
on me. I asked Fabio if she received any other treatment besides acupuncture
and he wrote:
Only acupuncture was used to treat her. In the weeks directly after the
treatment, she had some trouble getting used to the "changes" in her body,
but after this period, everything was fine and just normal, no limp, no
brace. I followed her case for roughly two years after the treatment, where
no negative change to her constitution happened.
What effect, if any, did the acupuncture have on this little girl? There is
no way for us to know. We don't have anything to compare this treatment to.
We have no way of knowing what would have happened had she received no
treatment at all. There are studies that indicate acupuncture can relieve
some of the pain associated with joint problems, but what relevance such
studies have for this case is uncertain. Also, apparently it is not uncommon
to have one leg shorter than the other, a fact that has been exploited by
some
chiropractors.
Acupuncture
supposedly works by unblocking
chi, a universal life force that the ancient Chinese believed runs
through everything in the universe. Many modern people still cling to this
ancient metaphysical notion, although there is no scientific evidence that
it exists. We can only guess how the ancient Chinese came up with this idea,
but the notion of a breathlike spiritual energy flowing through all things
also evolved in other pre-scientific cultures such as ancient India, where
it is called prana. George Dillman teaches martial arts and is a 9th
degree black belt in Ryukyu Kempo Tomari-te. He has trained heavyweight
champion Muhammad Ali and movie action hero Bruce Lee, among others. He
claims he can knock out anyone without touching them by hurling chi
at them. Many students are attracted to Dillman’s martial arts school
because of his reputation for possessing the amazing ability of delivering a
no-touch knockout. The students come, they see, and they end up believing
because they have seen with their own eyes a demonstration of his
superpower. On a National Geographic television program (“Is It Real?
Superpowers”), Dillman pushes his hands through the air at one of his
students while grunting loudly. After a few seconds, the student falls to
the ground and appears to be unconscious. Dillman repeats the demonstration
and knocks over a line of students without touching anyone. He rolls his
hands around an invisible ball in the air before him and pushes his hands
forward while grunting. Soon the line of students tumbles over. Dillman
brags that he once moved the line at Starbucks by using only his mind to
push auras around. Several young
martial arts students were interviewed on the National Geographic program.
Each told how they had come to the school because they had to see this
exhibition for themselves. Now that they had seen with their own eyes what
Dillman can do, they were convinced he was knocking people out with chi.
If you can’t believe your own eyes, what can you believe?
Luigi Garlaschelli, however, had his
doubts. Garlaschelli was not one of Dillman’s fawning students, but a
chemist at Pavia University in Italy. Garlaschelli and fellow skeptic
Massimo Polidoro have had extensive practice uncovering errors made by
people who do not consider that there might be another, better explanation
for their experiences. What appears supernatural or paranormal often has a
simple physical or psychological explanation. Garlaschelli was shown a film
of Dillman delivering his no-touch knockout. “I don’t think it is possible
for a person to transfer his or her energy to another person,” said
Garlaschelli. He agreed to put Dillman’s claim to the test. For some
reason, Dillman did not participate in the test. One of his top aides, Leon
Jay, an 8th degree black belt, was given the job of knocking out
Garlaschelli without touching him. Jay moved his hands in the air around
Garlaschelli’s head for some time but was unable to move the 125-pound
chemist even a fraction of an inch.
Dillman did participate in the
rationalization session that followed Jay’s failure. He claimed that the
skeptic is "a total non-believer," which was stating the obvious and,
ironically, is the key to understanding what is going on with Dillman’s
chi punch. Dillman was suggesting that one must believe in the power of
chi in order to be knocked out by it. That is exactly what the
skeptics were to maintain. Like many AK chiropractors, however, Dillman was
not going to let a controlled experiment destroy his belief. He went on to
claim that maybe Garlaschelli wasn't knocked out because his tongue was "in
the wrong position." He also claimed that he could not knock out someone if
they had one big toe pointing upward and the other pointing downward. He
didn’t elaborate on the requirements of tongue location or toe direction for
the flow of chi. Understandably, National Geographic didn’t do any
further tests on these claims, so we don’t know for sure whether they were
just rationalizations, which is what they appear to be, or legitimate
exceptions to whatever rules govern the manipulation of chi. A
reasonable person might be willing to consider Dillman’s excuses had he
revealed them to an experimenter prior to the test. Bringing them up
after the test had been completed only adds to their implausibility.
Polidoro offered a different
explanation for Dillman’s no-touch knockout and for Jay’s failure to put
down a featherweight Italian chemist using his superpower. Polidoro suspects
that Dillman’s students were being manipulated by the power of suggestion,
not by chi. It was their belief in Dillman’s powers that led
them to unconsciously suggest to themselves that they were being overpowered
by some force controlled by Dillman. The students weren’t pretending to be
knocked out, but they were conditioned to behave the way they did based on
what they believed was expected of them in that situation. The expected
reaction to the master’s moves had been reinforced by the community of
students in past demonstrations.
You can view the experiment and
decide for yourself which explanation for what happens is more plausible,
Dillman's or Polidoro's. Pay attention to how the students fall when they
are "knocked out." This slow, gentle falling is a key to what is going on.
I'll conclude this section with a very apropos quote from
Barry Beyerstein:
The pioneers of the scientific revolution were aware of
the large potential for error when informal reasoning joins forces with our
penchant for jumping to congenial conclusions. By systematizing
observations, studying large groups rather than a few isolated individuals,
instituting control groups, and trying to eliminate confounding variables,
these innovative thinkers hoped to reduce the impact of the frailties of
reasoning that lead to false beliefs about how the world works. None of
these safeguards exists when we base our decisions merely on a few satisfied
customers’ personal anecdotes—unfortunately, these stories are the
“alternative” practitioner’s stock in trade. Psychologists interested in
judgmental biases have repeatedly demonstrated that human inference is
especially vulnerable in complex situations, such as that of evaluating
therapeutic outcomes, which contain a mix of interacting variables and a
number of strong social pressures. Add a pecuniary interest in a particular
outcome, and the scope for self-delusion is immense.
Sense Perception
Some events are not subject to controlled studies, e.g.,
UFO sightings. Having some knowledge about how the visual system works is
essential to evaluating UFO claims. Since we tend to see things that conform
with our beliefs, it is inevitable that skeptics and those who believe we
have been visited by aliens from other worlds will disagree in their
evaluations of such things as the so-called
Phoenix
Lights. On the evening of March 13, 1997, at 8:30 p.m. many people witnessed lights
over Phoenix. At 8:15 p.m. the lights were seen over Prescott, Arizona, and
over Tucson at 8:45. Later, around 10 p.m. up to nine bright lights were
seen in the sky near Phoenix. They appeared to hover for several minutes and
then disappear southwest of Phoenix in the direction of the Sierra Estrella.
"Video cameras at points across the Valley caught the string of hovering
lights. All nine were visible from some locations, others saw fewer."*
The lights were reportedly also seen in Nevada and Sonora (Mexico).
Some
identify the lights as a massive spacecraft of unearthly origin. Others
identify the lights as illumination flares used by the military. Those who believe they saw an alien
spacecraft have no explanation as to where it came from or where it went to
after the time it was visible. Those who believe they saw flares note
that the United States Air Force has stated that A-10 Warthogs from the
Barry Goldwater Range at Luke Air Force Base were on a training exercise
that night that involved jettisoning slow-falling, long-burning illumination
flares over the area. (Air National Guard pilot Lt. Col. Ed Jones has
confirmed that he flew one of the aircraft in the formation that dropped
flares on the night in question.*)
These kinds of military flares can be visible for hundreds of miles. The
skeptics also note that the lights disappeared just as one would expect them
to if they dropped out of sight behind the Estrella mountain range.
One of those who believes he saw an aircraft is former
Arizona governor Fife Symington III. He is quoted as saying: "I'm a pilot
and I know just about every machine that flies. It was bigger than anything
that I've ever seen....And it couldn't have been flares because it was too
symmetrical. It had a geometric outline, a constant shape."
Brian Dunning has investigated the type of flare used
that night:
The illumination flare is ... called the LUU-2 air-deployed high intensity
illumination flare. It's made by defense contractor ATK Thiokol. The variant
in use at the time of the Phoenix Lights incident was the LUU-2B/B. It
weighs 30 pounds and its canister is three-feet long and 5 inches in
diameter. Once it ejects its parachute and ignites, it puts out 1.8 million
candela for 4 minutes, or 1.6 million
candela for 5 minutes. It falls in its
parachute at 8.3 feet-per-second. At 1000 feet above the ground, it lights
up an area half a kilometer wide at 5
lux. The LUU-2's pyrotechnic candle
burns magnesium, which produces an intense white light. Because it burns so
hot, it also ends up burning the aluminum canister, which adds an orange hue
to the light for most of the burn. About halfway through the burn, enough of
the canister has been burned away that it actually lightens the load and it
falls more and more slowly. Once it's almost completely out, an explosive
bolt disconnects the parachute and the flare drops, burning out completely
sometime hopefully before landing on someone's wood shingle roof.*
Despite the governor's protestation, it could have been
flares that he and others saw. Perceiving lights in the sky as symmetrical, geometric,
or constant does not exclude the possibility of the lights being objects
falling from the sky. Everything depends on perspective. Depending on the
angle of perception, the distance and size of the objects emitting the
light, atmospheric conditions, the brightness and constancy of the light,
and wind patterns, military flares could have produced an optical illusion
of a solid object. If one believes he is looking at a spacecraft, he could
easily perceive lights from falling flares as a solid object. The brain is
always filling in the blanks, so to speak, when we see things. We're not
taking videos or pictures of the external world; we're constructing and
projecting from the data available and from our expectations. I can't say that the governor didn't see
an aircraft that night but I am sure he is wrong to claim that it is
impossible that what he saw was flares that appeared to be an aircraft.
Another aspect of perception that cases like the
Phoenix lights invite us to explore is how what others say affects what we
see. Once we've been encouraged to see a spacecraft instead of some drifting
lights, our perception may change and we, too, may see a spacecraft where
moments ago we saw no such thing.
The governor saw the lights with his own eyes, without
the use of cameras. Video and still photos present special problems when
evaluating experience because their settings are not the same as those for
the human eye. Thus, evaluating photos is a different process from
evaluating eyewitness testimony. Things can show up in photos and videos
that are very misleading as to what was actually occurring in the real world
at the time the pictures
were taken. This is especially true when we are dealing with photos taken in
low light or of the night sky. Many people took photos and videos of the
Phoenix Lights but those who then filled in the blanks to produce spacecraft
were constructing a specific vision out of ambiguous data (pareidolia).
Another perceptual twist to this story is that
some people may have seen the lights from planes flying in formation, while
others saw flares. Some may have seen both and conflated them into a single
object. Tony Ortega investigated the story for the Phoenix New Times
and interviewed Mitch Stanley who reported seeing the vee formation from his
backyard. He was able to view the formation through his telescope and was
sure he saw planes in formation, each with two lights. The lights that
appeared as single lights to the naked eye, appeared as two lights in his
telescope.
In an article published a year after the 1997
sighting, Ortega wrote:
An alert owner of a home video camera caught the 8:30 vee
pattern on tape. Terry Proctor filmed the vee for several minutes. The
quality of the tape is poor, and even under enhancement the video shows
nothing joining the five lights of the pattern. However, the pattern of
lights changes over just a few seconds. The lights clearly move in relation
to each other, proving that the lights represent five separate objects,
rather than a solid body. This is consistent with witness reports from
Prescott, where one light trailed the others temporarily. ("The Hack and the
Quack,"
Phoenix New Times, March 5, 1998)
The pictures above show an image from video shot of the 10:30
lights and a drawing of the 8:30 lights that hammers home the vee shape.
Ortega has recently written that programs like Dateline continue to
interview people who saw the 8:30 lights while showing pictures of the 10:30
lights. The witnesses swear that what they saw was not a bunch of flares
hanging from balloons. "They are right," says Ortega. "They couldn't see
flares, they saw a formation of planes." ("The Phoenix Lights Explained
(Again)," Skeptic, Volume 14 Number 2, August, 2008).
MSNBC
considers the Phoenix Lights to be the top UFO story of all time!
The story doesn't end here, however. (In fact, these
kinds of stories never end, thanks to UFO fanatics and the ratings-hungry
media.) On April 21, 2008,
strange lights appeared over Phoenix again! This couldn't be coincidence,
right? But the military was not engaging in any exercises involving
illumination flares on that night. Well, that's what the Air Force
spokesperson said, but he may have been part of a conspiracy to keep the
public ignorant of alien visitation. Could it be that the skeptics were
wrong and that the spacecraft seen by so many people, including the governor
of the state, was making a return visit to the Phoenix area? Perhaps. But
another explanation is that the second lights were launched by a hoaxer. A
helicopter pilot with the Phoenix police department reported that the lights
looked like flares hanging from balloons. Furthermore, there was an
eyewitness to the launching of several helium balloons shortly before the
sightings of the mysterious lights. Lino Mailo said he saw his neighbor
launch balloons with flares attached to them from the back porch of his
north Phoenix home. Ben Radford noted that there are other problems with the
alien spacecraft hypothesis: the lights slowly drifted apart over time and
disappeared one by one, and nothing showed up on the radar screens of air
traffic controllers in the area. ("The Mysterious Phoenix Lights,"
Skeptical Inquirer, July/August 2008.)
Transcendental Experiences
Transcendental experiences are those that appear to be
grounded in some supernatural or paranormal dimension, quite distinct from
ordinary experience. Mystical experiences, out-of-body
and near-death experiences, some
energy healing experiences, and the like
deserve special attention because they often have monumental, life-altering
effects. Those who are skeptical of the supernatural or the paranormal often
derisively reject accounts of transcendental experiences as delusions,
hallucinations, or nothing but brain states. On the other hand, those who
experience the transcendental are often too eager to identify their
experiences as supernatural or paranormal and are unwilling to even consider
other possible explanations. If it is the truth that a person is after, then
it behooves that person to become knowledgeable of what psychology has
learned about unconscious processes that affect how we perceive, remember,
and develop beliefs. The first step each of us must take if we seek the
truth and if we desire to avoid self-deception is to recognize why it is
dangerous to take experience at face value. We must always keep in mind that
nobody is immune to the many perceptual, affective, and cognitive biases
that can lead us into error when evaluating personal experiences.
At the end of his fine little book on
The Psychology of Transcendence, Andrew Neher notes that researchers
have shied away from studying transcendental experiences. Today, more than
twenty-five years after the publication of Neher's book, there is more study
being done on transcendental experiences but overall the subject is
basically still ignored by skeptics* and pursued mostly by incompetents like
Gary Schwartz and
Rupert
Sheldrake on the believer side.* Neher believed that "transcendental
experience is of great importance to us all" (296). Maybe, but until
competent researchers, without axes to grind or points to prove, design and
carry out the research projects, I'm afraid we'll continue to be left
wondering whether there is anything behind the transcendental mask worth
exploring.
See also
Belief Armor,
Why Do People
Believe in the Palpably Untrue?,
Defending
Falsehoods, and
Why Woo-woo
Wins.
further reading
Why We
Need Science: “I saw it with my own eyes” Is Not Enough by Harriet Hall,
M.D.
Robert Todd Carroll
July 16, 2007
* There are a
few exceptions, e.g., Richard
Wiseman, Chris French,
and Peter
Brugger.
Evaluating Personal Experience
by
Robert Todd Carroll
"...the self is hardly a neutral observer of the world." --Daniel Schacter, The Seven Sins of Memory
Experts in the psychology of human error have long been aware that even highly trained experts are easily misled when they rely on personal experience and informal decision rules to infer the causes of complex events. --Barry Beyerstein
This article is dedicated to the memory of Barry L. Beyerstein
There are sound reasons for preferring the data from randomized, double-blind, controlled experiments to the data provided by anecdotes when we are searching for causes. Even well-educated, highly trained experts are subject to many perceptual, affective, and cognitive biases that lead us into error when evaluating personal experiences. The informal procedures most of us use to decide whether events are causally related are vastly inferior to formal rules such as Mill's methods. The formal rules aren't infallible and can be misapplied but they are orders of magnitude more reliable than naive sense perception, unaided memory, or so-called intuition. Formal methods of causal analysis are necessary even if we do not have an emotional, doctrinal, or monetary stake in the acceptance of a particular causal claim. Many of our beliefs are driven by our biases and are generated for their comfort-value rather than for their truth-value. Formal methods of causal analysis are especially necessary when the causal claim is not particularly comforting or attractive. All things being equal, the more impersonal and detached we are in evaluating potential causal events, the less likely error becomes.
Thus, when personal experience conflicts with the data of well designed and executed controlled studies, it behooves us to examine our personal experience for bias. We should be especially wary of any product promising beneficial effects that is marketed before controlled studies have been done. When there is no compelling supportive evidence from controlled studies, we should subject a causal claim to a detailed critical evaluation before considering that there might be a causal connection between two events. For example, what evidence there is from controlled studies indicates that the claims of applied kinesiology (AK) are bogus. This fact has not diminished the attractiveness of AK for many people. In fact, some chiropractors reject controlled experiments on the ground that they don't work, i.e., the results conflict with their beliefs about and their personal experience with AK. This response is irrational, however. Instead of rejecting formal, scientific testing, a rational person would inspect his conflicting belief and make every effort to discover the source of his error.
The benefits of formal analysis and the pitfalls of anecdotal evidence are illustrated in the following e-mail I received from "Mike":
Hello!
I can’t attest to the validity of David Hawkins’s assertions or opinions, but I will share my personal experience with applied kinesiology with you. [Hawkins is an AK practitioner, among other things.]
Twenty-five years ago (age 30) I began suffering from serious food allergies. I began reacting to common foods that I had eaten for most of my life with increasing severity including a couple of potentially life threatening anaphylactic shocks that resulted in my being rushed to nearby hospital emergency wards and saved with injections of epinephrine.
I exhausted the best traditional medical resources available in Boston without results. Out of desperation, I visited a practitioner of applied kinesiology on a recommendation from a local nutritionist. It was the strangest thing. This very professional female in a white lab coat had me lie down on a medical-style table in what looked like a traditional doctor’s office. She placed dozens of small amounts food substances enclosed within what looked like small plastic pill boxes on my stomach while I laid down face up. She asked me to extend my right arm and resist her efforts to press my arm down toward the ground. In most instances, I easily resisted her pressure. In some instances my arm collapsed, which was astounding to me at the time both physically and mentally. She meticulously created a list that included several food substances to eliminate from my diet. I did so.
From that day forward I stopped having allergic food reactions. After a few months I visited her again. She tested me for the items I showed sensitivity to during our first visit. I tested as cured. Cautiously, I reintroduced those food items back into my diet without any repercussions. I have been fine ever since.
I don’t know how or why this worked. I do know that very sophisticated lab work conducted in a highly regarded hospital in Boston by a renowned physician was unable to determine what I was allergic to and how to pursue relief.
I am a conservative executive, a practicing Catholic, and have undergraduate and graduate degrees in business.
Feel free to use my comments although I am not on a crusade for the credibility of applied kinesiology. If you do so, please withhold my last name.
Mike says he doesn't know how or why AK worked. However, from this anecdotal evidence alone we can't justifiably say that AK worked. Mike also implies that AK worked better than the best science available at the time. After all, AK found what caused his allergies and scientific medicine did not. The problem here is that at this point it is just an assumption that Mike suffered from food allergies identified by the AK lady.
It is understandable that a person suffering from some problem would infer that since the problem went away after the treatment that the treatment was effective. The only evidence we have that the treatment worked (i.e., was effective) is that the problem went away after the visit to the AK lady and after Mike stopped ingesting the foods on the list she made for him. The fact that scientific medical tests did not indicate that Mike's symptoms were due to an allergy is not evidence in favor of the view that AK is superior to conventional scientific methods. Had Mike not visited the AK lady, his symptoms might have ceased without any intervention at all. We have no way of knowing what would have happened to Mike had he not gone to the AK lady.
Mike's anecdote illustrates why scientists prefer controlled studies to the details of personal experience, even though those details are important for medical diagnosis. The first problem with the anecdote is that we have to rely on Mike's memory for details. The events happened twenty-five years ago. A lot of things can conspire to corrupt even a recent memory. The accuracy of the details may not be reliable. Memory can be selective and important details may not be recalled. Things that Mike experienced in the past twenty-five years might get mixed up with the old memory. Time sequences can get confused. A properly designed and executed controlled study can eliminate these problems.
Then there is the problem of interpreting the data. Mike seems convinced that he suffered from food allergies. It would be relatively easy to test AK's effectiveness at diagnosing food allergies. All we have to do is put the food or a known non-allergen (for the client) in a pill box and lay it on the client's stomach and subject him to the arm pressure test. As long as neither client nor AK practitioner knows what is in the box we should be able to do a fair test, expecting distinctly different reactions to the pressure on the arm routine depending on what's in the box. We could do a significant number of such tests with various clients, randomly assigning pill boxes and their contents, which would give us enough data to determine whether there is likely anything to the AK claims.
To prevent complaints from the AK person that the test wasn't fair, we'd let her do a few unblinded trials to make sure she thinks she can do what she claims. John Renish suggests that, to throw a monkey wrench into the mix, he would "put a compressive strain sensor between the practitioner's hand and the patient's for both the known and double-blinded materials: the record of force would demonstrate the variation due to the practitioner's ideomotor or unconscious effort, or possibly the practitioner's knowingly fraudulent actions."
There are a "number of social, psychological, and cognitive factors that can convince honest, intelligent, and well-educated people that scientifically-discredited [or untested] treatments have merit."--Barry Beyerstein
On the other hand, with the anecdote we have some problems. We can probably trust Mike's memory about the anaphylactic shocks and the treatment with injections of epinephrine. But we have no way of knowing whether the AK practitioner's pill box routine reliably identified food allergies. The list she gave Mike could have been determined by her pill box routine, but she could have just made it up based on general knowledge of foods that might be problematic. The amount of pressure she exerted might have been unconsciously determined by her beliefs about the allergenic potential of what she knew was in the pill box for each test. Even more problematic is the fact that there is no compelling scientific evidence to support the claim that this AK practitioner's method of identifying food allergens is reliable.
Furthermore, we can't be sure that the absence of further symptoms was due to the absence of the foods removed from Mike's diet. One thing happened after the other, but that is not enough to support a causal connection. For all we know, some X-factor caused the anaphylactic shocks, some event or substance not controlled for and completely independent of any of the foods Mike believes caused his symptoms. The best medical people in Boston couldn't identify the cause of Mike's symptoms, he tells us. It is possible but unlikely that such competent medical people couldn't identify a simple food allergy. The belief that food caused his medical problems seems based on little more than the claims of the AK practitioner and the fact that his symptoms did not recur after visiting her and he stopped eating the foods on her list.
I find it puzzling, however, that even though he received no therapy, he was told by the AK lady several months after his first visit with her that he was "cured" of his allergies and could now return to eating the foods that nearly killed him. He says he returned to eating these foods and hasn't had any more problems. This evidence is consistent with the claim that an X-factor, not the foods he avoided for a few months, caused his symptoms.
A properly controlled study would produce unambiguous results. This anecdote, however, can be interpreted in several ways. Perhaps Mike can now eat the killer foods because he never had an allergic reaction to them in the first place. Perhaps his symptoms weren't caused by an allergic reaction at all. Something else that we know nothing about may have actually caused his symptoms. Whatever it was, it was temporary. The symptoms might have been produced by voodoo or they may have been a punishment from God. Maybe stress or an infection caused the symptoms. Maybe aliens were experimenting on Mike, unbeknownst to him.
I am no expert on food allergies but from what I do know, it seems unlikely a person could both go thirty years without a reaction to certain foods that suddenly nearly kill him and then a few months later find that the killers have returned to their former gentle selves.
This anecdote illustrates the power of experience to deceive us into thinking we have identified a causal agent when, in fact, we have very little evidence in support of our belief. The astute reader will have noticed that I made no mention of the placebo effect to explain Mike's relief. In my Skeptic's Dictionary entry on homeopathy I review Wendy Kaminer's defense of the rationality of going to a homeopath even if there is no compelling scientific evidence that homeopathic remedies are effective. They may have some benefit due to the placebo effect, she argues, and so it would be irrational not to go to the homeopath. I won't repeat my arguments here, but I don't think Kaminer's position is defensible. Even if AK has some sort of placebo effect, that is not good enough reason to recommend it. It is possible, I suppose, that the AK lady was so charming and sweet that she completely disarmed Mike and relieved him of all stress, thereby relaxing him to such an extent that his own body was able to produce natural substances that brought him relief or shut down the production of the harmful chemicals that were causing his symptoms. However, even if Mike's recovery was due to a placebo effect, we have no way of knowing this without doing systematic formal testing of AK. With the amount of evidence we have here, the placebo effect is on par with voodoo or divine vengeance in terms of its supportive evidence.
It might seem that the evidence in Mike's anecdote that AK is an effective diagnostic and therapeutic procedure is on par with the evidence that the prescription a physician gives you for a diagnosed bladder infection was effective. It might appear that the only evidence you have is that the infection went away after you took the medicine. However, presumably the physician chose the drug to prescribe based on the evidence from scientific studies. The AK lady has no scientific studies to back up her method of diagnosis. What she probably has is some experience using AK where the method seems to work. She has probably seen demonstrations of AK, has received some positive feedback from her customers, and so "knows from experience" that it works.
The evidence from satisfied customers is anecdotal and consists of nothing more than that some customers claiming that after the treatment they felt better. Again, these customers have no way of knowing that it was the AK that brought about the desired outcome. All they know is that one thing happened after another. The demonstrations of AK that the practitioner has probably witnessed or produced herself were most likely not controlled demonstrations. The method of testing AK used by practitioners would probably involve knowing what substance is being tested and then behaving in an expected manner. If the substance is believed to be bad for you, then they press hard or produce little resistance (depending on what role they are playing). If the substance is believed to be good for you, then they don't press hard or don't offer much resistance. This may all take place at the unconscious level, however. So, we need not accuse the AK folks of fraud. Ideomotor action combined with a strong desire to succeed in providing support for their belief system is sufficient to explain how this kind of self-deception works. These factors also illustrate the importance of doing randomized, double-blind, controlled studies rather than relying on personal experience and anecdotes to establish causal connections.
While there haven't been any well designed controlled studies indicating the effectiveness of AK, there have been numerous studies that seem to provide support for the effectiveness of acupuncture. Evaluating treatment by an acupuncturist is a bit more tricky than evaluating AK treatment. I will try to illustrate some of the issues involved in evaluating personal experiences with acupuncture by using an e-mail sent to me from "Fabio." (English is not Fabio's first language but I have kept my editing to a minimum. I believe that Fabio expresses himself clearly enough to be understood and that is all that is required for our purposes here. The reader should understand that it is the ideas that we are interested in here, not the grammar.)
Regarding your article on acupuncture: I agree with you that there are many people who claim to do acupuncture and what they do is just junk. And I agree that there might be several studies that prove acupuncture is not working, at least not in the way it should, or that it is even harmful.
On the other hand, take this personal experience, which I witnessed with my own eyes, from a person I know: A little girl of six years has had big problems due to the fact that her left leg was 6 cm [2.4 inches] shorter than the right one since birth. After taking her to hospital, the doctors advised the parents [that they wanted] to operate [on] their daughter and remove 6 cm from the longer leg. Accidentally, they met an acupuncturist, who I know personally, and he offered ... to try to treat the girl before the operation [and] to try [to] avoid it [the operation]. And funny thing, it worked, since it wasn't the leg that was 6 cm shorter, but the hip, which was tilted relative to the backbone. No difficult operation, almost no pain, almost no cost, and no more problems in the years since the treatment up to now.
I don't say this to claim that acupuncture is THE great thing and can replace conventional medical treatment. I am aware that there are a lot of people around who are just frauds or even harm people. But this little first-hand experience and some others I had myself on my own body, convinced me that acupuncture definitely can work and can do something to your body. Of course it will not work if not applied correctly. And it's no wonder-healing. Just like conventional medicine is dependent on being applied correctly and cannot perform wonders.
Of course you don't need to believe my story and can say that I am just telling nonsense out of wishful thinking or whatever. My only advice, if you want to be a "real" skeptic: go out, try a lot of people [and] a lot of treatments and see what it can do... if it does not work, you just waste money, if it helps you with something it is fine and if it harms you, at least you know that it works, just not in the way you liked it to have... no, sorry I was just kidding.
Anyhow, from my opinion it is very very important to try out as much as one can do, and not only read studies and books. In my opinion, this is what a skeptic in the sense of being a scientist should do. What can happen? If it does not work, it simply does not work, and if something works, it's just a starting point for thinking, more investigations and maybe findings. And it can definitely be exciting.
If you want to post my mail as a reader comment on your page, feel free to do so.
Fabio advises me and other skeptics to personally experience acupuncture if we are to understand it. Experience can provide us with lessons that books and scientific studies cannot. Furthermore, Fabio and many others are satisfied customers in the sense that they have experienced the effectiveness of acupuncture.
First, I must remind Fabio and the reader that in my article on acupuncture I make a point to distinguish the claim that sticking needles into various parts of a human body induces physiological changes in that body and the claim that acupuncture works by unblocking chi in meridians. The concepts of chi and meridians are metaphysical concepts that evolved long before physiology and long before the discovery of nerve endings, endorphins, etc. The fact that millions of people have sworn by acupuncture or lucky charms or prayers or faith healers, etc. is irrelevant to whether these practices are causal agents in the way their adherents think they are. As I make clear in my article, appealing to unblocking chi in meridians is superfluous metaphysical gibberish but the claim that sticking needles in people has measurable physiological effects is an empirical claim that has been tested and is correct.
We can test the claim, for example, that sticking needles in a person's ear lobes effectively treats addiction. But we are deceiving ourselves if we think we have good evidence for such a claim by experiencing auricular acupuncture before we quit drinking, smoking, or using heroin. I understand why a person who went to an acupuncturist to help with some addictive behavior would think that the acupuncture was the most significant causal factor in her treatment if she did actually overcome the addiction. But this subjective certainty of a causal connection is of little relevance to the accuracy of the general claim: acupuncture is an effective causal agent in the treatment of addiction. To provide compelling evidence for such a claim we should require controlled studies, not anecdotes. In our individual experience we are unable to control for dozens of other factors that might be significant and we are likely to give too much credit to the one thing that actually had little or no effect.
Fabio might reply by asking: What difference does it make? You quit smoking! That's what you wanted. So what if the acupuncture actually had nothing to do with it? The treatment worked!
In fact, you don't know that the treatment worked, i.e., that the acupuncture was effective. You can't reasonably advise others to try it simply on the grounds that after you did the treatment you quit smoking. On the other hand, if dozens of your smoking friends and colleagues all tried it and all quit smoking for several years, we would have to admit that the odds of some X-factor being involved have diminished. Though the placebo effect might account for these successes, such a mass of anecdotal evidence would warrant a nice grant from your government to do a proper scientific study of the issue. Any lingering doubts we might have about the causal efficacy of acupuncture for treating nicotine addiction could be erased by a few well designed and executed controlled studies.
I was intrigued by the story about the little girl because I was told by my parents that when I was very young one of my legs was shorter than the other. I received neither surgery nor acupuncture, but a shoe with a lift. Eventually, I didn't need the shoe. As far as I can tell, my legs are the same length now. Maybe my hip was out of whack and the shoe with the lift slowly adjusted it in relation to my leg. I've never heard of acupuncture making legs grow or hips relocate, so that is a new one on me. I asked Fabio if she received any other treatment besides acupuncture and he wrote:
Only acupuncture was used to treat her. In the weeks directly after the treatment, she had some trouble getting used to the "changes" in her body, but after this period, everything was fine and just normal, no limp, no brace. I followed her case for roughly two years after the treatment, where no negative change to her constitution happened.
What effect, if any, did the acupuncture have on this little girl? There is no way for us to know. We don't have anything to compare this treatment to. We have no way of knowing what would have happened had she received no treatment at all. There are studies that indicate acupuncture can relieve some of the pain associated with joint problems, but what relevance such studies have for this case is uncertain. Also, apparently it is not uncommon to have one leg shorter than the other, a fact that has been exploited by some chiropractors.
Acupuncture supposedly works by unblocking chi, a universal life force that the ancient Chinese believed runs through everything in the universe. Many modern people still cling to this ancient metaphysical notion, although there is no scientific evidence that it exists. We can only guess how the ancient Chinese came up with this idea, but the notion of a breathlike spiritual energy flowing through all things also evolved in other pre-scientific cultures such as ancient India, where it is called prana. George Dillman teaches martial arts and is a 9th degree black belt in Ryukyu Kempo Tomari-te. He has trained heavyweight champion Muhammad Ali and movie action hero Bruce Lee, among others. He claims he can knock out anyone without touching them by hurling chi at them. Many students are attracted to Dillman’s martial arts school because of his reputation for possessing the amazing ability of delivering a no-touch knockout. The students come, they see, and they end up believing because they have seen with their own eyes a demonstration of his superpower. On a National Geographic television program (“Is It Real? Superpowers”), Dillman pushes his hands through the air at one of his students while grunting loudly. After a few seconds, the student falls to the ground and appears to be unconscious. Dillman repeats the demonstration and knocks over a line of students without touching anyone. He rolls his hands around an invisible ball in the air before him and pushes his hands forward while grunting. Soon the line of students tumbles over. Dillman brags that he once moved the line at Starbucks by using only his mind to push auras around. Several young martial arts students were interviewed on the National Geographic program. Each told how they had come to the school because they had to see this exhibition for themselves. Now that they had seen with their own eyes what Dillman can do, they were convinced he was knocking people out with chi. If you can’t believe your own eyes, what can you believe?
Luigi Garlaschelli, however, had his doubts. Garlaschelli was not one of Dillman’s fawning students, but a chemist at Pavia University in Italy. Garlaschelli and fellow skeptic Massimo Polidoro have had extensive practice uncovering errors made by people who do not consider that there might be another, better explanation for their experiences. What appears supernatural or paranormal often has a simple physical or psychological explanation. Garlaschelli was shown a film of Dillman delivering his no-touch knockout. “I don’t think it is possible for a person to transfer his or her energy to another person,” said Garlaschelli. He agreed to put Dillman’s claim to the test. For some reason, Dillman did not participate in the test. One of his top aides, Leon Jay, an 8th degree black belt, was given the job of knocking out Garlaschelli without touching him. Jay moved his hands in the air around Garlaschelli’s head for some time but was unable to move the 125-pound chemist even a fraction of an inch.
Dillman did participate in the rationalization session that followed Jay’s failure. He claimed that the skeptic is "a total non-believer," which was stating the obvious and, ironically, is the key to understanding what is going on with Dillman’s chi punch. Dillman was suggesting that one must believe in the power of chi in order to be knocked out by it. That is exactly what the skeptics were to maintain. Like many AK chiropractors, however, Dillman was not going to let a controlled experiment destroy his belief. He went on to claim that maybe Garlaschelli wasn't knocked out because his tongue was "in the wrong position." He also claimed that he could not knock out someone if they had one big toe pointing upward and the other pointing downward. He didn’t elaborate on the requirements of tongue location or toe direction for the flow of chi. Understandably, National Geographic didn’t do any further tests on these claims, so we don’t know for sure whether they were just rationalizations, which is what they appear to be, or legitimate exceptions to whatever rules govern the manipulation of chi. A reasonable person might be willing to consider Dillman’s excuses had he revealed them to an experimenter prior to the test. Bringing them up after the test had been completed only adds to their implausibility.
Polidoro offered a different explanation for Dillman’s no-touch knockout and for Jay’s failure to put down a featherweight Italian chemist using his superpower. Polidoro suspects that Dillman’s students were being manipulated by the power of suggestion, not by chi. It was their belief in Dillman’s powers that led them to unconsciously suggest to themselves that they were being overpowered by some force controlled by Dillman. The students weren’t pretending to be knocked out, but they were conditioned to behave the way they did based on what they believed was expected of them in that situation. The expected reaction to the master’s moves had been reinforced by the community of students in past demonstrations.
You can view the experiment and decide for yourself which explanation for what happens is more plausible, Dillman's or Polidoro's. Pay attention to how the students fall when they are "knocked out." This slow, gentle falling is a key to what is going on.
I'll conclude this section with a very apropos quote from Barry Beyerstein:
The pioneers of the scientific revolution were aware of the large potential for error when informal reasoning joins forces with our penchant for jumping to congenial conclusions. By systematizing observations, studying large groups rather than a few isolated individuals, instituting control groups, and trying to eliminate confounding variables, these innovative thinkers hoped to reduce the impact of the frailties of reasoning that lead to false beliefs about how the world works. None of these safeguards exists when we base our decisions merely on a few satisfied customers’ personal anecdotes—unfortunately, these stories are the “alternative” practitioner’s stock in trade. Psychologists interested in judgmental biases have repeatedly demonstrated that human inference is especially vulnerable in complex situations, such as that of evaluating therapeutic outcomes, which contain a mix of interacting variables and a number of strong social pressures. Add a pecuniary interest in a particular outcome, and the scope for self-delusion is immense.
Sense Perception
Some events are not subject to controlled studies, e.g.,
UFO sightings. Having some knowledge about how the visual system works is
essential to evaluating UFO claims. Since we tend to see things that conform
with our beliefs, it is inevitable that skeptics and those who believe we
have been visited by aliens from other worlds will disagree in their
evaluations of such things as the so-called
Phoenix
Lights. On the evening of March 13, 1997, at 8:30 p.m. many people witnessed lights
over Phoenix. At 8:15 p.m. the lights were seen over Prescott, Arizona, and
over Tucson at 8:45. Later, around 10 p.m. up to nine bright lights were
seen in the sky near Phoenix. They appeared to hover for several minutes and
then disappear southwest of Phoenix in the direction of the Sierra Estrella.
"Video cameras at points across the Valley caught the string of hovering
lights. All nine were visible from some locations, others saw fewer."*
The lights were reportedly also seen in Nevada and Sonora (Mexico).
Some identify the lights as a massive spacecraft of unearthly origin. Others identify the lights as illumination flares used by the military. Those who believe they saw an alien spacecraft have no explanation as to where it came from or where it went to after the time it was visible. Those who believe they saw flares note that the United States Air Force has stated that A-10 Warthogs from the Barry Goldwater Range at Luke Air Force Base were on a training exercise that night that involved jettisoning slow-falling, long-burning illumination flares over the area. (Air National Guard pilot Lt. Col. Ed Jones has confirmed that he flew one of the aircraft in the formation that dropped flares on the night in question.*) These kinds of military flares can be visible for hundreds of miles. The skeptics also note that the lights disappeared just as one would expect them to if they dropped out of sight behind the Estrella mountain range.
One of those who believes he saw an aircraft is former Arizona governor Fife Symington III. He is quoted as saying: "I'm a pilot and I know just about every machine that flies. It was bigger than anything that I've ever seen....And it couldn't have been flares because it was too symmetrical. It had a geometric outline, a constant shape."
Brian Dunning has investigated the type of flare used that night:
The illumination flare is ... called the LUU-2 air-deployed high intensity illumination flare. It's made by defense contractor ATK Thiokol. The variant in use at the time of the Phoenix Lights incident was the LUU-2B/B. It weighs 30 pounds and its canister is three-feet long and 5 inches in diameter. Once it ejects its parachute and ignites, it puts out 1.8 million candela for 4 minutes, or 1.6 million candela for 5 minutes. It falls in its parachute at 8.3 feet-per-second. At 1000 feet above the ground, it lights up an area half a kilometer wide at 5 lux. The LUU-2's pyrotechnic candle burns magnesium, which produces an intense white light. Because it burns so hot, it also ends up burning the aluminum canister, which adds an orange hue to the light for most of the burn. About halfway through the burn, enough of the canister has been burned away that it actually lightens the load and it falls more and more slowly. Once it's almost completely out, an explosive bolt disconnects the parachute and the flare drops, burning out completely sometime hopefully before landing on someone's wood shingle roof.*
Despite the governor's protestation, it could have been flares that he and others saw. Perceiving lights in the sky as symmetrical, geometric, or constant does not exclude the possibility of the lights being objects falling from the sky. Everything depends on perspective. Depending on the angle of perception, the distance and size of the objects emitting the light, atmospheric conditions, the brightness and constancy of the light, and wind patterns, military flares could have produced an optical illusion of a solid object. If one believes he is looking at a spacecraft, he could easily perceive lights from falling flares as a solid object. The brain is always filling in the blanks, so to speak, when we see things. We're not taking videos or pictures of the external world; we're constructing and projecting from the data available and from our expectations. I can't say that the governor didn't see an aircraft that night but I am sure he is wrong to claim that it is impossible that what he saw was flares that appeared to be an aircraft.
Another aspect of perception that cases like the Phoenix lights invite us to explore is how what others say affects what we see. Once we've been encouraged to see a spacecraft instead of some drifting lights, our perception may change and we, too, may see a spacecraft where moments ago we saw no such thing.
The governor saw the lights with his own eyes, without
the use of cameras. Video and still photos present special problems when
evaluating experience because their settings are not the same as those for
the human eye. Thus, evaluating photos is a different process from
evaluating eyewitness testimony. Things can show up in photos and videos
that are very misleading as to what was actually occurring in the real world
at the time the pictures
were taken. This is especially true when we are dealing with photos taken in
low light or of the night sky. Many people took photos and videos of the
Phoenix Lights but those who then filled in the blanks to produce spacecraft
were constructing a specific vision out of ambiguous data (pareidolia).
Another perceptual twist to this story is that some people may have seen the lights from planes flying in formation, while others saw flares. Some may have seen both and conflated them into a single object. Tony Ortega investigated the story for the Phoenix New Times and interviewed Mitch Stanley who reported seeing the vee formation from his backyard. He was able to view the formation through his telescope and was sure he saw planes in formation, each with two lights. The lights that appeared as single lights to the naked eye, appeared as two lights in his telescope.
In an article published a year after the 1997 sighting, Ortega wrote:
An alert owner of a home video camera caught the 8:30 vee pattern on tape. Terry Proctor filmed the vee for several minutes. The quality of the tape is poor, and even under enhancement the video shows nothing joining the five lights of the pattern. However, the pattern of lights changes over just a few seconds. The lights clearly move in relation to each other, proving that the lights represent five separate objects, rather than a solid body. This is consistent with witness reports from Prescott, where one light trailed the others temporarily. ("The Hack and the Quack," Phoenix New Times, March 5, 1998)
The pictures above show an image from video shot of the 10:30 lights and a drawing of the 8:30 lights that hammers home the vee shape. Ortega has recently written that programs like Dateline continue to interview people who saw the 8:30 lights while showing pictures of the 10:30 lights. The witnesses swear that what they saw was not a bunch of flares hanging from balloons. "They are right," says Ortega. "They couldn't see flares, they saw a formation of planes." ("The Phoenix Lights Explained (Again)," Skeptic, Volume 14 Number 2, August, 2008).
MSNBC considers the Phoenix Lights to be the top UFO story of all time!
The story doesn't end here, however. (In fact, these kinds of stories never end, thanks to UFO fanatics and the ratings-hungry media.) On April 21, 2008, strange lights appeared over Phoenix again! This couldn't be coincidence, right? But the military was not engaging in any exercises involving illumination flares on that night. Well, that's what the Air Force spokesperson said, but he may have been part of a conspiracy to keep the public ignorant of alien visitation. Could it be that the skeptics were wrong and that the spacecraft seen by so many people, including the governor of the state, was making a return visit to the Phoenix area? Perhaps. But another explanation is that the second lights were launched by a hoaxer. A helicopter pilot with the Phoenix police department reported that the lights looked like flares hanging from balloons. Furthermore, there was an eyewitness to the launching of several helium balloons shortly before the sightings of the mysterious lights. Lino Mailo said he saw his neighbor launch balloons with flares attached to them from the back porch of his north Phoenix home. Ben Radford noted that there are other problems with the alien spacecraft hypothesis: the lights slowly drifted apart over time and disappeared one by one, and nothing showed up on the radar screens of air traffic controllers in the area. ("The Mysterious Phoenix Lights," Skeptical Inquirer, July/August 2008.)
Transcendental Experiences
Transcendental experiences are those that appear to be grounded in some supernatural or paranormal dimension, quite distinct from ordinary experience. Mystical experiences, out-of-body and near-death experiences, some energy healing experiences, and the like deserve special attention because they often have monumental, life-altering effects. Those who are skeptical of the supernatural or the paranormal often derisively reject accounts of transcendental experiences as delusions, hallucinations, or nothing but brain states. On the other hand, those who experience the transcendental are often too eager to identify their experiences as supernatural or paranormal and are unwilling to even consider other possible explanations. If it is the truth that a person is after, then it behooves that person to become knowledgeable of what psychology has learned about unconscious processes that affect how we perceive, remember, and develop beliefs. The first step each of us must take if we seek the truth and if we desire to avoid self-deception is to recognize why it is dangerous to take experience at face value. We must always keep in mind that nobody is immune to the many perceptual, affective, and cognitive biases that can lead us into error when evaluating personal experiences.
At the end of his fine little book on The Psychology of Transcendence, Andrew Neher notes that researchers have shied away from studying transcendental experiences. Today, more than twenty-five years after the publication of Neher's book, there is more study being done on transcendental experiences but overall the subject is basically still ignored by skeptics* and pursued mostly by incompetents like Gary Schwartz and Rupert Sheldrake on the believer side.* Neher believed that "transcendental experience is of great importance to us all" (296). Maybe, but until competent researchers, without axes to grind or points to prove, design and carry out the research projects, I'm afraid we'll continue to be left wondering whether there is anything behind the transcendental mask worth exploring.
See also Belief Armor, Why Do People Believe in the Palpably Untrue?, Defending Falsehoods, and Why Woo-woo Wins.
further reading
Why We Need Science: “I saw it with my own eyes” Is Not Enough by Harriet Hall, M.D.
Robert Todd Carroll
July 16, 2007
* There are a few exceptions, e.g., Richard Wiseman, Chris French, and Peter Brugger.
