Bacon (The Advancement of Learning,
1605) on the tendency to publish only positive results that tell good stories:
For as knowledges are now delivered, there is a kind of contract of error
between the deliverer and the receiver; for he that delivereth knowledge
desireth to deliver it in such form as may be best believed, and not as
may be best examined; and he that receiveth knowledge desireth rather
present satisfaction than expectant inquiry.
Begg
and Berlin then cite several documented cases of publication bias. We
can hardly doubt, for example, that a correlation exists between socioeconomic
status and academic achievement, but the strength and nature of this association
represent important information for both political practice and social
theory. A 1982 study by K. R. White revealed a progressively increasing
intensity of correlation with the prestige and permanence of the published
source. Studies published in books reported an average correlation coefficient
of 0.51 between academic achievement and socioeconomic status; articles
in journals gave an average of 0.34, while unpublished studies yielded
a value of 0.24. Similarly, in a 1986 article, A. Coursol and E. E. Wagner
found publication bias both in the decision to submit an article at all,
and in the probability of its acceptance. In a survey of outcomes in psychotherapy,
they found that 82 percent of studies with positive results lead to submission
of papers to a journal, while only 43 percent of negative outcomes provoked
an attempt at publication. Of papers submitted, 80 percent reporting positive
outcomes were accepted for publication, versus only 50 percent of papers
claiming negative results.
My favorite study of publication bias is the book-length Myths of
Gender by Anne Fausto-Sterling, a unique and important contribution
to the literature of feminism for this reason. In tabulating claims in
the literature for consistent differences in cognitive and emotional styles
of men and women, Fausto-Sterling does not deny that genuine differences
often exist, and in the direction conventionally reported. But she then,
so to speak, surveys her colleagues' file drawers for studies not published,
or for negative results published and then ignored, and often finds that
a great majority report either a smaller and insignificant disparity between
sexes or find no differences at all. When all studies, those not published
as well as those published, are collated, the much-vaunted differences
often devolve into triviality. Natural history, after all (as I have argued
so often in these essays), is preeminently a study of relative frequency,
not of absolute yeses or noes. If a claim based on published literature
states that "women in all studies strongly..."--and the
addition of unpublished data changes that claim to "in a minority
of studies, a weak effect suggests that women..."--then meaning
is effectively reversed (even though positive outcomes, when rarely found,
show a consistent direction.)
For example, a recent favorite in pop psychology (although waning of
late, I think), has attributed different cognitive styles in men and women
to the less lateralized brains of women (less specialization between right
and left hemispheres of the cerebral cortex). Some studies have indeed
reported a small effect of greater male lateralization; none has found
more lateralized brains in women. But most experiments, Fausto-Sterling
found, detected no measurable differences in lateralization--and this
is the dominant relative frequency (even in published literature) that
should be prominently reported, but tends to be ignored as "no story."
Publication bias is serious enough in its promotion of a false impression
based on a small and skewed subset of the total number of studies. But
at least the right questions are being asked and negative results can
be conceptualized and obtained--even if they then tend to be massively
underreported. But consider the fur more insidious problem closer to Cordelia's
dilemma with her father: what if our conceptual world excludes the possibility
of acknowledging a negative result as a phenomenon at all? What if we
simply can't see, or even think about, a different and meaningful alternative?
Cordelia's plight is a dilemma in the literal sense--a choice between
two equally undesirable alternatives: she either remains honorable, says
nothing, and incurs |