R. A. Fisher is generally acknowledged as the father of modern statistical science. Fisher was both a mathematician and geneticist, so he was ideally equipped for developing and applying mathematical methods for the design and analysis of scientific studies. In the early 1900s Fisher introduced quantitative measures of statistical efficiency and information, which have become universally accepted as vital guides to the analysis of scientific data.

As statistics developed into a mathematical science in its own right, however, the search for quantitatively optimal statistical procedures inevitably became an end in itself to many researchers. Fisher and others became concerned that this would divert the focus of statistical scientists away from the primary goal of fostering inquiry in substantive scientific fields. Indeed, by the early 1970s, many statistical scientists had taken note of John Tukey's famous admonition "An appropriate answer to the right problem is worth a good deal more than an exact answer to [the wrong] problem." This idea, deliberately introduced by Tukey at the beginning of the computer revolution, has been tremendously influential—Efron's statistical "bootstrap" is but one prominent example. Nonetheless, the quest for "optimal" procedures regardless of their scientific relevance, although somewhat slowed by Tukey's warning, has continued to this day.

Our paper "The Emperor's New Tests" collects a series of simple examples that clearly demonstrate the possible scientific irrelevance of quantitatively optimal statistical procedures. We conclude that the standard optimality criteria themselves, such as the Neyman-Pearson criterion of a most powerful test, are not absolute but rather may be tangential to scientific inquiry. This conclusion is far from new—for example:

• "There is no statistical sense to significance levels." (Herman Rubin, 1969)
• "The difficulty is that the solution to this problem [finding the most powerful test] has no relevance per se to the problems of applied statistics..." (Oscar Kempthorne, 1977)
• "...the familiar optimality criteria of statistics are in fact in conflict with scientific principles..." (Donald Fraser and Nancy Reid, 1990)
• "[Neyman-Pearson theory] does not address the problem of representing and interpreting statistical evidence, and the decision rules derived from NP theory are not appropriate tools for interpreting data as evidence." (Royall, 1997, p.58)
• "This points to the difference between statistics as an effort to learn, to get at the truth, and decision theory—a difference that was emphasized by Fisher in some of his disputes with Neyman." (Erich Lehmann, 1998, after noting frequentist advantages of the NP formulation.)

We hope that "The Emperor's New Tests" will alert young statistical scientists to these essential issues and raise questions that will guide their research toward the direction that Fisher intended.

Michael D. Perlman, Professor
Department of Statistics
University of Washington
Seattle, WA, USA