12 Mar 95
Thomas C. Reeves

Ron Oliver's [10 Mar 95] contention that my recent ITForum paper is by my own standards a piece of pseudoscience is yet another example of Ron's marvelous sense of humor. In 1993, Ron and I spent days together traveling around remote areas of Western Australia while evaluating a distance education program. His jokes (as well as his driving) kept me in stitches. (By the way, in addition to being an accomplished comedian, Ron is by his own admission the most talented instructional designer in all of Australia. His patented multimedia course, "The Fundamentals of Sailboat Construction," was used to train this year's Australian competitors in the America's Cup Race.)

In any case, Ron knows very well that the 39 articles in Educational Technology Research & Development (ETR&D) and the 56 articles in Journal of Computer-Based Instruction (JCBI) that I classified as "empirical" in terms of their goals and "quantitative" in terms of their methods all used experimental or quasi-experimental designs to test hypotheses. Reviewing these experimental studies, I found that 62 of the total of 95 studies exhibited two or more of the characteristics of pseudoscience. That anyone could conclude that my paper is an example of pseudoscience when my review was neither experimental nor quasi-experimental is quite amusing. It sure made me chuckle.

On the other hand, Ron's suggestion that a literature review should focus on outcomes rather than processes is a bit worrisome. Is Ron pulling my leg again? How can anyone judge the outcomes reported in a research paper if he/she doesn't understand how they were obtained? The way I learned to conduct literature reviews was to examine the research question first, then the underlying theory, next the related literature, followed by the methods and analysis, and finally the results and discussion. I was taught that if I found the methods inappropriate to the questions or the analysis flawed, I had to ignore the results. My review of the papers in ETR&D and JCBI indicated that 65% of the studies exhibited two or more (usually many more) characteristics of pseudoscience. The remaining 35% were the only studies for which consideration of outcomes would have been valid.

What is really surprising is Ron's admission that he doesn't "have many problems with pseudoscience." Such humility is unbecoming, Ron! From my perspective, you don't have "any" problems with pseudoscience! You've mastered it! ;-)

All kidding aside, the real issue of whether as a research community, we can see the error of our ways and commit ourselves to socially responsible research. As Jim Quinn [10 Mar 95] wisely reminds us in his ITForum note, it is "a very fundamental responsibility for us to think deeply about (1) the goals of our research programs and (2) how successful are the paradigms we are using in achieving these goals." I contend that if our goals are "enhancing human communication, learning, and performance though the use of theory and technology," we should set aside empirical, quantitative studies for the time being, and focus instead on alternative approaches such as developmental research studies that involve mixed methods.

I wish I could say that I am hopeful that many of us in the IT research community will actually re-examine our goals and methods, but I'm not optimistic, especially given the lack of serious discussion of this issue on this listserv. As Phillips (1987) wrote, "New approaches... are being formulated that make the 'true experiment' seem like a lumbering dinosaur, yet some folk persist in thinking that dinosaurs are wonderful creatures" (p. viii). The dominance of the empirical, quantitative approach derives strength from the degree to which it is entrenched in the universities where many novice IT researchers are learning their craft. Many (most?) members of the tenured generation of educational psychology and instructional technology faculty who teach research methods were schooled in the positivist, quantitative "paradigm" and are unfamiliar with or uncomfortable with alternative approaches to inquiry. These same people populate the editorial boards that publish the pseudoscience identified in my review. This dual stranglehold on the future of socially responsible research in IT will not be easy to break.