23 Oct 96.a
Tom Reeves

Reasoning that, because we can do experimental research in physics and make better cars and planes, we can conduct experimental research of the kind you espouse with human learning and as a result invent better education and training seems to be just as fallacious as the "Quantum Leap" inference from Heisenberg's "uncertainty principle" to alternative methodologies in educational research. If Lloyd Rieber's critique of those of us who are tempted to leap from quantum physics to educational research paradigms has merit (and I think it does), your leap from the impact of physics experiments on engineering to the world of teaching and learning is similarly flawed.

IN RESPONSE TO MY STATEMENT THAT IT IS "THE CONDITIONS" THAT MAKE ALL THE DIFFERENCE IN DRAWING CONCLUSIONS FROM YOUR TYPE OF STUDY, YOU WROTE:

Come on, Steve. If you're going to be a positivist, at least play the game according to the rules. Do you really think your study and Furnham and Gunter's represent a sufficient body of research to support a principle of instructional design that text is better than audio for learning facts? Wow? If it was this simple, why did Lee Cronbach and Richard Snow (1977) as well as a whole generation of educational researchers give up on the quest for establishing an empirical basis for aptitude-treatment interactions?

In an earlier contribution to ITForum, I described nine signs of pseudoscience that are often found in your brand of empirical research studies. These are:

1. Specification error--Vague definitions of the primary independent variables (e.g., text versus audio).

2. Lack of linkage to robust theory--Little more than nominal attention to the underlying learning and instructional theories that are relevant to the investigation.

3. Inadequate literature review--Cursory literature review focused on the results of closely related studies with little or no consideration of alternative findings.

4. Inadequate treatment implementation--Infrequent (usually single) treatment implementation often averaging less than 30 minutes.

5. Measurement flaws--Precise measurement of easy-to-measure variables (e.g., time); insufficient effort to establish the reliability and validity of measures of other variables.

6. Inconsequential outcome measures--A lack of intentionality in the learning context, usually represented by outcome measures that have little or no relevance for the subjects in the study.

7. Inadequate sample sizes--Small samples of convenience, e.g., the ubiquitous undergraduate teacher education or psychology majors.

8. Inappropriate statistical analysis--Use of obscure statistical procedures in an effort to tease statistically significant findings out of the data.

9. Meaningless discussion of results--Rambling, often incoherent, rationales for failing to find statistically significant findings.

I have nothing against experimental and quasi-experimental research and evaluation designs per se, and I use them whenever they are appropriate. (For example, earlier this year some friends and I reported the results of a quasi-experimental study (Reeves, Laffey, & Marlino, 1996) at the annual meeting of the American Educational Research Association in New York.)

The problem I have with these methods is that they are so poorly applied in most research on media. For example, I suspect that a careful analysis of your study would indicate that your study suffers from at least pseudoscience signs numbers 1, 2, 3, 4, 5, 6, and 7. Some of this analysis has already been done by other contributors to this discussion, but it is difficult to do a more detailed analysis without a full report of your investigation.