[quoting Hart's paper] Do Tom Reeves' well-known arguments about pseudoscience apply equally to qualitative research? What kinds of qualitative outcomes could be branded as pseudo- ? (and what should be the root of this compound noun?)
In his otherwise brilliant paper, Ian claims that my arguments about pseudoscience are "well-known." Wrong! The continuing proliferation of pseudoscience reported at our conferences and published in our journals indicates the fallacy of this claim.
Ian is referring to a series of papers I have written that address the weaknesses of instructional technology research conducted by researchers with empirical goals using experimental or quasi-experimental methods (cf. Reeves, 1986, 93, 95). In the most recent of these papers, a version of which appeared on this forum two year's ago, I described nine signs of pseudoscience in these types of studies. (Interested parties can find versions of this paper at the InTRO
The nine signs of pseudoscience are:
1. Specification error: Vague or inappropriate definitions of the primary independent variables (e.g., learner control versus program control).
2. Lack of linkage to robust theory: Little more than nominal attention to the underlying learning and instructional theories relevant to the investigation.
3. Inadequate literature review: Cursory literature review focused on the results of closely related studies with little or no consideration of alternative findings.
4. Inadequate treatment implementation: Infrequent (usually single) treatment implementation often averaging 30 minutes or less.
5. Measurement flaws: Precise measurement of easy-to-measure variables (e.g., time) accompanied by insufficient efforts to establish the reliability and validity of measures of other variables (e.g., learning).
6. Inconsequential outcome measures: A lack of intentionality in the learning context, usually represented by outcome measures that have little or no relevance for the subjects in the study.
7. Inadequate sample sizes: Small samples of convenience, e.g., the ubiquitous undergraduate teacher education or psychology majors.
8. Inappropriate statistical analysis: Use of obscure statistical procedures in an effort to tease statistically significant findings out of the data.
9. Meaningless discussion of results: Rambling, often incoherent, rationales for failing to find statistically significant findings.
Given my past critiques of analytical research, I appreciate Ian's invitation to consider the signs of "pseudoscience" in systemic inquiry. Although there are fewer published studies to analyze, the qualitative research in our field can be just as flawed as empirical investigations using quantitative methods. However, I am not qualified to delineate a list of "Signs of Pseudoscience in Qualitative Research." (To identify these flaws, I need the help of my expert "qualitative" colleagues here at UGA plus some of you on this forum.) However, I can briefly describe three obvious flaws I have observed.
Weak or inappropriate data analysis strategies are probably the most common and most serious weaknesses in qualitative studies. There are several rigorous approaches to analyzing qualitative data, but the actual strategies used should be guided by the nature of the research questions informing the study. An example of this flaw is the researcher who will claim he/she is going to use "constant comparative analysis" when all the data has already been collected. This doesn't make sense.
A weakness qualitative studies share with their empirical, quantitative cousins is a failure to conduct a comprehensive literature review. Some qualitative researchers have the false notion that if they are going to observe technology in use in a given context they shouldn't read any of the research literature concerning that technology or that context because it will bias their observations. Actually, we are biased no matter what our preparation (or lack thereof), and it is better to be aware of what others have revealed about this phenomenon than to assume a false sense of naivete. There are strategies for dealing with bias in qualitative inquiry, but ignorance is not the answer.
The failure to consider alternative interpretations is another flaw. If qualitative researchers fail to find anything that seriously surprises them or runs counter to the expectations they had going into a study, there may be a problem. In my own experience, instructional technologies and other innovations rarely work as planned and it is usually the unintended outcomes that are the most notable. Qualitative research, done well, is more difficult than most quantitative studies, and it often challenges our most cherished assumptions about teaching, learning, and technology. Ironically, some researchers seem drawn to qualitative inquiry because they perceive it as easier (i.e., no statistics), but nothing could be further from "the truth."
I won't go into other flaws right now because I don't want to cloud the major issue that Ian Hart has set before us. Like Salomon (1991) before him, Ian is challenging us to view "non-quantitative methods as an indispensable monkey-wrench in the research toolbox; and to demonstrate that the outcomes of qualitative investigation can be equally productive, though often differently valid, to the selective, positivist, analytico-deductive, scientific, statistical research methods sanctioned by the gatekeepers of our discipline." I hope that many of us will accept this challenge.
Reeves, T.C. (1995). Questioning the questions of instructional technology research. In M.R. Simonson & M. Anderson (Eds.), Proceedings of the Annual Conference of the Association for Educational Communications and Technology, Research and Theory Division (pp. 459-470), Anaheim, CA.
Reeves, T.C. (1993). Pseudoscience in computer-based instruction: The case of learner control research. Journal of Computer-Based Instruction, 20(2), 39-46.
Reeves, T.C. (1986). Research and evaluation models for the study of interactive video. Journal of Computer-Based Instruction, 13, 102-106.
Salomon, G. (1991). Transcending the qualitative-quantitative debate: The analytic and systemic approaches to educational research. Educational Researcher, 20(6), 10-18.