I'd like to respond to the comments about my Questioning the Questions of Instructional Technology Research paper made by the Information Technology and the Teaching and Learning Process Group, Faculty of Education, Sydney University, NSW Australia. They wrote:
[quoting Harvey, 8 Mar 96] Do you consider your paper to be research? We are proceeding on the assumption that it is.
Actually, I don't consider my paper to be "research" at all, especially not in the sense that you seem to be using that term. I wrote this paper as a critical essay to address the question "Is research in IT socially responsible?" In my paper, I did not lay claim to any of the goals of research which I defined in Figure 1 of the paper as: Theoretical, Empirical, Interpretivist, Postmodern, Developmental, or Evaluation. Neither did I claim to be employing any of the methods of research which I defined in Figure 2 of my paper as: Quantitative, Qualitative, Critical Theory, Literature Review, and Mixed-methods. Therefore, your assumption is fallacious. Nevertheless, the substance of your comments warrants further discussion.
1. Consequently, we find it a touch ironic that you've employed a quantitative/empirical method, of which you seem to be so critical. For example we have some problems with the fact that your survey is only of two journals, and that the sample is a snapshot of a rather limited time-frame.
I don't understand why you think that I employed a "quantitative/empirical method" in this paper. The closest I came to using any type of research method in this paper is where I used "literature review" methods to support my critical analysis, i.e., I reviewed research papers presented in Educational Technology Research & Development (ETR&D) from 1989-94 and Journal of Computer-Based Instruction (JCBI) from 1988-93. In Figure 2, I defined "Literature Review" methods as: "various forms of research synthesis that primarily involve the analysis and integration of other forms of research, e.g., frequency counts and meta-analyses." Although I presented some "quantitative" data in terms of frequency counts and percentages, this hardly approximates an "quantitative/empirical method." Indeed, the primary substance of my literature review was qualitative in nature, e.g., based upon judgments I made in categorizing the goals of these research studies and the methods used in them.
2. We have applied your criteria for defining pseudoscience to your own paper. We feel that you meet at least two or more of the characteristics you outlined. For example, inadequate sample size and simplistic statistical analyses have been used to make broad generalizations.
Given that my critical essay is by no means "Empirical" in intent and "Quantitative" in method, the characteristics of pseudoscience that I identified in Figure 3 of my paper simply don't apply. However, let me examine your critique from the perspective of a literature review method. Concerning the issue of sample size, what would you recommend as an adequate sample size in a literature review of this type? For ETR&D, I reviewed all 104 articles in the research section of the journal published in the last six years, and for JCBI, I reviewed all 129 articles published in it between 1988 and 1993 when it ceased publication. Among IT researchers in the USA, these two journals have been considered to be respectable outlets for refereed research papers. I maintain that these 233 articles provide an adequate sampling of the state of IT research at this time, at least within the USA. (I encourage anyone to conduct a similar review of one or more related journals published in Australia or other countries). Concerning the issue of "simplistic statistical analysis," what statistical analysis would you recommend for this type of literature review other than frequency counts and percentages?
3. Looked at overall, your article neatly proves the point that systemic analysis would have been of more benefit in proving your point rather than the quantitative empirical methods you employed.
Salomon (1991) maintains that advocates of a "systemic" approach to studying a phenomenon assume that "each component, event, or action has the potential of affecting the unit as a whole; the whole is assumed to be more than the sum of its components and is characterized by the patterns and forms of the relations among them" (p. 14). By contrast, proponents of the "analytic" approach assume that "complex behaviors, settings, and internal events are additively and interactionally composed of more basic elements, the effects of which can be studied in isolation" (p. 13). Although my essay is not about "proving" anything, I consider the argument in my paper to have more in common with a systemic approach than an analytic one. I have tried to persuade readers that IT research lacks social relevance. This is not easily done in a brief essay because of the inherent complexity of the question of social relevance. For example, some of the factors that influence this lack of social relevance include the "publish or perish" phenomenon among academics, inadequate training in research methods, lack of understanding of the philosophy of science, incestuous peer reviews that undermine our research integrity, and the inability of many people to recognize or critique pseudoscience.
There is a tone to your critique that suggests that I have a bias against research that is "empirical" in intent and "quantitative" in method. I would like to clarify this. My view is summed up in the following quote from Lipsey (1993):
There can be little quarrel with this minimalist research paradigm when it is applied to relatively simple situations, such as input events that are basically indivisible molar wholes, a few well-defined outcome states, and input-output changes that are contiguous, immediate, and direct. Indeed, it is worth remembering that, in the Fisherian tradition, experimental design was invented for agricultural studies that had many of these properties. However, most causal phenomenon of practical interest are more complex. They involve multidimensional interactions that often are extended over time, complex multistep causal processes in which different individuals may react differently, and uncertain and potentially wide-ranging outcomes, not all necessarily desirable.
I maintain that most of us conducting "empirical, quantitative" research in IT ignore the enormous differences between studying plants and studying humans. The most serious weakness in our research relates to a characteristic of pseudoscience identified in my paper as the "lack of linkage to robust theory." What the empiricists among us conveniently forget is that the "empirical, quantitative" methodology demands that experiments are grounded in strong theory which:
1. clearly defines independent and dependent variables,
2. specifies the interrelationships among variables as well as their relationships with relevant contextual variables, and
3. predicts the direction and strength of outcomes and side effects.
Instead of playing the empirical game according to the rules, we frequently employ what Campbell (1986) and Lipsey (1993) label atheoretical "let's try this and see if it works" experiments. I contend that this is not only misguided, but ultimately unethical because of its lack of social relevance. I do believe that theory-based empirical research studies utilizing both experimental and quasi-experimental designs, implemented with fidelity, analyzed with appropriate inferential statistics, and concluded with meaningful discussion of the results, eventually may have value in terms of the research "complementarity" advocated by Salomon (1991). However, my critical analysis of the research reported in ETR&D and JCBI indicates that there is precious little research of this kind being conducted.
One last point. Although I maintain that pseudoscience and the lack of social relevance are major problems for IT research, I am not advocating qualitative research as "the" solution. As Cizek points out in a witty essay in this month's issue of Educational Researcher, the rush toward qualitative methods is often led by "wild-eyed storytelling mythmakers" who seem to be guided more by sociopolitical causes than by a genuine desire to know. There is probably just as much pseudoscience going on today in the name of interpretivist research as my analysis indicates is the problem with empirical research. Instead, I advocate IT research programs that have "developmental" goals and are "mixed" in terms of method.