My paper on qualitative research did not raise many hackles. The reason may be that where you are it's either a motherhood issue or it's of marginal interest. Nevertheless we stirred one or two 'possums such as the methods of teaching research, and perhaps provided food for thought which will permeate future discussions. As the author I get the chance to sum up, so in the absence of great controversy I will reiterate what I consider to be the most important issues.
One purpose of my paper was to argue that we put our trust in quantitative outcomes because they are comfortable and comprehensible. We're surrounded by (obsessed with?) statistics so we think we understand what it means when we're told that "Labor has a 26 point lead on the Conservatives" or "David Helfgot has a 180 IQ." We trust these numbers because they indicate that someone has been out there with an officially sanctioned and standardized measuring stick. Numbers are "objective" and the gatekeepers have defined this as "good." Qualitative outcomes, on the other hand, are "subjective" and not to be trusted.
I'm sure we all agree that neither are to be trusted. Just because a quantitative outcome has internal validity, does this mean it can be generalized to the population? In the same way, if a qualitative outcome has external validity can it be held to be representative? We see these uses and abuses of quant/qual methodology every day in newspapers.
Much of the weaknesses in educational technology research can be traced to a blinkered reliance on one or the other approach--until recently the dominant paradigm has been exclusively quantitative. Lloyd Rieber pointed out that there has been a tendency to decide on the methodology before formulating the question, or use the methodology to formulate the question. If you are a "quantitative" sort of person you will normally think of research in terms of laboratory-style experiments in which the ecology is reduced to as few as possible independent variables. If you are of "qualitative" disposition you will argue that there is no such thing as an independent variable and it is counterproductive to attempt to untangle the web of interlinked relationships--it is enough to describe it. Both Tom Reeves and Walter Wager have given us some thought-provoking opinions on these abuses.
It may seem that I show a lack of balance in my criticisms of the quantitative approach, but if this is so, it is only because positivist thinking dominates the IT field. I also belong to another discussion group devoted to qualitative research and I can assure you that Tom's sins of pseudoscience pale before the qualitative shibboleths of ethnography and postmodern cultural studies.
In order to be able to say anything useful about learning and technology we need a full toolbox of research methods. In my paper I described qualitative methodologies as a "monkey wrench" (aka adjustable spanner). I don't want to take this metaphor too far, but if you've ever tried to use this indispensable tool you will be aware that (a) a good quality monkey wrench works better than a cheap one, (b) there's an art to using it properly, and (c) metric spanners (let's call them quantitative tools) may do parts of the job a great deal more effectively and with less effort.
We need both and I would commend a re-reading of Lloyd Rieber's [24 Mar 97] example to see how this might be done. Lloyd describes an essentially qualitative approach to the design of a research project, in which there is no formal hypothesis, but what seems to me to be a quest to define or refine a question. A variety of research methods are applied, each of which contributes to the formulation of the idea. The eventual outcome is unlikely to be a number accompanied by a measure of significance, but rather a set of insights which can be generalized to the development of gaming and simulation software. (If this is a distortion, Lloyd, I apologize... but it's too bad because I get the last word!)
The difference between valuable and questionable research, no matter what the methodology, is the rigor applied to the analysis. Ethnographic research methods have suffered by comparison in the past, because of their reliance on huge and unmanageable quantities of data (not to mention miles of adhesive tape and strikingly original filing systems), with the result that qualitative researchers have tended to stress insight (even instinct) over scientific rigor.
The computer has come to the aid of ethnographic researchers in recent times, providing them/us with precision tools to classify and analyze data, to formulate theories and to test them recursively against the data. In my paper I described an Australian software called NUD*IST (Non-numeric Unstructured Data--Indexing Searching & Theorizing) which I have been using for some time now. One of the advantages of NUD*IST is that it makes it possible to open my analysis to outside scrutiny--to be able to describe and demonstrate just how I arrived at my conclusions. (Professor Lyn Richards, one of the developers of NUD*IST, is currently on a lecture/workshop tour of North American universities--catch her if you can, she's a most persuasive and knowledgeable advocate of this methodology).
Another unfair advantage which quantitative methods have always enjoyed is graphs. Only accountants can make sense of columns of figures, but we can all understand a good pie chart. Qualitative research has traditionally required narrative--sometimes at book length--to make sense of the data. Links between NUD*IST and concept-mapping software such as Inspiration now make it possible to provide graphic representations as persuasive as a histogram.
The final issue, raised by several corespondents, is time. In the restricted time available to our graduate students it appears easier to undertake a controlled, quantitative study based on a clear hypothesis than it is to follow the meandering, branching and possibly endless paths of qualitative research. This is a pity, because qualitative research is both dangerous and fun. It forces you to come down into the swamp where life is not simple, it makes you question the assumptions you've learned from text books, and it makes you confront your subjects as human beings.
I'll be happy to continue this discussion at EdMedia 97--I'll be in the Horseshoe bar behind the rodeo with my friend Professor J. Alfred Prufrock.
In the room the women come and go
Talking of Michaelangelo