[quoting Quinn (quoting Hart's paper), 17 Mar 97] According to Richards (1992), the test of the ultimate conclusion is to see how elegantly and methodically the evidence was shaped into the conclusion, how the conclusion was coaxed (never forced) to "emerge" from the data, "how evidence and grand account form a well-connected, seamless web of belief that illuminates and enriches our perceptions and understanding of phenomena we see every day. To be credible, the report must show these processes in action, and demonstrate how the conclusions were reached."I realize it's couched in careful terms, but it still feels like massaging the data rather than reading from it. I do recognize times when you do more exploratory work and do not have a particular hypothesis you are trying to support, but I also believe you can use qualitative research to support more explicit hypotheses. Then you use converging evidence from several forms of data. You do make an argument, but it's not coaxing an argument from the data, but making a compelling argument about the data. Perhaps I'm niggling, but I would worry that the above might undermine an otherwise compelling discussion.
Clark's problem is the same that all positivists will have with the qualitative approach--the belief that there is an absolute truth out there somewhere and it is our job to discover it. Therefore unstructured data can be used to "support ... explicit hypotheses." In my view (and Lyn Richards' too I'd guess) hypothesis-testing is a misuse of unstructured data.
Hypothesis-testing is the fatal flaw in much of the research that is described (by its authors) as "qualitative" (and I hope Tom Reeves might weigh in on this because it touches on the pseudoscience question). A researcher sets up a hypothesis, e.g., that children learn better from the Internet than from teachers, interviews them and their teachers a few times over the course of the study, then selects quotations which support his thesis. Or worse, does a word count of supportive versus unsupportive statements and presents them as pseudo-statistics.
The validity of qualitative studies relies on the methodology used to treat the data. Therefore the methodology must be explicitly described in the report of the study. As Lyn says: "To be credible, the report must show these processes in action, and demonstrate how the conclusions were reached."
To qualify it must be disciplined and it must be data-driven. And there are some good models to follow: e.g., Grounded Theory has gained a great deal of popularity in the health sciences, Miles and Hubermann list a dozen more [references in Hart's paper].
Thanks Clark, you are now indexed in the skeptic box (just joking!)