[quoting Rieber, 21 Mar 97] At the risk of being tarred and feathered by the general research community, I feel the best outcome of good research is not the results based on a certain methodology, but a greater clarity and understanding of what the question ought to be and the generation of a few good other questions. In fact, I don't think we spend enough time worrying what it takes for someone to get to a point where s/he knows an area well enough to ask good questions.
I've tried to keep this note brief, so pardon my oversimplifications. Don't interpret my words as suggesting that methods and results don't matter and don't see them as an excuse to do sloppy research. All I want to suggest is that we shift the emphasis back to where it truly belongs--on asking good questions.
And I commented--I think he is right on the mark. I also said I was weary of case study dissertation research. What I didn't say is that I am an advocate of good research, no matter what the methodology.
However, I am very suspicious of research like this: A study of how students use resources in a collaborative learning environment on the web, or phrased as a question: "How do students use resources in a collaborative learning environment on the web?" Even with detailed (read tedious) description of the learning environment, the types of learning activities, and transcriptions of individual verbal interactions, my first question is: "What, as a professional, can I learn from your study that might have some utility for future design decisions, or theory building?" My second, and I believe most important, question: "Can I believe the result(s) (observations triangulated with data from at least three sources) you found are generalizable to any persons beside the (pick a "small/manageable" number) ones you chose to analyze because you knew someone in that school who could get you access to observe and interview those students for a grand total of four hours?"
Now, if I am being too hard on what I see as an increasingly typical qualitative prospectus. I also want to come down hard on the quantitative types who want to teach some abstract topic in a single 50 minute period (in which the 200 general psychology student volunteers have not an iota of interest in the topic), using hypertext as one treatment and linear text as another, crossed with field dependent and field independent measures to prove one group doesn't learn in one way and the other doesn't learn in the other. PLEEAASE. The only difference is that the quantitative researcher has wasted less time in coming up with useless conclusions.
I guess what I am trying to say is good questions are necessary but not sufficient conditions for good research. One has to look at external validity and reliability questions that go beyond statistical calculations--ecological validity if you will.