Some anecdotal comments and opinions (perhaps not so humble) regarding my own search for understanding (i.e., research) that supports Ian's points:
I was pursuing my master's in IT from Utah State as a non-traditional student (3/4 of my course work was done by distance learning over their COM-NET system) and did not have the availability (demanding job, plus 75 mile commute) to do on-campus work. I was very interested in developing a "style guide" that could be provided to IMM/CBT development teams to increase the overall quality and effectiveness of the materials that they developed. (I was supervising six teams in my Air Force job, plus "buying" over $12M in custom development from outside vendors.)
I proposed creation of this "style guide" as my creative project (as a non-traditional student, I could not complete the prerequisites for the thesis track). As a creative project, I also could use less stringent research methods for my "meta-analysis" and the documentation of the results (I was not required to publish in formal APA standards).
[quoting Hart, 23 Mar 97] What's going on here? Who is allowing this poor research to flourish? Who is publishing it? Who is supervising it? Who is laying down the requirement that research is the only way to earn a university degree?
Having completed my prerequisite research methods and stat classes, my chairman, Dr. Alan Hofmeister, REQUIRED that I do two things--regardless of what direction of readings or review that I took from there: (1) begin with the infamous Clark papers and (2) broaden the base of my readings to include "effective teaching" regardless of the medium used. This proved incredibly valuable as I proceeded to review well over 1,000 sources (probably closer to 2,000) and selected close to 500 for inclusion in my final paper. The "devil's advocate" perspective provided by Clark immediately required me to discard much of the meta-analysis previously performed (Kulik et al., etc.), critique much of the "isolated research," and draw my own conclusions (i.e., I had to "dig deep" instead of simply "mouthing" the words of others--both into the instructional theories underlying the research and the results thereof).
If you are 20 years old the world is full of simple, black and white issues. Maybe a 20 year old can be excused for thinking that it is acceptable to poll a first year Psych class, test one or two variables, calculate a few standard deviations and earn a Ph.D. But someone must be giving out these Ph.D.s. So maybe universities are under pressure to turn out increasing numbers of graduates (and therefore do more and more research) in order to justify themselves? Are we pursuing a "research imperative" for its own sake?
... (b) is the methodology you plan to use likely to provide a level of insight and validity which will make your conclusions worth reading?
When I began the "literature search" I naively thought that I could end up with "rules" that would define qualitative differences in CBT/IMM design. By the time I ended, I realized that not only could I not (the research base was not either definitive enough nor the theories prescriptive enough), but that I had been incredibly naive to think so. It was both humbling and frustrating to go through this assessment of the "research base" for ID as it relates to IMM/CBT. My final paper included the Wager & Gagne quote that I used as my "signature tag line" when I first joined this forum: "To imagine that the design process can be reduced to a set of cookbook exercises is wishful thinking."
Yet, when it was time to present the final project and review those presented by my peers for either the thesis or non-thesis tracks, I was incredibly grateful to Al, as my chair, and for the entire experience. Rather than one more "isolated paper" that researched the effects of an "isolated design or presentation variable," I had delved deep into the underlying theories and research base of the field in general. To sum in up, the papers presented by all my peers appeared naive. Either they were on another isolated variable, used questionable quantitative/qualitative methods, or they were documenting yet another "isolated design or development project (non-thesis students)." I could have readily "questioned or invalidated" virtually any one of them by pointing out contradictory studies or methodological weaknesses. I learned MORE of substance/value from my own independent research than the rest of the entire, structured program of course work.
Also, several of my peers/friends whom I asked to review my drafts (during development), suggested that I was performing "doctoral work" not "master's work." I think not. I would challenge this group, since many of you "control" the hoops that graduate students are asked to jump through, with the following questions:
1. Do you need to "re-think" the requirements for master's versus doctoral work? I think much of the work being done at the master's level is NOT appropriate--but I disagree with my friend/peers perspective above.
2. What is MOST appropriate for master's students, who in most cases are intending to be practitioners (not researchers)? Wouldn't more meta-analysis or literature reviews/critiques to understand both the incomplete theoretical base and the generally weak research base of our field be much more valuable than one more isolated, flawed independent research project? Wouldn't they be better "grounded, in general," to practice within the field and to "independently evaluate and apply" the research results that are being published as they continue their practice?
3. If this perspective of "literature review or meta-analysis" were REQUIRED at the master's level, wouldn't those who went on to perform doctoral work be much better prepared to perform research that would truly contribute to our research base--since they would be familiar with how "weak and flawed" it truly is? Perhaps then, the doctoral students could make some significant contributions to it, rather than continuing to propose, execute, and document even more "weak research" that does not contribute significantly to our field. The continued "gap analysis" alone, would be very beneficial to strengthen our overall research base.
4. Wouldn't this broader-base understanding for the master's students be much more valuable, long-term, than a single research or developmental project (regardless of how successful it may have been)? If nothing else, they would have "learned how to learn." On a side note, I'm so tired of seeing the "results of my first project" being presented at conferences, that I've intentionally taken the tactic of staying home and using my hard-earned money to subscribe to as many sources of potential insight (including such things as HCI, human factors, etc.) instead of on conference fees.
In summary, I would value ANY research results, qualitative or quantitative, that would more predictably ensure the effectiveness of the designs that I execute. Sadly, I don't see very much of substance coming out of the recent research. Similarly, I don't see much extension of the theoretical base (with the possible exception of Dave Merrill's continued work). I think the ongoing debates over constructivism or media (ala Clark) only serve to validate this opinion. If there is significant substance being published, I'm not aware of it. If so, where do I, as a conscientious practitioner find it?