[quoting Wager, 21 Mar 97.b] Now, if I am being too hard on what I see as an increasingly typical qualitative prospectus. I also want to come down hard on the quantitative types who want to teach some abstract topic in a single 50 minute period (in which the 200 general psych student volunteers have not an iota of interest in the topic), using hypertext as one treatment and linear text as another, crossed with field dependent and field independent measures to prove one group doesn't learn in one way and the other doesn't learn in the other. PLEEAASE. The only difference is that the quantitative researcher has wasted less time in coming up with useless conclusions.
[quoting Hart, 22 Mar 97] Walter, in particular, has put his finger on one of the great weaknesses of much research that passes as qualitative: poor design and lack of generalizability. I recall a paper from Tom Reeves a year or more ago in which he described some awful dissertation project to do with collecting conversations in coffee shops. I just hope that Walter is not including the Kearins and Lam case studies in my paper in his sweeping generalization as I consider them to be fine work with important and generalizable results.
This is a place where I have some disagreements.
Depending on the research question, I see no reason why we should categorically exclude research that takes place in coffee houses or in 50 minute blocks conducted in the "laboratory." Also, as I have stated in previous postings to ITForum, even if there are imperfections in the way a study was conducted, one must go beyond criticizing the short-comings of any one study and instead look to see how it contributes to the general literature (however humble that contribution might be). If the questions are good and situated within good theory, I think coffee house/lab work has meaning and value. Some of the best research I know (done by people like John Bransford, Richard Mayer, and Jean Lave) fall into these categories. Some research is best accomplished in the field (systemic) and other in the laboratory (analytic) for lengths of time that are consistent with the goals of the research.
[quoting Reeves, 21 Mar 97] I would very much like to see some examples of "good questions" from Lloyd and Walt, and the methodologies they would recommend to address them. ... I completely agree with Lloyd and Walt that the research question is what should drive the methodology, but I do not have the benefit of their perspectives on what types of questions or problems we should be addressing in this field. The tone of their responses suggests that they think some of us are asking "better" questions than others, and so I respectfully ask for direction.
Naturally, the questions/issues I find the most compelling are the ones I am studying at the moment (shouldn't that be true for all of us if we feel what we are doing is worthwhile?) I hesitate to talk about my work for fear that it will come across as self-promotion, but since you asked. (By the way, I would not want to characterize my research questions as "better" than someone else's since that implies some absolute standard against which all are judged. However, I think it's fair to suggest that we all have different interests and values, therefore I will certainly value some research directions more than others.)
I almost put together a list of all the issues/questions that I am currently working on, but figured that would put too many people to sleep. Given that I still need to finish preparing for the AERA conference (and I leave in less than two days) I thought it appropriate to share a study I'm co-presenting there with David Noah, a doctoral student here at UGA. The research deals with the role of gaming and visual metaphors on learning physical science principles in an open-ended simulation. Here are some of the specific issues we studied:
Several theoretical frameworks have guided the research, especially Malone's theory of intrinsic motivation, mental models (i.e., use of metaphors as conceptual models), and play theory.
The study has both quantitative and qualitative elements (though such mixtures are considered "wrong" by at least one of the profs who teach educational research here at UGA). We started with the quantitative phase and followed up with qualitative (also considered the "wrong" order by most research gurus). All the sessions were short, about an hour (which seems to violate all the tenets raised here). However, I consider the work and results to be important (though not perfect). Most important, and here's the main point, even if others aren't impressed by our methods or their scope, the research has made it possible for David and I to understand better the issues we are raising and will lead us (and hopefully others) to ask even better questions than we have asked so far. The methodology worked for us and led us to good things (and required no special funding). It also is leading us to conduct some field research with these materials in the next couple of months.
I do both field work and laboratory work and don't apologize for either. The postings I have read during this discussion so far seem to be advocating abandoning laboratory research outright (regardless of its quality). I think that would be a mistake. Am I the only one?
Here's one example of how our research approach has helped our understanding thus far. David and I are clearly proponents of gaming. However, our results clearly indicated a negative consequence of integrating game elements to simulations (even though participants really like the game, they got so wrapped up in it that they forgot they were supposed to try to learn something from it!) Therefore, we were forced to confront our own biases favoring game design and think more deeply about our views. We also needed both methodologies to come to our conclusions. If we had only used qualitative methods, I fear our biases would have gotten the better of us and we would have concluded that games are great. Fortunately, the quantitative results clearly showed that the game interfered with explicit learning--we could not "get around" that. Instead, we used the qualitative sessions to help us understand why this was so (although we had predicted, based on previous research, that the game had the potential for distraction).
A fear of mine, as I have stated in past discussions, is that many talented people in our field will read this stuff and decide that doing research is a fool's paradise. You only get criticized for not following the latest dogma. I also often wonder about all of the people whose dissertation research was the last study they ever conducted. We would benefit so much, in my opinion, if more of us took on the "identity" of writer and researcher in our professional lives, no matter our status or role. It seems it actually takes courage to be a researcher these days!
By the way, there are many areas in which I have interest but not enough understanding to be able to ask good questions. For example, I have become fascinated with the influence/role of aesthetic elements (mainly images and music) within multimedia. I am also interested in the act of creative design. However, in neither case do I understand these issues well enough to ask good questions (I hope that will change in the next couple of years).