[quoting Norman/Spohrer paper] In analyzing the papers of this special issue, we find it useful to evaluate them along three dimensions: Engagement, effectiveness, and viability.
I'm not sure I consider these dimensions. Sure, you can have greater or lesser degrees of each, and the more the merrier. But I'm not sure they're at the same level. For instance, isn't engagement a component of effectiveness? I would put effectiveness and viability at a top level. To evaluate viability, you need to figure out what are the working conditions under which these tools can be employed, and under which they cannot. Effectiveness can be evaluated in an artificial setting, and it might be effective in one and not in another.
I'd also quibble on the descriptions: The issue of toy problems scaling is the need for transfer, which Don & Jim have lumped under viability, but I would argue is part of effectiveness. Our measures of effectiveness should capture this. Viability captures something else, about whether the mechanism engineering the transfer can be accomplished on real-world budgets, in hide-bound departments, with real kids.
But these are less interesting issues, really, at least to me. Important, but there's more here than is discussed. Specifically, the dimensions Don & Jim specify aren't the ones that they use to categorize the papers they're discussing! Subsequently they admit that all the papers rate high on engagement and don't even discuss effectiveness and viability. So what are the dimensions that distinguish these efforts? I think there are some interesting dimensions here.
One of the obvious categories is those grouped together as examples of collaborative projects. So one dimension is whether the tool/environment supports or requires collaboration, or is designed for more individual exploration. Increasingly we have the capability to support group work. There are all the standard CSCW issues, in resolution of competing revisions, in formats, in the cognitive research on what representations are useful and how to support them. And, of course, learners may have different cognitive capabilities than adults. However, I think that students can work together even on non-collaborative environments, so it's rather whether the collaboration is synchronous, or asynchronous in time or space.
Another important dimension, and one I'm still coming to grips with, is whether a system is exploratory or constructive. What I'm trying to say here (as both are constructive in one sense) is whether the system has the learner explore an environment and internalize relations, such as in a simulation, or whether the learner is given tools to build a knowledge base or a system, formalizing (and testing) the relations they believe exist. In the exploratory environment you create hypotheses and then act in the world to see if your predictions hold; there is an external model and the student is building an internal one. In the constructive (and perhaps I should choose a less "loaded" phrase :-) environment, the student has an internal model, and constructs an externalization. Cognitively, I believe I can justify both. The tension is between my interest in building exploratory environments (read: games) and Lloyd's in empowering the learner by supporting their building of games.
A third dimension that I've seen between previous construction kits is whether the construction is of static knowledge or a dynamic system. However, this seems orthogonal to the last two groups of papers, which seem to be both about building dynamic systems (games, microworlds, models of mathematical or scientific phenomena). My take on the distinction between them is the former is about building interactive applications while the second is about building models, but it may be that the latter has model-building tools built into an exploratory environment. I guess I'll have to wait till my copy of this issue of Communications of the ACM arrives (sorry for those of you that aren't in a Computer Science department, maybe you can annoy some colleagues).
So I'll propose different dimensions than Don and Jim raised:
at the top level...
Effectiveness and Viability
At a second level, within effectiveness there's...
Constructive or exploratory
synchronous or asynchronous
Within constructive there's...
static or dynamic (not relevant, necessarily, for the papers listed)