Since other "CBTer's" in industry have responded, I assume the invitation to comment is not limited to IMM in academia. I think many of the lessons learned may still be relevant.
By the way, does the "I" in IMM stand for "interactive" or "instructional?" Acronyms abound!
[quoting Phillips' paper] In framing your responses, give a brief overview of the project so that readers get a context in which to read your analysis. What is its place in the curriculum? What are the average hours of use by students?
Beginning in 1986, numerous (about 100, but I honestly don't recall) interactive videodisc courses (and later just CBT, once SVGA and CD-ROM became widely available) designed for highly technical training. Target audience for most of the courses was journeyman aircraft maintenance personnel in the U.S. Air Force. Content was advanced systems theory and operation with the end goal of improving maintenance troubleshooting processes. Beginning in 1991(having left the Air Force), the target audience was railroad employees, high-tech factory workers, oil pipeline pump-station operators, firefighters, etc., and the content was typically either the use of new technology, the performance of complex tasks in emergency conditions, or once-again troubleshooting. (Note that troubleshooting is most frequently not just the performance of complex procedures, but also problem-solving at very demanding levels. Recent emphasis has been teaching salespeople (several industries) either software operation or the product knowledge necessary to facilitate sales. The "normal" course was five to ten hours of average student contact (i.e., interaction) time, though a few were 20 to 40 and a few were three hours or less. All have been designed to introduce new material, tools, or concepts to experienced people.
Were your plans too ambitious? Initially, projects exceeded the anticipated schedule by 25% to 50%. After a half dozen projects or so, this margin of error was reduced to the 5% to 15% range. Experience teaches you where the highest risks are. I've successfully run a large commercial development group, bidding a 25% profit margin and living with 15% to 20% actual profit margins. Some you lose money on, some you do very well.
Did you bite off too much to chew? Only initially. Another technique learned quickly by experience (if you're accountable for time and budget) is "designing or producing to scope." There are numerous "tradeoffs" that can be made in both design and production that do not significantly reduce the instructional effectiveness, yet allow you to "cut corners" if necessary. The client typically will not know what's possible, so they do not know about "what could be." Other skills learned the hard way are "controlling expectations" of the client or the scope of the project through reviews, approvals, etc. The bottom line is that you can allow the client to define content, but you can't allow them to define the design, or they'll drive the project over schedule or budget almost every time.
If you did it again, what would you change? Perform much more rigorous reviews of early projects, using both peer reviews and structured walk-through methods. Finding weaknesses or flaws during the design phase is much more efficient than later.
Would you ever do it again? Every day!
Did you have enough funding? Never enough to do what I'd really like. Usually enough to produce adequate results....
Did it take longer than you thought? See above.
Did the project finish? Every one.
Is it in use? Virtually all of them. Some are now almost ten years in the field and still used daily. Those not in use are due to content changes rendering them "obsolete" and the ongoing need was judged insufficient to justify the expense of major updates or enhancements.
Are you satisfied with the result overall? Yes, overall. Some were national award winners, all "got the job done," and only one was embarrassing at the time.
Are you satisfied with the instructional design? Overall, yes. But there's things to improve too numerous to mention. A few trends stand out, as follows:
(1) Improve the overall quality of the interactions by requiring a "deeper level of mental processing" to respond correctly. (An interesting exercise for new designers is to require them to design a unit or module without using a "NEXT" or "CONTINUE" type of interaction.) Require the student to interact with the content, not the machine. Page-turners are far to easy to design and far too common. They give all CBT/MM a bad name.
(2) Improve the answer analysis, trying to differentiate between "Correct, Close, Not Very Close, and Clearly Incorrect" responses, and provide more meaningful feedback. Effective feedback to the students' mistakes is probably the most overlooked design element in CBT. Much CBT created by newcomers (or experienced people under severe time or budget constraints) has very shallow feedback, usually just addressing correct and incorrect answers with no further differentiation. If you want to perform a quick feedback...
(3) Make sure the content adequately addresses not only "What?" and "How?" but also "When and Why?" with a lot more emphasis on the latter. (Note that virtually ALL these courses implement a "situated learning" or "functional context training" approach, containing case studies or "word problems.")
Was there sufficient student interaction with the software? Yes, if sufficient means the number or frequency of interaction. Quantity wasn't typically an issue, only quality as described above. Much of this is attributable, I think, to the situated learning approach and "real-world" examples, not just automating a book. Quality of the interactions (as described above) improved dramatically over the years of experience, but early efforts sure could use some improvement as I look back.
Are you happy with the way it looks? Overall, yes, given the tools, time, and limitations. I started with 40-character CGA, remember.
Did parts of it turn out differently than you envisaged? Frequently. Media elements and interactions are never quite what you imagine when you read the scripts. Interactions are often easy to refine--media is always expensive to "re-do."
Were there arguments about who said what, when? Yes. Documentation is critical, both of the design and project management decisions.
Did design flaws surface late in the project? Almost always, but usually minor. Serious flaws decrease based upon the designer's experience. Yet, there's no substitute for reviews and intensive tryouts.
Are there bugs in the program? How many? How much effort did it take to get rid of them? Almost always, but again, usually minor. The authoring tools get better all the time. Ten years ago, the "logic or code" had to be embedded in each frame (page, display, card, or whatever metaphor the tools used). Today, more of the code can be isolated and called when needed, with some level of event-driven or object-oriented capability. This allows fewer opportunities for bugs. The problem is that the bugs, though far fewer in number, tend to be harder to duplicate consistently, troubleshoot, correct, and finally test. Examples include problems with specific graphic card drivers, operating system add-ons such as device players, memory leaks, resources not being freed up, etc. Typically, about 10% of the time is now spent in test/debug cycles, versus 20% to 25% ten years ago. This is largely due to improved authoring tools.
(1) Even today's "high-end" authoring tools may unintentionally foster a "page turning" design, with weak feedback to user input. Look at how many use metaphors of flowcharts, books, etc. What would be the difference if they used a "Socratic dialogue metaphor" where "listening to the student" and evaluating their response was equally as important (AND SUPPORTED BY THE TOOL) as the presentation of content?
(2) The traditional, linear or waterfall design and development process has been our friend in controlling scope, once we've gained some experience. However, a rapid-prototyping or iterative process would likely provide much more effective designs, if we would conduct tryouts using iteratively improved draft materials.