We seem to have some of the terminology problems in communication (due to perhaps different backgrounds) that Rob mentions as justification for full, detailed documentation during an IMM project. Note that I parenthetically defined iterative design as successive approximation. I meant this to apply to both design and development (or production), yet Rob clearly distinguishes between the two phases. I don't think the differences were clear to either of us, nor perhaps the forum at large--hence this long post.
Rob, please don't interpret ANY of my following comments as critical. They are NOT. You have done a remarkable job of addressing many of the critical issues of IMM design and development. (I look forward to your clearing the legal hurdles so that your book can be published. There's an empty spot in my bookcase waiting for it.) To the other less-experienced IMM designers, I also recommend that you carefully heed Rob's advice. It is obviously based upon lots of experience (i.e., lessons learned), and can help you avoid many of the worst pitfalls of trying to do this difficult task for a living (or even an enjoyable hobby).
I am merely trying to "push the envelope" in this professional forum and gather others' inputs or insight. I sincerely believe, given the state of today's design and development tools (4th generation tools, data-driven designs, etc.), that we are poised to make giant strides in our ability to develop very effective IMM very cost-effectively. I think that we have the tools today to iteratively develop IMM also, in addition to iteratively design it. But, how do we do that and predict budgets and schedules?
[quoting Phillips, 4 Oct 96] The particular incremental prototyping model that we use for IMM development starts with a definition of the problem, coupled to a feasibility study (discussed earlier), and then production proceeds through a cycle consisting of Design, Develop, Evaluate, until the project is finished and Implemented.
Experience has shown that this incremental prototyping model is appropriate for the production of quality educational interactive multimedia. Under this model all aspects of the project should be formatively evaluated and revised until the project team is satisfied with their effectiveness. The word "quality" here may differentiate between the commercial and academic developers (in a nice way).
But, is the client satisfied with the quality? The student? I admit, Rob, I'm somewhat confused here. It sounds like you iteratively prototype (Design, Develop Evaluate), then "lock in" the design prior to production. So do we, in our traditional linear or waterfall method, though I failed to describe that we typically present the prototype to the client about three times before we finish it, and "lock in" the design. Is this what you meant? I assume so, and the remainder of my comments come from that perspective. If I'm wrong again, I apologize in advance.
Our projects cover such a wide range of fields, and we feel that the range of topics dictates a different user interface for each topic. You can't shoehorn drug dosage calculations and scientific inquiry skills into a standard template. At least we don't. We try to find the most appropriate user interface for the topic at hand, given our particular subjective feelings about educational theory. The commercial folks probably don't have this luxury.
We also produce a different interface, "look and feel" for each client, but I think that you may be undervaluing templates, if they are designed and built correctly, and the textual content. While I was with Allen Communication (previous job) we produced a series of 21 different courses, averaging over 10 hours each (estimated 240 hours of total contact time) for the Air Force using a total of only 24 screen templates (23 pre-defined, and one free-form). I'll admit it was for the same client (one look and feel) but for about five different end-user groups as I recall. Many of those 23 pre-defined templates dealt with multiple choice questions with four distracters, five distracters, etc. and variations of those same multiple choice distracters but with a picture (it was interactive videodisc) or graphic supporting the text. I came to the conclusion that most of the variations in screen design needed were made to allow the student an opportunity to interact--not to present the content. This, combined with readings such as like M. David Merrill and his ID2 theory which at the time had defined some 17 or 19 (I don't honestly recall) different instructional transactions. This train of thought, combined with the intense levels of effort required to thoroughly document IMM design, led to my design of the first prototype of Designer's Edge--a series of tested interaction templates that could be easily customized to change the "look and feel" simply by substituting backgrounds, graphics or buttons--then to "feed" these templates with a database of content that also contains instructions to the "code" on which template to use next and which content to present. (This is truly not a "plug" for ANY Allen Communication product--we have IconAuthor, Authorware, Director, and Toolbook all at my current job and use all of them selectively, but the team I've assembled uses Quest daily since the least experienced of us has over five years experience with it.) Though the current functionality of Designer's Edge bears only limited resemblance to that first crude Visual Basic prototype, I honestly think these types of tools are where the future of IMM design and development lie. Also, I'm very encouraged by the recent announcement of a version of Designer's Edge supporting Toolbook, but that's a different issue. Let's leave tools as an aside and continue on with the process and effectiveness discussion.
However, a key point of our development methodology is to separate the Design Process from the Production Process. Both of these processes follow the design, develop, evaluate model described above, but the design process has a higher weighting of the design component, while the production process is mainly concerned with development. Evaluation is important in both cycles. If we get it right the production process is a spiral with very few turns.
It seems in Rob's discussion that he also is just as concerned with "locking in" an approved design (prototype plus specifications) and content (storyboards).
That's why we design the project as comprehensively as possible on paper, in order to produce a quality project within budget. After a comprehensive requirements specification of the overall structure of the program has been produced, we create a complete storyboard of all of the content of the program. Production should only commence after the storyboard has been agreed on by the development team. This milestone in the process is formalized by the signing of a contractual agreement that this stage has been reached. The signing off procedure is an important incentive to ensure that the design is complete, and that members of the team will not attempt to change some components of the design at a later stage.
I think that your design/development teams and the client are also likely left with a long list of enhancements that they would like to also implement, if only there was time or money left on the project. I always did.
Evaluation of the design usually consists of constructive criticism and reflection within the development team, and with other experts and end-users.
I've also found that the "best laid plans of mice and men" and the best thought out designs still must be modified once the final products are being formatively evaluated. Or, in military terms "no battle plan survives contact with the enemy"--not that learners are our enemies--they are our true customers!
The use of a wordprocessor simplifies revision of the storyboard, and minimizes spelling errors. Does anyone know a programmer who can spell?
Instead of using a word processor, we're using an ODBC database (created in Access) with a Quest compiled executable that reads that same database and presents it to the learner. I'll admit that we've not spent the time to try mapping the entire Designer's Edge database to use it as a storyboarding tool, though we've used it to prepare the first drafts of Design Documents (i.e., specifications in Rob's terms) and have found the large library of Quest templates valuable. Supposedly, Allen will be "opening up" the database specification for Designer's Edge in the next release, so that the content would "automatically" flow from the storyboards into the Quest lessons, with the templates in the library (.QLB file) serving as the "pipe" between the two. We're doing something very similar, using Access and ODBC, with a much "cruder" Access interface to those storyboards (though it does support the all-important spell-checking!). Some of our smaller IMM courses contain only six or seven different interaction templates.
However, I would get these people involved at the prototype stage, rather than during the final debugging.
In an interative development (as opposed to design) approach, this isn't final debugging, merely refinement of what was planned. Using our database tools, we can present the content (at its current state of development) in the actual interface of the final course at any time. We can perform what the top-down structure programming approach calls "structured walk-throughs" at any time. We can present the storyboards on-screen, including a textual definition of the graphics or media to be produced. The text content is as it will appear in the final product unless someone's desire to change it. If so, they edit the database and "Voila," the new text appears (both on screen and on any printed storyboard from the database). We can do this incrementally, for one lesson, one module or as much as we have defined at that point. All the interactions are already there, with the exception of "hot spot" graphics, there the user interacts with the graphic itself. (We try to limit this, if possible, to only simulation strategies.) The more complex media elements (especially animations and hot-spot graphics) may need to be "manually storyboarded" by sketching them on paper, and getting client review and signoff of these sketches prior to production.
Once the content has been thoroughly reviewed as story-boards, we go off and produce the media elements and incorporate them. Then we have the client review the course again. Now all the PLANNED content is included and is reviewed. If changes are needed (and they always are), we can make them on the spot as long as the graphics or audio elements do not need to be changed. We can change any text almost immediately. We can fine-tune the interactions by modifying the Quest templates (examples: Adding a "time-out" to every quiz question, adding "a second try" with a "try again" feedback message before we disclose the correct answer, etc.). This typically requires editing no more than two or three screen templates in Quest, yet the results are evident throughout the entire course. We often have learners in during these "Alpha" tests, and significantly refine the course (minor design "tweaks" and sometimes major content changes) based upon their input. This can happen even though the same students reviewed and approved the "on-screen storyboards" earlier. This is likely due to at least two reasons. Until the entire course is developed, you simply cannot get a good sense of continuity, depth of content coverage, the level of prompting/feedback needed to make the human/computer-based tutor dialogue more realistic, pacing, etc. There's also the issue that some people cannot "visualize" well, i.e., they cannot "picture in their minds" what the final product will look like. They will withhold comment until they really see it--then, on to Beta test (i.e., small-group tryouts) for even more refinement.
ASIDE #1: We've found that most "hot spot" graphics can be incorporated in a database simply by defining the XY coordinates of each of the hotspots in the database and otherwise treating them like a multiple-choice question. Only irregularly-shaped hotspots pose a problem. When we encounter them, we try for "maximum re-use" of that graphic, since it must be defined as a unique Quest template, though the content in the database can allow multiple uses of it for exploratory learning, quizzing, branching like a menu, etc.
The key points made in this post to efficiently produce educationally effective IMM materials are shown below:
- Use an incremental prototyping model, but
- Separate design from production
- Design the project as comprehensively as possible on paper through:
- a thorough Requirements Specification
- a comprehensive storyboard
- Use prototypes where appropriate
- Document all aspects of the development
However, Kent, I am not sure that any of this will help resolve your problem.
Though there were a few valuable insights, Rob, especially regarding teams and relationships, the process discussion didn't help much. Been there, done that, got that T-shirt! I'm trying to go further. Any further comments? Anyone else? Who else is trying to use databases directly? IconAuthor has supported them for quite a while. Toolbook support came somewhat later, and Authorware does in it's last release.
ASIDE #2 The linking of Quest and Designer's Edge is where my original prototype and the current, commercial product differ most. The original prototype contained this automated link, the current version does not. The original prototype was designed from the "authoring/production" perspective, designing instructional interactions first, and then working backwards into the design steps of storyboarding and flowcharting, with linking only back to the point of instructional objectives, where direct correlations broke down. The current version begins at step one of the design process (defining the mission or goals), proceeds to audience analysis, then objectives, etc., with loose correlations throughout, yet lacks the automated link of storyboards to the final, authored lesson. Why did I work backwards? In my experience, storyboarding was taking 25-30% of the total effort, with authoring another 25% or so, and debugging another 10-15%. Linking the storyboard automatically to automatically become the authored product theoretically yields at least a 25% time savings, with significantly decreased bugs to be found and corrected in the test/debug steps. It also allowed the content expert to do much of the final development, just as Rob uses them to produce storyboards. Programmers were needed only to develop the databases, interaction templates, and the links between the two.
ASIDE #3 We're now developing a Visual Basic Òquiz engineÓ that's entirely database driven. Why VB? It responds to the user quickly, is more Òdatabase friendly,Ó and has great add-in reporting capabilities, such as Crystal Reports. There's no reason why Borland's Delphi (for those who know Pascal) or the IBM Visual Age family of systems couldn't be used similarly (choice of Smalltalk, C, or Basic languages). PowerSoft's PowerBuilder also has the same potential, perhaps with less robust built-in scripting (I'm not sure) and perhaps several more tools that I'm not familiar with. Hints: (1) Focus on designing typical instructional interactions -- don't focus on content first. (2) Treat graphics as either interface elements or just another way of representing content. (3) If it's content, define it or a cross-reference to the final picture, for example, in a database. (4) Define the instructional sequencing also in the database.