I find the emerging thread of iterative design (successive approximation) an intriguing subject for two distinctly different reasons.
[quoting Quinn, 3 Oct 96] I'll just note that in HCI design, there is agreement about iterative design (the cycles) and uncertainty how to decide when to stop. I recognize the pragmatics of the indicated approach, but there is another proposal on a more theoretical basis. The proposed approach is to decide as part of the original specification just what are acceptable metrics that would indicate achieving usability goals. These metrics are then used to indicate that the cycle can stop. For instance, you might specify things like: eight out of ten transactions processed error free, a transaction takes no more than 30 seconds, etc.
Translating these into educational products might mean adding some educational goals: learners will successfully complete in 15 minutes, 95% of learners will successfully be able to identify causal factors in the system after completion, etc. (OK, these are off the top of my head, but I hope you get the idea.)
Of course, pragmatics may make this untenable, but it's a nice ideal!
Iterative design may propose significant process changes in the both the "mechanics" of the design/development process and in establishing and maintaining the client/provider relationship, regardless of whether this is a commercial, paying vendor-client relationship. I work for custom software company, and many of our IMM products are to teach users how to use the software. Though these are for commercial, paying customers, these insights may be of value in any client/provider relationship.
As I alluded to in my previous posting, our SALES process institutes a traditional linear (i.e., waterfall) process of a design phase at a fixed cost which produces a prototype and a set of functional requirements, client review/signoff, then a firm fixed-price bid for developing and implementing the custom software package defined by the prototype and functional requirements. This linear approach automatically makes any change to the "approved design" at any later point (including as a result of Alpha, Beta or Pilot tests) to be "out of the original scope" and subject to negotiation for additional cost. Most of the CBT/IMM I've designed and developed has also followed this same approach. It makes it much "safer" for those designing and developing the solution, by defining the scope of the project in terms of functionality and the look-and-feel. However, it clearly doesn't produce the most effective solution, in my experience, because the product almost always ships with a list of "deferred enhancements" that were defined as a result of user test and acceptance process. Typically, only the critical "enhancements," or the ones that can be implemented within the original estimated time/budget, will be incorporated. This negotiation process with the client at the critical, late stages in the process is often quite "stormy" at times, as we must reach agreement on the differences between bugs, design errors (of omission or commission), or true enhancements. Unless it is a bug, it isn't free. Even design errors are "joint responsibility" since the client approved the design. Bottom line is that it can "test" the provider/client relationship, and we never get to implement all the "lessons learned" on the project at hand.
Conversely, we often receive Request for Proposals (RFPs) or Request for Quotes (RFQs) with the client requesting a firm, fixed price (commitment to staffing, resources, and schedule) for both design and development. In fact, most of our clients are insisting upon this for CBT/IMM, and it is a real educational and sales challenge to convince them that a two-step, separately priced process is advantageous to everyone. Often our clients think that our design phase is merely an attempt to get them to pay to respond with a detailed proposal. They simply do not understand that until WE UNDERSTAND the content, the audience, and the client's expectations (often illustrated via a prototype), then we cannot accurately predict either cost or schedule. Our only alternatives are to set limits or "caps" on the number of hours to be expended, and price the "overruns" on an hourly basis, or to accept the risk and attempt to "design to scope." Frequently we will accept the risk on a 3-6 month $100,000 CBT/IMM project, but seldom if ever on a 12-18 month $1M-plus custom software project. The company's experience when doing this on software packages has almost invariably resulted in late delivery and over budget. The company will still strategically do so, if the client insists, and the company wants the client enough, essentially "buying the business."
However, we in the entire production side (both software and CBT/IMM) want to move to an iterative design and development process, based upon our limited experience with it (and the power of the new 4th generation software development environments and CBT/IMM authoring systems), but haven't quite settled on a way to "sell it" either to our own sales force or our client base. Clients are reluctant to enter a "trust me, you'll like what you get" relationship, without firm definition of what they'll get for their money. Our sales force is reluctant to enter client relationships that don't have acceptable boundaries on risk or exposure. Maybe we'll need to estimate and "cap" each successive iteration. So much for the changing client/provider relationship issue.
The other issue is effectiveness of the design, and the quality of the end products delivered by the iterative cycle. We've now developed a few CBT/IMM courses that are entirely data-driven, with all the content and selective logic being stored in a database. This allows several very powerful changes to the traditional, linear design and development approach. We can print a complete set of current, correct storyboards at any point and obtain client signoff and approval. We can rapidly implement significant changes to the course very, very quickly by updating only the database. Rather than reviewing scripts or storyboards, we have the client review the actual courses in development, and provide printed output only to document results. Typically the client will review each unit or module at least three times: with all text and no multimedia (only definitions of what is to be produced), again after the media elements are produced, and a final time for acceptance. As long as we do not need to produce additional graphics or sound files, we are essentially limited in making revisions only by our typing speed. As an example, we can make significant changes during individual tryouts overnight, and have them available for further testing the next day. However, we can get them even more involved in the design process.
We've successfully used the technique of one student reviewing the course on one machine and the ID making the desired changes on an adjacent machine in "real-time" and presenting them back to the reviewer to critique. We try to do this with at least three "novices" who have never been exposed to the content and three "subject matter experts." The content, especially for learner prompts and feedback, then has been re-designed and refined "on-the-fly" based upon live input from the end-users. (Note that I said the ID, not a programmer!) This has dramatically improved the "richness" of the dialogue between the learner and the computer-based learning experience. We now can capture and address subtle distinctions never before possible, add elaborations where proven needed, etc. We can correct the causes of misunderstandings with the feedback to user mistakes, distinguish and award "partial credit" based upon typical users' explanations for their actions, etc.
The outcome is significantly improved courses and intense "buy-in and ownership" from our clients. On our most recent project, at the end of small group tryouts the only "bug" reported was one typo. Replication masters were provided the next day and final client acceptance was immediate. Now if we could only figure out how to describe and "sell" this relationship to potential clients up front.