Some of you may notice a certain "cut-and-pasted"ness to this post. I think I'm getting worn out! This is some of the stuff that I might have posted originally, so thanks to T. Kent Thomas for raising the questions I wanted to answer!
[quoting Thomas, 3 Oct 96] I find the emerging thread of iterative design (successive approximation) an intriguing subject...
Iterative design may propose significant process changes in the both the "mechanics" of the design/development process and in establishing and maintaining the client/provider relationship, regardless of whether this is a commercial, paying vendor-client relationship.
I am not sure how well our model can be applied to the real world.
Perhaps it can, given your comments below....
As I alluded to in my previous posting, our SALES process institutes a traditional linear (i.e., waterfall) process of a design phase at a fixed cost which produces a prototype and a set of functional requirements, client review/signoff, then a firm fixed-price bid for developing and implementing the custom software package defined by the prototype and functional requirements.
The particular incremental prototyping model that we use for IMM development starts with a definition of the problem, coupled to a feasibility study (discussed earlier), and then production proceeds through a cycle consisting of Design, Develop, Evaluate, until the project is finished and Implemented.
Experience has shown that this incremental prototyping model is appropriate for the production of quality educational interactive multimedia. Under this model all aspects of the project should be formatively evaluated and revised until the project team is satisfied with their effectiveness. The word "quality" here may differentiate between the commercial and academic developers (in a nice way). Our projects cover such a wide range of fields, and we feel that the range of topics dictates a different user interface for each topic. You can't shoehorn drug dosage calculations and scientific inquiry skills into a standard template. At least we don't. We try to find the most appropriate user interface for the topic at hand, given our particular subjective feelings about educational theory. The commercial folks probably don't have this luxury.
However, a key point of our development methodology is to separate the Design Process from the Production Process. Both of these processes follow the design, develop, evaluate model described above, but the design process has a higher weighting of the design component, while the production process is mainly concerned with development. Evaluation is important in both cycles. If we get it right the production process is a spiral with very few turns.
It makes it much "safer" for those designing and developing the solution, by defining the scope of the project in terms of functionality and the look-and-feel. However, it clearly doesn't produce the most effective solution, in my experience, because the product almost always ships with a list of "deferred enhancements" that were defined as a result of user test and acceptance process. Typically, only the critical "enhancements," or the ones that can be implemented within the original estimated time/budget, will be incorporated. This negotiation process with the client at the critical, late stages in the process is often quite "stormy" at times, as we must reach agreement on the differences between bugs, design errors (of omission or commission), or true enhancements. Unless it is a bug, it isn't free. Even design errors are "joint responsibility" since the client approved the design. Bottom line is that it can "test" the provider/client relationship, and we never get to implement all the "lessons learned" on the project at hand.
That's why we design the project as comprehensively as possible on paper, in order to produce a quality project within budget. After a comprehensive requirements specification of the overall structure of the program has been produced, we create a complete storyboard of all of the content of the program. Production should only commence after the storyboard has been agreed on by the development team. This milestone in the process is formalized by the signing of a contractual agreement that this stage has been reached. The signing off procedure is an important incentive to ensure that the design is complete, and that members of the team will not attempt to change some components of the design at a later stage.
Let's look at the design process in more detail, thanks to Kent again:
Conversely, we often receive Request for Proposals (RFPs) or Request for Quotes (RFQs) with the client requesting a firm, fixed price (commitment to staffing, resources, and schedule) for both design and development. In fact, most of our clients are insisting upon this for CBT/IMM, and it is a real educational and sales challenge to convince them that a two-step, separately priced process is advantageous to everyone. Often our clients think that our design phase is merely an attempt to get them to pay to respond with a detailed proposal. They simply do not understand that until WE UNDERSTAND the content, the audience, and the client's expectations (often illustrated via a prototype), then we cannot accurately predict either cost or schedule....(Convince the boss!)
However, we in the entire production side (both software and CBT/IMM) want to move to an iterative design and development process, based upon our limited experience with it (and the power of the new 4th generation software development environments and CBT/IMM authoring systems), but haven't quite settled on a way to "sell it" either to our own sales force or our client base.
The overall structure and content of the project is designed in the Design Process. A technique we have found extremely useful is to conduct a brainstorming session with a large group of people with a range of experiences to start the production of a Requirements Specification. The purpose of the requirements specification is to clearly define the functionality and scope of the project. It is interlinked with some aspects of the feasibility study and evolves as the design process progresses.
After the initial design is produced, there is a focus on developing the requirements specification in finer and finer detail. Developments are reviewed by the team, and prototypes of the graphic design and of technically difficult programming areas may be developed.
As the requirements specification becomes more tightly defined, the group will start to develop a comprehensive description of the content of the project, called a storyboard. Inevitably, work on the storyboard will expose deficiencies in the requirements specification. For example, certain decisions about the way that content will be presented will cause modifications to the user interface.
With each pass through the design process, there should be fewer and fewer changes to the design, until it is eventually complete, and can be signed off.
The development phase of the design process includes production of prototypes of technically difficult programming components and the creation of the graphic design, but it may also involve constructing physical models to visualize the structure of the project, or parts of it. For example, the project team may use index cards and string to map out the project on a wall or table.
Evaluation of the design usually consists of constructive criticism and reflection within the development team, and with other experts and end-users.
After the design has been completed, the Production Process commences. Some aspects of production may have already been completed as part of the prototyping, such as technically difficult programming issues. In the production process, the resources, such as graphics and sounds are prepared, and the major part of the programming work is done. Work is monitored while it is progressing, but the major part of the evaluation at this stage is in testing of the program, at first by members of the development team, and subsequently by end-users.
The other issue is effectiveness of the design, and the quality of the end products delivered by the iterative cycle. We've now developed a few CBT/IMM courses that are entirely data-driven, with all the content and selective logic being stored in a database. This allows several very powerful changes to the traditional, linear design and development approach. We can print a complete set of current, correct storyboards at any point and obtain client signoff and approval. ... As long as we do not need to produce additional graphics or sound files, we are essentially limited in making revisions only by our typing speed. As an example, we can make significant changes during individual tryouts overnight, and have them available for further testing the next day. However, we can get them even more involved in the design process...
Yes, the data-driven approach is really the only way to go, if you have a choice. In current development, we develop storyboards on a wordprocessor. Text is then automatically "sucked in" to the authoring package. The use of a wordprocessor simplifies revision of the storyboard, and minimizes spelling errors. Does anyone know a programmer who can spell?
My colleagues Rod Kevill and Robyn Devenish have developed a package on the morphology of red and white blood cells in haematology. The program will accept different case studies of microscope slide images and appropriate data without any intervention from the programmer. The program is thus "infinitely" flexible to the haematologist with image acquisition skills.
We've successfully used the technique of one student reviewing the course on one machine and the ID making the desired changes on an adjacent machine in "real-time" and presenting them back to the reviewer to critique. We try to do this with at least three "novices" who have never been exposed to the content and three "subject matter experts." The content, especially for learner prompts and feedback, then has been re-designed and refined "on-the-fly" based upon live input from the end-users. (Note that I said the ID, not a programmer!) This has dramatically improved the "richness" of the dialogue between the learner and the computer-based learning experience. We now can capture and address subtle distinctions never before possible, add elaborations where proven needed, etc. We can correct the causes of misunderstandings with the feedback to user mistakes, distinguish and award "partial credit" based upon typical users' explanations for their actions, etc.
However, I would get these people involved at the prototype stage, rather than during the final debugging.
The outcome is significantly improved courses and intense "buy-in and ownership" from our clients. On our most recent project, at the end of small group tryouts the only "bug" reported was one typo. Replication masters were provided the next day and final client acceptance was immediate. Now if we could only figure out how to describe and "sell" this relationship to potential clients up front...
My earlier posts on Communication [ 1 Oct 96] and Roles [2 Oct 96.a] highlighted this point. However, how do you remove the "them and us" mentality when somebody is paying and somebody can go broke?
Put more formally: A continuing dialogue between content experts, designers, programmers and project managers and reflection and evaluation of the successes and failures of a project are important to improve the processes by which projects are developed.
The most important thing is not to assume anything. In our experience, when problems have occurred in projects, very often it has been that one team member has assumed that he or she knew what was required. Unfortunately, usually that assumption was proved to be false.
A very effective method of ensuring that all members of the team understand all facets of the design is to clearly document all of them.
As described above, the requirements specification is the first phase of the design process. In traditional software development, such as database programs and defense systems, the requirements specification details all aspects of the program. In these environments, the requirements specification is completed well before design starts.
However, it is not practical to produce such a rigorous and complete description for IMM, because it is so much more difficult to finalize the form of the content and interactions. Typically, the structure does not become clear until the design process is partially complete. (This is Kent's dilemma.) In many cases, it is not possible to determine whether a particular design strategy will be effective until after it has been developed to an advanced stage, implemented, and evaluated.
The purpose of the requirements specification in an IMM development is to clearly define the functionality and scope of the project. The requirements specification is interlinked with the feasibility study and evolves as the design cycle progresses. Evaluation of prototypes will expose problems in the design, which in turn will lead to changes in the requirements specification. While it is difficult to arrive at a finalized requirements specification, the process of creating the requirements specification is absolutely essential because it documents the design decisions made for all members of the team.
The requirements specification provides the project team with as clear as possible an understanding of the structure of the project, including an overview of the content. It is essential that all team members are involved in the design of the content from the earliest stages, because even though some team members do not always have content expertise, they generally have more experience in IMM, and can provide important insights into how the content may best be structured to suit the IMM medium.
It is important to write down what is agreed at team meetings about what is to be in the project. In many of our earlier projects, the requirements specification was mainly described orally at meetings. The danger of this approach is that different team members have a different understanding of what is agreed. This can lead to substantial content changes late in the project, or to complaints that the developers added features that were not agreed on. The written approach is especially important in IMM development, because team members come from different professional backgrounds with different world-views, and it is common for what one has said to be interpreted very differently by another.
Once the requirements specification has been completed, work can begin on the content of the program, i.e., what is actually going to be on each screen. The requirements specification may have specified that there will be a screen about topic X, but would not have specified anything about the actual content of the screens. The storyboard defines all the resources required for each screen. Obviously, this means the text, but also includes graphics, video, sound and other elements. Some storyboards are less formal than others, depending usually on the size of the project. The content expert is usually responsible for the preparation of the majority of the storyboard.
The development process requires that the storyboard be comprehensively reviewed several times by other team members. Production work should not start until the content is agreed to be final, and signed off so it cannot be changed. If a storyboard is accepted as complete, but requires subsequent revision, then the resultant changes to the programming can take hours to modify, and may even require redesign of much of the user interface.
The entire team should carefully read the storyboard from their point of view. It is very easy for inconsistencies and discrepancies to appear in the storyboard, and these may not be immediately obvious to the content expert who has written the material. Designers and programmers may discover things in the storyboard which don't match the requirements specification or which are simply not technically feasible.
It is also difficult for all team members to appreciate all aspects of the project unless they have a thorough understanding of the storyboard.
Whilst the requirements specification and prototypes of the screen design are essential for team members to visualize the project, as little content as possible should be entered into the project until the storyboard is complete. If possible, content experts should try to visualize the project from the prototypes when writing the storyboard. This is difficult, but can save considerable effort and money later.
The key points made in this post to efficiently produce educationally effective IMM materials are shown below:
However, Kent, I am not sure that any of this will help resolve your problem.
Thanks also to Bob Corderoy [4 Oct 96] for echoing my comments on teams. I am interested, Bob, in what sorts of documentation was used in the development of Exploring the Nardoo.
By the way, If anyone hasn't seen this package, then you should, real soon now. It is the only package I have seen which effectively meets the requirements of Laurillard's Guided Discovery Learning model. It just shades Just Grandma and Me as the best IMM package I have seen.