I hesitate a bit in responding in this forum. I am not an educator. Nor have I been involved in significant academic interactive multimedia development.
But then, I gain confidence from the belief that the purpose of interactive multimedia is to communicate. An educational IMM would have communication as its core objective. Its role in learning would be that of a facilitator for creation of the learner's own metacognitive maps. And so, here I go...
First, before images of a mystic from India are conjured, may I briefly describe where I come from.
For the past ten years, I have been involved in CBT and multimedia development with two commercial organizations in India. During this time, we have developed over 30 small and big projects--topics ranging from teaching programming languages, to a reference title on Buddha, to storytelling, to edutainment, to learning literature, to edutainment games.
My graduate work in Instructional Science and Technology formed the foundation for: training of team members, design approaches, optimum project processes, project management, evaluation philosophies, and most importantly, learning for future projects.
Yes, mistakes have been the best teachers. We have made many, and we continue to make more. The emphasis is, on not repeating the same mistakes (if we do, no learning has taken place!).
I thus respond to Rob's questions in the context of this collective experience.
Were your plans too ambitious? Yes. In the initial projects, I had no idea of the ground realities. I did not know that...
Did you bite off too much to chew? Yes, the time, people, and budget have been referred to in the earlier response. I know that in the initial projects, my estimate of the scope of content which could be covered in the given time & budget, was over ambitious. Also, my anticipation of design sophistication within the given time & budget was off the mark.
If you did it again, what would you change?
1. Insist on a clear brief: Something which could make me very unpopular with the client!! A lot of inaccuracy in project planning can be avoided if we are clear about what we are setting out to achieve. This means that preparation of the brief in itself could become a project. But clarity in objectives, content scope, overall design approaches, available resources, and so on, can go a long way in preventing future heartaches (yes, that's what they are!) for all concerned.
2. Spend time on team orientation: The team needs to share a commonalty of purpose. They need to have clarity in the specific roles they will perform on each project. They need to "understand" and "appreciate" each other's personalities, skills, and experience. So, I would spend a short time up front in having the team sit together in a sort of orientation program.
3. Build in more time for design: Many a time, we succumb to the client's need to "see" something quickly--and they don't like seeing paper prototypes! So, we cut short the initial design stage and jump right into production. The initial design stage needs to explore a variety of instructional approaches (has anyone out there discovered the single most right way for instruction?!!), a variety of graphics styles, user-interface philosophy and designs, media usage, etc. The eventual outcome of this stage would be the prototype (or two)--and this takes time. But, experience tells me that this is time well spent. It prevents eventual extensions of deadlines, frequent design changes, variation in content scope, and so on. Most importantly, it gives the project team confidence in what they are building. Rigor at the design stage is well worth everyone's time and effort. And maybe (just maybe), even the client will like you better at the end of it all!
4. Use an efficient project execution model: Over the years, a project execution model has evolved. I believe that this model inherently lends itself to efficient processes and effective products. But, the problem is in motivating the team to follow it. At the start, I would thus customize the model to the requirements of each project. This could be done in the "orientation" session.
5. Build in frequent project reviews: Have frequent project reviews which deal with team climate (along with progress, quality, and other such regular issues). The best teams are the ones where members feel comfortable in questioning, demanding, and challenging each other. And yet, they have basic trust and respect for each other. This is not as intimidating as it sounds.
6. Make quality assurance an ongoing process: Build in a continuous process of quality assurance (formative evaluation) and then find the will and means to implement it!
7. Take documentation seriously: The fact of life is that people forget and people leave. We have learned over years that documentation of changes in design decisions, to program design, and to bug reports, is critical for handling eventualities and also for posterity.
Would you ever do it again? I can't think of a single project I would not want to do. And hopefully, all of them would be done more efficiently. But, can I be 16 again???
Did you have enough funding? This is a tricky one. What comes first, the funding or the design? If it is the funding, the design has to be done within the constraints of available time and money. If design has the option of asking for funds, WOW!
I guess what is important is that there be a lowest common denominator for funds for specific design/development requirements. And, then, there is also the issue of efficiency. No amount of funding will be adequate if there are inefficiencies and wastage in project execution--thus the criticality of having a good project model to start with.
Did it take longer than you thought? In one of my first projects, we were off by more than a 100%. I think we have gotten better over the years so now the error margin is 5 to 10%. Have not yet arrived at the perfect formulae!
Did the project finish? Is it in use? The proof of the pudding is in the eating! Some projects occupy space on Magneto Optical Devices and have entered the annals of history, never to be seen again. But others are being used. The teams get the best kicks from seeing their work in use. The ones which are in use are not necessarily the best designed, but those which meet specific requirements of the user!
Are you satisfied with the result overall? As I mentioned earlier, if the product is in use, it is great. If the users say they like it (for example, when children who have been involved in formative evaluation, want to come back for more), it gives us fantastic kicks. But in all projects, we know we could have done better on most fronts.
Are you satisfied with the instructional design? Not really. All instructional projects I have been involved in, could have been more learner oriented than content and technology oriented. The closest anything comes to instructional design in India, is training for teachers. And I have experienced a real dearth of insightful writers with an appreciation for "instructional design." And so, involvement of an "instructional designer" on an educational IMM project seems critical.
Was there sufficient student interaction with the software? Is there ever? Probably, the issue is not of sufficient number, but rather, the quality of interactions. I have a dream of a package with interactions which go beyond the mundane navigation or simple response-feedback algorithms. How about a package which allows freedom to explore, analyzes responses "intelligently," allows discovery, facilitates individual metacognitive maps, etc.
OK, you can now raise issues of time, budget, expertise, evaluation, etc.
Are you happy with the way it looks? Except for the initial CBT packages which were text based (and in retrospect, rather boring), we have had exceptional graphics artists on our projects. So in most cases, the packages look and feel good.
Did parts of it turn out differently than you envisaged? Yes, very much so--especially when we did not spend enough effort on a clear brief and a rigorous design. While on one hand, this has time and budget implications, it also has an impact on the morale of the team. I am a proponent of Michael Allen's development model of Successive Approximation. However, this model has to be pyramidal so successive changes are finer and smaller. This means progress is continuous and very evident throughout the project cycle.
Were there arguments about who said what, when? Invariably ambiguity of roles, egos, confusion in directions, and so on, crept in. I doubt very much if they were totally avoidable. Probably, how they were dealt with is more important. Frequent reviews which encourage conflict resolution and open communication helped in most cases.
Did design flaws surface late in the project? Yes. Some were a consequence of bad initial planning (have I mentioned this enough times?) and others were discovered through formative evaluation. Most could be corrected in time (but often at the expense of frayed tempers and egos).
Are there bugs in the program? How many? How much effort did it take to get rid of them? Normally, in a six month project cycle, we set aside one month for the final quality control. In most cases, this is adequate but in one specific case, a particular technical bug took almost five months to fix. However, we also do frequent "bugs" evaluation during the development cycle. This reduces the piling up of bugs towards the end of the project.