[quoting Alessi, 1 Mar 96] On the topics of just in time learning and the empirical approach to instructional design, T. Kent Thomas had some good comments. To summarize some of his comments, I hope accurately, he takes the position that in industry, just in time learning is becoming the norm as is just in time manufacturing, that in industry time is money and time spent on education may even be wasted if (as in the example of IBM) employees go to your competitors, and that there is neither time nor money to try multiple instructional methods or use control groups in designing instruction.
Steve accurately captured most of my points, with one need expansion needed. In the case of IBM, their investment in training can and is actually being "used against them" by their competition. Many of their former employees now work for the competition, or their "intellectual property" or expertise is no longer the "property of IBM," but available in the general marketplace. As an example, my current "boss" is an exceptionally well-trained and experienced ex-IBM'er who took an early retirement. We're a software company, a Microsoft Partner, and an IBM Solution Provider--IBM is now buying our services, as is their competitor, Compaq.
First a minor point, I was never suggesting control groups. Even in research I'm beginning to be more in favor of the developmental methodology as Tom Reeves has recommended. But the more critical thing is this. Kent, are you saying that because these are the typical business attitudes and norms that they are necessarily CORRECT, or are you saying that these are UNFORTUNATELY the typical business attitudes and instructional technologists should at least be aware of them? I would agree with the latter, not the former.
I didn't mean to imply that Steve recommended the use of control groups. I used this only as an example of the "classical" scientific study. I also support and use the developmental methodology, but am daily confronted with its current limitations. First, our tools to create IT-based learning materials (such as CBT, EPSS, distance learning, etc.) are generally not conducive to "rapid-prototyping" types of development methods. It is simply too labor-intensive to make constant changes in an evolutionary design. Second, the client who provides the subject-matter expertise has very limited time, and cannot or will not make the time available for frequent, in-depth reviews of the evolving design. The single greatest challenge is to "capture" the content from a very busy client in the least amount of time possible. Third, there's the perceptions of the client that do not support a developmental approach, such as "if you're the 'experts' then why do you need so much of my time" or "why do I need to pay for all these revisions, when you didn't 'get it right' the first time," etc., etc., etc.
But, just as Kent thinks I have simplified things by being concerned only with academia, I think he has simplified the variety of goals and needs in industry. First another minor point, the majority of students in our Instructional Design and Technology program at Iowa go into business and industry training jobs and not into academia, so I am more concerned with training than Kent thinks. But the more important point is the great variety of goals and needs in industry training. In an earlier posting I discussed the issue of error consequences. The consequences of error by an airline pilot are more severe than, for example, a manufacturing error which causes a batch of sweaters to be the wrong color. While textile manufacturing may deserve training at the lowest possible cost, airline pilot training does not.
Just as Steve acknowledges over-simplification to make a point, I acknowledge over-simplification also. I fully acknowledge the variety of goals and needs in training. Please note that I said "we need results" at the lowest possible cost. I stand by that statement. Steve spoke of his perspective at Iowa--my background is 12 years experience as an Air Force training systems analyst, and now five years in industry. I fully understand the issue of error consequences--the Air Force task analysis methods used for media selection require that the tasks be rated on frequency of performance, difficulty of learning, and task criticality (to address the issue of non-performance). Branson, in the "original ISD-model" addressed all these issues, in addition to the requirement to select the least expensive method from alternative, effective methods (Branson, R.K., Rayner, G.I., Cox, J.L., Furman, J.P., King, F.J., & Hannum, W.H. (1975). Inter-service Procedures for Instructional System Development. Ft Monroe, VA: US Army Training and Doctrine Command).
To "counter" his specific example of "airline pilot" I'd also point out the ongoing debates in military and aviation training related to the issues of "fidelity" and "realism," which are the primary determinants of cost is a simulation-based training method. As an example, a computer-driven mockup of a cockpit (including real switches and gauges) to teach cockpit procedures training costs about six to ten times the amount of an equivalent multimedia-based CBT that simulates exactly the same functionality on-screen. The training has been proven (to most who will objectively analyze the data from extensive tests) to be equally effective. Consequently, most cockpit procedures training now being developed is now being done by CBT, because it is much more cost-effective. (The much more expensive flight simulators are "reserved" for teaching the more complex tasks that require refined psycho-motor skills.)
Furthermore, the phrase "lowest possible cost" assumes we have some way of knowing what that lowest possible cost is. We rarely do. Rather, we make estimates and deal with probabilities. For some training it is safe to err in the direction of spending too little money. For other training we should err in the direction of spending a little too much. The issue of error consequence is just one example of my contention that industry training has many variables, probably more than academic education. A wider variety of skills and knowledge are taught in industry, the learning environments vary more, the learners vary more, the trainers vary more, and so on. So to look for a one-size-fits-all training solution would be unwise. I'm concerned that some companies look to just-in-time learning as such a solution.
Steve, I agree with your evaluation of the challenges of designing training for industry, both in variety and complexity of the needs and in the expectations. My clients fully expect (and perhaps rightly so) that I provide accurate estimates and probabilities. That's just my point. I develop materials that are "guaranteed to be effective" on firm, fixed-price bids. Time-and-materials contracts are virtually a thing of the past, and the clients require vendors to compete against defined requirements (usually very vague ones, however) and then select a vendor based upon either least cost or overall "best value" (which at least allows for some design variables other than cost).
By the way, where did the "one-size-fits all" come from? I didn't recommend one, nor do I think this is an accurate characterization of any of the instructional methods or technologies being discussed. I think this is an over-simplification also. My view of them follows:
"Just-in-time" training (JITT) is more of a concept or philosophy, not a specific technology. It focuses on providing the training as close to the point of need as possible, and doesn't identify a specific delivery method or medium. Doesn't this make sense, given what we know about retention, transfer, and practice? Provide the training as close to the need (increasing retention and transfer) and provide almost immediate application (i.e., practice) in a "real-world" environment.
An EPSS (which I characterize as a subset of JITT) is more a design perspective or technique, not a technology. An EPSS is of necessity a computer-based solution that is designed to provide all the necessary support (job aid, reference information, or training) on-line, at the point of performance. If there's a representative model for an EPSS, I haven't found one yet, and I've fielded two and have two more in design now. While I don't think an EPSS can fully replace other training methods or provide equivalent results in all cases, I still recommend them as a "best value" type of solution. If training and support funds are limited (and they always seem to be) the combination of training and job aids offered by an EPSS seems to yield a good combination of cost, effectiveness, efficiency, ongoing availability, and continued use.
CBT can obviously implement a variety of instructional strategies and teach a wide variety of content, as Steve so aptly describes in his book, Computer-based Training: Methods and Development. (I have both the first and second editions and recommend it frequently as the overall best basic book on CBT design available--is there a third edition available or coming soon?)
As for having as scientific a method as possible, I agree with you, Kent. But we probably disagree as to HOW MUCH is possible. I believe that designers tend to place too much confidence in the prescriptions of various instructional design models and too little emphasis on evaluation and revision, thus turning out instruction that is not ready. You are right that evaluation and revision is expensive and I certainly don't suggest is should occur until training is perfect. But I do believe there is a middle ground between perfect and awful, and that more attention to evaluation and revision is what will get you to that middle ground.
I agree that evaluation and revision are desirable, but overall I've found that industrial clients view this cycle as "mistakes that need correcting that they shouldn't be required to pay for." As for how much is possible, I don't know, either. I just know that I'd like more accuracy in my projections and more confidence in my designs (which are pretty eclectic, by-the-way) since "my paycheck depends upon them." I also believe there's a middle ground--perhaps I spent too long in the military, where "it's close enough for government work" was a prevalent paradigm. I'm accurate enough to continue making a living doing it, and "effective enough" to get repeat business from Fortune 100 companies--but that's about it--I certainly haven't gotten "rich and famous" doing it. Perhaps all I'm suggesting is a methodology to capture what revisions are made, and why, (based upon evaluation) and to hypothesize those "lessons learned" into more prescriptive guidance. If we can't "turn IT or ID into a science" can we at least move it closer to a "craft" where there are guidelines, generally accepted practices, and pretty consistent quality? (If you want a challenge, try explaining to a novice client the difference between "good CBT" and "bad CBT" that they've heard so much about.)
As far as my concerns for efficiency (effectiveness in the least amount of time), I'd refer you to the Nov 95 issue of Training magazine, whose cover story was "No Time To Train" and leave you with a paraphasiztation of Peter Senge (The Fifth Discipline, 1990):
The only sustainable source of competitive advantage is your organization's ability to learn faster than the competition.