Factors in the Design of Experimental Graphic Displays

William A. Kealy
University of South Florida

A couple months ago, when I was invited to share some of my recent research with the IT Forum, my thought was to present the findings of research on graphic organizers (GO) being prepared for journal submission. In a nutshell, the two experiments show that arranging the elements of a GO according to the gestalt principle of proximity and their real-world spatial relationship results in significantly better recall—with respectively greater accuracy and speed—of accompanying text information. Along with an earlier article on GOs (Kealy, 2000), I use three experiments to argue that these displays, and perhaps all instructional graphics in general, should be designed with three factors in mind that have psychological consequences: perceptual, cognitive, and sociocultural or constructionist.

Instead of my original idea for the Forum, I’m going to use this opportunity to “step out on a limb” and present a bolder proposition: some research employing graphic materials may be confounded due to a failure to consider the three factors previously mentioned. I won’t venture a guess as to how much research has been conducted with questionable findings because of the quality of the experimental treatments used, but it’s nevertheless a possibility worth exploring. Apart from the standard recommendations one receives in Research 101 to norm one’s material (we always do that, right?), there doesn’t seem to be much said about how one can undermine or strengthen a study by virtue of how its graphic treatments are designed.

There are several reasons why graphic-based experimental treatments are especially vulnerable to being poorly implemented. One is that there are much fewer standards for controlling for extraneous variables with graphics compared to text. In designing an experimental text it’s common to create alternate versions that put target information in different locations as a way of controlling for order effects. What corresponding standards and measures exist for the development of graphical experimental treatments?

A second reason for why it may be harder to create valid experimental graphics is that people have less skill in creating visuals than prose. Even when the graphic task is fairly easy, such as making a fictitious map, novices might lack the domain specific knowledge needed to produce a suitable experimental treatment. Ormrod, Ormrod, and Wagner (1988), for example, criticized the maps many researchers use in experiments on the grounds of their implausibility. Singling out the map used by Thorndyke and Stasz (1980) to make their point, they observed:

[The map] showed roads and railways that untypically followed mutually exclusive routes, contained road and rail network junctions that were unexpectedly mutually exclusive, and lacked any indication of a normal settlement hierarchy. (p. 425)

In this case, the deficiency of the map in question was not due to the researchers’ lack of artistic ability but rather because they lacked a cartographer’s sensitivity for how the arrangement of a map’s elements influences its interpretation.

Employing professional artists or draftsmen to create experimental graphic treatments doesn’t guarantee that subjects will interpret the materials as researchers expect. A case in point is the urban map used by Kulhavy, Schwartz and Shaha (1982) that displays six city blocks as a grid containing 36 pictorially represented, or mimetic, features. Here, the researchers hired a professional artist to carefully render each feature as a line drawing. As an additional precautionary measure, three judges evaluated the images to ensure that each was visually distinct and accurate in its portrayal. Yet, as Figure 1 shows, the resulting display bears little resemblance to a typical street map. In Experiment 2 of the study, where the street grid appeared on all treatment maps, none of the anticipated differences in feature recall resulted between displays that sequentially presented features by category, by spatial distribution, or randomly. Do you think there might be different results obtained using a redesigned version of the map in Figure 1?

Figure 1
Figure 1. Map used in the Kulhavy, Schwartz, and Shaha (1982) study and its redsign to the right

A third reason why graphics materials are harder to control in an experimental setting is that, as a symbol system, they have a greater capacity for repletness (Salomon, 1994) than text. This means that subtle changes to the surface features of a graphic have a much greater impact on how it is interpreted compared to a text. To use an extreme example, changing the color of the sky in a photograph may render the scene more pastoral or ominous whereas changing the font used in a prose passage does little to alter its meaning. Consequently, graphic media are inherently more vulnerable to misinterpretation than the written word.

Finally, I think graphic treatments have a greater probability of having a confounding influence in research simply because they have a shorter history of use in psychological research than verbal materials. We were well into the cognitive era, for instance, before it became acceptable to speak of imaginal coding as a process distinctly separate from linguistic encoding. With the emergence and acceptance of dual coding theory (Paivio, 1971, 1986), the past decade has seen a remarkable increase in the amount of research that incorporates graphic materials such as maps (Kulhavy, Stock, & Kealy, 1993), diagrams (Winn, 1988, 1993), graphic organizers (Robinson & Schraw, 1994), and multimedia (Mayer & Sims, 1994).

Types of Threats Graphics Can Pose for Researchers

One way that graphics potentially undermine research outcomes is when they do a poor job of activating the particular psychological state being investigated. For instance, suppose a researcher is studying the effect that prior knowledge for a map has on its effectiveness as an aid for learning a related text. Two maps are created for the study, a U.S. map and one that’s fictitious representing the “no prior knowledge” condition. If the latter map unintentionally reminds subjects of a particular country, can one really claim that the variable of prior knowledge has been examined?  In most cases, however, researchers probably are aware of this sort of problem and take appropriate measures in norming the materials.  

In this paper, by contrast, I discuss a different category of flaws in experimental treatments that I believe can easily be hidden from a researcher’s scrutiny. Within this class of threats to graphic treatments, the elements, rules, and field of reference of a graphic-based symbol system unintentionally impinge on how a representation of the system is interpreted. Unlike the earlier example, these influences are not the result of faulty employment of the graphics system per se. Instead, they exist as artifacts of the system itself. Such artifacts, I contend, are products of the interaction between the cognitive capacities (Kozma, 1991) of a symbol system and the stages involved in mentally processing the various aspects of the system. For the sake of clarity, I’ll mention symbol systems and mental processing first and then move on to what I mean by system artifacts.

A symbol system consists of symbolic elements, symbolic rules, and a field of reference or context of use within which elements and rules manifest themselves in practical ways (Salomon, 1994). Within the context of cartography, for example, there are symbolic elements such as map features and symbolic rules that guide how these elements should be organized; features on maps are spatially arranged to correspond with how they would occur in the world being represented. Personally, I find that the idea of symbol systems offers an interesting way of considering the graphic variables that influence learning (Kealy, 1996). To illustrate what I mean, here are three experiments dealing with maps from the point of view of the display as a symbol system:

Component
 Symbolic Elements
 Symbolic Rules
 Symbolic Field of Reference
Research variable 
 Discrimination of map features as mimetic pictures or verbal labels
 Configuration of map features into a list or into a spatially dispersed array.
 Representation of a display in either a map-like or diagram-like interpretive context.
Experiment 
 Griffin & Robinson, 2000
 Abel & Kulhavy, 1989
 Kealy & Webb (1995)

These three aspects of a symbol system come into play when a person uses a representative display to get information for accomplishing a given task. Since, in most cases, even static instructional displays are accompanied by verbal material, it’s legitimate to call such graphics “multimedia” according to part of Mayer’s (2001) definition. When learners are presented with a semantically related set of graphics and text, they typically first attend to critical parts of the visual and verbal information. Learners next try to form mental models of the two forms of information for encoding. In the last stage of this three-stage encoding process (Mayer, 1999), learners try to integrate these imaginal and linguistic mental models for long-term storage.

Perceptual Artifacts. At last I come to the artifacts of symbols systems that present themselves as extraneous variables when research subjects examine graphics displays. I think these artifacts fall into three categories: perceptual, cognitive, and constructionist. Perceptual artifacts are typically encountered early in processing (i.e., the first stage of Mayer’s [1999] model) and can involve either low-level or high-level perception—a distinction I won’t get into now for the sake of brevity.
One example of how perception influences a display might be the tendency for subjects to process the whole of a display before its parts or vice versa. Many factors influence whether the whole or part is attended to first such as the viewing angle and the number of objects being viewed. Nevertheless, the point is that if the outcome of an experiment relies on mental processing, and processing effectiveness relies on attention allocation, and attention is affected by whether one sees the whole before the parts—well, you see what I mean. If a display isn’t created to perceptually support the location of target information, it will probably hinder subjects’ learning performance without the researcher knowing it, because perception wasn’t the independent variable of the study.

I first became aware of the possibility of unplanned perceptual influences on experimental treatments while reading the outcome of Robinson and Schraw’s (1994) study on graphic organizers (GO). One of the treatments in the study consisted of a GO that presented subjects with a matrix of fish characteristics with three levels of water depth presented across the top as major headings and categories of fish name, diet, size, socialization, and color along the left-hand edge as row labels. For each water depth, there were two columns, resulting in six fish types and corresponding characteristics (e.g., six colors, six sizes).

In looking at the experimental display, I was struck by the fact that the elements of the matrix were arranged equidistant to one another. If, by contrast, the GO experimental display used the gestalt principle of proximity to signal membership to a major column category of water depth, it seemed an entirely different result might have occurred. Our research team redesigned the original GO (Robinson & Schraw, 1994), slightly rearranging the display’s elements so that the matrix formed suggested three columns consisting of two columns each. A comparison of the original GO with the redesigned display indicated the latter yielded significantly better comparative judgments by subjects on the information contained in the GO.

Cognitive Artifacts. Cognitive influences of a symbol system are those that affect learners’ ability to effectively form linguistic or imaginal mental models of the information at hand. Hence, cognitive artifacts of the symbol system are associated with the second, or model-building, phase of Mayer’s (1999) model. Aspects of information processing, such as the limits of working memory, task expectations, and availability of relevant prior knowledge, are especially important during model-building. So, this leads me to ask, in what ways can graphic experimental treatments inadvertently color how subjects form mental representations? From the standpoint of symbolic elements, suppose map features are discriminated between subjects so that one version activates prior knowledge more than the other. Assume also that, while prior knowledge correlates with performance on the dependent measure, prior knowledge isn’t the psychological construct under investigation. In this hypothetical situation, do you think the graphic treatments might portray an intervening variable that a researcher wouldn’t be aware of?

The potential influence of cognitive artifacts is reflected in an experiment I ran a few years ago (Kealy, 2000) using the GO and accompanying text from the Robinson and Schraw (1994) study. I observed that GO used by the two researchers was designed so that “depth” served as the major category for semantically organizing the display. Meanwhile, the accompanying text was semantically organized according to the socialization of fish (school, small group, or solitary); socialization was the topic sentence for every paragraph in the text. Because the semantic structure of the graphic treatment was incongruous with its accompanying text, it seemed likely that such a discrepancy would have an influence on what subjects remembered from the information presented to them. A redesigned GO, with socialization as the major heading, produced better recall for information dealing with the socialization behavior of fish than the original GO.

Constructionist Artifacts. I use the term constructionist to define the last category of unintended influences graphics treatment can have on experimental results. I prefer constructionism over constructivism as a term to describe this category of intervening variables because the former implies a kind of knowledge-building that’s more serendipitous and less self-conscious than the latter (Papert & Harel, 1991; see Cole & Wertsch, 1996 and Wink, 2002 for a fuller discussion of this distinction). Further, my use of constructionist is with a sociocultural context in mind. Specifically, constructionist artifacts of graphics are preconscious responses rooted in the cultural conventions of using a particular symbol system.

To illustrate, in my graduate courses at USF, the preconscious influence of constructionist artifacts on learning, I present a relevant study by Winn and Holliday (1982). In the experiment, subjects studied a diagram on dinosaur evolution (See Figure 2) in which creatures faced to the right and the sequence of evolution proceeded from left-to-right. A second group received the same diagram but with dinosaurs facing left and evolving right-to-left. I ask my classes to speculate what group showed significantly better recall performance for the related information and why the result occurred. What outcome would you guess? Virtually everyone picks the former group and speculates the reason for the performance difference is that both groups “map” their experience of reading words (i.e., from left to right) on the symbol system peculiar to diagrams. Consequently, the latter group is at a disadvantage due to the incongruity between the verbal and diagrammatic symbol systems without ever being aware that something was wrong with their diagram. As an aside, some students invariably ask if the study has even been replicated in, say, China or Israel (an interesting question—does anyone out there know?).
Figure 2
Figure 2. Dinosaur diagrams used in the Winn and Holliday (1982) study

The potential threats that cultural conventions of symbol systems pose for graphics treatments are especially insidious because, like water to a fish, they are parts of the environment that escape the awareness of both experimenters and subjects alike. This hidden aspect of graphic treatments was demonstrated in a study we conducted that presented subjects with two types of graphic organizers. One GO (the display used in the first experiment that followed the principle of proximity in arranging array elements) had three levels of the category “depth” arranged across the top of the chart. A second GO had the depth category arranged vertically along the left side so that levels became progressively deeper from top to bottom—as they would appear in nature. Although accuracy of subjects’ ability to make comparative judgments on fish characteristics was the same for both groups, the participants with a naturally congruous GO made their responses significantly faster.

Research with Graphic Displays: A Call to Action

Despite the cautions presented here on the many ways graphic treatments can introduce intervening variables into research, I believe that empirical studies remain a viable method for improving the effectiveness of instructional media. First, there’s still an important place for well-run scientific experiments in educational research (Mayer, 2000)—and for understanding how learners can better use instructional graphics and how we, the designers of these adjuncts, can make them better.
Second, there is much we’re only now beginning to understand about the role of instructional media within learning environments. Most instructional technologists today would, I think, agree that learning is the result of a complex mix between the task, the learner, and the nature of the learning material involved. Over the past few decades our awareness of how people learn has matured to the point where we’re able to start making useful prescriptions for the design and delivery of instructional media.

Third, technological strides achieved in just the last few years, such as the increase in available memory and bandwidth, have profoundly altered the nature of instructional media both in who produces it and in its power to communicate. Developments like zoomable-user interfaces and interactive 3-D models invite researchers to revisit Kozma’s (1991) notion about the “cognitive capacities” of media.

Finally, authoring software such as SuperLab, E-Prime, and MediaLab give researchers the tools for designing computer-based experimental treatments that can gather data previously very difficult or impossible to obtain. Using Authorware, for instance, our research team has been able to create experimental treatments that accurately determine when and for how long learners access adjunct graphic displays in a computer-based instruction (Kealy et al, in press). This capability offers exciting possibilities for determining, for instance, the types of task expectancies and learning contexts that motivate the study of some graphic displays more than others.

These possibilities, however, can’t be fully realized unless researchers exercise care in designing and developing the graphic treatments they use in experiments. Whatever the technology employed, we would ideally want graphic treatments to be designed so that they effectively tap the psychological construct being investigated. What should we expect of graphics used in research? Perhaps one desirable quality of experimental displays is that they don’t inadvertently misdirect subjects’ attention either by failure to activate perception (e.g., by portraying symbolic elements too faintly to be seen) or by accidentally cueing visual interest for irrelevant details.

Might it also be a good idea for experimental displays to adhere to basic rules of good graphic design (e.g., simplicity, balance, unity) and, when presented in concert with text or audio narration, that they conform to the design recommendations gained from research on learning with dually-encoded multimedia? Should researchers have greater awareness of the symbolic rules pertaining to the instructional medium that they are using in a study? For instance, could the vertical stacking of graphic elements unintentionally suggest a hierarchical relationship or, if the same items were placed side by side, would the arrangement suggest a state of causality (Winn & Solomon 1993)? To what degree do instructional technologists need to sharpen their awareness of the preconscious knowledge that’s socially constructed in the daily use of the media under examination?

What else…your thoughts?

References

Abel, R. R., & Kulhavy, R. W. (1989). Associating map features and related prose in memory. Contemporary Educational Psychology, 14, 33 48.

Cole, M. & Wertsch, J.V. (1996). Beyond the individual-social antinomy in discussions of Piaget and Vygotsky. Human-Development. 39(5): 250-256.

Griffin, M. M., & Robinson, D. H. (2000). Role of mimeticism and spatiality in textual recall. Contemporary Educational Psychology, 25, 125–149.

Kealy, W. A. (1996). The effect of picture position on learning from an accompanying text. Reading Psychology, 17(4), 309-331.

Kealy, W. A.  (2000). The Role of Semantic Congruency in the Design of Graphic Organizers. Quarterly Review of Distance Education, 1(4), 205-214.]

Kealy, W. A. & Webb, J.M. (1995). Verbal Learning with maps and diagrams. Contemporary Educational Psychology, 20, 340-358.

Kealy, W. A., Bakriwala, D. J., & Sheridan, P. B. (in press). When tactics collide: Counter effects between an adjunct map and prequestions. Educational Technology Research and Development.

Kozma, R. B. (1991). Learning with media. Review of Educational Research, 61(2), 179-211.

Kulhavy, R. W., Schwartz, N. H., & Shaha, S. H. (1982). Interpretative framework and memory for map features. American Cartographer, 9, 141-147.

Kulhavy, R. W., Stock, W. A. & Kealy, W. A. (1993). How geographic maps increase recall of instructional text. Educational Technology, Research and Development, 41(4), 47-62.

Mayer, R. E. (1999). Designing instruction for constructivist learning. In C. M. Reigeluth (Ed.), Instructional-design theories and models: A new paradigm of instructional theory. (pp. 141-159). Orlando, FL: Academic Press.

Mayer, R. E. (2000). What is the Place of Science in Educational Research? Educational Researcher, 29(6), 38-39.

Mayer, R. E, (2001). Multimedia learning. New York: Cambridge University Press.

Mayer, R. E., & Sims, V. K. (1994). For whom is a picture worth a thousand words? Extensions of a dual-coding theory of multimedia learning. Journal of Educational Psychology. 86(3), 389-401.

Ormrod, J. E., Ormrod, R. K., Wagner, E. D.,& McCallin, R. C. (1988). Reconceptualizing map learning. American Journal of Psychology. 101(3), 425-433.

Paivio, A. (1971). Imagery and verbal processes. New York: Holt, Rinehart, and Winston.

Paivio, A. (1986). Mental representations: A dual coding approach. New York: Oxford University Press.
Papert, S. & Harel, I, (1991). Constructionism. Norwood, NJ: Ablex Publishing.

Robinson, D. H., & Schraw, G. (1994). Computational efficiency through visual argument: do graphic organizers communicate relations in text too effectively? Contemporary Educational Psychology, 19(4), 399-415.

Salomon, G.  (1994). Interaction of media, cognition, and learning: An exploration of how symbolic forms cultivate mental skills and affect knowledge acquisition (2nd ed.). Hillsdale, NJ: Erlbaum.

Thorndyke, P. W., & Stasz, C. (1980). Individual differences in procedures for knowledge acquisition from maps. Cognitive Psychology, 12, 137-175.

Wink, J. (2002). Constructivist – Constructionist. Retrieved February 24, 2003 from the World Wide Web: http://www.joanwink.com/newsletter/news1002-construct.html.

Winn, W. D. (1988). Recall of the pattern, sequence, and names of concepts presented in instructional diagrams. Journal of Research in Science Teaching, 25(5), 375-386.

Winn, W.D. (1993). An account for how people search for information in diagrams. Contemporary Educational Psychology, 18, 162-185.

Winn, W. D. & Holliday, W. G. (1982). Design principles for diagrams and charts. In D. H. Jonassen (Ed.), The technology of text (pp. 277 299). Englewood Cliffs, NJ: Educational Technology Publications.

Winn, W. D., & Solomon, C. (1993). The effect of the spatial arrangement of simple diagrams on the interpretation of English and nonsense sentences. Educational Technology, Research and Development, 41(1), 29-41.


ITFORUM PAPER #69 - Factors in the Design of Experimental Graphic Displays by William A. Kealy. Posted on ITFORUM on March 28, 2003. The author retains all copyrights of this work. Used on ITFORUM by permission of the author. Visit the ITFORUM WWW Home Page at http://it.coe.uga.edu/itforum/