«Running Head: Analyzing A Novel Expertise Analyzing a Novel Expertise: An Unmarked Road Wayne D. Gray George Mason University & Susan S. Kirschenbaum ...»
We recognized from the beginning that the AO’s problem is not simply goal-driven in that the problem state changes without the intervention of the person. Some part of what the AO does is clearly an event-driven process. The issue is not event-driven versus goal-driven, but what combination of the two controls the AO's problem-solving behavior and how this combination can be represented in a cognitively plausible manner.
As we were struggling with the goal-driven versus event-driven interpretations, we were having problems supporting another early idea – localizing a target as a diagnostic process. The literature suggests that an important first stage in solving many tasks is diagnosing the problem.
This stage ends when the correct schema is selected (VanLehn, 1989).
We fully expected that the first stage of localizing the target would entail diagnosing the situation to determine the correct target-finding schema. Indeed, for about the first 18 months, our encodings had a goal called IDENTIFY-POSSIBLE-INITIAL-SCHEMA. Believing that the “absence of evidence is not evidence of absence,” we struggled mightily to find support for a diagnostic process. However, we could not find evidence in our protocols or elsewhere (e.g., from any of the other AOs and experts whom we consulted) that we could interpret as evidence that schema selection required extended deliberation. Our tenacity in clinging to the notion of schema selection is, in part, attributable to evidence that suggested that AOs used different schemas when the target was a submarine than when the target was a merchant. Unfortunately, this
In brief, our analyses indicate that when LOCATE-MERCHANT was the goal, fewer subgoals were pushed and popped than when LOCATE-SUBMARINE was the goal (see the left side of Figure 3). More important, most of the subgoals for LOCATE-MERCHANT were similar to, but qualitatively different from, those for LOCATE-SUBMARINE. As discussed later, what turned out to be bogus was the apparent qualitative difference between similar subgoals for different goals. Once this spurious qualitative difference was eliminated, we concluded that the same LOCALIZE-THE-TARGET schema guided the processing of localizing either a hostile submarine or a friendly merchant. This conclusion converged with our failure to find any evidence of schema selection. We currently believe that schema selection is not a problem. Whether the goal is LOCATE-SUBMARINE or LOCATE-MERCHANT, our expert AOs have one schema, LOCALIZE-THE-TARGET, that is automatically chosen.
Schema-directed Problem Solving Conceptualizing the AOs’ task as schema instantiation, not selection, was the key to a parsimonious account of the general structure of the AOs’ cognitive processes. Localizing the target was not driven by goals or events, but was directed by a schema. The data gathered were used to instantiate attributes of this schema. On each cycle of problem solving, the schema was reevaluated, the currently most critical attribute-value pair was identified, and instantiating this pair became the current subgoal. New events were incorporated into the schema and affect problem solving only to the extent that they affect the identification of the currently most critical
Figure 3: Mean number of subgoals per goal (left two) and mean number of operators per subgoal (right two) for LOCALIZE-MERC and LOCALIZE-SUB. Also shown are the 95% confidence intervals for the standard error of the mean.
This view of schema instantiation as schema-directed problem solving fits in nicely with our emerging awareness of shallow subgoaling. Each time the AO returned to the top-level goal (LOCATE-SUBMARINE in Figure 2), the schema was reevaluated. The subgoal chosen was one that would return information regarding one attribute-value pair. Hence, localizing a target is a wide and shallow task. The width is represented by a well-learned schema. The shallowness is represented by shallow subgoaling. Localizing the target involves dozens of iterations of schema evaluation and shallow subgoaling. These iterations continue until the AO has confidence that the
Data Analysis: Levels of Analysis and Limits to the Data Changes in our taxonomy of goals, subgoals, and operators were driven by three sources: our changing understanding of the general structure of AO cognition (discussed earlier), our failure to understand the limits to our data (discussed next), and our tendency to embed specifics of interacting with the simulation in our analysis of the task – that is, a failure to distinguish between task and artifact (discussed later).
Distinctions Not Supported Our final encodings resulted in nine categories of operators (Gray, Kirschenbaum, & Ehret, 1997a; Kirschenbaum, Gray, & Ehret, 1997). Each of these categories can be considered as representing many subcategories. For example, at one time, we tried to distinguish subcategories of things that could be queried and sources from which information could be received. An AO could query a display, the own ship operator5, his own memory, an instrument reading, and so on. Likewise, information could be received from long-term memory, from short-term memory, from reading a table, from viewing a graphic, from inferring a relationship among other information, from the own ship operator, and so on. At one time or another, each of these subcategories was considered – if only briefly – as a candidate for inclusion in the analysis.
In deriving our encoding categories, we focused on two transcripts, one from each of two AOs.
New encoding categories could be proposed by any of our three encoders. Typically, the encoder would present a rationale for why the subcategory was needed, along with a particular instance from the transcript that the encoder felt represented that subcategory. If the other two encoders were convinced by the instance, much discussion would be spent defining criteria by which other
two sample transcripts. New instances of the encoding category would be identified and debated until consensus was achieved. After this, two or three of the encoders would independently go through the other sample transcript in an attempt to identify instances of the new category.
More often than not, each independent encoding would yield a handful of instances (out of about 300 encoded operators) that could be considered members of the new category. However, there would be little or no agreement about which encodings were exemplars of the new category and which were not. At this point, the candidate would be abandoned. We considered it a plausible distinction, but a distinction for which the data were at too coarse of a level to support.
Problem Solving the Tool as Opposed to Problem Solving the Task We collected data from AOs who localized targets presented on a dynamic simulation of the ocean environment – CSEAL (for more information, see Gray et al., 1997a; or Kirschenbaum et al., 1997). However, our task was not to describe how AOs’ used the simulation (the artifact or tool) to localize the target (the task), but to separate the specifics of the simulation from the more general aspects of their problem-solving process (i.e., a functional level of analysis). Time and again, we found ourselves encoding specifics of using the simulator to localize targets rather than simply localizing targets.
We attempted to encode every segment of the transcript, but realized early on that a significant number of segments simply had nothing to do with the problem. The fact that many of the segments in a verbal protocol have nothing to do with the problem being solved is not news and, indeed, we expected this. These NAs (not applicable) were easy to identify because
weather, the room in which the simulation was run, and so on. Of the 2,882 segments encoded, 946 were in this NA category.
More difficult to distinguish were the segments that did not refer to solving the problem but to some aspect of the simulation. An analogy to this is talking to a co-author about how to format a table in Microsoft Word™ as opposed to discussing the data that would go into the table. A simple example is represented by the following query from the AO to the own ship operator asking about the units (relative or true) in which a particular display represented the data.
At some point, we realized that what we were seeing was a ubiquitous phenomenon that was not limited to our paradigm, but that represented a type of usability issue. We saw this as problem solving in the tool space, as opposed to the task space, and proposed the tool:task ratio as a usability metric (Kirschenbaum, Gray, Ehret & Miller, 1996). Following this insight, our transcripts were recoded with a new operator, instrumentation. As per the prior example, most of the instrumentation operators were found in groups of one to three amid a series of task operators. These small groups of instrumentation operators are essentially asides that are embedded among operators concerned with a particular task goal. Occasionally, both the AO and the own ship operator abandoned the task of localizing the target and became engaged in an episode of collaborative problem solving – attempting to figure out how to get the simulation to take a particular input or to display a particular type of data. Such episodes were recoded as a
Over our entire set of encoded data, 421 goals and 2,882 operators were required to encode the nine scenarios from our six AOs (for more details, see Gray, Kirschenbaum, et al., 1997 or Kirschenbaum et al., 1997). After removing all supervisory goals, their operators, all instrumentation operators, and all NA operators, we were left with 397 goals and 1,269 operators. We refer to this remainder as our clean set. The clean set of encodings was used in all subsequent analyses.
The significance of this reduction cannot be overstated; over half of our encoded utterances had nothing to do with localizing per se. Having these in the analyses confounded our efforts to make sense of the data. Once these were removed, regularities that had been obscured became apparent. For example, the scenarios we studied involved two targets: a hostile submarine and a friendly merchant. Given the shallow subgoaling we were beginning to believe in, it made sense to us that there would be more subgoals involved in localizing the quiet submarine than in localizing the noisy merchant. Indeed, this is what our encodings suggested (see the left two data points in Figure 3). However, the same subgoal seemed to involve many more steps if its supergoal was LOCATE-SUBMARINE rather than LOCATE-MERCHANT.
More steps were needed for a subgoal such as determine the signal-to-noise ratio (determineSNR) when determine-SNR was a subgoal of LOCATE-SUBMARINE than when it was a subgoal of LOCATE-MERCHANT, and this did not seem unreasonable. Indeed, as discussed earlier, such differences supported our belief that different schemas were used for different targets. By inference, this finding supported the belief that schema selection was an important
However, once the instrumentation operators and supervisory goals were removed, regularities appeared. As shown in the right two data points of Figure 3, the subgoals used in LOCATEMERCHANT required the same number of operators as the subgoal of LOCATESUBMARINE. (An extended discussion of this point is provided in both Gray et al., 1997a; and Kirschenbaum et al., 1997.) Completed versus Successful Goals Another part of our struggle to define the number of levels and depth of the goal stack resulted from an implicit assumption that a completed goal was synonymous with a successful goal. In our initial attempts to shoehorn reality, we viewed a completed goal as one that returned the information queried. For example, if the AO queried the bearing rate on a particular target, this goal would not be completed until the target’s bearing rate was determined. Attempting to trace the path from initial query to completion led us to postulate a tangled web of semi-infinite subgoaling. Stepping back and listening to the data led to a different conclusion. We discovered that, for the AO, knowing that, at a given point in time under current conditions, the bearing rate (or course, speed, etc.) cannot be determined is an important and complete piece of information.
The insight that a goal can be considered completed without being considered successful supports the shallow subgoaling component of schema-directed problem-solving. The AO launches a continuing stream of short queries. Each query returns some information. In the early stages of localizing (immediately after the target has been detected), the information typically is something such as the data are too noisy to answer that question (the SNR is literally too low).
Summary of Issues Understanding the control structure of the AOs’ cognition, together with understanding the nature and limits of the data, were major obstacles in our attempts to do a cognitive analysis of the AOs’ task. Before beginning this project, we knew that understanding the control structure of cognition would be the key to the cognitive task analysis (this concern was reflected in the original proposal to ONR).
At each iteration around the loop (see Figure 1), each component of our analysis seemed plausible. What troubled us were our efforts to fit the parts together into a coherent whole.
Although during this period we were not building ACT-R models, we were constantly asking ourselves how the disparate parts could fit into an ACT-R model. It was the failure to answer this question positively that kept driving us around the loop and deeper into the data.
At this point, we have exited the loop (Figure 1) and moved onto the next phase of the project. Although the current hypothesis – schema-directed problem solving with shallow and adaptive subgoaling – is coherent, we are collecting additional data in the hopes of capturing finer grained data on key aspects of AO problem solving in a dynamic environment.