FREE ELECTRONIC LIBRARY - Online materials, documents

Pages:     | 1 || 3 |

«Running Head: Analyzing A Novel Expertise Analyzing a Novel Expertise: An Unmarked Road Wayne D. Gray George Mason University & Susan S. Kirschenbaum ...»

-- [ Page 2 ] --

We recognized from the beginning that the AO’s problem is not simply goal-driven in that the problem state changes without the intervention of the person. Some part of what the AO does is clearly an event-driven process. The issue is not event-driven versus goal-driven, but what combination of the two controls the AO's problem-solving behavior and how this combination can be represented in a cognitively plausible manner.

As we were struggling with the goal-driven versus event-driven interpretations, we were having problems supporting another early idea – localizing a target as a diagnostic process. The literature suggests that an important first stage in solving many tasks is diagnosing the problem.

This stage ends when the correct schema is selected (VanLehn, 1989).

We fully expected that the first stage of localizing the target would entail diagnosing the situation to determine the correct target-finding schema. Indeed, for about the first 18 months, our encodings had a goal called IDENTIFY-POSSIBLE-INITIAL-SCHEMA. Believing that the “absence of evidence is not evidence of absence,” we struggled mightily to find support for a diagnostic process. However, we could not find evidence in our protocols or elsewhere (e.g., from any of the other AOs and experts whom we consulted) that we could interpret as evidence that schema selection required extended deliberation. Our tenacity in clinging to the notion of schema selection is, in part, attributable to evidence that suggested that AOs used different schemas when the target was a submarine than when the target was a merchant. Unfortunately, this

–  –  –

In brief, our analyses indicate that when LOCATE-MERCHANT was the goal, fewer subgoals were pushed and popped than when LOCATE-SUBMARINE was the goal (see the left side of Figure 3). More important, most of the subgoals for LOCATE-MERCHANT were similar to, but qualitatively different from, those for LOCATE-SUBMARINE. As discussed later, what turned out to be bogus was the apparent qualitative difference between similar subgoals for different goals. Once this spurious qualitative difference was eliminated, we concluded that the same LOCALIZE-THE-TARGET schema guided the processing of localizing either a hostile submarine or a friendly merchant. This conclusion converged with our failure to find any evidence of schema selection. We currently believe that schema selection is not a problem. Whether the goal is LOCATE-SUBMARINE or LOCATE-MERCHANT, our expert AOs have one schema, LOCALIZE-THE-TARGET, that is automatically chosen.

Schema-directed Problem Solving Conceptualizing the AOs’ task as schema instantiation, not selection, was the key to a parsimonious account of the general structure of the AOs’ cognitive processes. Localizing the target was not driven by goals or events, but was directed by a schema. The data gathered were used to instantiate attributes of this schema. On each cycle of problem solving, the schema was reevaluated, the currently most critical attribute-value pair was identified, and instantiating this pair became the current subgoal. New events were incorporated into the schema and affect problem solving only to the extent that they affect the identification of the currently most critical

–  –  –

Figure 3: Mean number of subgoals per goal (left two) and mean number of operators per subgoal (right two) for LOCALIZE-MERC and LOCALIZE-SUB. Also shown are the 95% confidence intervals for the standard error of the mean.

This view of schema instantiation as schema-directed problem solving fits in nicely with our emerging awareness of shallow subgoaling. Each time the AO returned to the top-level goal (LOCATE-SUBMARINE in Figure 2), the schema was reevaluated. The subgoal chosen was one that would return information regarding one attribute-value pair. Hence, localizing a target is a wide and shallow task. The width is represented by a well-learned schema. The shallowness is represented by shallow subgoaling. Localizing the target involves dozens of iterations of schema evaluation and shallow subgoaling. These iterations continue until the AO has confidence that the

–  –  –

Data Analysis: Levels of Analysis and Limits to the Data Changes in our taxonomy of goals, subgoals, and operators were driven by three sources: our changing understanding of the general structure of AO cognition (discussed earlier), our failure to understand the limits to our data (discussed next), and our tendency to embed specifics of interacting with the simulation in our analysis of the task – that is, a failure to distinguish between task and artifact (discussed later).

Distinctions Not Supported Our final encodings resulted in nine categories of operators (Gray, Kirschenbaum, & Ehret, 1997a; Kirschenbaum, Gray, & Ehret, 1997). Each of these categories can be considered as representing many subcategories. For example, at one time, we tried to distinguish subcategories of things that could be queried and sources from which information could be received. An AO could query a display, the own ship operator5, his own memory, an instrument reading, and so on. Likewise, information could be received from long-term memory, from short-term memory, from reading a table, from viewing a graphic, from inferring a relationship among other information, from the own ship operator, and so on. At one time or another, each of these subcategories was considered – if only briefly – as a candidate for inclusion in the analysis.

In deriving our encoding categories, we focused on two transcripts, one from each of two AOs.

New encoding categories could be proposed by any of our three encoders. Typically, the encoder would present a rationale for why the subcategory was needed, along with a particular instance from the transcript that the encoder felt represented that subcategory. If the other two encoders were convinced by the instance, much discussion would be spent defining criteria by which other

–  –  –

two sample transcripts. New instances of the encoding category would be identified and debated until consensus was achieved. After this, two or three of the encoders would independently go through the other sample transcript in an attempt to identify instances of the new category.

More often than not, each independent encoding would yield a handful of instances (out of about 300 encoded operators) that could be considered members of the new category. However, there would be little or no agreement about which encodings were exemplars of the new category and which were not. At this point, the candidate would be abandoned. We considered it a plausible distinction, but a distinction for which the data were at too coarse of a level to support.

Problem Solving the Tool as Opposed to Problem Solving the Task We collected data from AOs who localized targets presented on a dynamic simulation of the ocean environment – CSEAL (for more information, see Gray et al., 1997a; or Kirschenbaum et al., 1997). However, our task was not to describe how AOs’ used the simulation (the artifact or tool) to localize the target (the task), but to separate the specifics of the simulation from the more general aspects of their problem-solving process (i.e., a functional level of analysis). Time and again, we found ourselves encoding specifics of using the simulator to localize targets rather than simply localizing targets.

We attempted to encode every segment of the transcript, but realized early on that a significant number of segments simply had nothing to do with the problem. The fact that many of the segments in a verbal protocol have nothing to do with the problem being solved is not news and, indeed, we expected this. These NAs (not applicable) were easy to identify because

–  –  –

weather, the room in which the simulation was run, and so on. Of the 2,882 segments encoded, 946 were in this NA category.

More difficult to distinguish were the segments that did not refer to solving the problem but to some aspect of the simulation. An analogy to this is talking to a co-author about how to format a table in Microsoft Word™ as opposed to discussing the data that would go into the table. A simple example is represented by the following query from the AO to the own ship operator asking about the units (relative or true) in which a particular display represented the data.

–  –  –

At some point, we realized that what we were seeing was a ubiquitous phenomenon that was not limited to our paradigm, but that represented a type of usability issue. We saw this as problem solving in the tool space, as opposed to the task space, and proposed the tool:task ratio as a usability metric (Kirschenbaum, Gray, Ehret & Miller, 1996). Following this insight, our transcripts were recoded with a new operator, instrumentation. As per the prior example, most of the instrumentation operators were found in groups of one to three amid a series of task operators. These small groups of instrumentation operators are essentially asides that are embedded among operators concerned with a particular task goal. Occasionally, both the AO and the own ship operator abandoned the task of localizing the target and became engaged in an episode of collaborative problem solving – attempting to figure out how to get the simulation to take a particular input or to display a particular type of data. Such episodes were recoded as a

–  –  –

Over our entire set of encoded data, 421 goals and 2,882 operators were required to encode the nine scenarios from our six AOs (for more details, see Gray, Kirschenbaum, et al., 1997 or Kirschenbaum et al., 1997). After removing all supervisory goals, their operators, all instrumentation operators, and all NA operators, we were left with 397 goals and 1,269 operators. We refer to this remainder as our clean set. The clean set of encodings was used in all subsequent analyses.

The significance of this reduction cannot be overstated; over half of our encoded utterances had nothing to do with localizing per se. Having these in the analyses confounded our efforts to make sense of the data. Once these were removed, regularities that had been obscured became apparent. For example, the scenarios we studied involved two targets: a hostile submarine and a friendly merchant. Given the shallow subgoaling we were beginning to believe in, it made sense to us that there would be more subgoals involved in localizing the quiet submarine than in localizing the noisy merchant. Indeed, this is what our encodings suggested (see the left two data points in Figure 3). However, the same subgoal seemed to involve many more steps if its supergoal was LOCATE-SUBMARINE rather than LOCATE-MERCHANT.

More steps were needed for a subgoal such as determine the signal-to-noise ratio (determineSNR) when determine-SNR was a subgoal of LOCATE-SUBMARINE than when it was a subgoal of LOCATE-MERCHANT, and this did not seem unreasonable. Indeed, as discussed earlier, such differences supported our belief that different schemas were used for different targets. By inference, this finding supported the belief that schema selection was an important

–  –  –

However, once the instrumentation operators and supervisory goals were removed, regularities appeared. As shown in the right two data points of Figure 3, the subgoals used in LOCATEMERCHANT required the same number of operators as the subgoal of LOCATESUBMARINE. (An extended discussion of this point is provided in both Gray et al., 1997a; and Kirschenbaum et al., 1997.) Completed versus Successful Goals Another part of our struggle to define the number of levels and depth of the goal stack resulted from an implicit assumption that a completed goal was synonymous with a successful goal. In our initial attempts to shoehorn reality, we viewed a completed goal as one that returned the information queried. For example, if the AO queried the bearing rate on a particular target, this goal would not be completed until the target’s bearing rate was determined. Attempting to trace the path from initial query to completion led us to postulate a tangled web of semi-infinite subgoaling. Stepping back and listening to the data led to a different conclusion. We discovered that, for the AO, knowing that, at a given point in time under current conditions, the bearing rate (or course, speed, etc.) cannot be determined is an important and complete piece of information.

The insight that a goal can be considered completed without being considered successful supports the shallow subgoaling component of schema-directed problem-solving. The AO launches a continuing stream of short queries. Each query returns some information. In the early stages of localizing (immediately after the target has been detected), the information typically is something such as the data are too noisy to answer that question (the SNR is literally too low).

–  –  –

Summary of Issues Understanding the control structure of the AOs’ cognition, together with understanding the nature and limits of the data, were major obstacles in our attempts to do a cognitive analysis of the AOs’ task. Before beginning this project, we knew that understanding the control structure of cognition would be the key to the cognitive task analysis (this concern was reflected in the original proposal to ONR).

At each iteration around the loop (see Figure 1), each component of our analysis seemed plausible. What troubled us were our efforts to fit the parts together into a coherent whole.

Although during this period we were not building ACT-R models, we were constantly asking ourselves how the disparate parts could fit into an ACT-R model. It was the failure to answer this question positively that kept driving us around the loop and deeper into the data.

At this point, we have exited the loop (Figure 1) and moved onto the next phase of the project. Although the current hypothesis – schema-directed problem solving with shallow and adaptive subgoaling – is coherent, we are collecting additional data in the hopes of capturing finer grained data on key aspects of AO problem solving in a dynamic environment.

Pages:     | 1 || 3 |

Similar works:

«Research Report Disability Living Allowance and work: Exploratory research and evidence review by Andrew Thomas and Rita Griffiths Department for Work and Pensions Research Report No 648 Disability Living Allowance and work: Exploratory research and evidence review Andrew Thomas and Rita Griffiths A report of research carried out by Insite Research and Consulting on behalf of the Department for Work and Pensions © Crown Copyright 2010. Published for the Department for Work and Pensions under...»

«A Look At Sponsor-Backed Going Private Transactions May 2013 Table of Contents Introduction Research Methodology United States Key Conclusions New and Noteworthy Highlights of 2012 Europe Key Conclusions Highlights of 2012 Asia-Pacific Key Conclusions Highlights of 2012 Representative Global Private Equity Partners About Weil Introduction Welcome to our sixth survey of sponsor-backed going private transactions prepared by Weil, Gotshal & Manges LLP. We hope that you will find this information...»

«Unsolvable Problems Outline for Today Recap from Last Time ● Wow, we covered a lot. Where are we again? ● R and RE Languages ● What does it mean to solve a problem? ● Encodings ● Computing over general objects. ● The Universal Turing Machine ● One machine to run them all. ● Impossible Problems ● Discovering truly impossible problems. ● Recap from Last Time What problems can we solve with a computer? What kind of computer? The Church-Turing Thesis claims that every effective...»

«www.kidology.org/opensource/           Kidology Open Source Curriculum is a teaching tool you download, remix, and share with others. It consists of three 1-year programs: Old Testament Chronological Study, New Testament Chronological Study, and Value Keepers of the Bible. Originally written by Susan Harper of Caring Hands Ministries, this comprehensive Bible curriculum was provided to share and distribute on Kidology.org. This curriculum is made available as is and may contain legacy...»

«Defacing the map: Cartographic vandalism in the digital commons arXiv:1404.3341v1 [cs.CY] 13 Apr 2014 Andrea Ballatore Department of Computer Science National University of Ireland, Maynooth andrea.ballatore@nuim.ie Author copy. Cartographic Journal, 2014. Abstract This article addresses the emergent phenomenon of carto-vandalism, the intentional defacement of collaborative cartographic digital artefacts in the context of volunteered geographic information. Through a qualitative analysis of...»

«HEADLINES July 2016 Dear Parents and Friends One of the best parts of my job (which incidentally is the best job in the world!) is spending time with the girls. Some highlights during this term have been travelling to Wellington to watch the girls perform as part of the Sheilah Winn Shakespeare Festival and spending time with The Bai Trio as they enjoyed a weekend chamber music master class at the New Zealand Music School at Victoria University. Another proud principal moment was sharing in the...»

«CBI Product Factsheet: Tea in Poland CBI | Market Intelligence Product Factsheet Cloves in Germany | 1 Introduction Poland has amongst the highest per capita tea-consumption rates in Europe. Although Polish consumers regard tea (particularly black tea) as a necessity, consumer preferences are changing. As mainstream black and green teas are losing market share, demand is increasing for new premium tea products (e.g. teas with distinctive fruit and herbal flavours and high-quality speciality...»

«2009 2014 EUROPEAN PARLIAMENT Committee on Budgetary Control 2010/0807(NLE) 25.2.2010 DRAFT REPORT on the nomination of Szabolcs Fazakas as a Member of the Court of Auditors (C7-0019/2010 – 2010/0807(NLE)) Committee on Budgetary Control Rapporteur: Inés Ayala Sender PR\805785EN.doc PE431.020v02-00 EN EN United in diversity PR_NLE_art108 CONTENTS Page PROPOSAL FOR A EUROPEAN PARLIAMENT DECISION ANNEX 1: CURRICULUM VITAE OF SZABOLCS FAZAKAS ANNEX 2: ANSWERS BY SZABOLCS FAZAKAS TO THE...»

«Papakura Anglican to worship, serve and love like Jesus
 About Us Thanks for joining us here at Papakura Anglican. This booklet tells a little about our story: our ministry and mission for 2014. ! Hopefully you’ll find within this something that inspires you. When you do, get in touch with me and I can guide you through what the next steps are to get involved. ! In Christ’s service together, Dion. ! Introduction Angela, Daniel, Caitlin, Annalia, Isaac and I have been here 8years now. We...»

«1. Editorial 2. Interview with Dr Santrupt Misra Editors 3. A New Look at Performance Management System Dr T V Rao 4. Prerequisites for an Effective Performance Management System Devika Dharmaraj 5. On Making the PMS Count R Shantaram 6. Miles to go before we sleep Nandini Chawla 7. EFT Performance Coaching David Childerley 8. Performance Appraisal 360 Degree way Marcel Parker 9. Performance Oriented Culture The Covansys Story Sruthi S and Poornima C V 10. The Performance Management Saga The...»

«Hirar Live At State: {o}arin Amirka A Yankin Saharan Jawabin: Linda Thomas-Greenfield, Mataimakiyar Sakatariyar Ofishin Harkokin Afrika MS. JENSEN: Muna ma ku maraba, a zauren hirar LiveAtState, yanar-gizon gwamnatin Amirka, domin gudanar da hira da manema labarun }asashen duniya. Ina mai farin cikin marharbin da wa]anda suka halarci wannan hirar, daga sauran ~angarorin nahiyar Afrika, a yau. Ina kuma son aikewa da sa}o na musamman, ga wa]anda ke tare da mu, daga }asashen Jumhuriyar...»

«Allen Watson’s Commentary on the Text of A Course in Miracles Study Guide and Commentary ACIM® Text, Chapter 9, Section VIII Grandeur versus Grandiosity Legend: blue text = Material from ACIM 3rd edition (FIP) bold blue text = words emphasized in all caps in Urtext red text = alternate or omitted material from the Urtext light blue text = editorial comments strikethrough blue text = Not in Urtext, in FIP edition The previous section spoke of how “lofty” our true Self is (T-9.VII.8:4),...»

<<  HOME   |    CONTACTS
2017 www.thesis.dislib.info - Online materials, documents

Materials of this site are available for review, all rights belong to their respective owners.
If you do not agree with the fact that your material is placed on this site, please, email us, we will within 1-2 business days delete him.