The area of knowledge acquisition research concerned with the development of knowledge acquisition (KA) tools is in need of a methodological approach toevaluation. E orts such as the Sisyphus experiments have been useful to illustrate particular approaches, but have not served in practice as testbeds for comparing and evaluating di erent alternative approaches. This paper describes our experimental methodology to conduct studies and experiments of users modifying knowledge bases with KA tools. We also report the lessons learned from several experiments that we have performed. Our hope is that it will help others design or improve future user evaluations of KA tools. We found that performing these experiments is particularly hard because of di culties in controlling factors that are unrelated to the particular claims being tested. We discuss our ideas for improving our current methodology and some open issues that remain. 1
|
3215
|
C4.5: Programs for machine learning
– Quinlan
- 1993
|
|
233
|
Software Metrics - A Rigorous and Practical Approach" (2nd edition
– Fenton, Pfleeger
- 1997
|
|
126
|
Building Expert Systems
– Hayes-Roth, Waterman, et al.
- 1983
|
|
107
|
Experimentation in Software Engineering
– Basili, Selby, et al.
- 1986
|
|
102
|
Task Modeling with Reusable Problem-Solving Methods
– Eriksson, Shahar, et al.
- 1995
|
|
65
|
SALT: A Knowledge Acquisition Language for Propose-andRevise Systems
– Marcus, McDermott
- 1989
|
|
58
|
The DARPA high-performance knowledge base project
– Cohen, Schrag, et al.
- 1998
|
|
46
|
Interactive Transfer of Expertise: Acquisition of New Inference Rules
– Davis
- 1979
|
|
45
|
Explicit Representations of Problem-Solving Strategies to Support Knowledge Acquisition
– Gil, Melz
- 1996
|
|
45
|
Prototyping an Intelligent Agent through Wizard of Oz
– Maulsby, Greenberg, et al.
- 1993
|
|
44
|
Linkability: Examining causal link commitments in partial-order planning
– Veloso, Blythe
- 1994
|
|
36
|
Acquiring Domain Knowledge for Planning by Experimentation
– Gil
- 1992
|
|
29
|
Control knowledge to improve plan quality
– Perez, Carbonell
- 1994
|
|
25
|
A script-based approach to modifying knowledge-based systems
– Tallis
- 1998
|
|
19
|
Deriving expectations to guide knowledge base creation
– Kim, Gil
- 1999
|
|
19
|
The Sisyphus-VT initiative
– Schreiber
- 1996
|
|
16
|
Designing scripts to guide users in modifying knowledge-based systems
– Tallis, Gil
- 1999
|
|
15
|
An experimental evaluation of integrating machine learning with knoweldge acquisition
– Webb, Wells, et al.
- 1999
|
|
14
|
Models Exposed: An Empirical Study
– Corbridge, Major, et al.
- 1995
|
|
13
|
Knowledge acquisition in the small: Building knowledge-acquisition tools from pieces
– Runkel, Birmingham
- 1993
|
|
12
|
Knowledge re nement in a re ective architecture
– Gil
- 1994
|
|
11
|
User Studies of an Interdependency-Based Interface for Acquiring Problem-Solving Knowledge
– Kim, Gil
|
|
10
|
Partial-order planning: Evaluating possible e ciency gains
– Barrett, Weld
- 1994
|
|
9
|
A framework for investigating production system formulations with polynomially bounded match
– Tambe, Rosenbloom
- 1990
|
|
7
|
Using Multi-Strategy Learning to Improve Planning Efficiency and Quality
– Estlin
- 1998
|
|
5
|
Knowledge-acquisition tools with explicit problem-solving methods
– Birmingham, Klinker
- 1993
|
|
5
|
Special issue on evaluation
– Self
- 1993
|
|
5
|
TAQL: A Problem Space Tool for Expert System Development
– Yost
- 1992
|
|
4
|
Knowledge acquisition in Soar
– Yost
- 1993
|
|
3
|
EXPECT: Explicit representations for exible acquisition
– Swartout
- 1995
|
|
2
|
The Nature of Expertise. Lawrence Erlbaum
– Chi, Glaser
- 1988
|
|
2
|
Learning e cient rules by maintaining the explanation structure
– Rosenbloom
- 1996
|
|
2
|
Special issue on sisyphus: Models of problem solving
– Linster
- 1992
|
|
2
|
Learning as Knowledge Integration
– Murray
- 1995
|
|
1
|
A speci cation of process planning for PRODIGY
– Gil
- 1991
|
|
1
|
Dimensions to analyze applications
– Gil
- 1995
|
|
1
|
EXPECT: A Re ective Architecture for Knowledge Acquisition
– Gil
- 1994
|
|
1
|
Knowledge Acquisition for Visually Oriented Planning
– Joseph
- 1992
|
|
1
|
aips planning competition. ftp.cs.yale.edu/pub/mcdermott/aipscomp-results.html
– McDermott
- 1998
|
|
1
|
Special issue on experimental comparisions of usability evaluation methods
– Olson
- 1998
|
