• Summary
• Related Documents
  • Active Bibliography
  • Co-citation
• Version History

Abstract:

Carrying out empirical studies is slowly becoming widely held to be of importance by the software engineering community. A view perhaps less widely held is that experiments should be replicated externally to both verify and generalise the original results. This paper serves a number of purposes. The need for external replications is established and the role of replication in experimental software engineering is discussed. Without the con rming power of external replications, results in experimental software engineering should only provisionally be accepted, if at all. The paper then draws heavily on the authors ' experiences in externally replicating three software engineering experiments ([Kor86, PVB95, KL96])to provide guidance on three, relatively neglected areas: the improving of experimental `recipes ' during replication to either focus or generalise results, the use of alternative data analysis techniques, speci cally rule induction, to seek alternative explanations when the results of replications di er, and packaging of experiments for replication. The facilitation of replication through properly constructed replication packages places formidable but largely unrewarding demands on the original experimenter. Yet without appropriately constructed and documented replication packages systematic and cumulative empirical investigation will be di cult to achieve. It is demonstrated that a modest generalisation of the characterisation scheme, or framework, proposed by Lott and Rombach [LR96], will serve well as the basis for the reporting, packaging and recipe improvement of software engineering experiments for the purposes of external replication. It also demonstrates the insights that can be achieved from seeking explanations beyond those provided by purely statistical analyses. 1

Citations