Producing wrong data without doing anything obviously wrong (2009)
Cached
Download Links
| Venue: | In Proc. of Int’l Conf. on Architectural Support for Programming Languages and Operating Systems |
| Citations: | 84 - 7 self |
BibTeX
@INPROCEEDINGS{Mytkowicz09producingwrong,
author = {Todd Mytkowicz and Amer Diwan and Matthias Hauswirth and Peter F. Sweeney},
title = {Producing wrong data without doing anything obviously wrong},
booktitle = {In Proc. of Int’l Conf. on Architectural Support for Programming Languages and Operating Systems},
year = {2009},
pages = {265--276},
publisher = {ACM}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
This paper presents a surprising result: changing a seemingly innocuous aspect of an experimental setup can cause a systems researcher to draw wrong conclusions from an experiment. What appears to be an innocuous aspect in the experimental setup may in fact introduce a significant bias in an evaluation. This phenomenon is called measurement bias in the natural and social sciences. Our results demonstrate that measurement bias is significant and commonplace in computer system evaluation. By significant we mean that measurement bias can lead to a performance analysis that either over-states an effect or even yields an incorrect conclusion. By commonplace we mean that measurement bias occurs in all architectures that we tried (Pentium 4, Core 2, and m5 O3CPU), both compilers that we tried (gcc and Intel’s C compiler), and most of the SPEC CPU2006 C programs. Thus, we cannot ignore measurement bias. Nevertheless, in a literature survey of 133 recent papers from ASPLOS, PACT, PLDI, and CGO, we determined that none of the papers with experimental results adequately consider measurement bias. Inspired by similar problems and their solutions in other sciences, we describe and demonstrate two methods, one for detecting (causal analysis) and one for avoiding (setup randomization) measurement bias.
Keyphrases
measurement bias wrong data experimental setup innocuous aspect setup randomization wrong conclusion social science computer system evaluation literature survey incorrect conclusion causal analysis spec cpu2006 program performance analysis surprising result similar problem experimental result recent paper significant bias system researcher
