Producing wrong data without doing anything obviously wrong

Producing wrong data without doing anything obviously wrong (2009)

Cached

Download Links

by Todd Mytkowicz , Amer Diwan , Matthias Hauswirth , Peter F. Sweeney

Venue:	In Proc. of Int’l Conf. on Architectural Support for Programming Languages and Operating Systems
Citations:	25 - 3 self

Versions

Version History

Metadata Version 1

Datum	Value	Source
TITLE	Producing wrong data without doing anything obviously wrong	INFERENCE
AUTHOR NAME	Todd Mytkowicz	SVM HeaderParse 0.2
AUTHOR AFFIL	Department of Computer Science; University of Colorado	SVM HeaderParse 0.2
AUTHOR ADDR	Boulder, CO, USA	SVM HeaderParse 0.2
AUTHOR NAME	Amer Diwan	SVM HeaderParse 0.2
AUTHOR AFFIL	Department of Computer Science; University of Colorado	SVM HeaderParse 0.2
AUTHOR ADDR	Boulder, CO, USA	SVM HeaderParse 0.2
AUTHOR NAME	Matthias Hauswirth	SVM HeaderParse 0.2
AUTHOR AFFIL	Faculty of Informatics; University of Lugano	SVM HeaderParse 0.2
AUTHOR ADDR	Lugano, CH	SVM HeaderParse 0.2
AUTHOR NAME	Peter F. Sweeney	SVM HeaderParse 0.2
AUTHOR AFFIL	IBM Research	SVM HeaderParse 0.2
AUTHOR ADDR	Hawthorne, NY, USA	SVM HeaderParse 0.2
ABSTRACT	This paper presents a surprising result: changing a seemingly innocuous aspect of an experimental setup can cause a systems researcher to draw wrong conclusions from an experiment. What appears to be an innocuous aspect in the experimental setup may in fact introduce a significant bias in an evaluation. This phenomenon is called measurement bias in the natural and social sciences. Our results demonstrate that measurement bias is significant and commonplace in computer system evaluation. By significant we mean that measurement bias can lead to a performance analysis that either over-states an effect or even yields an incorrect conclusion. By commonplace we mean that measurement bias occurs in all architectures that we tried (Pentium 4, Core 2, and m5 O3CPU), both compilers that we tried (gcc and Intel’s C compiler), and most of the SPEC CPU2006 C programs. Thus, we cannot ignore measurement bias. Nevertheless, in a literature survey of 133 recent papers from ASPLOS, PACT, PLDI, and CGO, we determined that none of the papers with experimental results adequately consider measurement bias. Inspired by similar problems and their solutions in other sciences, we describe and demonstrate two methods, one for detecting (causal analysis) and one for avoiding (setup randomization) measurement bias.	SVM HeaderParse 0.2
YEAR	2009	INFERENCE
VENUE	In Proc. of Int’l Conf. on Architectural Support for Programming Languages and Operating Systems	INFERENCE
VENUE TYPE	CONFERENCE	INFERENCE
PAGES	265--276	INFERENCE
CITATIONS	18 found	ParsCit 1.0