Statistically rigorous Java performance evaluation

DMCA

Statistically rigorous Java performance evaluation (2007)

Cached

Download Links

Other Repositories/Bibliography

DBLP

by Andy Georges , Dries Buytaert , Lieven Eeckhout

Venue:	In Proceedings of the ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA
Citations:	107 - 5 self

BibTeX

@INPROCEEDINGS{Georges07statisticallyrigorous,
    author = {Andy Georges and Dries Buytaert and Lieven Eeckhout},
    title = {Statistically rigorous Java performance evaluation},
    booktitle = {In Proceedings of the ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA},
    year = {2007},
    pages = {57--76},
    publisher = {ACM}
}

OpenURL

Abstract

Java performance is far from being trivial to benchmark because it is affected by various factors such as the Java application, its input, the virtual machine, the garbage collector, the heap size, etc. In addition, non-determinism at run-time causes the execution time of a Java program to differ from run to run. There are a number of sources of non-determinism such as Just-In-Time (JIT) compilation and optimization in the virtual machine (VM) driven by timerbased method sampling, thread scheduling, garbage collection, and various system effects. There exist a wide variety of Java performance evaluation methodologies used by researchers and benchmarkers. These methodologies differ from each other in a number of ways. Some report average performance over a number of runs of the same experiment; others report the best or second best performance observed; yet others report the worst. Some iterate the benchmark multiple times within a single VM invocation; others consider multiple VM invocations and iterate a single benchmark execution; yet others consider multiple VM invocations and iterate the benchmark multiple times. This paper shows that prevalent methodologies can be misleading, and can even lead to incorrect conclusions. The reason is that the data analysis is not statistically rigorous. In this paper, we present a survey of existing Java performance evaluation methodologies and discuss the importance of statistically rigorous data analysis for dealing with non-determinism. We advocate approaches to quantify startup as well as steady-state performance, and, in addition, we provide the JavaStats software to automatically obtain performance numbers in a rigorous manner. Although this paper focuses on Java performance evaluation, many of the issues addressed in this paper also apply to other programming languages and systems that build on a managed runtime system.

Keyphrases

rigorous java performance evaluation virtual machine benchmark multiple time multiple vm invocation java performance evaluation methodology thread scheduling java performance evaluation wide variety report average performance various system effect garbage collection programming language data analysis performance number incorrect conclusion rigorous manner execution time heap size java program timerbased method sampling garbage collector prevalent methodology java performance rigorous data analysis runtime system java application steady-state performance single benchmark execution various factor javastats software single vm invocation run-time cause

DMCA