M. Seltzer, D. Krinsky, K. Smith, X. Zhang, The Case for Application-Specific Benchmarking, Proceedings of the The Seventh Workshop on Hot Topics in Operating Systems, 1998.  Home/Search   Document Details and Download   Summary   Related Articles   Check

....chapters. While effort has been taken to choose a wider and interesting range of benchmarks, there are some deficiencies that hamper generalization. First, there are not enough benchmark. As Seltzer et al. points out in applicationspecific benchmarks are much more informative than micro benchmarks [66]. 110 However, the very specificity of a particular application makes future predictions difficult unless one is lucky enough to hit on a close or perfect match. Nonetheless, with a suitably large suite of application and an algorithm which clearly does not unfairly take advantage of some quirk ....

Margo Seltzer. The case for application-specific benchmarking. In Proceedings of the Workshop on Hot Topics in Operating Systems (HotOS VII), March 1999.

....these two challenges are difficult ones, we believe that they can be overcome, and we will now outline our ideas on how to do so. We believe that the first challenge, that of the dependence of maintainability on workload, can be addressed by applying the ideas of application specific benchmarking [42]. The idea behind application specific benchmarking is to characterize the system of interest along as many different axes as possible, producing a large set of results that capture all of the fundamental behaviors of the system and any important interactions between the behaviors. Once this ....

....of the entire performance space, and do so with a static workload representative of only a single environment. While this limiting of scope has led to a lot of debate about the validity of performance benchmarks, and to proposals to restore environment application specificity to benchmarks [42], it has clearly simplified the benchmarking process and allowed performance benchmarks to become pervasive. As long as humans are involved in the experiments, we will never be able to achieve the same level of ubiquity with maintainability benchmarks as we see with performance benchmarks. ....

[Article contains additional citation context not shown here]

M. Seltzer, D. Krinsky, et al. The Case for Application-Specific Benchmarking. Proceedings of the 1999.

.... to the abstraction process, basically consisting of modeling the execution of the benchmarked system as a mixture of primitive operations, whose execution times combined yield the execution times of the benchmarked system, was suggested elsewhere, mostly in the operating systems research domain [22][23] 24] Two basic questions that need to be answered when using this approach are, what should be the primitive operations used, and what is the usage of a particular primitive operation by the benchmarked system. Typically, these were answered through analysis of the system calls or machine ....

Seltzer, M., Krinsky, D., Smith, K., Zhang, X., The Case for Application-Specific Benchmarking, Proceedings of the Seventh Workshop on Hot Topics in Operating Systems, 1998

....are easily correlated with implementation features. Brown and Seltzer have pointed out, however, that creating a reproducible and statistically valid microbenchmark is not as trivial as it seems [5] Perhaps more important, I know of no systematic work (except for prelimary work by Seltzer et al. [19]) that has measured the correlation between an operating system s microbenchmark results and that system s performance for complex applications. In other words, the predictive value of microbenchmarks is questionable. Many of the file systems benchmarks are also, in effect, sets of ....

....(especially electronic commerce) use mostly dynamic content, which is far less efficient. The ability to predict the performance of servers generating dynamic content would be a major victory, but today it s not even a well defined problem. Can we 4 Margo Seltzer has students working on this [19], although I believe their approach discounts the need for environmental realism. characterize the performance requirements of a typical commerce site There is often a tension between realism and reproducibility; real world effects can make it hard to repeat experiments exactly. We should ....

M. Seltzer, D. Krinsky, K. Smith, and X. Zhang. The Case for Application-Specific Benchmarking. In Proc. HotOS-VII. Rio Rico, AZ, Mar., 1999.

....amount of bytes transferred) can easily lead to queue overflow and memory starvation [6] 3. When constructing synthetic workloads for use in file system design and benchmarking we need to ensure that the infinite variance characteristics are properly modeled in the file system test patterns. In [22], Seltzer et al. argue for application specific file system benchmarking, which already allows more focused testing, but for each test application we need to ensure that the input parameters from the file system under test and the ON OFF activity pattern of the application is modeled after the ....

Seltzer, Margo, David Krinsky, Keith Smith and Xiaolan Zhang, "The Case for Application-Specific Benchmarking", in Proceedings of the 1999 Workshop on Hot Topics in Operating Systems, Rico, AZ, 1999

....must be fetched from disk. In this sense, our work is more closely related to the studies of read ahead [23] and the heuristics used by FFS [13] 3 Benchmarking File Systems and I O Performance There has been much work in the development of accurate workload and micro benchmarks for file systems [22, 24]. There exists a bewildering variety of benchmarks, and there have been calls for still more [16, 19, 27] Nearly all benchmarks can be loosely grouped into one of two categories: micro benchmarks, such as lmbench [14] or bonnie [1] which measure specific lowlevel aspects of system performance ....

Margo I. Seltzer, David Krinsky, Keith A. Smith, and Xiaolan Zhang. The Case for Application-Specific Benchmarking. In Workshop on Hot Topics in Operating Systems, pages 102--107, 1999.

.... Designers of file systems and network storage devices should embrace this philosophy of application specific design, as has already happened with database and WWW servers, rather than adhering to the specious ideal that monolithic file systems can perform well under widely divergent workloads [7]. 2. Antiquated and inefficient file system usage can create an enormous load. Many applications use the file system in antiquated and arguably inappropriate ways. The use of lock files to provide resource locking in a distributed environment is a particularly widespread example. It is ....

Margo I. Seltzer, David Krinsky, Keith A. Smith, and Xiaolan Zhang. The case for application-specific bench- marking. In Workshop on Hot Topics in Operating Systems, pages 102-107, 1999.

....regarding which JVM is the best for the application of interest. In comparison, HBench:Java is a general benchmarking framework that can be applied to any specific workload. 3. HBENCH:JAVA DESIGN 3. 1 Overview HBench:Java is based on the vector based methodology of the HBench framework [14]. The principle behind the vector based methodology is the observation that a system s performance is determined by the performance of the individual primitive operations that it supports, and that an application s performance is determined by how much it utilizes the primitive operations of the ....

Seltzer, M., Krinsky, D., Smith, K., and Zhang X. The Case for Application-Specific Benchmarking. In Proceedings of the 1999 Workshop on Hot Topics in Operating Systems (HotOS VII), pages 102-107, Rio Rico, AZ, March 29-30, 1999.

No context found.

M. Seltzer, D. Krinsky, K. Smith, X. Zhang, The Case for Application-Specific Benchmarking, Proceedings of the The Seventh Workshop on Hot Topics in Operating Systems, 1998.

No context found.

Seltzer, M., Krinsky, D., Smith, K., Zhang, X., The Case for Application-Specific Benchmarking, Proceedings of the Seventh Workshop on Hot Topics in Operating Systems, 1998

No context found.

Margo Seltzer, David Krinsky, Keith Smith, and Xialan Zhang. The case for application-specific benchmarking. In The 7th Workshop of Hot Topics in Operating Systems, Rio Rico, Arizona, March 1999. IEEE.

No context found.

Margo Seltzer,David Krinsky, Keith Smith, and Xiolan Zhang, "The case for application-specific benchmarking"inProceedings of the 1999.

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC