Brown, Aaron B and Margo I. Seltzer. Operating system benchmarking in the wake of lmbench: A case study of the performance of NetBSD on the Intel x86 architecture. Technical report, Harvard University, 1997.  Home/Search   Document Not in Database   Summary   Related Articles   Check

....tools, and test programs. Section 5 presents our measurement results and the corresponding analysis of measured timing and scheduling services. The paper concludes with Section 6. 2 Related Work In recent years, many models have been proposed for OS and integrated ESW performance analysis [1 3, 7]. All these models suggested a modeling hierarchy with the OS performance model as a separate layer between hardware and the application, and required the measurement of OS services for model construction and performance analysis. There have also been numerous general techniques for measuring the ....

....measurement of OS services for model construction and performance analysis. There have also been numerous general techniques for measuring the different aspects of computer systems [5, 9] and bechmarks for OS level measurements, such as Rhealstone [6] Hartstone [15] lmbench [10] and hbench OS [3]. Although these techniques are useful in measuring the performance of individual operations and determining performance bottlenecks, the effects of application structures and interactions which may make significant performance differences of ESW are ignored. The information measured with ....

[Article contains additional citation context not shown here]

A. B. Brown and M. I. Seltzer. Operating system benchmarking in the wake of lmbench: A case study of the performance of netbsd on the intel x86 architecture. In Proceedings of the 1997.

....hurts the performance of ordinary applications. The second is comparing the performance of the resulting thread system with existing thread systems to demonstrate the merits of the scheduler activations approach. The first measurements were done with the HBench OS package from Harvard University [3]. HBench OS focuses on getting many individual measurements to explore the performance of a system in detail. Five of the tests measure system call latency; the results from the NetBSD kernel before and after the implementation of scheduler activations are shown here, as measured on a 500 MHz ....

A. Brown and M. Seltzer. Operating system benchmarking in the wake of lmbench: A case study of the performance of NetBSD on the Intel x86 architecture. In Proceedings of the 1997.

....systems we have compared. In Section 4, we describe in detail our experiments and what we have learned. We conclude the paper in Section 6 with a summary and an outlook on future work. 2 Related work There is a large body of work on microbenchmarking IPC or interrupt latencies (see for example [1, 12, 19]) and many worst case evaluations for real time operating systems (e.g. 23] However, we are unaware of any work that specifically addresses the worst case overhead of using address spaces in real time systems. In this paper, we concentrate on hybrid real time and time sharing operating ....

....memory. We have modified the Mbuff driver for RTLinux such that it hands out physical contiguous memory pages which where reserved using the bigphysarea mechanism [2] In DROPS, physical memory is offered by a memory server. For code coverage we use a benchmarking suite for UNIX, hbench [1]. This benchmark provides excellent coverage for Linux and, in our experience, also for L and the Fiasco microkernel, as it makes use of almost all of the Fiasco microkernel s services. We can make this statement with confidence because of Fiasco s small size and limited amount of functionality ....

A. B. Brown and M. I. Seltzer. Operating system benchmarking in the wake of lmbench: A case study of the performance of NetBSD on the Intel x86 architecture. In ACM SIGMETRICS Conference on Measurement and Modeling of Computer Systems, pages 214--224, Seattle, WA, June 1997.

....in the MESI protocol, making inclusion unnecessary [Int99] As a result, inclusion is not required on x86 CPUs. Because the cache flooding program uses more memory pages than TLB entries, it has the side effect of flushing the TLB. For code coverage I used a benchmarking suite for UNIX, hbench [BS97] This benchmark is designed to provide excellent coverage for Unix systems such as Linux and L Linux, and I have verified using a profiling tool that it also provides wide coverage for the Fiasco microkernel. As a measurable and reliable interrupt source, I have used the x86 CPU s built in ....

A. B. Brown and M. I. Seltzer. Operating system benchmarking in the wake of lmbench: A case study of the performance of NetBSD on the Intel x86 architecture. In ACM SIGMETRICS Conference on Measurement and Modeling of Computer Systems, pages 214--224, Seattle, WA, June 1997.

....problems that arise from applying throughput sensitive performance measurement techniques to interactive systems. Conventional benchmarks, whether designed to measure throughput or latency, concentrate on obtaining stable, repeatable results excluding data points that deviate wildly from the norm [4][7] 21] when in fact, these data points are exactly the ones that we need to understand in order to improve user perceived performance. To measure interactive system throughput, benchmark programs commonly saturate the system s input request stream by injecting requests into the system as quickly ....

....to releasing the product. The benchmarks most often used are sensitive to changes in system throughput rather than latency, and even in cases where the benchmarks measure latency, the program is geared toward measuring average latency, often throwing away data points that vary wildly from the norm [4][5] 21] 29] 45] These benchmarks help the developers remove performance bugs that affect system throughput but do little to help the developers identify and remove latency related performance problems, particularly those that affect latency variability. A measurement technique such as ours ....

[Article contains additional citation context not shown here]

Aaron B. Brown and Margo I. Seltzer, "Operating System Benchmarking in the Wake of Lmbench: A Case Study of the Performance of NetBSD on Intel x86 Architecture," Proceedings of the 1997.

....operating system specific (it runs on nearly every version of UNIX) has freely available source code and most importantly, is used by other Linux developers. With it we were able to compare results with Linux on other platforms and with other Linux developers. 10 11 We chose not to use HBench [3] since it offered little more than LmBench [16] and is not as widespread or accepted. We also used the informal Linux benchmark of compiling the kernel, which is a traditional measure of Linux performance. The mix of process creation, file I O, and computation in the kernel compile is a good ....

A. B. Brown and M. I. Seltzer. Operating system benchmarking in the wake of lmbench: A case study of the performance of netbsd on intel x86 architecture. In ACM SIGMETRICS Conference on Measurement and Modeling of Computer Systems, 1997.

....such as le systems, network protocols, and remote procedure call; the performance of these services constrains the performance of the application. To understand server application performance, we must understand the performance of the Java interface to system services. Several previous studies [12, 3, 14] have presented microbenchmarks as tools to help analyze performance of operating system services. In this paper, we apply this methodology to Java, and report the results. Speci cally, we describe a suite of Java microbenchmarks from The jMocha Microbenchmark Framework and Suite for Java, and ....

....and network protocols as crucial factors in OS performance. McVoy and Staelin [12] developed lmbench, a portable set of operating system benchmarks focusing on issues including memory system, IPC, cached I O, system call, signal handling, network, and process thread performance. Brown and Seltzer [3] built further on this work with hbench:OS, which improved some methodological and practical issues of lmbench. We build directly on this line of research, comparing benchmarks similar to those of hbench:OS with comparable Java versions. Other related work, pre Java, includes microbenchmarks ....

[Article contains additional citation context not shown here]

Aaron B. Brown and Margo I. Seltzer. Operating system benchmarking in the wake of lmbench: A case study of the performance of NetBSD on the Intel x86 architecture. Performance Evaluation Review, 25(1):214-224, June 1997.

....confidence that the Drops system is indeed completely preemptible as we claimed in Section 2.1. To avoid missing critical code paths because of pathologic timer synchronization, we varied the time between triggering two interrupts. For code coverage we use a benchmarking suite for Unix, hbench [1]. This benchmark provides excellent coverage for Linux and, in our experience, also for L 4 Linux and the Fiasco microkernel. As a reliable and measurable interrupt source, we have used the x86 CPU s built in interrupt controller (Local APIC 3 ) This unit o#ers a timer interrupt that can be ....

A. B. Brown and M. I. Seltzer. Operating system benchmarking in the wake of lmbench: A case study of the performance of NetBSD on the Intel x86 architecture. In ACM SIGMETRICS Conference on Measurement and Modeling of Computer Systems, pages 214--224, Seattle, WA, June 1997.

....high performance database software mostly avoids the operating system. 2.2.2. Research benchmarks Researchers and experimenters have developed a number of benchmarks, for various topic areas, including: Kernel microbenchmark suites: the Ousterhout benchmarks [17] lmbench [15] hbench:OS [5]. File systems: the Modified Andrew Benchmark (MAB) 17] Chen and Patterson s self scaling benchmark [8] Bonnie, IObench, IOZone. Network throughput and latency: netperf [14] ttcp. Microbenchmarks are widely employed because they are easy to use, they provide simple numeric results ....

....they provide simple numeric results (seldom requiring statistical analysis) and because their results are easily correlated with implementation features. Brown and Seltzer have pointed out, however, that creating a reproducible and statistically valid microbenchmark is not as trivial as it seems [5]. Perhaps more important, I know of no systematic work (except for prelimary work by Seltzer et al. 19] that has measured the correlation between an operating system s microbenchmark results and that system s performance for complex applications. In other words, the predictive value of ....

A. Brown and M. Seltzer. Operating System Benchmarking in the Wake of Lmbench: A Case Study of the Performance of NetBSD on the Intel x86 Architecture. In Proc. 1997 ACM SIGMETRICS Conf. on Measurement and Modeling of Computer Systems, pp. 214-224. Seattle, WA, June, 1997.

.... of operating system performance improvements to track performance improvements in hardware technology [142] More recently, efforts have focused on developing a suite of portable operating system benchmarks for cross platform performance comparisons [115] as well as detailed system analysis [24]. In all these efforts, however, the performance profiling is geared towards performance comparisons and the impact of OS hardware interactions on operating system primitives. As such, there is no need to equip the operating system with information regarding the performance of individual ....

....overhead, but also limits the resolution of the real time clock. For example, for our platform timestamps using the real time clock cost 15 s. While this overhead is relatively high, it can be reduced drastically by using hardware performance counters provided in several modern processors [24]. Hardware cycle counters provide timestamps with resolutions of the order of just a few nanoseconds. The kernel level profiling described in Chapter 7 is performed using such hardware counters. For platforms with significant timestamp cost, the generated profile sample must be adjusted ....

[Article contains additional citation context not shown here]

A. Brown and M. Seltzer, "Operating system benchmarking in the wake of lmbench: A case study of the performance of NetBSD on the Intel x86 architecture," in Proc. of ACM SIGMETRICS, pp. 214--224, June 1997.

.... improvements to track performance improvements in hardware technology [30] More recently, efforts have focused on developing a suite of portable operating system benchmarks for cross platform performance comparisons, such as lmbench [13] as well as for detailed system analysis, such as hbench:OS [31]. In all these efforts, however, the performance profiling is geared towards performance comparisons and the impact of OS hardware interactions on operating system primitives. As such, there is no need to equip the operating system with information regarding the performance of individual ....

....overhead, but also limits the resolution of the real time clock. For example, for our platform timestamps using the real time clock cost 15 s. While this overhead is relatively high, it can be reduced drastically by using hardware performance counters provided in several modern processors [31]. Hardware cycle counters provide timestamps with resolutions of the order of just a few nanoseconds. The kernel level profiling described in [39] is performed using such hardware counters. For platforms with significant timestamp cost, the generated profile sample must be adjusted appropriately. ....

[Article contains additional citation context not shown here]

Aaron Brown and Margo Seltzer, "Operating system benchmarking in the wake of lmbench: A case study of the performance of NetBSD on the Intel x86 architecture," in Proc. of ACM SIGMETRICS, June 1997, pp. 214--224.

.... Given that for each URL miss the proxy should create and delete a file, the proxy will only be able to serve one URL miss every 60 milliseconds, or equivalently, less than 1 To quantify performance limitations of file system operations, we experimented with the HBENCH OS file system benchmark [6]. HBENCH OS evaluates file creation deletion cost by creating 64 files of a given size in the same directory and then deleting them in reverseof creation order. We run HBENCH OS on an UltraSparc 1 running Solaris 5.6, for file sizes of 0 to 10 Kbytes. Our results, plotted in Figure 2, suggest ....

A. Brown and M. Seltzer. Operating System Benchmarking in the Wake of Lmbench: A Case Study of the Performance of NetBSD on the Intel x86 Architecture. In Proc. of the 1997 ACM SIGMETRICS Conference, pages 214--224, 1997.

....calls, context switches, memory accesses, and pipe operations. This benchmark has been developed to compare different hardware from the operating system s perspective and therefore also includes a variety of OS independent benchmarks, in particular measuring the hardware memory system and the disk [3]. Since we always use the same hardware for our experiments, we only present selected operating system dependent parts. The hardware related measurements indeed reported the same results on all systems. Figure 4 shows a chart of slowdowns for various operations for monolithic Linux, RT Linux, L ....

A. B. Brown and M. I. Seltzer. Operating system benchmarking in the wake of lmbench: A case study of the performance of NetBSD on the Intel x86 architecture. In ACM SIGMETRICS Conference on Measurement and Modeling of Computer Systems, pages 214--224, Seattle, WA, June 1997.

....an earlier version of Panda without this scatter gather interface, Manta achieved a maximum throughput of only 20.6 MByte s. The difference is due to two extra memory copies that the earlier version needed for serialization and deserialization. Since memory copies are expensive on a Pentium Pro [Brown and Seltzer 1997], they decrease throughput significantly. Unfortunately, dereferencing the scatter gather vector involves extra processing, so the null latency of the current Manta RMI system is slightly higher than that for the earlier system (34 versus 37 s) Maassen et al. 1999] The throughput for Sun ....

BROWN, A. AND SELTZER, M. 1997. Operating System Benchmarking in the Wake of Lmbench: A Case Study of the Performance of NetBSD on the Intel x86 Architecture. In Proc. of the 1997 Conf. on Measurement and Modeling of Computer Systems (SIGMETRICS) (Seattle, WA, June 1997), pp. 214-- 224.

.... a more robust, Presented at the USENIX Third Symposium on Operating Systems Design and Implementation (OSDI 99) 1999 Intel Corporation Page 3 of 14 flexible and accurate suite, hbench:OS, which utilizes the performance counters on the Pentium and Pentium Pro processors to instrument the OS [3]. For the purposes of characterizing real time performance, all of these benchmarks share a common problem in that they measure a subset of the OS overhead that an actual application would experience during normal operation. For example, Brown and Seltzer revise the lmbench measurement of context ....

A.B. Brown and M.I. Seltzer, "Operating System Benchmarking in the Wake of Lmbench: A Case Study of the Performance of NetBSD on the Intel x86 Architecture", Proc. 1997 Sigmetrics Conf., Seattle, WA, June 1997.

....hardware. Our measurements in [21] yield context switch times for Linux of 2.60 sec (lowest reported) and 9.72 sec (highest reported) and for Solaris of 11.2 sec (lowest reported) and 131 (highest reported) on an Intel Pentium II 450 MHz. hbench benchmarks: The hbench benchmark suite [38], based on the lmbench package [37] measures the interactions between OS and hardware architecture, i.e. Intel. It measures the scalability of OS primitives, e.g. process creation and cached file read, and hardware capabilities, e.g. memory bandwidth and latency. In addition, hbench measures ....

A. B. Brown and M. I. Seltzer, "Operating System Benchmarking in the Wake of Lmbench: Case Study of the Performance of NetBSD on the Intel Architecture," in Proceedings of the 1997 Sigmetrics Conference, June 1997.

....about the performance of such systems [4] 5] If the machine is destined to be a Web server, the right answer may be different yet. The bottom line is that fastest machine is context sensitive; it depends on the use to which the machine will be applied. Even collections of system microbenchmarks [2][10] which are useful for comparing the behavior of low level primitives, lack any means for providing an indication of what aspects of the system s behavior are important for a particular application. Let us consider a second, and potentially simpler task, Find the fastest file server. There ....

....abstractions used, the total execution time should be the sum of these components. An example of this technique was Brown s performance analysis of the Apache web server [3] in which he applied vector based benchmarking to the operating system domain. First, using the hBench benchmarking suite [2], he measured the performance of a number of operating system abstractions on the NetBSD operating system. Then, using system call tracing facilities, he characterized Apache s usage of operating system primitives as it served a single HTTP request. The components of the two resulting vectors did ....

Brown, A., Seltzer, M., "Operating System Benchmarking in the Wake of Lmbench: Case Study of the Performance of NetBSD on the Intel Architecture," Proceedings of the 1997 Sigmetrics Conference, Seattle, WA, June 1997, 214--224.

No context found.

Brown, Aaron B and Margo I. Seltzer. Operating system benchmarking in the wake of lmbench: A case study of the performance of NetBSD on the Intel x86 architecture. Technical report, Harvard University, 1997.

No context found.

A. B. Brown and M. I. Seltzer. Operating system benchmarking in the wake of lmbench: A case study of the performance of NetBSD on the Intel x86 architecture. In Proceedings of the 1997.

No context found.

A. Brown and M. Seltzer. Operating System Benchmarking in the Wake of Lmbench: A Case Study of the Performance of NetBSD on the Intel x86 Architecture. In Proceedings of the 1997.

No context found.

A. Brown and M. Seltzer. Operating System Benchmarking in the Wake of Lmbench: A Case Study of the Performance of NetBSD on the Intel x86 Architecture. In Proceedings of the 1997.

No context found.

A. Brown and M. Seltzer. Operating System Benchmarking in the Wake of Lmbench: A Case Study of the Performance of NetBSD on the Intel x86 Architecture. In Proceedings of the 1997.

No context found.

A. B. Brown and M. I. Seltzer. Operating system benchmarking in the wake of lmbench: A case study of the performance of NetBSD on the Intel x86 architecture. In ACM SIGMETRICS Conference on Measurement and Modeling of Computer Systems, pages 214--224, Seattle, WA, June 1997.

No context found.

A. B. Brown and M. I. Seltzer, "Operating system benchmarking in the wake of lmbench: A case study of the performance of NetBSD on the Intel x86 architecture," in ACM SIGMETRICS Conference on Measurement and Modeling of Computer Systems, (Seattle, WA), pp. 214--224, June 1997.

No context found.

Aaron Brown and Margo Seltzer, `Operating System Benchmarking in the Wake of Lmbench: A Case Study of the Performance of NetBSD on the Intel x86 Architecture', ACM SIGMETRICS, June 1997. http://www.eecs.harvard.edu/~vino/perf/hbench/index.html.

First 50 documents

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC