Developers and users of high-performance distributed systems often observe performance problems such as unexpectedly low throughput or high latency. To determine the source of these performance problems, detailed end-to-end monitoring data from applications, networks, operating systems, and hardware must be correlated across time and space. Researchers need to be able to view and compare this very detailed monitoring data from a variety of angles. To solve this problem, we propose a relational monitoring data archive that is designed to efficiently handle high-volume streams of monitoring data. In this paper we present an instrumentation and event archive service that can be used to collect and aggregate detailed end-to-end monitoring information from distributed applications. We also show how the archive fits into the Global Grid Forum's "Grid Monitoring Architecture".

The current state-of-the-art techniques are not sufficient to debug, understand and characterize multithreaded and distributed systems. In this report, we present a software monitoring framework for distributed and multithreaded systems which are built upon component technology, as the attempt to explore software development tools to address this need. Our monitoring framework captures multidimensional system beuho/oov'...+ hffyvph#v'+rh#vp+##vvtyh#rp'hq+uh...rq...r+'...pr+htr#Xrhq'f# the approach of compiler-assisted auto-generation of instrumentation probes into the stub and the skeleton. With our system-wide causal tracing techniques, global propagation of resource consumption and timing is captured from locally recorded monitoring data. Furthermore, the monitoring framework allows users to select different types of system behaviors and different subsets of candidate components for monitoring. As the result, not only the amount of monitoring data can be reduced, but also the monitoring associated interference can be eliminated or mitigated. System behavior characterization tools are also developed to support this monitoring framework, including reconstruction and visualization of dynamic system call graphs, and characterization of end-to-end timing latency for function invocation. 1.

Writing efficient parallel programs for a massively parallel system like the CRAY T3E is still a difficult task because such programs are typicallyveryly 4 and complex, not trivially parallelizable and their dynamic behavior is difficult9 understand or predict. Therefore, runtime performance analysis tools are41225 on such systems in addition to the normal programming environment tools like editors - debuggers. For the CRAY T3E, Silicon Graphics/Cray Research implemented and provides two performance analysis tools, Apprentice and PAT. Apprentice is a profiling tool ng 4 uses source code instrumentation through compiler switches andprovides38590 9 on the level of functions and basic blocks. PAT, the Performance Analysis Tool,s actually several tools in one. It provides profiling through sampling and accesstos 41896 performance information. It also includes an object code instrumenterwhichme be used for detailed call site profiling and gathering of functionlevel 37162 performance statisti...

As computer systems continue to grow rapidly in both complexity and scale, developers need tools to help them understand the behavior and performance of these systems. While information visualization is a promising technique, most existing computer systems visualizations have focused on very specific problems and data sources, limiting their applicability. This dissertation introduces Rivet, a general-purpose environment for the development of computer systems visualizations. Rivet can be used for both real-time and post-mortem analyses of data from a wide variety of sources. The modular architecture of Rivet enables sophisticated visualizations to be assembled using simple building blocks representing the data, the visual representations, and the mappings between them. The implementation of Rivet enables the rapid prototyping of visualizations through a scripting language interface while still providing high-performance graphics and data management. The effectiveness of Rivet as a tool for computer systems analysis is demonstrated through a collection of case studies. Visualizations created using Rivet have been used to display: (a) line-by-line execution data from the SUIF Explorer interactive parallelizing compiler, enabling programmers to maximize the parallel speedups of their applications; (b) detailed memory system utilization data from the FlashPoint memory profiler, providing insights on both sequential and parallel program bottlenecks; (c) the behavior of applications running on superscalar processors, allowing developers to take full advantage of these complex CPUs; and (d) the real-time performance of computer systems and clusters, drawing attention to interesting or anomalous behavior. In addition to these focused examples, Rivet has been also used in co...

Abstract—In message-passing applications, the temporal or spatial distance between cause and symptom of a performance problem constitutes a major difficulty in deriving helpful conclusions from performance data. Just knowing the locations of wait states in the program is often insufficient to understand the reason for their occurrence. We present a method for verifying hypotheses on causality between temporally or spatially distant performance phenomena in message-passing applications without altering the application itself. The verification is accomplished by modifying MPI event traces and using them to simulate the hypothetical message-passing behavior. By performing a parallel real-time reenactment of the communication to be simulated using the original execution configuration, we can achieve high scalability and good predictive accuracy in relation to the measured behavior. Not relying on a potentially complex model of the message-passing subsystem, our method is also platform independent. I.

: This paper describes a performance tuning tool, named SVMview, for DSMbased parallel computers. SVMview is a tool for doing a post-mortem analysis of page movements generated during the execution of Fortran-S programs. This tool is able to analyze some particular phenomena such as false-sharing which occurs when several processors write to the same page simultaneously. Such behavior entails badly the performance of DSM systems, and thus are particularly important to be detected. Key-words: (R'esum'e : tsvp) Unit de recherche INRIA Rennes IRISA, Campus universitaire de Beaulieu, 35042 RENNES Cedex (France) Tlphone : (33) 99 84 71 00 -- Tlcopie : (33) 99 84 71 71 SVMview: un outil de "performance tuning" pour MVP R'esum'e : Ce papier d'ecrit un outil de "performance tuning", appel'e SVMview, pour des architectures parall`eles supportant une m'emoire virtuelle partag'ee. SVMview est un outil permettant une analyse post-mortem des mouvements de page g'en'er'es durant l'ex'ecution d'...

Efficient execution of a scientific computing application requires insights into how system features impact the performance of the application. A distributed system consists of heterogeneous components, such as networks, processors, run-time systems, operating systems, etc. This heterogeneity complicates the task of gaining

Efficient performance tuning of parallel programs is often hard. We present a performance prediction and visualization tool called VPPB. Based on a monitored uni-processor execution, VPPB shows the (predicted) behaviour of a multithreaded program using any number of processors and the program behaviour is visualized as a graph. The first version of VPPB was unable to handle I/O operations. This version has, by an improved tracing technique, added the possibility to trace activities at the kernel level as well. Thus, VPPB is now able to trace various I/O activities, e.g., manipulation of OS internal buffers, physical disk I/O, socket I/O, and RPC. VPPB allows flexible performance tuning of parallel programs developed for shared memory multiprocessors using a standardized environment; C/C++ programs that uses the thread package in Solaris 2.X. 1.

Current software development techniques and tools lack the capability to characterize function call chains in multithreaded and distributed applications built upon component technologies like CORBA, COM and J2EE. The root cause is that causal linkage information necessary to trace end-to-end call chains is private to each vendor’s runtime and often unavailable for logging or analysis. We propose and demonstrate a mechanism for maintaining and correlating global causality information of component-based applications, and using this information to expose and characterize function call chains and their associated behaviors in such multithreaded and distributed applications. Our approach relies on a global virtual tunnel facilitated by the instrumented stubs and skeletons. This tunnel maintains and correlates causal information throughout the end-toend call chains spanning threads, processes and processors. As a result, monitoring data captured locally can be correlated and system-wide propagation of timing latency and CPU utilization becomes perceivable. 1.

This paper demonstrates the use of a model-based evaluation approach for instrumentation systems (ISs). The overall objective of this study is to provide early feedback to tool developers regarding IS overhead and performance; such feedback helps developers make appropriate design decisions about alternative system configurations and task scheduling policies. We consider three types of system architectures: network of workstations (NOW), symmetric multiprocessors (SMP), and massively parallel processing (MPP) systems. We develop a Resource OCCupancy (ROCC) model for an on-line IS for an existing tool and parameterize it for an IBM SP-2 platform. This model is simulated to answer several `what if' questions regarding two policies to schedule instrumentation data forwarding: collect-and-forward (CF) and batch-and-forward (BF). In addition, this study investigates two alternatives for forwarding the instrumentation data: direct and binary tree forwarding for an MPP system. Simulati...