Abstract. Modern enterprise applications and systems are characterized by complex underlying software structures, constantly evolving feature sets, and frequent changes in the data on which they operate. The dynamic nature of these applications and systems poses substantial challenges to their use and management, suggesting the need for automated solutions. This paper considers a specific set of dynamic changes, large data updates that reflect changes in the current state of the business, where the frequency of such updates can be multiple times per day. The paper then presents techniques and their middleware implementation for automatically managing requests streams directed at server applications subjected to dynamic data updates, the goal being to improve application reliability in face of evolving feature sets and business data. These techniques (1) automatically detect input patterns that lead to performance degradation or failures and then (2) use these detections to trigger application-specific methods that control input patterns to avoid or at least, defer such undesirable phenomena. Lab experiments using actual traces from Worldspan show a 16 % decrease in frequency of server restarts when using these techniques, at negligible costs in additional overheads and within delays suitable for the rates of changes experienced by this application. 1

Today’s enterprise systems and applications implement functionality that is critical to the ability of society to function. These complex distributed applications, therefore, must meet dynamic criticality objectives even when running on shared heterogeneous and dynamic computational and communication infrastructures. Focusing on the broad class of applications structured as distributed information flows, the premise of our research is that it is difficult, if not impossible, to meet their dynamic service requirements unless these applications exhibit autonomic or self-adjusting behaviors that are ‘vertically’ integrated with underlying distributed systems and hardware. Namely, their autonomic functionality should extend beyond the dynamic load balancing or request routing explored in current web-based software infrastructures to (1) exploit the ability of middleware or systems to be aware of underlying resource availabilities, (2) dynamically and jointly adjust the behaviors of interacting elements of the software stack being used, and even (3) dynamically extend distributed platforms with enterprise functionality (e.g., network-level business rules for data routing and distribution),. The resulting vertically integrated systems can meet stringent criticality or performance requirements, reduce potentially conflicting behaviors across applications, middleware, systems, and resources, and prevent breaches of the

Existing stream processing systems are designed for clustered deployments, and cannot adequately meet the scalability and adaptivity requirements of Internet-scale monitoring applications. Furthermore, these systems commonly optimize for a specific QoS metric, which may limit their applicability to diverse applications and environments. This paper presents XFlow, a generic distributed data collection, processing, and dissemination system that addresses these limitations. XFlow integrates a pub-sub model with data flows for stream processing. The underlying pub-sub model decouples sources and clients, as well as the processing operators, leading to a loosely-coupled architecture that can gracefully scale, adapt to churn in system membership and workload, and facilitate sophisticated optimizations. We first provide an overview of XFlow’s architecture. We then describe XFlow’s optimization model that changes the placement and implementation of operators to meet application-specific performance goals and constraints. Finally, we demonstrate the flexibility and the effectiveness using real-world streams and experimental results obtained from our PlanetLab deployment. 1.

stream processing, overlay, quality-of-service