D.G. Feitelson and A.M. Weil. Utilization and Predictability in Scheduling the IBM SP2 with Backfilling. In Procedings of IPPS/SPDP 1998, pages 542--546. IEEE Computer Society, 1998.  Home/Search   Document Details and Download   Summary   Related Articles   Check

.... performance that is reasonably competitive with high performance (but more di#cult to implement) preemptive policies such as gang scheduling or spatial equi partitioning [5] This relatively high performance is achieved in spite of the fact that user requested runtimes are often highly inaccurate [6 9]. For example, analysis of the NCSA O2K logs shows that 30 of the jobs that request 200 or more hours of runtime terminate in under ten hours [9] The key open question addressed in this paper is whether the high performance backfill policies can be further improved with more accurate requested ....

....in this paper is whether the high performance backfill policies can be further improved with more accurate requested runtimes. Several previous simulation studies of FCFS backfill show that more accurate requested runtimes have only minimal impact on the average wait time or average slowdown [6, 10 12, 7]. We briefly revisit the question for FCFSbackfill, using workloads from recent months on the O2K that have significantly heavier system load (e.g. up to 100 cpu demand) and using a more complete set of performance measures. More importantly, we investigate the question of whether more accurate ....

[Article contains additional citation context not shown here]

Feitelson, D.G., Weil, A.M.: Utilization and predictability in scheduling the IBM SP2 with backfilling. In: Proc. 12th Int'l. Parallel Processing Symp., Orlando (1998) 542--546

....the utilization of the machine. Hence, much work has been done in the field of improving or developing new scheduling algorithms and policies in general. Some examples to mention are: gang scheduling [2] combined with migration and backfilling [20] several backfilling variants (conservative [4], EASY [13] or slack based [18] or a tool for predicting job runtimes [14] Also, research for explicit machines was done, e.g. for the IBM SP2 and the LoadLeveler System [18, 11] or the IBM ASCI Blue [5, 10] It is common to use simulation environments for evaluating scheduling algorithms. ....

....scheduler is presented. 3.1 The Basic dynP Scheduler We started our work with two job sets which were derived from traces of our 96 node hpcLine cluster. This machine is managed by CCS which is a long term running project at the PC . All policies are combined with (conservative) backfilling [4]. Currently, CCS is configured to use FCFS for scheduling jobs. Now the question is, has performance su#ered, because we have used FCFS instead of SJF or LJF. So we have developed a simulation framework for evaluating the three scheduling policies with two trace based job sets from our machine. ....

[Article contains additional citation context not shown here]

D. G. Feitelson and A. Weil. Utilization and Predictability in Scheduling the IBM SP2 with Backfilling. In Proceedings of the 1st Merged International Paral lel Processing Symposium and Symposium on Parallel and Distributed Processing (IPPS/SPDP-98), pages 542--547, Los Alamitos, March 1998. IEEE Computer Society.

....these software systems is the jobscheduler. In a production environment the performance of the scheduler is primarily responsible for a good or bad overall machine performance and user acceptance. Developing and improving job schedulers is often done with the aid of simulations (Feitelson et al. [3, 9, 12] for the IBM SP2, or by Franke, Moreira et al. 4, 8] for the ASCI BluePacific) This paper presents a similar approach: improving the job scheduler of CCS (Computing Center Software) for our Fujitsu Siemens hpcLine Cluster. CCS [6, 7] was developed at the PC over the last years to manage and ....

D. G. Feitelson and A. Weil. Utilization and Predictability in Scheduling the IBM SP2 with Backfilling. In Proceedings of the 1st Merged International Parallel Processing Symposium and Symposium on Parallel and Distributed Processing (IPPS/SPDP-98), pages 542--547, Los Alamitos, March 1998. IEEE Computer Society.

....This effort has started with the identification of resource scheduler features that benefit the performance of application schedulers. We then built (Space Shared Scheduler) to provide such features, and an AppLeS to use them. S implements a strategy called conservative backfilling [Feitelson 98] and provides unique support for application schedulers that target supercomputers. 4.1. Design Goals A resource scheduler with good support for application scheduling still has to perform well according to its traditional metrics, otherwise no administrator will install it. Therefore, our ....

....calls for submission, reservation, cancellation, and status inquiry (see details in the next Section) However, currently no provision is made for priorities. All applications are treated equally. 4.3. S Space Shared Scheduler The heart of S is the conservative backfilling algorithm [Feitelson 98] Conservative backfilling uses an allocation list that keeps, for any given time, which processors are already committed to which applications. The allocation list can be implemented as a linked list whose nodes represent time frames in which all processors in system are allocated in the same ....

[Article contains additional citation context not shown here]

D. G. Feitelson and A. Mu'alem Weil. Utilization and predictability in scheduling the IBM SP2 with backfilling. In 12 Intl. Parallel Processing Symp., pp. 542-546, Apr 1998.

....section. 2.2. SA over Conservative Backfilling When the supercomputer scheduler is known, it may be possible to optimize SA by avoiding the simulation of each request r . This section describes SAcb, a version of SA that assumes the supercomputer scheduler to be conservative backfilling [43]. Conservative backfilling was chosen as the target for a faster version of SA because it can be viewed as an idealized representative of today s supercomputer schedulers. In practice the behavior of supercomputer schedulers varies from machine to machine. Even when the same scheduling software is ....

....to reduce unnecessary idle time, making conservative backfilling a good representative of current practice in supercomputer scheduling. 2.2.1. Conservative Backfilling Conservative backfilling uses an allocation list that maintains, for any given time, which processors are assigned to which jobs [43]. The allocation list can be im plemented as a linked list whose nodes represent time periods in which all processors in system are allocated in the same way. Arriving jobs are processed using the first fit strategy, i.e. they are put in the first slot they fit. For example, Figure 3 depicts the ....

[Article contains additional citation context not shown here]

Dror Feitelson and A. Mu'alem Weil. Utilization and predictability in scheduling the IBM SP2 with backfilling. In 12 Intl. Parallel Processing Symp., pp. 542-546, Apr 1998.

....ones) 7 iv) Some mechanism to avoid starvation of large jobs. In this work, we employ conservative backfilling as an idealized representative of today s supercomputer schedulers. Conservative backfilling displays all four characteristics listed above, has been shown to attain good performance [14], and it is simple to understand. Conservative backfilling allows for out of order job start up, provided that the job that jumps ahead in the queue does not delay the expected start time of any other job [14] Conservative backfilling uses an allocation list that maintains, for any given time, ....

....all four characteristics listed above, has been shown to attain good performance [14] and it is simple to understand. Conservative backfilling allows for out of order job start up, provided that the job that jumps ahead in the queue does not delay the expected start time of any other job [14]. Conservative backfilling uses an allocation list that maintains, for any given time, which processors are assigned to which jobs. Arriving jobs are processed using the first fit strategy, i.e. they are put in the first slot they fit. Whenever a job finishes using less time than it had requested, ....

[Article contains additional citation context not shown here]

D. Feitelson and A. M. Weil. "Utilization and predictability in scheduling the IBM SP2 with backfilling". Proc. 12th Intl. Parallel Processing Symp., pp. 542-546, Apr 1998.

....system directed adaptation of application parallelism. The cumulative improvements of these techniques decrease the bounded slowdown, our primary metric, to less then 15 of conservative backfilling. 1. Introduction We present a study of batch schedulers based on the conservative backfilling [2] algorithm. A batch scheduler is a scheduler for non interactive jobs. Such a scheduler manages resources on large cluster(s) of dedicated computer nodes interconnected with high speed networks that must be shared among many end users. Jobs have a large variety of shapes. Some are highly parallel, ....

....with high speed networks that must be shared among many end users. Jobs have a large variety of shapes. Some are highly parallel, while others are sequential. Some last a few seconds, while others can last a day. Scheduling those jobs optimally is known to be hard. Conservative backfilling [2], a commonly used scheduling algorithm, performs much better than a basic first come first serve (FCFS) policy. FCFS suffers from fragmentation, and therefore low utilization, when used alone. Backfilling allows smaller jobs to move forward in the schedule as long as such movement does not cause ....

[Article contains additional citation context not shown here]

D.G. Feitelson and A. Well, "Utilization and Predictability in Scheduling the IBM SP2 with Backfill- ing," in 12 tn International Parallel Processing Sym- posium, 1998.

....Longest Job First (LJF) each combined with conservative backfilling. As the scheduler has a strong influence on the system performance, it is worth while to improve the performance of the scheduling strategy. In the past preliminary evaluations based on simulations were done by Feitelson et al. [1, 10, 12] for the IBM SP2, or by Franke, Moreira et al. 4, 9] for the ASCI Blue Pacific. The remainder of this paper is organized as follows. Section 2 describes the scheduling strategies and backfilling in detail. The two job sets for the evaluation are described in Section 3. After the performance ....

....delaying other already scheduled jobs. For that, users must provide an estimated runtime for their jobs, so that the scheduler can predict when jobs are finished and others can be started. Jobs are cancelled, if they run longer than estimated. Two backfilling variants are known: conservative [1] and aggressive [8] The aggressive variant is also known as EASY (Extensible Argonne Scheduling sYstem [11] backfilling. Backfilling works as follows: IF (first job j of the queue can not be started now) FORALL (job i j in the queue) IF (conservative) IF (start of job i does not delay ....

D. Feitelson and A. Weil. Utilization and Predictability in Scheduling the IBM SP2 with Backfilling. In Proceedings of the 1st Merged International Parallel Processing Symposium and Symposium on Parallel and Distributed Processing (IPPS/SPDP98) , pages 542--547, Los Alamitos, March 1998. IEEE Computer Society.

....the earliest time when the required nodes will be available) In our previous example, this was 5 hours. A Dynamic Backfilling (DBF) algorithm, rather than fixing the time at the earliest time that it can, will instead determine the most appropriate time to start the job within a starting window [4, 11]. The earliest time possible may in some cases not lead to the best result. Using our previous example, let s assume that we also had a 125 node job (for 5 hours 30 Jul.94 Jul.95 0 20 40 60 80 100 Available Node Utilization Weekly Node Utilization (FCFS FF DBF) Weekly Node Utilization ....

Dror Feitelson and A. Mu'alem Weil, Utilization and Predictability in Scheduling the IBM SP2 With Backfilling, in Proceedings of 12th International Parallel Processing Symposium., pp. 542-546, April 1998.

....are unutilized despite a waiting queue of jobs. Furthermore, the wait and response times for jobs with an exclusively space sharing strategy can be relatively high. We analyze three approaches to alleviate the problems with space sharing scheduling. The first is a technique called backfilling [6, 14], which attempts to assign unutilized nodes to jobs that are behind in the priority queue (of waiting jobs) rather than keep them idle. To prevent starvation for larger jobs, conservative) backfilling requires that a job selected out of order completes before the jobs that are ahead of it in the ....

....limit of their allocated time. The rest of the jobs (1 Gamma Phi) are distributed such that the distribution of jobs that end at a certain fraction of their estimated time is uniform. This distribution is shown in Figure 1. It has been shown to represent well actual job behavior in large systems [6]. To obtain the desired distribution for execution times in the Phi model, in our simulations we compute the overestimation factor as follows: Let y be a uniformly distributed random number in the range 0 y 1. If y Phi, then the overestimation factor is 1 (i.e. estimated time = execution ....

[Article contains additional citation context not shown here]

D. G. Feitelson and A. M. Weil. Utilization and predictability in scheduling the IBM SP2 with backfilling. In 12th International Parallel Processing Symposium, pages 542--546, April 1998.

No context found.

D. G. Feitelson and A. Mu'alem Weil, "Utilization and predictability in scheduling the IBM SP2 with backfilling ". In 12th Intl. Parallel Processing Symp., pp. 542--546, Apr 1998.

.... it may actually delay the high priority full con guration jobs (even if the conservative version is used) As a side issue, even if back lling is used, it is not clear what runtime estimates should be given, as accurate estimates are not necessarily the best, but normal user estimates are worse [5]. While favoring machines with checkpoint and restart is reasonable for a system level metric, the e ect in this case may be too extreme. A more balanced approach would be to rst run the workload with no special requirements, and allowing all the features that exist in the scheduler. Then a ....

D. G. Feitelson and A. Mu'alem Weil, \Utilization and predictability in scheduling the IBM SP2 with back lling". In 12th Intl. Parallel Processing Symp., pp. 542-546, Apr 1998. 76

.... parallel supercomputers is variable partitioning, meaning that each job receives a partition of the machine with its desired number of processors [5] Such partitions are allocated in a rst come rst serve (FCFS) This paper supercedes the preliminary version published in IPPS SPDP 98 [7]. c 2001 IEEE. Personal use of this material is permitted. However, permission to reprint republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in ....

....same amount of back lling, which matches the prediction of equal performance according to this model. Using the Jann model, we nd that EASY back lls a slightly larger percentage of the jobs By chance, the preliminary version of this paper used the combinations that predict equal performance [7]. trace mon load jobs EASY cons di erence EASY cons di erence CTC jul 0.539 7950 11394 11605 1.9 4.9 5.5 12.2 aug 0.584 7273 11558 11728 1.5 3.3 4.6 39.4 sep 0.566 6167 14950 15360 2.7 5.8 5.9 1.7 oct 0.547 7257 9963 10298 3.4 3.0 3.6 20.0 nov 0.531 7917 10621 10684 0.6 2.0 ....

D. G. Feitelson and A. Mu'alem Weil, \Utilization and predictability in scheduling the IBM SP2 with back lling". In 12th Intl. Parallel Processing Symp., pp. 542-546, Apr 1998.

....[10] now part of IBM s LoadLeveler, uses an aggresive strategy that backfills a job if it does not delay the first job in the queue. We recently showed that a more conservative scheduler, which only backfills a jobs that doesn t delay any job in the queue, retains roughly the same performance [5]. Here we present a further improvement based on the notion c 1999 IEEE. Personal use of this material is permitted. However, permission to reprint republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or ....

D. G. Feitelson and A. Mu'alem Weil, "Utilization and predictability in scheduling the IBM SP2 with backfilling ". In 12th Intl. Parallel Processing Symp., pp. 542--546, Apr 1998.

....a slightly larger percentage of the jobs than conservative backfilling. However, the simulations based on the traces suggest that the amount of backfilling performed is similar and, in 1. By chance, the preliminary version of this paper used the combinations that predict equal performance [7]. one case (SDSC) conservative even performs more backfilling but achieves worse results. Thus, it is not a question of how much backfilling is done, but more of which jobs are backfilled. We are therefore left with a unique situation in which the workloads dictate the results (the only ....

D.G. Feitelson and A. Mu'alem Weft, "Utilization and Predict- ability in Scheduling the IBM SP2 with Backfilling," Proc. 12th Int'l Parallel Processing Symp., pp. 542-546, Apr. 1998.

No context found.

D.G. Feitelson and A.M. Weil. Utilization and Predictability in Scheduling the IBM SP2 with Backfilling. In Procedings of IPPS/SPDP 1998, pages 542--546. IEEE Computer Society, 1998.

No context found.

D. G. Feitelson and A. M. Weil. Utilization and predictability in scheduling the ibm sp2 with backfilling. In 12th Intl. Parallel Processing Symp., pages 542--546, Apr 1998.

No context found.

Dror G. Feitelson and Ahuva Mu'alem Weil. Utilization and predictability in scheduling the IBM SP2 with backfilling. In Proc. 12th Int'l. Parallel Processing Symp., pages 542-- 546, Orlando, March 1998.

No context found.

D. G. Feitelson and A. M. Weil. Utilization and predictability in scheduling the IBM SP2 with backfilling. In 12th Intl. Parallel Processing Symp., pages 542--546, 1998.

No context found.

D. G. Feitelson, A. M. Weil, "Utilization and predictability in scheduling the IBM SP2 with backfilling ", In 12th International Parallel Processing Symposium, pp. 542-546, April 1998.

No context found.

D. G. Feitelson and A. M. Weil. Utilization and predictability in scheduling the IBM SP2 with backfilling. In 12th International Parallel Processing Symposium, pages 542--546, April 1998.

No context found.

D.G. Feitelson and A.M. Weil. Utilization and Predictability in Scheduling the IBM SP2 with Backfilling. In Procedings of IPPS/SPDP 1998.

No context found.

D. G. Feitelson and A. M. Weil. Utilization and predictability in scheduling the IBM SP2 with backfilling. In 12th Intl. Parallel Processing Symp., pages 542--546, 1998.

No context found.

D. G. Feitelson and A. M. Weil. Utilization and predictability in scheduling the IBM SP2 with backfilling. In 12th International Parallel Processing Symposium, pages 542--546, April 1998.

No context found.

D. G. Feitelson, A. M. Weil, "Utilization and predictability in scheduling the IBM SP2 with backfilling", In 12th International Parallel Processing Symposium, pp. 542-546, April 1998.

First 50 documents

Online articles have much greater impact   More about CiteSeer.IST   Add search form to your site   Submit documents   Feedback  

CiteSeer.IST - Copyright Penn State and NEC