Citation Context

...inistic, and so much of our experimental methodology does not apply. However, we do find that AE similarly overfits to input tests (Section 4.2). GenProg [32, 62] uses a genetic programming heuristic =-=[31]-=- to search the space of candidate repairs. Given a buggy program and a set of tests, GenProg generates a population of random patches by using statistical fault localization to identify which program ...

Citation Context

...verfitting is also a well-studied problem in machine learning [41]. Our experiments suggest that minimization and overfitting are unrelated, which is consistent with prior results in machine learning =-=[52]-=-. To the best of our knowledge, ours is the first consideration of this relationship in the program repair domain. G&V approaches fall in the space of search-based software engineering [25], which ada...

Citation Context

...independent test suites: a black-box test suite written by the course instructor to the natural-language specification, and a white-box test suite constructed using the symbolic execution engine Klee =-=[12]-=- on a reference solution. This dataset is publicly available: http:// repairbenchmarks.cs.umass.edu. Our study admittedly uses small programs written by novice programmers, which threatens the general...

Citation Context

..., a group vote on the behavior may outperform each individual patch. N-version patches may therefore provide an avenue to mitigate overfitting. Human-written code typically lacks sufficient diversity =-=[30]-=- to enable true n-version programming [15], but randomized G&V repair may not. We created the n-version program Pn in the following way: For each buggy version-test suite subset pair Pb, run GenProg o...

Citation Context

...orm each individual patch. N-version patches may therefore provide an avenue to mitigate overfitting. Human-written code typically lacks sufficient diversity [30] to enable true n-version programming =-=[15]-=-, but randomized G&V repair may not. We created the n-version program Pn in the following way: For each buggy version-test suite subset pair Pb, run GenProg on Pb 20 times. If fewer than three of the ...

Citation Context

...ch-based software engineering [25], which adapts search methods, such as genetic programming, to software engineering tasks. Search-based software engineering has been used for developing test suites =-=[40,59]-=-, finding safety violations [3], refactoring [53], and project management and effort estimation [8]. Good fitness functions are critical to searchbased software engineering. Our findings indicate that...

Citation Context

...PROGRAM REPAIR Automatic repair techniques can be classified broadly into two classes: (1) Generate-and-validate (G&V) techniques create candidate patches (often via search-based software engineering =-=[25]-=-) and then validate them, typically through testing (e.g., [4, 13, 14, 17, 18, 28, 29, 37, 39, 43, 47, 50, 55, 58, 61, 62]. (2) Synthesis-based techniques use constraints to build correct-by-construct...

Citation Context

...ing other approaches, such as synthesis-based repair [26, 46, 60] techniques, is also of great value, but is outside the scope of this paper. Our contribution is a controlled investigation of GenProg =-=[35, 62]-=- and TrpAutoRepair [49], both test-case-guided, search-based automatic program repair tools with freely available implementations that scale to large programs. The evaluation identifies the circumstan...

Citation Context

...ell as evaluated the monetary and time costs of automatic repair [32], the relationship between operator choices and test execution parameters and success [33,61], and human-rated patch acceptability =-=[1,29]-=- and maintainability [21]. However, these evaluations have generally not used an objective metric of correctness independent of patch construction. Our evaluation measures patch correctness independen...

Citation Context

...ddress software quality in existing systems written in legacy languages. Since legacy codebases often are often idiosyncratic to the point of not adhering to the specifications of their host language =-=[9]-=-, it might not be possible even to add contracts to such projects. G&V repair works by generating multiple candidate patches that might address a particular bug and then validating the candidates to d...

Citation Context

...o improve repair by helping designers select change operators and search strategies [27, 64]. Understanding how automated repair handles particular classes of errors, such as security vulnerabilities =-=[35, 47]-=- can guide tool design. For this reason, some automated repair techniques focus on a particular defect class, such as buffer overruns [54, 57], unsafe integer use in C programs [17], single-variable a...

Citation Context

...stractly about the program specification. However, while humans can reason about program faults abstractly above the level of a repair tool, they are also subject to a large array of cognitive biases =-=[2, 38]-=- that can hamper their debugging effort. Repair tools have no such biases, and will mechanically explore the solution space as guided by an objective function, without becoming irrationally fixated on...

Citation Context

...ch-based software engineering [25], which adapts search methods, such as genetic programming, to software engineering tasks. Search-based software engineering has been used for developing test suites =-=[40,59]-=-, finding safety violations [3], refactoring [53], and project management and effort estimation [8]. Good fitness functions are critical to searchbased software engineering. Our findings indicate that...

Citation Context

...echniques provide the benefit of provable correctness for patches, but require contracts, so they are unsuitable for legacy systems. Synthesis techniques can also construct new features from examples =-=[16, 23]-=-, rather than address existing bugs. Our work has focused on G&V approaches, and investigating overfitting and patch quality in synthesisbased techniques is a complementary and worthwhile pursuit. The...

Citation Context

...r evaluates existing techniques in a new way to expose previously hidden limitations to G&V program repair. Our findings may extend to other search-based or test suite-guided repair techniques (e.g., =-=[6,18,29,39,43,44,47,61]-=-). Monperrus [42] has recently discussed the challenges of experimentally comparing program repair techniques. For example, the selection of test subjects (defects) can introduce evaluation bias [10, ...

Citation Context

...d requires paying special attention to the sample sizes, statistical tests, crossvalidation, and uses of bootstrapping. Our work is consistent with the guidelines for evaluating randomized algorithms =-=[5]-=- to enhance the credibility of our findings. Specifically, we used a large sample of 998 buggy student programs, controlled for a variety of potential influencers in our experiments, and used fixed-ef...

Citation Context

...cular classes of errors, such as security vulnerabilities [35, 47] can guide tool design. For this reason, some automated repair techniques focus on a particular defect class, such as buffer overruns =-=[54, 57]-=-, unsafe integer use in C programs [17], single-variable atomicity violations [26], deadlock and livelock defects [36], concurrency errors [37], and data input errors [4]. Other techniques tackle gene...

Citation Context

...g/10.1145/2786805.2786825. The most common prior evaluations of automatic repair provide evidence of techniques’ feasibility with respect to this test-casebased definition of patch correctness (e.g., =-=[17, 35, 46, 60]-=-). However, in practice, test suites are rarely exhaustive [51], and repair techniques must avoid breaking undertested functionality. When evaluations of repair techniques use the same test cases to b...

Citation Context

...ting tools General Terms: Experimentation Keywords: automated program repair, empirical evaluation, independent evaluation, GenProg, TrpAutoRepair, INTROCLASS 1. INTRODUCTION Automated program repair =-=[4,13,17,18,26,28,29,32,36,37,39,39, 43,46,47,50,55,58,60,61]-=- holds great potential to reduce debugging costs and improve software quality. For example, GenProg quickly and cheaply generated patches for 55 out of 105 C bugs [32], while PAR showed comparable res...

Citation Context

...ting tools General Terms: Experimentation Keywords: automated program repair, empirical evaluation, independent evaluation, GenProg, TrpAutoRepair, INTROCLASS 1. INTRODUCTION Automated program repair =-=[4,13,17,18,26,28,29,32,36,37,39,39, 43,46,47,50,55,58,60,61]-=- holds great potential to reduce debugging costs and improve software quality. For example, GenProg quickly and cheaply generated patches for 55 out of 105 C bugs [32], while PAR showed comparable res...

Citation Context

...7,61]). Monperrus [42] has recently discussed the challenges of experimentally comparing program repair techniques. For example, the selection of test subjects (defects) can introduce evaluation bias =-=[10, 48]-=-. Our evaluation focuses precisely on the limits and potential of repair techniques on a large dataset of defects, and controls for a variety of potential influencers, addressing some of Monperrus’ co...

Citation Context

...rch methods, such as genetic programming, to software engineering tasks. Search-based software engineering has been used for developing test suites [40,59], finding safety violations [3], refactoring =-=[53]-=-, and project management and effort estimation [8]. Good fitness functions are critical to searchbased software engineering. Our findings indicate that using test cases alone as the fitness function l...

Citation Context

...tch construction, it is fundamentally different from the correctness criterion we use in our evaluation, as it is often difficult for humans to spot bugs even when told exactly where to look for them =-=[45]-=-. Meanwhile, our recent evaluation of SearchRepair uses the same methodology as the evaluation we present here [28]. Our work evaluates automated repair so that it can be improved. Empirical studies o...

Citation Context

...ting tools General Terms: Experimentation Keywords: automated program repair, empirical evaluation, independent evaluation, GenProg, TrpAutoRepair, INTROCLASS 1. INTRODUCTION Automated program repair =-=[4,13,17,18,26,28,29,32,36,37,39,39, 43,46,47,50,55,58,60,61]-=- holds great potential to reduce debugging costs and improve software quality. For example, GenProg quickly and cheaply generated patches for 55 out of 105 C bugs [32], while PAR showed comparable res...

Citation Context

...r evaluates existing techniques in a new way to expose previously hidden limitations to G&V program repair. Our findings may extend to other search-based or test suite-guided repair techniques (e.g., =-=[6,18,29,39,43,44,47,61]-=-). Monperrus [42] has recently discussed the challenges of experimentally comparing program repair techniques. For example, the selection of test subjects (defects) can introduce evaluation bias [10, ...

Citation Context

...ting tools General Terms: Experimentation Keywords: automated program repair, empirical evaluation, independent evaluation, GenProg, TrpAutoRepair, INTROCLASS 1. INTRODUCTION Automated program repair =-=[4,13,17,18,26,28,29,32,36,37,39,39, 43,46,47,50,55,58,60,61]-=- holds great potential to reduce debugging costs and improve software quality. For example, GenProg quickly and cheaply generated patches for 55 out of 105 C bugs [32], while PAR showed comparable res...

Citation Context

...oadly into two classes: (1) Generate-and-validate (G&V) techniques create candidate patches (often via search-based software engineering [25]) and then validate them, typically through testing (e.g., =-=[4, 13, 14, 17, 18, 28, 29, 37, 39, 43, 47, 50, 55, 58, 61, 62]-=-. (2) Synthesis-based techniques use constraints to build correct-by-construction patches via formal verification or inferred or programmer-provided contracts or specifications (e.g., [26, 46, 60]. Th...

Citation Context

...ting tools General Terms: Experimentation Keywords: automated program repair, empirical evaluation, independent evaluation, GenProg, TrpAutoRepair, INTROCLASS 1. INTRODUCTION Automated program repair =-=[4,13,17,18,26,28,29,32,36,37,39,39, 43,46,47,50,55,58,60,61]-=- holds great potential to reduce debugging costs and improve software quality. For example, GenProg quickly and cheaply generated patches for 55 out of 105 C bugs [32], while PAR showed comparable res...

Citation Context

...stractly about the program specification. However, while humans can reason about program faults abstractly above the level of a repair tool, they are also subject to a large array of cognitive biases =-=[2, 38]-=- that can hamper their debugging effort. Repair tools have no such biases, and will mechanically explore the solution space as guided by an objective function, without becoming irrationally fixated on...

Citation Context

...re engineering tasks. Search-based software engineering has been used for developing test suites [40,59], finding safety violations [3], refactoring [53], and project management and effort estimation =-=[8]-=-. Good fitness functions are critical to searchbased software engineering. Our findings indicate that using test cases alone as the fitness function leads to patches that may not generalize to the pro...

Citation Context

...r evaluates existing techniques in a new way to expose previously hidden limitations to G&V program repair. Our findings may extend to other search-based or test suite-guided repair techniques (e.g., =-=[6,18,29,39,43,44,47,61]-=-). Monperrus [42] has recently discussed the challenges of experimentally comparing program repair techniques. For example, the selection of test subjects (defects) can introduce evaluation bias [10, ...

Citation Context

...7,61]). Monperrus [42] has recently discussed the challenges of experimentally comparing program repair techniques. For example, the selection of test subjects (defects) can introduce evaluation bias =-=[10, 48]-=-. Our evaluation focuses precisely on the limits and potential of repair techniques on a large dataset of defects, and controls for a variety of potential influencers, addressing some of Monperrus’ co...

Citation Context

... which adapts search methods, such as genetic programming, to software engineering tasks. Search-based software engineering has been used for developing test suites [40,59], finding safety violations =-=[3]-=-, refactoring [53], and project management and effort estimation [8]. Good fitness functions are critical to searchbased software engineering. Our findings indicate that using test cases alone as the ...

Citation Context

...the same program. Finally, as a challenge that applies to GenProg in particular, genetic programming is known to lead to bloat, in which solutions contain more code than necessary to maximize fitness =-=[24]-=-. GenProg minimizes code bloat post facto; prior work has claimed that minimization reduces patches overfitting to the training tests [35]. TrpAutoRepair only attempts single-edit patches, and thus do...

Citation Context

...trols for a variety of potential influencers, addressing some of Monperrus’ concerns [42]. Genetic programming tends to produce extraneous code that does not contribute to the fitness of the solution =-=[24, 56]-=-. GenProg attempts to mitigate this through solution minimization, which may reduce the chances of breaking undertested functionality. Overfitting is also a well-studied problem in machine learning [4...

Citation Context

...new way to expose previously hidden limitations to G&V program repair. Our findings may extend to other search-based or test suite-guided repair techniques (e.g., [6,18,29,39,43,44,47,61]). Monperrus =-=[42]-=- has recently discussed the challenges of experimentally comparing program repair techniques. For example, the selection of test subjects (defects) can introduce evaluation bias [10, 48]. Our evaluati...

Citation Context

... consider independent quality measures, though less extensively than we do here. And while some evaluations have used humans to independently measure repair acceptability [19, 29] and maintainability =-=[21]-=-, unlike our work, they neither directly nor objectively evaluate functional patch correctness. Meanwhile, our recent evaluation of SearchRepair uses the same methodology as the evaluation we propose ...

Citation Context

... search algorithms, but comparable to previous program repair evaluations. More attempts may have revealed more solutions. Finally, we used the recommended GenProg parameters defined in previous work =-=[33]-=-; a full parameter sweep is outside the scope of this investigation. Our INTROCLASS dataset — http://repairbenchmarks.cs. umass.edu— includes all the buggy versions, student-written solutions, and tes...

Citation Context

...number of bugs in medium-sized programs, as well as evaluated the monetary and time costs of automatic repair [32], the relationship between operator choices and test execution parameters and success =-=[33,61]-=-, and human-rated patch acceptability [1,29] and maintainability [21]. However, these evaluations have generally not used an objective metric of correctness independent of patch construction. Our eval...

Citation Context

... in an attempt to fix it. This describes the fix space of a particular program repair problem. GenProg and TrpAutoRepair tackle this challenge using the observation that programs are often repetitive =-=[7, 22]-=- and logic implemented with a bug in one place is likely to be implemented correctly elsewhere in the same program. GenProg and TrpAutoRepair therefore limit the code changes to deleting constructs an...

Citation Context

...ting tools General Terms: Experimentation Keywords: automated program repair, empirical evaluation, independent evaluation, GenProg, TrpAutoRepair, INTROCLASS 1. INTRODUCTION Automated program repair =-=[4,13,17,18,26,28,29,32,36,37,39,39, 43,46,47,50,55,58,60,61]-=- holds great potential to reduce debugging costs and improve software quality. For example, GenProg quickly and cheaply generated patches for 55 out of 105 C bugs [32], while PAR showed comparable res...

Citation Context

... in an attempt to fix it. This describes the fix space of a particular program repair problem. GenProg and TrpAutoRepair tackle this challenge using the observation that programs are often repetitive =-=[7, 22]-=- and logic implemented with a bug in one place is likely to be implemented correctly elsewhere in the same program. GenProg and TrpAutoRepair therefore limit the code changes to deleting constructs an...

Citation Context

...ting tools General Terms: Experimentation Keywords: automated program repair, empirical evaluation, independent evaluation, GenProg, TrpAutoRepair, INTROCLASS 1. INTRODUCTION Automated program repair =-=[4,13,17,18,26,28,29,32,36,37,39,39, 43,46,47,50,55,58,60,61]-=- holds great potential to reduce debugging costs and improve software quality. For example, GenProg quickly and cheaply generated patches for 55 out of 105 C bugs [32], while PAR showed comparable res...

Citation Context

...(e.g., [50, 64]) has begun to consider independent quality measures, though less extensively than we do here. And while some evaluations have used humans to independently measure repair acceptability =-=[19, 29]-=- and maintainability [21], unlike our work, they neither directly nor objectively evaluate functional patch correctness. Meanwhile, our recent evaluation of SearchRepair uses the same methodology as t...

Citation Context

...automated repair so that it can be improved. Empirical studies of fixes of real bugs in open-source projects can also improve repair by helping designers select change operators and search strategies =-=[27, 64]-=-. Understanding how automated repair handles particular classes of errors, such as security vulnerabilities [35, 47] can guide tool design. For this reason, some automated repair techniques focus on a...

Citation Context

...as synthesis-based repair [26, 46, 60] techniques, is also of great value, but is outside the scope of this paper. Our contribution is a controlled investigation of GenProg [35, 62] and TrpAutoRepair =-=[49]-=-, both test-case-guided, search-based automatic program repair tools with freely available implementations that scale to large programs. The evaluation identifies the circumstances under which these t...

Citation Context

...ting tools General Terms: Experimentation Keywords: automated program repair, empirical evaluation, independent evaluation, GenProg, TrpAutoRepair, INTROCLASS 1. INTRODUCTION Automated program repair =-=[4,13,17,18,26,28,29,32,36,37,39,39, 43,46,47,50,55,58,60,61]-=- holds great potential to reduce debugging costs and improve software quality. For example, GenProg quickly and cheaply generated patches for 55 out of 105 C bugs [32], while PAR showed comparable res...

Citation Context

...cular classes of errors, such as security vulnerabilities [35, 47] can guide tool design. For this reason, some automated repair techniques focus on a particular defect class, such as buffer overruns =-=[54, 57]-=-, unsafe integer use in C programs [17], single-variable atomicity violations [26], deadlock and livelock defects [36], concurrency errors [37], and data input errors [4]. Other techniques tackle gene...

Citation Context

...ting tools General Terms: Experimentation Keywords: automated program repair, empirical evaluation, independent evaluation, GenProg, TrpAutoRepair, INTROCLASS 1. INTRODUCTION Automated program repair =-=[4,13,17,18,26,28,29,32,36,37,39,39, 43,46,47,50,55,58,60,61]-=- holds great potential to reduce debugging costs and improve software quality. For example, GenProg quickly and cheaply generated patches for 55 out of 105 C bugs [32], while PAR showed comparable res...

Citation Context

...specifications. As examples, Clearview [47], GenProg, Par, and Debroy and Wong [18] have successfully fixed bugs in legacy software. Although new projects appear to be increasingly adopting contracts =-=[20]-=-, their penetration into existing systems and languages remains limited. Few maintained contract implementations exist for widely-used languages such as C. As an example, as of March 2014, in the Debi...

Citation Context

...er 4, 2015, Bergamo, Italy . 978-1-4503-3675-8/15/08...$15.00 http://dx.doi.org/10.1145/2786805.2786825 532 respect to some specification representation test suites. We produce the INTROCLASS dataset =-=[34]-=- for our evaluation by collecting 998 student-written programs with defects, submitted as homework in a freshman programming class, and all with student-written, bugfixing patches. Each program is acc...

Citation Context

...ting tools General Terms: Experimentation Keywords: automated program repair, empirical evaluation, independent evaluation, GenProg, TrpAutoRepair, INTROCLASS 1. INTRODUCTION Automated program repair =-=[4,13,17,18,26,28,29,32,36,37,39,39, 43,46,47,50,55,58,60,61]-=- holds great potential to reduce debugging costs and improve software quality. For example, GenProg quickly and cheaply generated patches for 55 out of 105 C bugs [32], while PAR showed comparable res...

Citation Context

...echniques provide the benefit of provable correctness for patches, but require contracts, so they are unsuitable for legacy systems. Synthesis techniques can also construct new features from examples =-=[16, 23]-=-, rather than address existing bugs. Our work has focused on G&V approaches, and investigating overfitting and patch quality in synthesisbased techniques is a complementary and worthwhile pursuit. The...

Citation Context

...tions of automated repair techniques that relied on test cases or workloads to validate candidate patches failed to evaluate those patches independently of patch construction. More recent work (e.g., =-=[50, 64]-=-) has begun to consider independent quality measures, though less extensively than we do here. And while some evaluations have used humans to independently measure repair acceptability [19, 29] and ma...

Citation Context

...ting tools General Terms: Experimentation Keywords: automated program repair, empirical evaluation, independent evaluation, GenProg, TrpAutoRepair, INTROCLASS 1. INTRODUCTION Automated program repair =-=[4,13,17,18,26,28,29,32,36,37,39,39, 43,46,47,50,55,58,60,61]-=- holds great potential to reduce debugging costs and improve software quality. For example, GenProg quickly and cheaply generated patches for 55 out of 105 C bugs [32], while PAR showed comparable res...

Citation Context

...ting tools General Terms: Experimentation Keywords: automated program repair, empirical evaluation, independent evaluation, GenProg, TrpAutoRepair, INTROCLASS 1. INTRODUCTION Automated program repair =-=[4,13,17,18,26,28,29,32,36,37,39,39, 43,46,47,50,55,58,60,61]-=- holds great potential to reduce debugging costs and improve software quality. For example, GenProg quickly and cheaply generated patches for 55 out of 105 C bugs [32], while PAR showed comparable res...

Citation Context

...ting tools General Terms: Experimentation Keywords: automated program repair, empirical evaluation, independent evaluation, GenProg, TrpAutoRepair, INTROCLASS 1. INTRODUCTION Automated program repair =-=[4,13,17,18,26,28,29,32,36,37,39,39, 43,46,47,50,55,58,60,61]-=- holds great potential to reduce debugging costs and improve software quality. For example, GenProg quickly and cheaply generated patches for 55 out of 105 C bugs [32], while PAR showed comparable res...