Abstract. Machine learning can be used to build models that predict the runtime of search algorithms for hard combinatorial problems. Such empirical hardness models have previously been studied for complete, deterministic search algorithms. In this work, we demonstrate that such models can also make surprisingly accurate predictions of the run-time distributions of incomplete and randomized search methods, such as stochastic local search algorithms. We also show for the first time how information about an algorithm’s parameter settings can be incorporated into a model, and how such models can be used to automatically adjust the algorithm’s parameters on a per-instance basis in order to optimize its performance. Empirical results for Novelty + and SAPS on structured and unstructured SAT instances show very good predictive performance and significant speedups of our automatically determined parameter settings when compared to the default and best fixed distribution-specific parameter settings. 1

Abstract. Advances in local-search SAT solvers have traditionally been presented in the context of local search solvers only. The most recent and rather comprehensive comparisons between UnitWalk and several versions of WalkSAT demonstrate that neither solver dominates on all benchmarks. QingTing2 (a ‘dragonfly ’ in Mandarin) is a SAT solver script that relies on a novel switching strategy to invoke one of the two local search solvers: WalkSAT or QingTing1. The local search solver Qing-Ting1 implements the UnitWalk algorithm with a new unit-propagation technique. The experimental methodology we use not only demonstrates the effectiveness of the switching strategy and the efficiency of the new unit-propagation implementation – it also supports, on the very same instances, statistically significant performance evaluation between local search and other state-of-the-art DPLL-based SAT solvers. The resulting comparisons show a surprising pattern of solver dominance, completely unanticipated when we began this work. 1

The low-autocorrelation binary sequence (LABS) problem represents a major challenge to all search algorithms, with the evolutionary algorithms claiming the best results so far. However, the termination criteria for these types of stochastic algorithms are not well-defined and no claims have been made about optimality. Our approach to find the optima of the LABS problem is based on (1) experiments with problem sizes for which optimal solutions are known, (2) an asymptotic analysis of statistics generated by such experiments, (3) reliable predictions of the cost required to find optimal solutions for larger problem sizes. The proposed methodology provides a well-defined termination criterion for evolutionary and alternative search algorithms alike.

Combinatorial optimization problems that may be expressed as ‘Boolean constraint satisfaction problems ’ (BCSPs) are being solved by different communities under different formulations and in different formats. If results of experimentation are reported, these can be seldom compared and replicated. We propose a pragmatic approach to reconcile these issues: (1) use the familiar LP model that naturally expresses the constraints as well as the goals of the optimization task to formulate an optimization instance, (2) assemble and translate a number of hard-to-solve instances from different domains into the.lpx format parsed by at least two BCSP solvers: lp solve in public domain, and cplex, (3) expose the intrinsic variability of BCSP solvers by constructing instance isomorphs as an equivalence class of randomized replicas of a reference instance; (4) use isomorph classes for the design of reproducible experiments with BCSP solvers that includes performance testing hypotheses; (5) release (on the web) all data sets, reported results, and software utilities used to prepare the data, invoke experiments, and post-process the results. 1.

Combinatorial optimization problems expressed as Boolean constraint satisfaction problems (BCSPs) arise in several contexts, ranging from the classical unate set-packing problems to the binate minimum cover problems, including the Haplotype Inference by Pure Parsimony (HIPP) problem. These problems are being solved under different formulations and in different formats. Results of experiments that are reported can be seldom compared and replicated. This paper is not about ‘the best BCSP solver’. Rather, it is a case study of how the scientific method can be applied to comparing the performance of not only BCSP solvers but also other solvers that address NP-hard problems. The approach is founded on two premises: (1) the introduction of instance isomorphs as families of equivalence classes, based on randomized replicas of a given reference instance, and (2) the use of isomorph classes for the design of reproducible experiments with BCSP solvers that includes performance testing hypotheses. We introduce a number of BCSP reference instances from different domains, generate isomorph classes and use various versions of cplex to characterize the solver performance and the isomorph classes themselves. This methodology may make it easier to (1) reliably improve the performance of combinatorial solvers and, (2) report results of experiments under the proposed schema. Categories and Subject Descriptors:

Abstract. Preconditioning of matrices, with the objective to solve large systems of linear equations more efficiently, is an active area of research. In contrast, there is no comparable systematic effort to precondition graph-based instances before solving them with a combinatorial solver. This paper asks the question: Does an existing preconditioning technique with known merits in solving systems of linear equations also improves the efficiency and effectiveness for a class of solvers on instances of combinatorial problems? We propose an experimental approach to evaluate merits of the Fiedler permutation when solving instances of the maximal independent set (MaxIS) problem with several solvers. Preliminary results are not only encouraging, they also demonstrate the value of Fiedler permutation when characterizing fundamental structural properties of graph instances themselves. Version: 2008-TR Preconditioning v1-Brglez, 28-Jan-2008 1

The low-autocorrelation binary sequence (LABS) problem represents a major challenge to all search algorithms, with the evolutionary algorithms claiming the best results so far. However, the termination criteria for these types of stochastic algorithms are not well-defined and no reliable claims have been made about optimality. Our approach to find the optima of the LABS problem is based on combining three principles into a single method: (1) solver performance experiments with problem sizes for which optimal solutions are known, (2) an asymptotic statistical analysis of such experiments, (3) reliable predictions of the computational cost required to find optimal solutions for larger problem sizes. The proposed methodology provides a well-defined termination criterion for evolutionary and alternative search algorithms alike. Version: 2008-TR labs v2-Brglez, 12-Feb-2008

Summary. This document gives an overview of how to use the SATbed software for managing various aspects of experiments with satisfiability solvers. The encapsulated solvers whose output is post-processed by SATbed include chaff [MMZ + 01], OpenSat [LeB03], QingTing (two versions) [LSB03], sato [Zha97], UnitWalk (versions 0.944 and 0.98) [HK01, HK03], and WalkSat [SK02]. Background Materials. (included with the SATbed release) • A SATbed tutorial in PowerPoint (file 2003-SATbed-Brglez).

Abstract. Analysis of our recent experiments for a group of SAT solvers and several classes of problem instances suggests a common mathematical framework with experiments in component reliability. In the latter, we observe the distribution of component lifetime; in the former, we observe the distribution of execution time (runtime). A lifetime distribution for hardware components is frequently found to have an exponential, Weibull, Pareto, or gamma distribution. Our experiments with state-of-the-art SAT solvers reveal normal distributions and exponential distributions of runtime and other related random variables, as well as other distributions commonly observed in reliability applications. SATbed, an experimental testbed for SAT solvers, emulates the reliability framework: equivalence classes of isomorphic problem instances (replicated hardware components), subjected to tests with specific SAT solvers (specifically controlled environments), observations of runtime, implications, etc. (lifetime), statistical analysis and modeling, based on random variable samples. The testbed not only facilitates systematic study and reliable improvement of any SAT solver but also supports the introduction and validation of new problem instance classes. 1

Abstract. In several fields, Satisfiability being one, there are regular competitions to compare multiple solvers in a common setting. Due to the fact some benchmarks of interest are too difficult for all solvers to complete within available time, time-outs occur and must be considered. Through some strange evolution, time-outs became the only factor that was considered in evaluation. Previous work in SAT 2010 observed that this evaluation method is unreliable and lacks a way to attach statistical significance to its conclusions. However, the proposed alternative was quite complicated and is unlikely to see general use. This paper describes a simpler system, called careful ranking, that permits a measure of statistical significance, and still meets many of the practical requirements of an evaluation system. It incorporates one of the main ideas of the previous work: that outcomes had to be freed of assumptions about timing distributions, so that non-parametric methods were necessary. Unlike the previous work, it incorporates ties.