Abstract not found

Mixture of experts (ME) is a modular neural network architecture for supervised learning. A double-loop Expectation-Maximization (EM) algorithm has been introduced to the ME architecture for adjusting the parameters and the iteratively reweighted least squares (IRLS) algorithm is used to perform maximization in the inner loop [Jordan, M.I., Jacobs, R.A. (1994). Hierarchical mixture of experts and the EM algorithm, Neural Computation, 6(2), 181--214]. However, it is reported in literature that the IRLS algorithm is of instability and the ME architecture trained by the EM algorithm, where IRLS algorithm is used in the inner loop, often produces the poor performance in multiclass classification. In this paper, the reason of this instability is explored. We find out that due to an implicitly imposed incorrect assumption on parameter independence in multiclass classification, an incomplete Hessian matrix is used in that IRLS algorithm. Based on this finding, we apply the Newton--Raphson met...

Enquiries about copyright, reproduction and requests for

fter a general treatment of primal and dual active set strategies, we present the Dual m Active Set Algorithm for linear programming and prove its convergence. An efficient impleentation is developed using proximal point approximations. This implementation involves a b primal/dual proximal iteration similar to one introduced by Rockafellar, and a new iteration ased on optimization of a proximal vector parameter. This proximal parameter optimization , w problem is well conditioned, leading to rapid convergence of the conjugate gradient method hile the original proximal function is terribly conditioned, leading to almost undetectable conz vergence of the conjugate gradient method. Limits as a proximal scalar parameter tends to ero are evaluated. Intriguing numerical results are presented for Netlib test problems. t s Key Words. Linear programming, quadratic programming, active sets, dual method, leas quares, proximal point, extrapolation, conjugate gradients, successive over-relexation ...

We propose an active set algorithm to solve the convex quadratic programming (QP) problem which is the core of the support vector machine (SVM) training. The underlying method is not new and have been used by the optimization community to solve moderately sized convex QPs. However, its application to large scale SVM problems is new. The special structure of SVM problems makes it possible to adapt this algorithm to work eciently for large problems. We also incorporate several \tricks" (such as shrinking) used by other advanced SVM implementations. We present computational results comparing our method with Joachims' SVM [11]. The results show that our method has overall similar or better behavior on many SVM problems. It has particularly strong advantage in problems that either have bad separability in the feature space or have a lot of noise; i.e., problems where the solution contains a lot of vectors violating the margin condition (outliers). Another advantage of the method is that it does not require the user to \guess" the chunk size and that it typically produces more accurate solution in similar time. The algorithm also naturally extends to the incremental mode.

We consider numerical methods for finding (weak) second-order critical points for large-scale non-convex quadratic programming problems. We describe two new methods. The first is of the active-set variety. Although convergent from any starting point, it is intended primarily for the case where a good estimate of the optimal active set can be predicted. The second is an interior-point trust-region type, and has proved capable of solving problems involving up to half a million unknowns and constraints. The solution of a key equality constrained subproblem, common to both methods, is described. The results of comparative tests on a large set of convex and non-convex quadratic programming examples are given.

Abstract. We present a two-phase algorithm for solving large-scale quadratic programs (QPs). In the first phase, gradient-projection iterations approximately minimize a bound-constrained augmented Lagrangian function and provide an estimate of the optimal active set. In the second phase, an equality-constrained QP defined by the current active set is approximately minimized in order to generate a second-order search direction. A filter determines the required accuracy of the subproblem solutions and provides an acceptance criterion for the search directions. The resulting algorithm is globally and finitely convergent. The algorithm is suitable for large-scale problems with many degrees of freedom, and provides an alternative to interior-point methods when iterative methods must be used to solve the underlying linear systems. Numerical experiments on a subset of the CUTEr QP test problems demonstrate the effectiveness of the approach. Key words. Large-scale optimization, quadratic programming, gradient-projection, active-set methods, filter methods, augmented Lagrangian. AMS subject classifications. 65K05, 90C06, 90C20, 90C26, 90C52 1. Introduction. Quadratic programs (QPs) play a fundamental role in optimization. They are useful across a rich class of applications, such as the simulation

We propose an active set algorithm to solve the convex quadratic programming (QP) problem which is the core of the support vector machine (SVM) training. The underlying method is not new and is based on the extensive practice of the Simplex method and its variants for convex quadratic problems.

QP is the optimization of a quadratic function subject to linear equality and inequality constraints. It arises in multiple objective decision making where the departure of the actual decisions from their corresponding ideal, or bliss, value can be evaluated using a weighted quadratic norm as a measure of deviation. The formulation of mean-variance optimization of uncertain systems also leads to QP. An important application of mean-variance is in simple optimal portfolio problems where the constraints are linear and the objective function is quadratic (Markowitz, 1959). The decision maker has to reconcile the conflicting desires of maximizing expected portfolio return, represented by the linear term, and minimizing the portfolio risk, represented by the quadratic (variance) term, in the objective function. Sequential QP algorithms require the solution of QP subproblems to generate descent directions for general nonlinear optimization and minimax.

Several ways of implementing methods for solving nonlinear optimization problems involving linear inequality and equality constraints using numerically stable matrix factorizations are described. The methods considered all follow an active constraint set approach and include quadratic programming, variable metric, and modified Newton methods. 1.