this paper, we consider two approaches to computing the desired implicitly defined derivative x

Mixture of experts (ME) is a modular neural network architecture for supervised learning. A double-loop Expectation-Maximization (EM) algorithm has been introduced to the ME architecture for adjusting the parameters and the iteratively reweighted least squares (IRLS) algorithm is used to perform maximization in the inner loop [Jordan, M.I., Jacobs, R.A. (1994). Hierarchical mixture of experts and the EM algorithm, Neural Computation, 6(2), 181--214]. However, it is reported in literature that the IRLS algorithm is of instability and the ME architecture trained by the EM algorithm, where IRLS algorithm is used in the inner loop, often produces the poor performance in multiclass classification. In this paper, the reason of this instability is explored. We find out that due to an implicitly imposed incorrect assumption on parameter independence in multiclass classification, an incomplete Hessian matrix is used in that IRLS algorithm. Based on this finding, we apply the Newton--Raphson met...

Abstract not found

There exist redundant, irrelevant and noisy data. Using proper data to train a network can speed up training, simplify the learned structure, and improve its performance. A two-phase training algorithm is proposed. In the first phase, the number of input units of the network is determined by using an information base method. Only those attributes that meet certain criteria for inclusion will be considered as the input to the network. In the second phase, the number of hidden units of the network is selected automatically based on the performance of the network on the training data. One hidden unit is added at a time only if it is necessary. The experimental results show that this new algorithm can achieve a faster learning time, a simpler network and an improved performance.

In this survey paper, an overview on the different approaches for solving nonlinear optimization problems is given. The presentation includes theory, numeric, interval and symbolic methods.

Surrogate-based-optimization methods provide a means to minimize expensive highfidelity models at reduced computational cost. The methods are useful in problems for which two models of the same physical system exist: a high-fidelity model which is accurate and expensive, and a low-fidelity model which is less costly but less accurate. A number of model management techniques have been developed and shown to work well for the case in which both models are defined over the same design space. However, many systems exist with variable fidelity models for which the design variables are defined over different spaces, and a mapping is required between the spaces. Previous work showed that two mapping methods, corrected space mapping and POD mapping, used in conjunction with a trust-region model management method, provide improved performance over conventional non-surrogate-based optimization methods for unconstrained problems. This paper extends that work to constrained problems. Three constraint-management methods are demonstrated with each of the mapping methods: Lagrangian minimization, an sequential quadratic programming-like surrogate method, and MAESTRO. The methods are demonstrated on a fixed-complexity analytical test problem and a variable-complexity wing design problem. The SQP-like method consistently outperformed optimization in the high-fidelity space and the other variable complexity methods. Corrected space mapping performed slightly better on average than POD mapping. On the wing design problem, the combination of the SQP-like method and corrected space mapping achieved 58 % savings in high-fidelity function calls over optimization directly in the high-fidelity space. I.

Abstract — We investigate nonlinear pre-equalization techniques for the downlink of multiuser systems from a transmitter equipped with multiple antennas to noncooperative single antenna mobile receivers. Besides linear Multiuser Transmission (MUT), known as Transmit Zero Forcing or Transmit Wiener Filters, recently nonlinear Minimum Bit Error Rate (BER) MUT was proposed. The latter computes the transmit signal by minimizing the mean BER at the detectors by a constrained numerical optimization method. This paper presents a new framework for an unconstrained optimization for systems with a fixed transmit power constraint. Furthermore, the Minimum BER MUT is extended to higher-order modulation. Simulation results reveal a superior performance of the proposed scheme over linear MUT while the computational complexity is significantly reduced compared to the constrained Minimum BER MUT approach. I.

Abstract. A new method for quasi-Newton minimization outperforms BFGS by combining least-change updates of the Hessian with step sizes estimated from a Wishart model of uncertainty. The Hessian update is in the Broyden family but uses a negative parameter, outside the convex range, that is usually regarded as the safe zone for Broyden updates. Although full Newton steps based on this update tend to be too long, excellent performance is obtained with shorter steps estimated from the Wishart model. In numerical comparisons to BFGS the new statistical quasi-Newton (SQN) algorithm typically converges with about 25 % fewer iterations, functions, and gradient evaluations on the top 1/3 hardest unconstrained problems in the CUTE library. Typical improvement on the 1/3 easiest problems is about 5%. The framework used to derive SQN provides a simple way to understand differences among various Broyden updates such as BFGS and DFP and shows that these methods do not preserve accuracy of the Hessian, in a certain sense, while the new method does. In fact, BFGS, DFP, and all other updates with nonnegative Broyden parameters tend to inflate Hessian estimates, and this accounts for their observed propensity to correct eigenvalues that are too small more readily than eigenvalues that are too large. Numerical results on three new test functions validate these conclusions.

The objective of this research has been to investigate the effectiveness of nonlinear optimization techniques to optimize the performance of hydrocarbon producing wells. The performance of a production well is a function of several variables. Examples of these variables are tubing size, choke size, and perforation density. Changing any of the variables will alter the performance of the well. There are several ways to optimize the function of well performance. An insensible way to optimize the function of well performance is by exhaustive iteration: optimizing a single variable by trial and error while holding all other variables constant, and repeating the procedure for different variables. This procedure is computationally expensive and slow to converge--especially if the variables are interrelated. A prudent manner to optimize the function of well performance is by numerical optimization, particularly nonlinear optimization. Nonlinear optimization finds the combination of these varia...

Cost functions formulated in four-dimensional variational data assimilation (4DVAR) are nonsmooth in the presence of discontinuous physical processes (i.e., the presence of "on--off" switches in NWP models). The adjoint model integration produces values of subgradients, instead of gradients, of these cost functions with respect to the model's control variables at discontinuous points. Minimization of these cost functions using conventional differentiable optimization algorithms may encounter difficulties. In this paper an idealized discontinuous model and an actual shallow convection parameterization are used, both including on--off switches, to illustrate the performances of differentiable and nondifferentiable optimization algorithms. It was found that (i) the differentiable optimization, such as the limited memory quasi-Newton (L-BFGS) algorithm, may still work well for minimizing a nondifferentiable cost function, especially when the changes made in the forecast model at switching points to the model state are not too large; (ii) for a differentiable optimization algorithm to find the true minimum of a nonsmooth cost function, introducing a local smoothing that removes discontinuities may lead to more problems than solutions due to the insertion of artificial stationary points; and (iii) a nondifferentiable optimization algorithm is found to be able to find the true minima in cases where the differentiable minimization failed. For the case of strong smoothing, differentiable minimization performance is much improved, as compared to the weak smoothing cases.