Results 1  10
of
12
Portfolio Allocation for Bayesian Optimization
"... Bayesian optimization with Gaussian processes has become an increasingly popular tool in the machine learning community. It is efficient and can be used when very little is known about the objective function, making it popular in expensive blackbox optimization scenarios. It uses Bayesian methods t ..."
Abstract

Cited by 23 (14 self)
 Add to MetaCart
(Show Context)
Bayesian optimization with Gaussian processes has become an increasingly popular tool in the machine learning community. It is efficient and can be used when very little is known about the objective function, making it popular in expensive blackbox optimization scenarios. It uses Bayesian methods to sample the objective efficiently using an acquisition function which incorporates the posterior estimate of the objective. However, there are several different parameterized acquisition functions in the literature, and it is often unclear which one to use. Instead of using a single acquisition function, we adopt a portfolio of acquisition functions governed by an online multiarmed bandit strategy. We propose several portfolio strategies, the best of which we call GPHedge, and show that this method outperforms the best individual acquisition function. We also provide a theoretical bound on the algorithm’s performance. 1
Anytime planning for decentralized POMDPs using expectation maximization
 IN UAI
, 2010
"... Decentralized POMDPs provide an expressive framework for multiagent sequential decision making. While finitehorizon DECPOMDPs have enjoyed significant success, progress remains slow for the infinitehorizon case mainly due to the inherent complexity of optimizing stochastic controllers representi ..."
Abstract

Cited by 15 (7 self)
 Add to MetaCart
Decentralized POMDPs provide an expressive framework for multiagent sequential decision making. While finitehorizon DECPOMDPs have enjoyed significant success, progress remains slow for the infinitehorizon case mainly due to the inherent complexity of optimizing stochastic controllers representing agent policies. We present a promising new class of algorithms for the infinitehorizon case, which recasts the optimization problem as inference in a mixture of DBNs. An attractive feature of this approach is the straightforward adoption of existing inference techniques in DBNs for solving DECPOMDPs and supporting richer representations such as factored or continuous states and actions. We also derive the Expectation Maximization (EM) algorithm to optimize the joint policy represented as DBNs. Experiments on benchmark domains show that EM compares favorably against the stateoftheart solvers.
Multistage stochastic programming: A scenario tree based approach to planning under uncertainty
 APPLICATIONS IN ARTIFICIAL INTELLIGENCE: CONCEPTS AND SOLUTIONS, CHAPTER 6
, 2011
"... In this chapter, we present the multistage stochastic programming framework for sequential decision making under uncertainty and stress its differences with Markov Decision Processes. We describe the main approximation technique used for solving problems formulated in the multistage stochastic progr ..."
Abstract

Cited by 8 (6 self)
 Add to MetaCart
In this chapter, we present the multistage stochastic programming framework for sequential decision making under uncertainty and stress its differences with Markov Decision Processes. We describe the main approximation technique used for solving problems formulated in the multistage stochastic programming framework, which is based on a discretization of the disturbance space. We explain that one issue of the approach is that the discretization scheme leads in practice to illposed problems, because the complexity of the numerical optimization algorithms used for computing the decisions restricts the number of samples and optimization variables that one can use for approximating expectations, and therefore makes the numerical solutions very sensitive to the parameters of the discretization. As the framework is weak in the absence of efficient tools for evaluating and eventually selecting competing approximate solutions, we show how one can extend it by using machine learning based techniques, so as to yield a sound and generic method to solve approximately a large class of multistage decision problems under uncertainty. The framework and solution techniques presented in the chapter are explained and illustrated on several examples. Along the way, we describe notions from decision theory that are relevant to sequential decision making under uncertainty in general.
MCMC for continuoustime discretestate systems
"... We propose a simple and novel framework for MCMC inference in continuoustime discretestate systems with pure jump trajectories. We construct an exact MCMC sampler for such systems by alternately sampling a random discretization of time given a trajectory of the system, and then a new trajectory giv ..."
Abstract

Cited by 6 (3 self)
 Add to MetaCart
(Show Context)
We propose a simple and novel framework for MCMC inference in continuoustime discretestate systems with pure jump trajectories. We construct an exact MCMC sampler for such systems by alternately sampling a random discretization of time given a trajectory of the system, and then a new trajectory given the discretization. The first step can be performed efficiently using properties of the Poisson process, while the second step can avail of discretetime MCMC techniques based on the forwardbackward algorithm. We show the advantage of our approach compared to particle MCMC and a uniformizationbased sampler. 1
Bayesian MultiScale Optimistic Optimization
"... Bayesian optimization is a powerful global optimization technique for expensive blackbox functions. One of its shortcomings is that it requires auxiliary optimization of an acquisition function at each iteration. This auxiliary optimization can be costly and very hard to carry out in practice. M ..."
Abstract

Cited by 6 (3 self)
 Add to MetaCart
Bayesian optimization is a powerful global optimization technique for expensive blackbox functions. One of its shortcomings is that it requires auxiliary optimization of an acquisition function at each iteration. This auxiliary optimization can be costly and very hard to carry out in practice. Moreover, it creates serious theoretical concerns, as most of the convergence results assume that the exact optimum of the acquisition function can be found. In this paper, we introduce a new technique for efficient global optimization that combines Gaussian process confidence bounds and treed simultaneous optimistic optimization to eliminate the need for auxiliary optimization of acquisition functions. The experiments with global optimization benchmarks and a novel application to automatic information extraction demonstrate that the resulting technique is more efficient than the two approaches from which it draws inspiration. Unlike most theoretical analyses of Bayesian optimization with Gaussian processes, our finitetime convergence rate proofs do not require exact optimization of an acquisition function. That is, our approach eliminates the unsatisfactory assumption that a difficult, potentially NPhard, problem has to be solved in order to obtain vanishing regret rates. 1
Analyzing and Escaping Local Optima in Planning as Inference for Partially Observable Domains
"... Abstract. Planning as inference recently emerged as a versatile approach to decisiontheoretic planning and reinforcement learning for single and multiagent systems in fully and partially observable domains with discrete and continuous variables. Since planning as inference essentially tackles a no ..."
Abstract

Cited by 5 (3 self)
 Add to MetaCart
(Show Context)
Abstract. Planning as inference recently emerged as a versatile approach to decisiontheoretic planning and reinforcement learning for single and multiagent systems in fully and partially observable domains with discrete and continuous variables. Since planning as inference essentially tackles a nonconvex optimization problem when the states are partially observable, there is a need to develop techniques that can robustly escape local optima. We investigate the local optima of finite state controllers in single agent partially observable Markov decision processes (POMDPs) that are optimized by expectation maximization (EM). We show that EM converges to controllers that are optimal with respect to a onestep lookahead. To escape local optima, we propose two algorithms: the first one adds nodes to the controller to ensure optimality with respect to a multistep lookahead, while the second one splits nodes in a greedy fashion to improve reward likelihood. The approaches are demonstrated empirically on benchmark problems. 1
An entropy search portfolio for bayesian optimization. arXiv:1406.4625
, 2014
"... Bayesian optimization is a sampleefficient method for blackbox global optimization. However, the performance of a Bayesian optimization method very much depends on its exploration strategy, i.e. the choice of acquisition function, and it is not clear a priori which choice will result in superi ..."
Abstract

Cited by 2 (2 self)
 Add to MetaCart
(Show Context)
Bayesian optimization is a sampleefficient method for blackbox global optimization. However, the performance of a Bayesian optimization method very much depends on its exploration strategy, i.e. the choice of acquisition function, and it is not clear a priori which choice will result in superior performance. While portfolio methods provide an effective, principled way of combining a collection of acquisition functions, they are often based on measures of past performance which can be misleading. To address this issue, we introduce the Entropy Search Portfolio (ESP): a novel approach to portfolio construction which is motivated by information theoretic considerations. We show that ESP outperforms existing portfolio methods on several real and synthetic problems, including geostatistical datasets and simulated control tasks. We not only show that ESP is able to offer performance as good as the best, but unknown, acquisition function, but surprisingly it often gives better performance. Finally, over a wide range of conditions we find that ESP is robust to the inclusion of poor acquisition functions. 1
Inference strategies for solving semiMarkov decision processes
"... SemiMarkov decision processes (SMDPs) generalize standard MDPs to domains where time is not discretized equally between every set of states and actions [3]. Instead we can define a jumpMarkov process where the amount of time spent in each state is a stochastic random variable. This formulation giv ..."
Abstract

Cited by 1 (0 self)
 Add to MetaCart
SemiMarkov decision processes (SMDPs) generalize standard MDPs to domains where time is not discretized equally between every set of states and actions [3]. Instead we can define a jumpMarkov process where the amount of time spent in each state is a stochastic random variable. This formulation gives us an intuitive way to reason about actions where it is also necessary to take into account how long these actions will take to perform. Formally we can define an SMDP as a continuoustime controlled stochastic process (x(t), u(t)) consisting, respectively, of states and actions at every point in time t where state transitions occur at random arrival times Tn. In particular, the process is stationary in between jumps, i.e. x(t) = xn and u(t) = un
Probabilistic Inference Techniques for Scalable Multiagent Decision Making
, 2015
"... Decentralized POMDPs provide an expressive framework for multiagent sequential decision making. However, the complexity of these models—NEXPComplete even for two agents—has limited their scalability. We present a promising new class of approximation algorithms by developing novel connections betwe ..."
Abstract
 Add to MetaCart
Decentralized POMDPs provide an expressive framework for multiagent sequential decision making. However, the complexity of these models—NEXPComplete even for two agents—has limited their scalability. We present a promising new class of approximation algorithms by developing novel connections between multiagent planning and machine learning. We show how the multiagent planning problem can be reformulated as inference in a mixture of dynamic Bayesian networks (DBNs). This planningasinference approach paves the way for the application of efficient inference techniques in DBNs to multiagent decision making. To further improve scalability, we identify certain conditions that are sufficient to extend the approach to multiagent systems with dozens of agents. Specifically, we show that the necessary inference within the expectationmaximization framework can be decomposed into processes that often involve a small subset of agents, thereby facilitating scalability. We further show that a number of existing multiagent planning models satisfy these conditions. Experiments on large planning benchmarks confirm the benefits of our approach in terms of runtime and scalability with respect to existing techniques.
Revision
, 2010
"... Comparative evaluation of approaches in T.4.14.3 and working definition of adaptive module Authors: ..."
Abstract
 Add to MetaCart
(Show Context)
Comparative evaluation of approaches in T.4.14.3 and working definition of adaptive module Authors: