The aim of this chapter is to provide guidance to the prospective user of the Adaptive Critic / Approximate Dynamic Programming methods for designing the action device in certain kinds of control systems. While there are currently various different successful “camps ” in

Abstract not found

Abstract—Some three decades ago, certain computational in-telligence methods of reinforcement learning were recognized as implementing an approximation of Bellman’s Dynamic Pro-gramming method, which is known in the controls community as an important tool for designing optimal control policies for nonlinear plants and sequential decision making. Significant theoretical and practical developments have occurred within this arena, mostly in the past decade, with the methodology now usually referred to as Adaptive Dynamic Programming (ADP). The objective of this paper is to provide a retrospective of selected threads of such developments. In addition, a com-mentary is offered concerning present status of ADP, and threads for future research and development within the con-trols field are suggested. I. HISTORICAL BACKGROUND HILE existence of control devices dates back to antiq-uity (call it Phase 1 for the controls field), it may be said Maxwell’s use of differential equations to analyze the dynamics of a flyball governor (ca. 1870) [34] that had been invented by James Watt in 1788 [65] ushered in a new phase for the controls field (call it PHASE 2). Mathematics has played a fundamental role in Phase 2, progressing through Fourier and Laplace transforms, state space methods, stochastic methods, Hilbert space methods, and more recent-ly, algebraic and geometric topological methods. The advent of modern computers with their fantastic evolution the past few decades has also been significant, not only from the im-plementation point of view, but also as a driver and motivator for various mathematical and algorithmic developments as well. An important aspect of the Phase 2 methods, however, is that the controller so designed is placed in service with no associated mechanism for modifying its design in response to context changes, be they in the plant or its environment. This phase includes at least the following well known design me-