OPTIMAL CONTROL
Optimal Control, Third Edition Frank L. Lewis, Draguna L. Vrabie and Vassilis L. Syrmos
Copyright © 2012 John Wiley & Sons, Inc.
OPTIMAL CONTROL
Third Edition
FRANK L. LEWIS
Department of Electrical Engineering
Automation & Robotics Research Institute
University of Texas at Arlington
Arlington, Texas
DRAGUNA L. VRABIE
United Technologies Research Center
East Hartford, Connecticut
VASSILIS L. SYRMOS
Department of Electrical Engineering
University of Hawaii at Manoa
Honolulu, Hawaii
JOHN WILEY & SONS, INC.
This book is printed on acid-free paper.
Copyright © 2012 by John Wiley & Sons, Inc. All rights reserved.
Published by John Wiley & Sons, Inc., Hoboken, New Jersey.
Published simultaneously in Canada.
No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any
form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise,
except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without
either the prior written permission of the Publisher, or authorization through payment of the
appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA
01923, (978) 750-8400, fax (978) 646-8600, or on the web at www.copyright.com. Requests to the
Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons,
Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at
www.wiley.com/go/permissions.
Limit of Liability/Disclaimer of Warranty: While the publisher and the author have used their best
efforts in preparing this book, they make no representations or warranties with respect to the
accuracy or completeness of the contents of this book and specifically disclaim any implied
warranties of merchantability or fitness for a particular purpose. No warranty may be created or
extended by sales representatives or written sales materials. The advice and strategies contained
herein may not be suitable for your situation. You should consult with a professional where
appropriate. Neither the publisher nor the author shall be liable for any loss of profit or any other
commercial damages, including but not limited to special, incidental, consequential, or other
damages.
For general information about our other products and services, please contact our Customer Care
Department within the United States at (800) 762-2974, outside the United States at (317)
572-3993 or fax (317) 572-4002.
Wiley publishes in a variety of print and electronic formats and by print-on-demand. Some
material included with standard print versions of this book may not be included in e-books or in
print-on-demand. If this book refers to media such as a CD or DVD that is not included in the
version you purchased, you may download this material at http://booksupport.wiley.com. For more
information about Wiley products, visit www.wiley.com.
Library of Congress Cataloging-in-Publication Data:
Lewis, Frank L.
Optimal control / Frank L. Lewis, Draguna L. Vrabie, Vassilis L. Syrmos.—3rd ed.
p. cm.
Includes bibliographical references and index.
ISBN 978-0-470-63349-6 (cloth); ISBN 978-1-118-12263-1 (ebk); ISBN 978-1-118-12264-8 (ebk);
ISBN 978-1-118-12266-2 (ebk); ISBN 978-1-118-12270-9 (ebk); ISBN 978-1-118-12271-6 (ebk);
ISBN 978-1-118-12272-3 (ebk)
1. Control theory. 2. Mathematical optimization. I. Vrabie, Draguna L. II. Syrmos, Vassilis L. III.
Title.
QA402.3.L487 2012
629.8’312–dc23
2011028234
Printed in the United States of America
10
9
8
4
3
7
6
5
2
1
To Galina, Roma, and Chris, who make every day exciting
—Frank Lewis
To my mother and my grandmother, for teaching me my potential and
supporting my every choice
—Draguna Vrabie
To my father, my first teacher
—Vassilis Syrmos
CONTENTS
PREFACE
1 STATIC OPTIMIZATION
1.1
1.2
1.3
Optimization without Constraints / 1
Optimization with Equality Constraints / 4
Numerical Solution Methods / 15
Problems / 15
xi
1
2 OPTIMAL CONTROL OF DISCRETE-TIME SYSTEMS
19
2.1
2.2
2.3
2.4
2.5
Solution of the General Discrete-Time Optimization Problem / 19
Discrete-Time Linear Quadratic Regulator / 32
Digital Control of Continuous-Time Systems / 53
Steady-State Closed-Loop Control and Suboptimal Feedback / 65
Frequency-Domain Results / 96
Problems / 102
3 OPTIMAL CONTROL OF CONTINUOUS-TIME
SYSTEMS
3.1
3.2
3.3
The Calculus of Variations / 110
Solution of the General Continuous-Time Optimization
Problem / 112
Continuous-Time Linear Quadratic Regulator / 135
110
vii
viii
CONTENTS
3.4
3.5
Steady-State Closed-Loop Control and Suboptimal Feedback / 154
Frequency-Domain Results / 164
Problems / 167
4 THE TRACKING PROBLEM AND OTHER
LQR EXTENSIONS
177
4.1
4.2
4.3
4.4
4.5
4.6
The Tracking Problem / 177
Regulator with Function of Final State Fixed / 183
Second-Order Variations in the Performance Index / 185
The Discrete-Time Tracking Problem / 190
Discrete Regulator with Function of Final State Fixed / 199
Discrete Second-Order Variations in the Performance Index / 206
Problems / 211
5 FINAL-TIME-FREE AND CONSTRAINED
INPUT CONTROL
5.1
5.2
Final-Time-Free Problems / 213
Constrained Input Problems / 232
Problems / 257
6 DYNAMIC PROGRAMMING
6.1
6.2
6.3
Bellman’s Principle of Optimality / 260
Discrete-Time Systems / 263
Continuous-Time Systems / 271
Problems / 283
213
260
7 OPTIMAL CONTROL FOR POLYNOMIAL SYSTEMS
287
7.1
7.2
Discrete Linear Quadratic Regulator / 287
Digital Control of Continuous-Time Systems / 292
Problems / 295
8 OUTPUT FEEDBACK AND STRUCTURED CONTROL
297
8.1
8.2
8.3
8.4
8.5
8.6
Linear Quadratic Regulator with Output Feedback / 297
Tracking a Reference Input / 313
Tracking by Regulator Redesign / 327
Command-Generator Tracker / 331
Explicit Model-Following Design / 338
Output Feedback in Game Theory and Decentralized Control / 343
Problems / 351
9 ROBUSTNESS AND MULTIVARIABLE
FREQUENCY-DOMAIN TECHNIQUES
Introduction / 355
9.1
9.2 Multivariable Frequency-Domain Analysis / 357
9.3
9.4
9.5
9.6 H∞ DESIGN / 430
Robust Output-Feedback Design / 380
Observers and the Kalman Filter / 383
LQG/Loop-Transfer Recovery / 408
Problems / 435
10 DIFFERENTIAL GAMES
CONTENTS
ix
355
438
10.1 Optimal Control Derived Using Pontryagin’s Minimum Principle
and the Bellman Equation / 439
10.2 Two-player Zero-sum Games / 444
10.3 Application of Zero-sum Games to H∞ Control / 450
10.4 Multiplayer Non-zero-sum Games / 453
11 REINFORCEMENT LEARNING AND OPTIMAL ADAPTIVE
CONTROL
461
11.1 Reinforcement Learning / 462
11.2 Markov Decision Processes / 464
11.3 Policy Evaluation and Policy Improvement / 474
11.4 Temporal Difference Learning and Optimal Adaptive Control / 489
11.5 Optimal Adaptive Control for Discrete-time Systems / 490
11.6
Integral Reinforcement Learning for Optimal Adaptive Control of
Continuous-time Systems / 503
11.7 Synchronous Optimal Adaptive Control for Continuous-time
Systems / 513
APPENDIX A REVIEW OF MATRIX ALGEBRA
518
Partitioned Matrices / 519
A.1 Basic Definitions and Facts / 518
A.2
A.3 Quadratic Forms and Definiteness / 521
A.4 Matrix Calculus / 523
A.5
The Generalized Eigenvalue Problem / 525
REFERENCES
INDEX
527
535