Numerical Optimization
Jorge Nocedal
Stephen J. Wright
Springer
Springer Series in Operations Research
Editors:
Peter Glynn
Stephen M. Robinson
Springer
New York
Berlin
Heidelberg
Barcelona
Hong Kong
London
Milan
Paris
Singapore
Tokyo
Jorge Nocedal
Stephen J. Wright
Numerical Optimization
With 85 Illustrations
1 3
Jorge Nocedal
ECE Department
Northwestern University
Evanston, IL 60208-3118
USA
Stephen J. Wright
Mathematics and Computer
Science Division
Argonne National Laboratory
9700 South Cass Avenue
Argonne, IL 60439-4844
USA
Series Editors:
Peter Glynn
Department of Operations Research
Stanford University
Stanford, CA 94305
USA
Stephen M. Robinson
Department of Industrial Engineering
University of Wisconsin–Madison
1513 University Avenue
Madison, WI 53706-1572
USA
Cover illustration is from Pre-Hispanic Mexican Stamp Designs by Frederick V. Field, courtesy of Dover Publi-
cations, Inc.
Library of Congress Cataloging-in-Publication Data
Nocedal, Jorge.
Numerical optimization / Jorge Nocedal, Stephen J. Wright.
p.
cm. — (Springer series in operations research)
Includes bibliographical references and index.
ISBN 0-387-98793-2 (hardcover)
1. Mathematical optimization.
I. Wright, Stephen J., 1960–
.
II. Title.
QA402.5.N62
519.3—dc21
III. Series.
1999
99–13263
© 1999 Springer-Verlag New York, Inc.
All rights reserved. This work may not be translated or copied in whole or in part without the written permission
of the publisher (Springer-Verlag New York, Inc., 175 Fifth Avenue, New York, NY 10010, USA), except for brief
excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage
and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or
hereafter developed is forbidden.
The use of general descriptive names, trade names, trademarks, etc., in this publication, even if the former are
not especially identified, is not to be taken as a sign that such names, as understood by the Trade Marks and
Merchandise Marks Act, may accordingly be used freely by anyone.
ISBN 0-387-98793-2 Springer-Verlag New York Berlin Heidelberg SPIN 10764949
To Our Parents:
Ra´ul and Concepci´on
Peter and Berenice
Preface
This is a book for people interested in solving optimization problems. Because of the wide
(and growing) use of optimization in science, engineering, economics, and industry, it is
essential for students and practitioners alike to develop an understanding of optimization
algorithms. Knowledge of the capabilities and limitations of these algorithms leads to a better
understanding of their impact on various applications, and points the way to future research
on improving and extending optimization algorithms and software. Our goal in this book
is to give a comprehensive description of the most powerful, state-of-the-art, techniques
for solving continuous optimization problems. By presenting the motivating ideas for each
algorithm, we try to stimulate the reader’s intuition and make the technical details easier to
follow. Formal mathematical requirements are kept to a minimum.
Because of our focus on continuous problems, we have omitted discussion of important
optimization topics such as discrete and stochastic optimization. However, there are a great
many applications that can be formulated as continuous optimization problems; for instance,
finding the optimal trajectory for an aircraft or a robot arm;
identifying the seismic properties of a piece of the earth’s crust by fitting a model of
the region under study to a set of readings from a network of recording stations;
viii
P r e f a c e
designing a portfolio of investments to maximize expected return while maintaining
an acceptable level of risk;
controlling a chemical process or a mechanical device to optimize performance or
meet standards of robustness;
computing the optimal shape of an automobile or aircraft component.
Every year optimization algorithms are being called on to handle problems that are
much larger and complex than in the past. Accordingly, the book emphasizes large-scale
optimization techniques, such as interior-point methods, inexact Newton methods, limited-
memory methods, and the role of partially separable functions and automatic differentiation.
It treats important topics such as trust-region methods and sequential quadratic program-
ming more thoroughly than existing texts, and includes comprehensive discussion of such
“core curriculum” topics as constrained optimization theory, Newton and quasi-Newton
methods, nonlinear least squares and nonlinear equations, the simplex method, and penalty
and barrier methods for nonlinear programming.
THE AUDIENCE
We intend that this book will be used in graduate-level courses in optimization, as of-
fered in engineering, operations research, computer science, and mathematics departments.
There is enough material here for a two-semester (or three-quarter) sequence of courses.
We hope, too, that this book will be used by practitioners in engineering, basic science, and
industry, and our presentation style is intended to facilitate self-study. Since the book treats
a number of new algorithms and ideas that have not been described in earlier textbooks, we
hope that this book will also be a useful reference for optimization researchers.
Prerequisites for this book include some knowledge of linear algebra (including nu-
merical linear algebra) and the standard sequence of calculus courses. To make the book as
self-contained as possible, we have summarized much of the relevant material from these ar-
eas in the Appendix. Our experience in teaching engineering students has shown us that the
material is best assimilated when combined with computer programming projects in which
the student gains a good feeling for the algorithms—their complexity, memory demands, and
elegance—and for the applications. In most chapters we provide simple computer exercises
that require only minimal programming proficiency.
EMPHASIS AND WRITING STYLE
We have used a conversational style to motivate the ideas and present the numerical
algorithms. Rather than being as concise as possible, our aim is to make the discussion flow
in a natural way. As a result, the book is comparatively long, but we believe that it can be
read relatively rapidly. The instructor can assign substantial reading assignments from the
text and focus in class only on the main ideas.