Solving Least
Squares Problems
SIAM's Classics in Applied Mathematics series consists of books that were previously
allowed to go out of print. These books are republished by SIAM as a professional
service because they continue to be important resources for mathematical scientists.
Editor-in-Chief
Robert E. O'Malley, Jr., University of Washington
Editorial Board
Richard A. Brualdi, University of Wisconsin-Madison
Herbert B. Keller, California Institute of Technology
Andrzej Z. Manitius, George Mason University
Ingram Olkin, Stanford University
Stanley Richardson, University of Edinburgh
Ferdinand Verhulst, Mathematisch Instituut, University of Utrecht
Classics in Applied Mathematics
C. C. Lin and L. A. Segel, Mathematics Applied to Deterministic Problems in the
Natural Sciences
Johan G. F. Belinfante and Bernard Kolman, A Survey of Lie Groups and Lie Algebras
with Applications and Computational Methods
James M. Ortega, Numerical Analysis: A Second Course
Anthony V. Fiacco and Garth P. McCormick, Nonlinear Programming: Sequential
Unconstrained Minimization Techniques
F. H. Clarke, Optimization and Nonsmooth Analysis
George F. Carrier and Carl E. Pearson, Ordinary Differential Equations
Leo Breiman, Probability
R. Bellman and G. M. Wing, An Introduction to Invariant Imbedding
Abraham Berman and Robert J. Plemmons, Nonnegative Matrices in the Mathemat-
ical Sciences
Olvi L. Mangasarian, Nonlinear Programming
*Carl Friedrich Gauss, Theory of the Combination of Observations Least Subject
to Errors: Part One, Part Two, Supplement. Translated by G. W. Stewart
Richard Bellman, Introduction to Matrix Analysis
U. M. Ascher, R. M. M. Mattheij, and R. D. Russell, Numerical Solution of Boundary
Value Problems for Ordinary Differential Equations
K. E. Brenan, S. L. Campbell, and L. R. Petzold, Numerical Solution of Initial-Value
Problems in Differential-Algebraic Equations
Charles L. Lawson and Richard J. Hanson, Solving Least Squares Problems
J. E. Dennis, Jr. and Robert B. Schnabel, Numerical Methods for Unconstrained
Optimization and Nonlinear Equations
Richard E. Barlow and Frank Proschan, Mathematical Theory of Reliability
Cornelius Lanczos, Linear Differential Operators
Richard Bellman, Introduction to Matrix Analysis, Second Edition
Beresford N. Parlett, The Symmetric Eigenvalue Problem
*First time in print.
Classics in Applied Mathematics (continued)
Richard Haberman, Mathematical Models: Mechanical Vibrations, Population
Dynamics, and Traffic Flow
Peter W. M. John, Statistical Design and Analysis of Experiments
Tamer Basar and Geert Jan Olsder, Dynamic Noncooperative Game Theory, Second
Edition
Emanuel Parzen, Stochastic Processes
Petar Kokotovic, Hassan K. Khalil, and John O'Reilly, Singular Perturbation Methods
in Control* Analysis and Design
Jean Dickinson Gibbons, Ingram Olkin, and Milton Sobel, Selecting and Ordering
Populations: A New Statistical Methodology
James A. Murdock, Perturbations: Theory and Methods
Ivar Ekeland and Roger T6mam, Convex Analysis and Variational Problems
Ivar Stakgold, Boundary Value Problems of Mathematical Physics, Volumes I and II
J. M. Ortega and W. C. Rheinboldt, Iterative Solution of Nonlinear Equations in
Several Variables
David Kinderlehrer and Guido Stampacchia, An Introduction to Variational
Inequalities and Their Applications
F. Natterer, The Mathematics of Computerized Tomography
Avinash C. Kak and Malcolm Slaney, Principles of Computerized Tomographic Imaging
R. Wong, Asymptotic Approximations of Integrals
O. Axelsson and V. A. Barker, Finite Element Solution of Boundary Value Problems:
Theory and Computation
David R. Brillinger, Time Series: Data Analysis and Theory
Joel N. Franklin, Methods of Mathematical Economics: Linear and Nonlinear
Programming, Fixed-Point Theorems
Philip Hartman, Ordinary Differential Equations, Second Edition
Michael D. Intriligator, Mathematical Optimization and Economic Theory
Philippe G. Ciarlet, The Finite Element Method for Elliptic Problems
Jane K. Cullum and Ralph A. Willoughby, Lanc^os Algorithms for Large Symmetric
Eigenvalue Computations, Vol. I: Theory
M. Vidyasagar, Nonlinear Systems Analysis, Second Edition
Robert Mattheij and Jaap Molenaar, Ordinary Differential Equations in Theory and
Practice
Shanti S. Gupta and S. Panchapakesan, Multiple Decision Procedures: Theory and
Methodology of Selecting and Ranking Populations
Eugene L. Allgower and Kurt Georg, Introduction to Numerical Continuation Methods
Heinz-Otto Kreiss and Jens Lorenz, Initial-Boundary Value Problems and the Navier-
Sto/ces Equations
This page intentionally left blank
Solving Least
Squares Problems
Charles L. Lawson
San Clemente, California
Richard J. Hanson
Rice University
Houston, Texas
siam.
Society for Industrial and Applied Mathematics
Philadelphia
Copyright © 1995 by the Society for Industrial and Applied Mathematics. This SIAM
edition is an unabridged, revised republication of the work first published by Prentice-Hall,
Inc., Englewood Cliffs, New Jersey, 1974.
10 9 8 7 6 5
All rights reserved. Printed in the United States of America. No part of this book may be
reproduced, stored, or transmitted in any manner without the written permission of the
publisher. For information, write to the Society for Industrial and Applied Mathematics,
3600 University City Science Center, Philadelphia, PA 19104-2688.
Library of Congress Cataloging-in-Publication Data
Lawson, Charles L.
Hanson.
p.
Solving least squares problems / Charles L. Lawson, Richard J.
cm. - (Classics in applied mathematics; 15)
"This SIAM edition is an unabridged, revised republication of the
work first published by Prentice-Hall, Inc., Englewood Cliffs, New
Jersey, 1974"--T.p. verso.
Includes bibliographical references and index.
ISBN 0-89871-356-0 (pbk.)
1. Least squares-Data processing. I. Hanson, Richard J., 1938-
II. Title.
III. Series.
QA275.L38 1995
511'.42--dc20
95-35178
Siam. is a registered trademark.
Contents
Preface to the Classics Edition .»„.»......»...»....„...«...„..,.„....»...».»........„.„.......»»........... ix
xi
Preface
Chapter 1 Introduction ...........................1
Chapter 2 Analysis of the Least Squares Problem ............................................... 5
Chapter 3 Orthogonal Decomposition by Certain Elementary
Orthogonal Transformations................................................................. 9
Chapter 4 Orthogonal Decomposition by Singular
Value Decomposition .....................................................
...—........... 18
Chapter 5
Perturbation Theorems for Singular Values............................... 23
Chapter 6 Bounds for the Condition Number of a Triangular Matrix..... 28
Chapter 7 The Pseudoinverse..........................^...................^...^....^..................... 36
Chapter 8
Perturbation Bounds for the Pseudoinverse............................. 41
Chapter 9
Perturbation Bounds for the Solution of Problem LS................ 49
Chapter 10 Numerical Computations Using Elementary
Orthogonal Transformations ....................«..^.».^««..................... 53
Chapter 11 Computing the Solution for the Overdetermined
or Exactly Determined Full Rank Problem
63
Chapter 12 Computation of the Covariance Matrix
or the solution Parameters »»«...........»«.........................................................67
Chapter 13 Computing the Solution for the Underdetermined
Full Rank Problem
Chapter 14 Computing the Solution for Problem LS
with Possibly Deficient Pseudorank
74
77
Chapter 15 Analysis of Computing Errors
for Householder Transformations...................................................... 83
Chapter 16 Analysis of Computing Errors for the Problem LS .................... 90
vii