Linear Algebra and Its Applications
Fourth Edition
Gilbert Strang
yxyzzAxbb0Ayb0Az0
Contents
Preface
iv
1 Matrices and Gaussian Elimination
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
1
1
1.1
1.2 The Geometry of Linear Equations
4
1.3 An Example of Gaussian Elimination . . . . . . . . . . . . . . . . . . . . 13
1.4 Matrix Notation and Matrix Multiplication . . . . . . . . . . . . . . . . . 21
1.5 Triangular Factors and Row Exchanges . . . . . . . . . . . . . . . . . . . 36
1.6
Inverses and Transposes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
1.7 Special Matrices and Applications . . . . . . . . . . . . . . . . . . . . . . 66
Review Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
2 Vector Spaces
77
2.1 Vector Spaces and Subspaces . . . . . . . . . . . . . . . . . . . . . . . . . 77
2.2 Solving Ax = 0 and Ax = b . . . . . . . . . . . . . . . . . . . . . . . . . . 86
2.3 Linear Independence, Basis, and Dimension . . . . . . . . . . . . . . . . 103
2.4 The Four Fundamental Subspaces . . . . . . . . . . . . . . . . . . . . . . 115
2.5 Graphs and Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
2.6 Linear Transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
Review Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
3 Orthogonality
159
3.1 Orthogonal Vectors and Subspaces . . . . . . . . . . . . . . . . . . . . . . 159
3.2 Cosines and Projections onto Lines
. . . . . . . . . . . . . . . . . . . . . 171
3.3 Projections and Least Squares . . . . . . . . . . . . . . . . . . . . . . . . 180
3.4 Orthogonal Bases and Gram-Schmidt
. . . . . . . . . . . . . . . . . . . . 195
3.5 The Fast Fourier Transform . . . . . . . . . . . . . . . . . . . . . . . . . . 211
Review Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
i
ii
CONTENTS
4 Determinants
224
4.1
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
4.2 Properties of the Determinant
. . . . . . . . . . . . . . . . . . . . . . . . 226
4.3 Formulas for the Determinant
. . . . . . . . . . . . . . . . . . . . . . . . 235
4.4 Applications of Determinants . . . . . . . . . . . . . . . . . . . . . . . . . 246
Review Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
5 Eigenvalues and Eigenvectors
258
5.1
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
5.2 Diagonalization of a Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . 271
5.3 Difference Equations and Powers Ak . . . . . . . . . . . . . . . . . . . . . 281
5.4 Differential Equations and eAt
. . . . . . . . . . . . . . . . . . . . . . . . 294
5.5 Complex Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
5.6 Similarity Transformations . . . . . . . . . . . . . . . . . . . . . . . . . . 323
Review Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338
6 Positive Definite Matrices
342
6.1 Minima, Maxima, and Saddle Points . . . . . . . . . . . . . . . . . . . . . 342
6.2 Tests for Positive Definiteness . . . . . . . . . . . . . . . . . . . . . . . . 349
6.3 Singular Value Decomposition . . . . . . . . . . . . . . . . . . . . . . . . 364
6.4 Minimum Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373
6.5 The Finite Element Method . . . . . . . . . . . . . . . . . . . . . . . . . . 381
7 Computations with Matrices
387
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387
7.1
7.2 Matrix Norm and Condition Number
. . . . . . . . . . . . . . . . . . . . 388
7.3 Computation of Eigenvalues . . . . . . . . . . . . . . . . . . . . . . . . . 396
Iterative Methods for Ax = b . . . . . . . . . . . . . . . . . . . . . . . . . 405
7.4
8 Linear Programming and Game Theory
414
8.1 Linear Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414
8.2 The Simplex Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419
8.3 The Dual Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431
8.4 Network Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
8.5 Game Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 448
456
A Intersection, Sum, and Product of Spaces
. . . . . . . . . . . . . . . . . . . 456
A.1 The Intersection of Two Vector Spaces
A.2 The Sum of Two Vector Spaces
. . . . . . . . . . . . . . . . . . . . . . . 457
A.3 The Cartesian Product of Two Vector Spaces . . . . . . . . . . . . . . . . 458
A.4 The Tensor Product of Two Vector Spaces
. . . . . . . . . . . . . . . . . 458
A.5 The Kronecker Product A⊗ B of Two Matrices . . . . . . . . . . . . . . . 459
CONTENTS
B The Jordan Form
C Matrix Factorizations
D Glossary: A Dictionary for Linear Algebra
E MATLAB Teaching Codes
F Linear Algebra in a Nutshell
iii
463
470
472
481
483
Preface
Revising this textbook has been a special challenge, for a very nice reason. So many
people have read this book, and taught from it, and even loved it. The spirit of the book
could never change. This text was written to help our teaching of linear algebra keep up
with the enormous importance of this subject—which just continues to grow.
One step was certainly possible and desirable—to add new problems. Teaching for all
these years required hundreds of new exam questions (especially with quizzes going onto
the web). I think you will approve of the extended choice of problems. The questions are
still a mixture of explain and compute—the two complementary approaches to learning
this beautiful subject.
I personally believe that many more people need linear algebra than calculus. Isaac
Newton might not agree! But he isn’t teaching mathematics in the 21st century (and
maybe he wasn’t a great teacher, but we will give him the benefit of the doubt). Cer-
tainly the laws of physics are well expressed by differential equations. Newton needed
calculus—quite right. But the scope of science and engineering and management (and
life) is now so much wider, and linear algebra has moved into a central place.
May I say a little more, because many universities have not yet adjusted the balance
toward linear algebra. Working with curved lines and curved surfaces, the first step is
always to linearize. Replace the curve by its tangent line, fit the surface by a plane,
and the problem becomes linear. The power of this subject comes when you have ten
variables, or 1000 variables, instead of two.
You might think I am exaggerating to use the word “beautiful” for a basic course
in mathematics. Not at all. This subject begins with two vectors v and w, pointing in
different directions. The key step is to take their linear combinations. We multiply to
get 3v and 4w, and we add to get the particular combination 3v + 4w. That new vector
is in the same plane as v and w. When we take all combinations, we are filling in the
whole plane. If I draw v and w on this page, their combinations cv + dw fill the page
(and beyond), but they don’t go up from the page.
In the language of linear equations, I can solve cv + dw = b exactly when the vector
b lies in the same plane as v and w.
iv
Matrices
v
I will keep going a little more to convert combinations of three-dimensional vectors into
linear algebra. If the vectors are v = (1,2,3) and w = (1,3,4), put them into the columns
of a matrix:
matrix =
2 3
3 4
.
1 1
1 1
2 3
3 4
c
d
1
+ d
1
.
3
4
= c
2
3
To find combinations of those columns, “multiply” the matrix by a vector (c,d):
Linear combinations cv + dw
Those combinations fill a vector space. We call it the column space of the matrix. (For
these two columns, that space is a plane.) To decide if b = (2,5,7) is on that plane, we
have three components to get right. So we have three equations to solve:
1 1
2 3
3 4
c
d
=
2
5
7
means
c + d = 2
2c + 3d = 5
3c + 4d = 7
.
I leave the solution to you. The vector b = (2,5,7) does lie in the plane of v and w.
If the 7 changes to any other number, then b won’t lie in the plane—it will not be a
combination of v and w, and the three equations will have no solution.
Now I can describe the first part of the book, about linear equations Ax = b. The
matrix A has n columns and m rows. Linear algebra moves steadily to n vectors in m-
dimensional space. We still want combinations of the columns (in the column space).
We still get m equations to produce b (one for each row). Those equations may or may
not have a solution. They always have a least-squares solution.
The interplay of columns and rows is the heart of linear algebra. It’s not totally easy,
but it’s not too hard. Here are four of the central ideas:
1. The column space (all combinations of the columns).
2. The row space (all combinations of the rows).
3. The rank (the number of independent columns) (or rows).
4. Elimination (the good way to find the rank of a matrix).
I will stop here, so you can start the course.
vi
Web Pages
PREFACE
It may be helpful to mention the web pages connected to this book. So many messages
come back with suggestions and encouragement, and I hope you will make free use
of everything. You can directly access http://web.mit.edu/18.06, which is continually
updated for the course that is taught every semester. Linear algebra is also on MIT’s
OpenCourseWare site http://ocw.mit.edu, where 18.06 became exceptional by including
videos of the lectures (which you definitely don’t have to watch...). Here is a part of
what is available on the web:
1. Lecture schedule and current homeworks and exams with solutions.
2. The goals of the course, and conceptual questions.
3. Interactive Java demos (audio is now included for eigenvalues).
4. Linear Algebra Teaching Codes and MATLAB problems.
5. Videos of the complete course (taught in a real classroom).
The course page has become a valuable link to the class, and a resource for the students.
I am very optimistic about the potential for graphics with sound. The bandwidth for
voiceover is low, and FlashPlayer is freely available. This offers a quick review (with
active experiment), and the full lectures can be downloaded.
I hope professors and
students worldwide will find these web pages helpful. My goal is to make this book as
useful as possible with all the course material I can provide.
Other Supporting Materials
Student Solutions Manual 0-495-01325-0 The Student Solutions Manual provides
solutions to the odd-numbered problems in the text.
Instructor’s Solutions Manual 0-030-10588-4 The Instructor’s Solutions Man-
ual has teaching notes for each chapter and solutions to all of the problems in the text.
Structure of the Course
The two fundamental problems are Ax = b and Ax = λx for square matrices A. The first
problem Ax = b has a solution when A has independent columns. The second problem
Ax = λx looks for independent eigenvectors. A crucial part of this course is to learn
what “independence” means.
I believe that most of us learn first from examples. You can see that
1 1 2
1 2 3
1 3 4
A =
does not have independent columns.