Foundations of Signal Processing
Martin Vetterli
´Ecole Polytechnique F´ed´erale de Lausanne
Jelena Kovaˇcevi´c
Carnegie Mellon University
Vivek K Goyal
Massachusetts Institute of Technology & Boston University
May 31, 2014
Copyright (c) 2014 Martin Vetterli, Jelena Kovaˇcevi´c, and Vivek K Goyal.
These materials are protected by copyright under the
Attribution-NonCommercial-NoDerivs 3.0 Unported License
from Creative Commons.
FoundationsofSignalProcessingCopyright2014M.Vetterli,J.Kovaˇcevi´c,andV.K.GoyalCambridgeUniversityPress(ISBN110703860X)v1.1[May2014][freeversion]Commentstobook-errata@FourierAndWavelets.org
To Marie-Laure, for her ∞ patience and many other qualities,
Thomas and No´emie, whom I might still convince of the beauty of this material,
and my parents, who gave me all the opportunities one can wish for.
— MV
To Danica and Giovanni, who make life beautiful.
To my parents, who made me who I am.
— JK
To Allie, Sundeep, and my family, who encourage me unceasingly, and
to the educators who made me want to be one of them.
— VKG
FoundationsofSignalProcessingCopyright2014M.Vetterli,J.Kovaˇcevi´c,andV.K.GoyalCambridgeUniversityPress(ISBN110703860X)v1.1[May2014][freeversion]Commentstobook-errata@FourierAndWavelets.org
The cover illustration captures an experiment first described by Isaac Newton in
Opticks in 1730, showing that white light can be split into its color components and
then synthesized back into white light. It is a physical implementation of a decom-
position of white light into its Fourier components – the colors of the rainbow –
followed by a synthesis to recover the original.
FoundationsofSignalProcessingCopyright2014M.Vetterli,J.Kovaˇcevi´c,andV.K.GoyalCambridgeUniversityPress(ISBN110703860X)v1.1[May2014][freeversion]Commentstobook-errata@FourierAndWavelets.org
Contents
Quick reference
Acknowledgments
Preface
1 On rainbows and spectra
2 From Euclid to Hilbert
Introduction
2.1
2.2 Vector spaces
Inner product
2.2.1 Definition and properties
2.2.2
2.2.3 Norm
2.2.4 Standard spaces
2.3 Hilbert spaces
2.3.1 Convergence
2.3.2 Completeness
2.3.3 Linear operators
2.4 Approximations, projections, and decompositions
2.4.1 Projection theorem
2.4.2 Projection operators
2.4.3 Direct sums and subspace decompositions
2.4.4 Minimum mean-squared error estimation
2.5 Bases and frames
2.5.1 Bases and Riesz bases
2.5.2 Orthonormal bases
2.5.3 Biorthogonal pairs of bases
2.5.4 Frames
2.5.5 Matrix representations of vectors and linear operators
2.6 Computational aspects
2.6.1 Cost, complexity, and asymptotic notations
2.6.2 Precision
2.6.3 Conditioning
2.6.4 Solving systems of linear equations
2.A Elements of analysis and topology
2.A.1 Basic definitions
ix
page xv
xxi
xxiii
1
9
10
18
18
23
27
30
35
36
37
40
50
51
54
60
63
69
69
76
86
101
109
119
120
123
126
129
135
135
FoundationsofSignalProcessingCopyright2014M.Vetterli,J.Kovaˇcevi´c,andV.K.GoyalCambridgeUniversityPress(ISBN110703860X)v1.1[May2014][freeversion]Commentstobook-errata@FourierAndWavelets.org
x
3
2.A.2 Convergence
2.A.3 Interchange theorems
2.A.4 Inequalities
2.A.5 Integration by parts
2.B Elements of linear algebra
2.B.1 Basic definitions and properties
2.B.2 Special matrices
2.C Elements of probability
2.C.1 Basic definitions
2.C.2 Standard distributions
2.C.3 Estimation
2.D Basis concepts
Chapter at a glance
Historical remarks
Further reading
Exercises with solutions
Exercises
Sequences and discrete-time systems
3.1
3.2
Infinite-length sequences
Introduction
Sequences
3.2.1
3.2.2 Finite-length sequences
3.2.3 Two-dimensional sequences
Systems
3.3.1 Discrete-time systems and their properties
3.3.2 Difference equations
3.3.3 Linear shift-invariant systems
3.3
3.4 Discrete-time Fourier transform
3.4.1 Definition of the DTFT
3.4.2 Existence and convergence of the DTFT
3.4.3 Properties of the DTFT
3.4.4 Frequency response of filters
z-transform
3.5.1 Definition of the z-transform
3.5.2 Existence and convergence of the z-transform
3.5.3 Properties of the z-transform
3.5.4
z-transform of filters
3.5
3.6 Discrete Fourier transform
3.6.1 Definition of the DFT
3.6.2 Properties of the DFT
3.6.3 Frequency response of filters
3.7 Multirate sequences and systems
3.7.1 Downsampling
3.7.2 Upsampling
3.7.3 Combinations of downsampling and upsampling
3.7.4 Combinations of downsampling, upsampling, and filtering
3.7.5 Polyphase representation
Stochastic processes and systems
3.8.1 Stochastic processes
3.8
Contents
136
138
139
140
141
141
147
151
151
154
155
159
161
162
162
163
169
181
182
185
185
192
193
195
195
202
205
216
216
218
221
227
233
234
235
240
249
252
252
255
259
264
265
268
270
272
278
285
285
FoundationsofSignalProcessingCopyright2014M.Vetterli,J.Kovaˇcevi´c,andV.K.GoyalCambridgeUniversityPress(ISBN110703860X)v1.1[May2014][freeversion]Commentstobook-errata@FourierAndWavelets.org
Contents
3.8.2 Systems
3.8.3 Discrete-time Fourier transform
3.8.4 Multirate sequences and systems
3.8.5 Minimum mean-squared error estimation
3.9 Computational aspects
3.9.1 Fast Fourier transforms
3.9.2 Convolution
3.9.3 Multirate operations
3.A Elements of analysis
3.A.1 Complex numbers
3.A.2 Difference equations
3.A.3 Convergence of the convolution sum
3.A.4 Dirac delta function
3.B Elements of algebra
3.B.1 Polynomials
3.B.2 Vectors and matrices of polynomials
3.B.3 Kronecker product
Chapter at a glance
Historical remarks
Further reading
Exercises with solutions
Exercises
4 Functions and continuous-time systems
Introduction
4.1
4.2 Functions
4.3
4.2.1 Functions on the real line
4.2.2 Periodic functions
Systems
4.3.1 Continuous-time systems and their properties
4.3.2 Differential equations
4.3.3 Linear shift-invariant systems
4.4 Fourier transform
4.4.1 Definition of the Fourier transform
4.4.2 Existence and inversion of the Fourier transform
4.4.3 Properties of the Fourier transform
4.4.4 Frequency response of filters
4.4.5 Regularity and spectral decay
4.4.6 Laplace transform
4.5 Fourier series
4.5.1 Definition of the Fourier series
4.5.2 Existence and convergence of the Fourier series
4.5.3 Properties of the Fourier series
4.5.4 Frequency response of filters
Stochastic processes and systems
4.6.1 Stochastic processes
4.6.2 Systems
4.6.3 Fourier transform
4.6
Chapter at a glance
Historical remarks
xi
288
292
294
300
303
303
307
311
313
313
315
316
316
318
318
321
324
325
328
328
329
336
343
344
345
345
351
351
352
355
355
359
359
360
365
373
374
379
380
381
383
385
394
395
395
397
399
401
403
FoundationsofSignalProcessingCopyright2014M.Vetterli,J.Kovaˇcevi´c,andV.K.GoyalCambridgeUniversityPress(ISBN110703860X)v1.1[May2014][freeversion]Commentstobook-errata@FourierAndWavelets.org
xii
Contents
Further reading
Exercises with solutions
Exercises
5
Sampling and interpolation
5.1
5.2 Finite-dimensional vectors
Introduction
5.3
5.2.1 Sampling and interpolation with orthonormal vectors
5.2.2 Sampling and interpolation with nonorthogonal vectors
Sequences
5.3.1 Sampling and interpolation with orthonormal sequences
5.3.2 Sampling and interpolation for bandlimited sequences
5.3.3 Sampling and interpolation with nonorthogonal sequences
5.4 Functions
5.4.1 Sampling and interpolation with orthonormal functions
5.4.2 Sampling and interpolation for bandlimited functions
5.4.3 Sampling and interpolation with nonorthogonal functions
5.5 Periodic functions
5.5.1 Sampling and interpolation with orthonormal periodic functions
5.5.2 Sampling and interpolation for bandlimited periodic functions
5.6 Computational aspects
5.6.1 Projection onto convex sets
Chapter at a glance
Historical remarks
Further reading
Exercises with solutions
Exercises
6 Approximation and compression
Introduction
6.1
6.2 Approximation of functions on finite intervals by polynomials
6.2.1 Least-squares approximation
6.2.2 Lagrange interpolation: Matching points
6.2.3 Taylor series expansion: Matching derivatives
6.2.4 Hermite interpolation: Matching points and derivatives
6.2.5 Minimax polynomial approximation
6.2.6 Filter design
6.3 Approximation of functions by splines
403
404
406
411
412
420
421
425
429
430
437
442
447
449
452
470
477
477
481
489
491
496
498
498
500
504
507
508
513
514
517
520
522
523
529
537
538
541
548
6.3.1 Splines and spline spaces
6.3.2 Bases for uniform spline spaces
6.3.3 Strang–Fix condition for polynomial representation
6.3.4 Continuous-time operators in spline spaces implemented with discrete-
time processing
6.4 Approximation of functions and sequences by series truncation
6.4.1 Linear and nonlinear approximations
6.4.2 Linear approximation of random vectors and stochastic processes
6.4.3 Linear and nonlinear diagonal estimators
6.5 Compression
6.5.1 Lossless compression
6.5.2 Scalar quantization
554
560
560
566
571
576
577
579
FoundationsofSignalProcessingCopyright2014M.Vetterli,J.Kovaˇcevi´c,andV.K.GoyalCambridgeUniversityPress(ISBN110703860X)v1.1[May2014][freeversion]Commentstobook-errata@FourierAndWavelets.org
Contents
6.5.3 Transform coding
6.6 Computational aspects
6.6.1 Huffman algorithm for lossless code design
6.6.2
6.6.3 Estimating from quantized samples
Iterative design of quantizers
Chapter at a glance
Historical remarks
Further reading
Exercises with solutions
Exercises
7
Localization and uncertainty
7.1
7.2 Localization for functions
Introduction
7.2.1 Localization in time
7.2.2 Localization in frequency
7.2.3 Uncertainty principle for functions
7.3 Localization for sequences
7.3.1 Localization in time
7.3.2 Localization in frequency
7.3.3 Uncertainty principle for sequences
7.3.4 Uncertainty principle for finite-length sequences
7.4 Tiling the time–frequency plane
7.4.1 Localization for structured sets of functions
7.4.2 Localization for structured sets of sequences
7.5 Examples of local Fourier and wavelet bases
7.5.1 Local Fourier and wavelet bases for functions
7.5.2 Local Fourier and wavelet bases for sequences
7.6 Recap and a glimpse forward
7.6.1 Tools
7.6.2 Adapting tools to real-world problems
7.6.3 Music analysis, communications, and compression
Chapter at a glance
Historical remarks
Further reading
Exercises with solutions
Exercises
Image and quote attribution
References
Index
xiii
584
591
591
593
594
597
598
599
601
607
615
616
619
620
622
624
627
629
630
633
637
638
638
640
645
645
651
656
657
658
659
666
667
667
669
671
673
675
681
FoundationsofSignalProcessingCopyright2014M.Vetterli,J.Kovaˇcevi´c,andV.K.GoyalCambridgeUniversityPress(ISBN110703860X)v1.1[May2014][freeversion]Commentstobook-errata@FourierAndWavelets.org