A
COURSE IN
PROBABILITY
THEORY
THIRD EDITION
A
COURSE IN
PROBABILITY
THEORY
THIRD EDITION
Kai Lai Chung
Stanford University
ACADEMIC PRESS
A l-iorcc_lurt Science and Technology Company
San Diego
San Francisco
New York
Boston
London Sydney
Tokyo
This book is printed on acid-free paper .
COPYRIGHT © 2001, 1974, 1968 BY ACADEMIC PRESS
ALL RIGHTS RESERVED.
NO PART OF THIS PUBLICATION MAY BE REPRODUCED OR TRANSMITTED IN ANY FORM OR BY ANY
MEANS, ELECTRONIC OR MECHANICAL, INCLUDING PHOTOCOPY, RECORDING, OR ANY INFORMATION
STORAGE AND RETRIEVAL SYSTEM . WITHOUT PERMISSION IN WRITING FROM THE PUBLISHER .
Requests for permission to make copies of any part of the work should be mailed to the
following address : Permissions Department, Harcourt, Inc ., 6277 Sea Harbor Drive, Orlando,
Florida 32887-6777 .
ACADEMIC PRESS
A Harcourt Science and Technology Company
525 B Street, Suite 1900, San Diego, CA 92101-4495, USA
h ttp ://www .academicpress .co m
ACADEMIC PRESS
Harcourt Place, 32 Jamestown Road, London, NW 17BY, UK
http ://www .academicpress .com
Library of Congress Cataloging in Publication Data : 00-106712
International Standard Book Number: 0-12-174151-6
PRINTED IN THE UNITED STATES OF AMERICA
00 01 02 03 IP 9 8 7 6 5 4 3 2 1
Contents
Preface to the third edition
Preface to the second edition
Preface to the first edition
ix
xi
xiii
1 J Distribution function
1 .1 Monotone functions
1 .2 Distribution functions
1 .3 Absolutely continuous and singular distributions
7
1
11
2 I Measure theory
2 .1 Classes of sets
2.2 Probability measures and their distribution
16
functions
21
3 1 Random variable . Expectation . Independence
3.1 General definitions
3.2 Properties of mathematical expectation
3.3 Independence
34
53
41
4
Convergence concepts
4.1 Various modes of convergence
4.2 Almost sure convergence ; Borel-Cantelli lemma
68
75
vi
I CONTENTS
4 .3 Vague convergence
4 .4 Continuation
91
4 .5 Uniform integrability ; convergence of moments
84
99
5 1 Law of large numbers . Random series
106
5.1 Simple limit theorems
5.2 Weak law of large numbers
5.3 Convergence of series
121
5.4 Strong law of large numbers
5.5 Applications
138
Bibliographical Note
148
112
129
6 1 Characteristic function
150
6.1 General properties ; convolutions
6.2 Uniqueness and inversion
160
6.3 Convergence theorems
6.4 Simple applications
6.5 Representation theorems
6.6 Multidimensional case ; Laplace transforms
169
187
175
196
Bibliographical Note
204
7 1 Central limit theorem and its ramifications
205
7.1 Liapounov's theorem
7.2 Lindeberg-Feller theorem
7.3 Ramifications of the central limit theorem
7.4 Error estimation
7 .5 Law of the iterated logarithm
7 .6 Infinite divisibility
214
242
235
250
Bibliographical Note
261
8 1 Random walk
224
263
270
8 .1 Zero-or-one laws
8 .2 Basic notions
8 .3 Recurrence
8 .4 Fine structure
8 .5 Continuation
288
298
Bibliographical Note
278
308
9 1 Conditioning . Markov property. Martingale
CONTENTS I
vii
9.1 Basic properties of conditional expectation
9.2 Conditional independence ; Markov property
9.3 Basic properties of smartingales
9.4 Inequalities and convergence
9.5 Applications
334
346
360
Bibliographical Note
373
310
322
Supplement : Measure and Integral
375
1 Construction of measure
2 Characterization of extensions
3 Measures in R
4 Integral
395
5 Applications
407
387
380
General Bibliography
413
Index
415
Preface to the third edition
In this new edition, I have added a Supplement on Measure and Integral .
The subject matter is first treated in a general setting pertinent to an abstract
measure space, and then specified in the classic Borel-Lebesgue case for the
real line. The latter material, an essential part of real analysis, is presupposed
in the original edition published in 1968 and revised in the second edition
of 1974 . When I taught the course under the title "Advanced Probability"
at Stanford University beginning in 1962, students from the departments of
statistics, operations research (formerly industrial engineering), electrical engi-
neering, etc . often had to take a prerequisite course given by other instructors
before they enlisted in my course . In later years I prepared a set of notes,
lithographed and distributed in the class, to meet the need . This forms the
basis of the present Supplement . It is hoped that the result may as well serve
in an introductory mode, perhaps also independently for a short course in the
stated topics .
The presentation is largely self-contained with only a few particular refer-
ences to the main text. For instance, after (the old) §2 .1 where the basic notions
of set theory are explained, the reader can proceed to the first two sections of
the Supplement for a full treatment of the construction and completion of a
general measure ; the next two sections contain a full treatment of the mathe-
matical expectation as an integral, of which the properties are recapitulated in
§3 .2. In the final section, application of the new integral to the older Riemann
integral in calculus is described and illustrated with some famous examples .
Throughout the exposition, a few side remarks, pedagogic, historical, even
x I PREFACE TO THE THIRD EDITION
judgmental, of the kind I used to drop in the classroom, are approximately
reproduced .
In drafting the Supplement, I consulted Patrick Fitzsimmons on several
occasions for support . Giorgio Letta and Bernard Bru gave me encouragement
for the uncommon approach to Borel's lemma in §3, for which the usual proof
always left me disconsolate as being too devious for the novice's appreciation .
A small number of additional remarks and exercises have been added to
the main text .
Warm thanks are due: to Vanessa Gerhard of Academic Press who deci-
phered my handwritten manuscript with great ease and care ; to Isolde Field
of the Mathematics Department for unfailing assistence ; to Jim Luce for a
mission accomplished . Last and evidently not least, my wife and my daughter
Corinna performed numerous tasks indispensable to the undertaking of this
publication.