O T H ER T I T L ES P U B L I S H ED IN T HE S E R I ES
(FORMERLY PERGAMON SCIENCE SERIES ELECTRONICS A ND W A V E S)
Vol.
1 Signal Noise and Resolution
in Nuclear Counter
Amplifiers
by A. B. GILLESPIE
Vol.
2 Scintillation
Counters
by J. B. B I R KS
Vol.
4 Physics and Applications
of Secondary Electron
Emission
Vol.
5 Millimicrosecond
Pulse Techniques
(2nd
edition)
by H. B R U I N I NG
6 Introduction
by I. A. D. L E W IS and F. H. W E L LS
Computers
to Electronic Analogue
Vol.
by C. A. A. W A SS
Vol.
7 Scattering and Diffraction
of Radio Waves
by J. R. M E N T Z ER
Vol.
8 Space-charge Waves and Slow Electromagnetic
Waves
by A. H. W. B E CK
Vol.
9 Statistical Theory of Signal Detection
Vol. 10 Laplace Transforms
by C A RL W. H E L S T R OM
for Electronic
Engineers
by J. G. H O L B R O OK
Vol. 11 Frequency Modulation
Theory—Application
to Microwave
Links
by J. FAGOT and P H. M A G NE
Vol. 12 Theory of Microwave
Valves
by S. D. G V O Z D O V ER
Vol. 13 Electronic
Computers
by A. I. K I T OV and N. A. K R I N I T S K II
Vol. 14 Topics
in Engineering
Logic
by M. N A D L ER
Vol. 15 Environmental
Testing
Techniques
by G. W. A. D U M M ER and N. B. G R I F F IN
Vol. 16 Fundamentals
of Microwave
Electronics
by V. N. SHEVCHIK
Vol. 17 Static Electromagnetic
Vol. 18 Problems
Changers
Frequency
by L. L. ROZHANSKII
in the Design and Development of 750 MW
by V. P. ANEMPODISTOV, E. G. K A S H A R S K II and
U R U S OV
Turbogenerators
I. D.
Vol. 19 Controlled-Delay
Devices
by S. A. DOGANOVSKII and V. A. I V A N OV
Vol. 20 High Sensitivity Counting
Techniques
by D. E. W A TT and D. R A M S D EN
Vol. 21 Asynchronised
Synchronous
Machines
by M. M. B O T V I N N IK
Vol. 22 Sampling Systems Theory and its Application,
Vol, 1
PROBABILITY AND INFORMATION
THEORY, WITH APPLICATIONS
TO RADAR
By
P. M. W O O D W A R D, M.A.
Mathematics Division, Royal Radar
Establishment, Ministry of Aviation
SECOND EDITION
P E R G A M ON P R E SS
O X F O RD
• L O N D ON • E D I N B U R GH • N EW Y O RK
P A R IS • F R A N K F U RT
PERGAMON PRESS LTD.
Headington Hill Hall, Oxford
4 & 5 Fitzroy Square, London W .l
PERGAMON PRESS (SCOTLAND) LTD.
2 & 3 Teviot Place, Edinburgh 1
PERGAMON PRESS INC.
122 East 55th Street, New York 22, N.Y.
GAUTHIER-VILLARS ED.
55 Quai des Grands-Augustins, Paris 6
PERGAMON PRESS G.m.b.H.
Kaiserstrasse 75, Frankfurt am Main
Copyright 1953
Pergamon Press Ltd.
First Published 1953
Second Impression 1957
Reprinted 1963
Second Edition 1964
Library of Congress Catalog Card Number 64-7502
Printed in Great Britain by Latimer Trend & Co. Ltd., Whitstable.
EDITOR'S PREFACE
THE aim of these monographs is to report upon research carried
out in electronics and applied physics. Work in these fields continues
to expand rapidly, and it is recognised that the collation and dis
semination of information in a usable form is of the greatest im
portance to all those actively engaged in them. The monographs
will be written by specialists in their own subjects, and the time
required for publication will be kept to a minimum in order that
these accounts of new work may be made quickly and widely
available.
Wherever it is practical the monographs will be kept short in
length to enable all those interested in electronics to find the
essentials necessary for their work in a condensed and concentrated
form.
D. W. F RY
AUTHOR'S PREFACE
THE first two chapters of this short monograph are concerned with
established mathematical techniques rather than with fresh ideas.
They provide the code in which so much of the mathematical theory
of electronics and radar is nowadays expressed. Information theory
is the latest extension of this code, and I hope that it will not be
considered improper that I have tried in Chapter 3 to summarise
so much of C. E. SHANNON'S original work, which already exists in
book-form (The Mathematical Theory of Communication, by CLAUDE
SHANNON and WARREN WEAVER). The account which is given in
Chapter 3 may perhaps spur the reader who has not studied the
original literature into doing so.
Chapters 4 and 5 deal with some of the fascinating problems,
which have been discussed so often in recent years, of detecting
signals in noise. The present approach was suggested to me by
SHANNON'S work on communication theory and is based on inverse
probability; it is my opinion that of all statistical methods, this one
comes closest to expressing intuitive notions in the precise language
ix
EDITOR'S PREFACE
THE aim of these monographs is to report upon research carried
out in electronics and applied physics. Work in these fields continues
to expand rapidly, and it is recognised that the collation and dis
semination of information in a usable form is of the greatest im
portance to all those actively engaged in them. The monographs
will be written by specialists in their own subjects, and the time
required for publication will be kept to a minimum in order that
these accounts of new work may be made quickly and widely
available.
Wherever it is practical the monographs will be kept short in
length to enable all those interested in electronics to find the
essentials necessary for their work in a condensed and concentrated
form.
D. W. F RY
AUTHOR'S PREFACE
THE first two chapters of this short monograph are concerned with
established mathematical techniques rather than with fresh ideas.
They provide the code in which so much of the mathematical theory
of electronics and radar is nowadays expressed. Information theory
is the latest extension of this code, and I hope that it will not be
considered improper that I have tried in Chapter 3 to summarise
so much of C. E. SHANNON'S original work, which already exists in
book-form (The Mathematical Theory of Communication, by CLAUDE
SHANNON and WARREN WEAVER). The account which is given in
Chapter 3 may perhaps spur the reader who has not studied the
original literature into doing so.
Chapters 4 and 5 deal with some of the fascinating problems,
which have been discussed so often in recent years, of detecting
signals in noise. The present approach was suggested to me by
SHANNON'S work on communication theory and is based on inverse
probability; it is my opinion that of all statistical methods, this one
comes closest to expressing intuitive notions in the precise language
ix
X
PREFACE
of mathematics. Chapters 5, 6 and 7 are devoted to radar, which is
simple enough (ideally) to lend itself to fairly exact mathematical
treatment along the lines suggested in the previous chapters. This
material is based on papers which have appeared in technical
journals, Chapter 6 in particular being a revised account of work
originally carried out at T.R.E. in partnership with I.L.DAVIES. It
was this work which led to the present monograph, but it is hoped
that the first four chapters—originally conceived as an introduction
to the special problems of radar—may find an independent usefulness
for the reader whose interests are not so narrowly confined.
I have to thank the Chief Scientist, Ministry of Supply, for per
mission to publish this book.
Malvern
March, 1953.
P. M. W.
PREFACE TO SECOND EDITION
OVER the last ten years, authoritative works of monumental propor
tions on radar and communication theory have appeared, with
bibliographies almost as long as the text of this little Monograph. To
attempt, in the second edition, a major enlargement of what was only
an extended research paper would be to destroy its purpose, and I
have thus left the original text substantially unchanged. One new
chapter is now added, describing in the briefest possible terms, the
way in which direct probabilities are usually applied to the decision
problem in radar design. To research workers, however, I would
suggest that Chapter 7 still poses some interesting practical problems.
P. M. W.
Malvern
February, 1964
X
PREFACE
of mathematics. Chapters 5, 6 and 7 are devoted to radar, which is
simple enough (ideally) to lend itself to fairly exact mathematical
treatment along the lines suggested in the previous chapters. This
material is based on papers which have appeared in technical
journals, Chapter 6 in particular being a revised account of work
originally carried out at T.R.E. in partnership with I.L.DAVIES. It
was this work which led to the present monograph, but it is hoped
that the first four chapters—originally conceived as an introduction
to the special problems of radar—may find an independent usefulness
for the reader whose interests are not so narrowly confined.
I have to thank the Chief Scientist, Ministry of Supply, for per
mission to publish this book.
Malvern
March, 1953.
P. M. W.
PREFACE TO SECOND EDITION
OVER the last ten years, authoritative works of monumental propor
tions on radar and communication theory have appeared, with
bibliographies almost as long as the text of this little Monograph. To
attempt, in the second edition, a major enlargement of what was only
an extended research paper would be to destroy its purpose, and I
have thus left the original text substantially unchanged. One new
chapter is now added, describing in the briefest possible terms, the
way in which direct probabilities are usually applied to the decision
problem in radar design. To research workers, however, I would
suggest that Chapter 7 still poses some interesting practical problems.
P. M. W.
Malvern
February, 1964
1
AN INTRODUCTION TO PROBABILITY
THEORY
1.1 THE RULES OF PROBABILITY
IF n possibilities are equally likely and exactly m of them have some
attribute A, we say that the probability of A is mjn. Strictly, this is
not a definition of probability because it assumes that the notion of
equally likely possibilities is understood in the first place. From a
purely mathematical point of view, however, no definition is
required. All we need is a set of rules for adding and multiplying
probabilities, which are then taken as the basic postulates of the
theory. But the study of probability is made easier and the rules
become intuitive rather than arbitary when from the beginning there
is an obvious practical interpretation, and this the opening remark
supplies.
Since probability is a fraction of equally likely possibilities, it is
often helpful to set these out in tabular form. Thus
A A A B B B BG
(1)
signifies that P(A), the probability of A, is f and so on. It is immedi
ately evident that the probability of A or B is P(A) + P(B). This
is the sum rule and it applies only when A and B cannot simul
taneously be true, in other words, when they are mutually exclusive.
When all mutually exclusive attributes have been taken into account,
their probabilities will naturally add up to unity.
It frequently happens that two sets of attributes, each mutually
exclusive among themselves, have to be considered together. Sup
pose, for instance, that we have eight pencils, three red (^4), four
black (B) and one blue (C). The scheme (1) represents the equally
likely possibilities when one pencil is selected randomly. But the
same pencils may also be hard (J) and soft (K) as follows,
A A A B B B BC
J J K J J J KK
1
)