Handbook of Face Recognition
Stan Z. Li
Anil K. Jain
Editors
Handbook of
Face Recognition
With 210 Illustrations
Stan Z. Li
Center for Biometrics Research and Testing &
Anil K. Jain
Department of Computer Science
National Lab of Pattern Recognition
Institute of Automation
Chinese Academy of Sciences
Beijing 100080
China
szli@nlpr.ia.ac.cn
& Engineering
Michigan State University
East Lansing, MI 48824-1226
USA
jain@cse.msu.edu
Library of Congress Cataloging-in-Publication Data
Handbook of face recognition / editors, Stan Z. Li & Anil K. Jain.
p. cm.
Includes bibliographical references and index.
ISBN 0-387-40595-X (alk. paper)
1. Human face recognition (Computer science
Anil K., 1948–
TA1650.H36
006.4′2—dc22
2004
I. Li, S. Z., 1958–
II. Jain,
2004052453
Printed on acid-free paper.
ISBN 0-387-40595-X
© 2005 Springer Science+Business Media, Inc.
All rights reserved. This work may not be translated or copied in whole or in part
without the written permission of the publisher (Springer Science+Business Media,
Inc., 233 Spring Street, New York, NY 10013, USA), except for brief excerpts in con-
nection with reviews or scholarly analysis. Use in connection with any form of infor-
mation storage and retrieval, electronic adaptation, computer software, or by similar
or dissimilar methodology now known or hereafter developed is forbidden.
The use in this publication of trade names, trademarks, service marks, and similar
terms, even if they are not identified as such, is not to be taken as an expression of
opinion as to whether or not they are subject to proprietary rights.
Printed in the United States of America.
(MP)
9 8 7 6 5 4 3 2 1
SPIN 10946602
springeronline.com
Preface
Face recognition has a large number of applications, including security, person verification, In-
ternet communication, and computer entertainment. Although research in automatic face recog-
nition has been conducted since the 1960s, this problem is still largely unsolved. Recent years
have seen significant progress in this area owing to advances in face modeling and analysis
techniques. Systems have been developed for face detection and tracking, but reliable face
recognition still offers a great challenge to computer vision and pattern recognition researchers.
There are several reasons for recent increased interest in face recognition, including ris-
ing public concern for security, the need for identity verification in the digital world, and the
need for face analysis and modeling techniques in multimedia data management and computer
entertainment. Recent advances in automated face analysis, pattern recognition, and machine
learning have made it possible to develop automatic face recognition systems to address these
applications.
This book was written based on two primary motivations. The first was the need for highly
reliable, accurate face recognition algorithms and systems. The second was the recent research
in image and object representation and matching that is of interest to face recognition re-
searchers.
The book is intended for practitioners and students who plan to work in face recognition or
who want to become familiar with the state-of-the-art in face recognition. It also provides ref-
erences for scientists and engineers working in image processing, computer vision, biometrics
and security, Internet communications, computer graphics, animation, and the computer game
industry. The material fits the following categories: advanced tutorial, state-of-the-art survey,
and guide to current technology.
The book consists of 16 chapters, covering all the subareas and major components nec-
essary for designing operational face recognition systems. Each chapter focuses on a specific
topic or system component, introduces background information, reviews up-to-date techniques,
presents results, and points out challenges and future directions.
Chapter 1 introduces face recognition processing, including major components such as face
detection, tracking, alignment, and feature extraction, and it points out the technical challenges
of building a face recognition system. We emphasize the importance of subspace analysis and
learning, not only providing an understanding of the challenges therein but also the most suc-
VI
Preface
cessful solutions available so far. In fact, most technical chapters represent subspace learning-
based techniques for various steps in face recognition.
Chapter 2 reviews face detection techniques and describes effective statistical learning
methods. In particular, AdaBoost-based learning methods are described because they often
achieve practical and robust solutions. Techniques for dealing with nonfrontal face detection
are discussed. Results are presented to compare boosting algorithms and other factors that af-
fect face detection performance.
Chapters 3 and 4 discuss face modeling methods for face alignment. These chapters de-
scribe methods for localizing facial components (e.g., eyes, nose, mouth) and facial outlines
and for aligning facial shape and texture with the input image. Input face images may be ex-
tracted from static images or video sequences, and parameters can be extracted from these input
images to describe the shape and texture of a face. These results are based largely on advances
in the use of active shape models and active appearance models.
Chapters 5 and 6 cover topics related to illumination and color. Chapter 5 describes recent
advances in illumination modeling for faces. The illumination invariant facial feature repre-
sentation is described; this representation improves the recognition performance under varying
illumination and inspires further explorations of reliable face recognition solutions. Chapter 6
deals with facial skin color modeling, which is helpful when color is used for face detection
and tracking.
Chapter 7 provides a tutorial on subspace modeling and learning-based dimension reduction
methods, which are fundamental to many current face recognition techniques. Whereas the col-
lection of all images constitutes high dimensional space, images of faces reside in a subspace of
that space. Facial images of an individual are in a subspace of that subspace. It is of paramount
importance to discover such subspaces so as to extract effective features and construct robust
classifiers.
Chapter 8 addresses problems of face tracking and recognition from a video sequence of
images. The purpose is to make use of temporal constraints present in the sequence to make
tracking and recognition more reliable.
Chapters 9 and 10 present methods for pose and illumination normalization and extract
effective facial features under such changes. Chapter 9 describes a model for extracting illu-
mination invariants, which were previously presented in Chapter 5. Chapter 9 also presents a
subregion method for dealing with variation in pose. Chapter 10 describes a recent innovation,
called Morphable Models, for generative modeling and learning of face images under changes
in illumination and pose in an analysis-by-synthesis framework. This approach results in algo-
rithms that, in a sense, generalize the alignment algorithms described in Chapters 3 and 4 to the
situation where the faces are subject to large changes in illumination and pose. In this work, the
three-dimensional data of faces are used during the learning phase to train the model in addition
to the normal intensity or texture images.
Chapters 11 and 12 provide methods for facial expression analysis and synthesis. The
analysis part, Chapter 11, automatically analyzes and recognizes facial motions and facial fea-
ture changes from visual information. The synthesis part, Chapter 12, describes techniques on
three-dimensional face modeling and animation, face lighting from a single image, and facial
expression synthesis. These techniques can potentially be used for face recognition with vary-
ing poses, illuminations, and facial expressions. They can also be used for human computer
interfaces.
Preface
VII
Chapter 13 reviews 27 publicly available databases for face recognition, face detection, and
facial expression analysis. These databases provide a common ground for development and
evaluation of algorithms for faces under variations in identity, face pose, illumination, facial
expression, age, occlusion, and facial hair.
Chapter 14 introduces concepts and methods for face verification and identification perfor-
mance evaluation. The chapter focuses on measures and protocols used in FERET and FRVT
(face recognition vendor tests). Analysis of these tests identifies advances offered by state-of-
the-art technologies for face recognition, as well as the limitations of these technologies.
Chapter 15 offers psychological and neural perspectives suggesting how face recognition
might go on in the human brain. Combined findings suggest an image-based representation
that encodes faces relative to a global average and evaluates deviations from the average as an
indication of the unique properties of individual faces.
Chapter 16 describes various face recognition applications, including face identification,
security, multimedia management, and human-computer interaction. The chapter also reviews
many face recognition systems and discusses related issues in applications and business.
Acknowledgments
A number of people helped in making this book a reality. Vincent Hsu, Dirk Colbry, Xiaoguang
Lu, Karthik Nandakumar, and Anoop Namboodiri of Michigan State University, and Shiguang
Shan, Zhenan Sun, Chenghua Xu and Jiangwei Li of the Chinese Academy of Sciences helped
proofread several of the chapters. We also thank Wayne Wheeler and Ann Kostant, editors at
Springer, for their suggestions and for keeping us on schedule for the production of the book.
This handbook project was done partly when Stan Li was with Microsoft Research Asia.
December 2004
Stan Z. Li
Beijing, China
Anil K. Jain
East Lansing, Michigan
Contents
Chapter 1. Introduction
Stan Z. Li, Anil K. Jain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1
Chapter 2. Face Detection
Stan Z. Li . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Chapter 3. Modeling Facial Shape and Appearance
Tim Cootes, Chris Taylor, Haizhuang Kang, Vladimir Petrovi´c . . . . . . . . . . . . . . . . . . . . . . 39
Chapter 4. Parametric Face Modeling and Tracking
J¨orgen Ahlberg, Fadi Dornaika . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Chapter 5. Illumination Modeling for Face Recognition
Ronen Basri, David Jacobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Chapter 6. Facial Skin Color Modeling
J. Birgitta Martinkauppi, Matti Pietik¨ainen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
Color Plates for Chapters 6 and 15
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
Chapter 7. Face Recognition in Subspaces
Gregory Shakhnarovich, Baback Moghaddam . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
Chapter 8. Face Tracking and Recognition from Video
Rama Chellappa, Shaohua Kevin Zhou . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
Chapter 9. Face Recognition Across Pose and Illumination
Ralph Gross, Simon Baker, Iain Matthews, Takeo Kanade . . . . . . . . . . . . . . . . . . . . . . . . . 193
Chapter 10. Morphable Models of Faces
Sami Romdhani, Volker Blanz, Curzio Basso, Thomas Vetter
. . . . . . . . . . . . . . . . . . . . . . 217