Elements of Information Theory.pdf

发布时间：2022-06-09 发布人：admin 分类：说明书资料大小：10.12M 资料格式：pdf 举报版权申诉

u013050755-6661521-4744302542855682769.pdf-第1页.png

第1页 / 共774页

u013050755-6661521-4744302542855682769.pdf-第2页.png

第2页 / 共774页

u013050755-6661521-4744302542855682769.pdf-第3页.png

第3页 / 共774页

u013050755-6661521-4744302542855682769.pdf-第4页.png

第4页 / 共774页

u013050755-6661521-4744302542855682769.pdf-第5页.png

第5页 / 共774页

u013050755-6661521-4744302542855682769.pdf-第6页.png

第6页 / 共774页

u013050755-6661521-4744302542855682769.pdf-第7页.png

第7页 / 共774页

u013050755-6661521-4744302542855682769.pdf-第8页.png

第8页 / 共774页

文本预览

ELEMENTS OF INFORMATION THEORY Second Edition THOMAS M. COVER JOY A. THOMAS A JOHN WILEY & SONS, INC., PUBLICATION

ELEMENTS OF INFORMATION THEORY

ELEMENTS OF INFORMATION THEORY Second Edition THOMAS M. COVER JOY A. THOMAS A JOHN WILEY & SONS, INC., PUBLICATION

Copyright  2006 by John Wiley & Sons, Inc. All rights reserved. Published by John Wiley & Sons, Inc., Hoboken, New Jersey. Published simultaneously in Canada. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 750-4470, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permission. Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and speciﬁcally disclaim any implied warranties of merchantability or ﬁtness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of proﬁt or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002. Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic formats. For more information about Wiley products, visit our web site at www.wiley.com. Library of Congress Cataloging-in-Publication Data: Cover, T. M., 1938– Elements of information theory/by Thomas M. Cover, Joy A. Thomas.–2nd ed. p. cm. “A Wiley-Interscience publication.” Includes bibliographical references and index. ISBN-13 978-0-471-24195-9 ISBN-10 0-471-24195-4 1. Information theory. I. Thomas, Joy A. II. Title. Q360.C68 2005 003 .54–dc22 Printed in the United States of America. 10 9 8 7 6 5 4 3 2 1 2005047799

CONTENTS Contents Preface to the Second Edition Preface to the First Edition Acknowledgments for the Second Edition Acknowledgments for the First Edition Introduction and Preview 1.1 Preview of the Book 5 1 2 13 20 16 19 Entropy, Relative Entropy, and Mutual Information 2.1 2.2 2.3 2.4 Entropy Joint Entropy and Conditional Entropy Relative Entropy and Mutual Information Relationship Between Entropy and Mutual Information Chain Rules for Entropy, Relative Entropy, and Mutual Information Jensen’s Inequality and Its Consequences Log Sum Inequality and Its Applications Data-Processing Inequality Sufﬁcient Statistics 35 Fano’s Inequality 25 30 2.5 22 34 2.6 2.7 2.8 2.9 2.10 Summary Problems Historical Notes 54 41 43 37 v xv xvii xxi xxiii 1 13 v

57 71 vi 3 4 CONTENTS Asymptotic Equipartition Property Theorem 58 Consequences of the AEP: Data Compression High-Probability Sets and the Typical Set 62 Asymptotic Equipartition Property 3.1 3.2 3.3 Summary Problems Historical Notes 64 64 69 60 71 Entropy Rates of a Stochastic Process 4.1 4.2 4.3 Markov Chains Entropy Rate 74 Example: Entropy Rate of a Random Walk on a Weighted Graph Second Law of Thermodynamics Functions of Markov Chains 84 78 81 4.4 4.5 Summary Problems Historical Notes 87 88 100 5 Data Compression 103 5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9 5.10 103 107 110 Examples of Codes Kraft Inequality Optimal Codes Bounds on the Optimal Code Length Kraft Inequality for Uniquely Decodable Codes 115 Huffman Codes Some Comments on Huffman Codes 123 Optimality of Huffman Codes Shannon–Fano–Elias Coding 127 Competitive Optimality of the Shannon Code 118 130 112 120 5.11 Generation of Discrete Distributions from Fair Coins 134 141 Summary Problems 142 Historical Notes 157

分享到：

赞收藏

资料库

Elements of Information Theory.pdf

相关推荐

课程资源

热门标签

最新资料