logo资料库

Advanced Analytics with Spark, 2nd Edition.pdf

发布时间：2022-06-03 发布人：admin 分类：说明书 资料大小：2.91M 资料格式：pdf 举报 版权申诉
第1页 / 共273页
第2页 / 共273页
第3页 / 共273页
第4页 / 共273页
第5页 / 共273页
第6页 / 共273页
第7页 / 共273页
第8页 / 共273页
资料共273页，剩余部分请下载后查看
    Copyright
    Table of Contents
    Foreword
    Preface
    What’s in This Book
    The Second Edition
    Using Code Examples
    O’Reilly Safari
    How to Contact Us
    Acknowledgments
    Chapter 1. Analyzing Big Data
    The Challenges of Data Science
    Introducing Apache Spark
    About This Book
    The Second Edition
    Chapter 2. Introduction to Data Analysis with Scala and Spark
    Scala for Data Scientists
    The Spark Programming Model
    Record Linkage
    Getting Started: The Spark Shell and SparkContext
    Bringing Data from the Cluster to the Client
    Shipping Code from the Client to the Cluster
    From RDDs to Data Frames
    Analyzing Data with the DataFrame API
    Fast Summary Statistics for DataFrames
    Pivoting and Reshaping DataFrames
    Joining DataFrames and Selecting Features
    Preparing Models for Production Environments
    Model Evaluation
    Where to Go from Here
    Chapter 3. Recommending Music and the Audioscrobbler Data Set
    Data Set
    The Alternating Least Squares Recommender Algorithm
    Preparing the Data
    Building a First Model
    Spot Checking Recommendations
    Evaluating Recommendation Quality
    Computing AUC
    Hyperparameter Selection
    Making Recommendations
    Where to Go from Here
    Chapter 4. Predicting Forest Cover with Decision Trees
    Fast Forward to Regression
    Vectors and Features
    Training Examples
    Decision Trees and Forests
    Covtype Data Set
    Preparing the Data
    A First Decision Tree
    Decision Tree Hyperparameters
    Tuning Decision Trees
    Categorical Features Revisited
    Random Decision Forests
    Making Predictions
    Where to Go from Here
    Chapter 5. Anomaly Detection in Network Traffic with K-means Clustering
    Anomaly Detection
    K-means Clustering
    Network Intrusion
    KDD Cup 1999 Data Set
    A First Take on Clustering
    Choosing k
    Visualization with SparkR
    Feature Normalization
    Categorical Variables
    Using Labels with Entropy
    Clustering in Action
    Where to Go from Here
    Chapter 6. Understanding Wikipedia with Latent Semantic Analysis
    The Document-Term Matrix
    Getting the Data
    Parsing and Preparing the Data
    Lemmatization
    Computing the TF-IDFs
    Singular Value Decomposition
    Finding Important Concepts
    Querying and Scoring with a Low-Dimensional Representation
    Term-Term Relevance
    Document-Document Relevance
    Document-Term Relevance
    Multiple-Term Queries
    Where to Go from Here
    Chapter 7. Analyzing Co-Occurrence Networks with GraphX
    The MEDLINE Citation Index: A Network Analysis
    Getting the Data
    Parsing XML Documents with Scala’s XML Library
    Analyzing the MeSH Major Topics and Their Co-Occurrences
    Constructing a Co-Occurrence Network with GraphX
    Understanding the Structure of Networks
    Connected Components
    Degree Distribution
    Filtering Out Noisy Edges
    Processing EdgeTriplets
    Analyzing the Filtered Graph
    Small-World Networks
    Cliques and Clustering Coefficients
    Computing Average Path Length with Pregel
    Where to Go from Here
    Chapter 8. Geospatial and Temporal Data Analysis on New York City Taxi Trip Data
    Getting the Data
    Working with Third-Party Libraries in Spark
    Geospatial Data with the Esri Geometry API and Spray
    Exploring the Esri Geometry API
    Intro to GeoJSON
    Preparing the New York City Taxi Trip Data
    Handling Invalid Records at Scale
    Geospatial Analysis
    Sessionization in Spark
    Building Sessions: Secondary Sorts in Spark
    Where to Go from Here
    Chapter 9. Estimating Financial Risk Through Monte Carlo Simulation
    Terminology
    Methods for Calculating VaR
    Variance-Covariance
    Historical Simulation
    Monte Carlo Simulation
    Our Model
    Getting the Data
    Preprocessing
    Determining the Factor Weights
    Sampling
    The Multivariate Normal Distribution
    Running the Trials
    Visualizing the Distribution of Returns
    Evaluating Our Results
    Where to Go from Here
    Chapter 10. Analyzing Genomics Data and the BDG Project
    Decoupling Storage from Modeling
    Ingesting Genomics Data with the ADAM CLI
    Parquet Format and Columnar Storage
    Predicting Transcription Factor Binding Sites from ENCODE Data
    Querying Genotypes from the 1000 Genomes Project
    Where to Go from Here
    Chapter 11. Analyzing Neuroimaging Data with PySpark and Thunder
    Overview of PySpark
    PySpark Internals
    Overview and Installation of the Thunder Library
    Loading Data with Thunder
    Thunder Core Data Types
    Categorizing Neuron Types with Thunder
    Where to Go from Here
    Index
    About the Authors
    Colophon
    www.allitebooks.com SECOND EDITION Advanced Analytics with Spark Patterns for Learning from Data at Scale Sandy Ryza, Uri Laserson, Sean Owen, and Josh Wills Beijing Beijing Boston Boston Farnham Sebastopol Farnham Sebastopol Tokyo Tokyo
    www.allitebooks.com Advanced Analytics with Spark by Sandy Ryza, Uri Laserson, Sean Owen, and Josh Wills Copyright © 2017 Sanford Ryza, Uri Laserson, Sean Owen, Joshua Wills. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com/safari). For more information, contact our corporate/insti‐ tutional sales department: 800-998-9938 or corporate@oreilly.com. Editor: Marie Beaugureau Production Editor: Melanie Yarbrough Copyeditor: Gillian McGarvey Proofreader: Christina Edwards June 2017: Second Edition Revision History for the Second Edition 2017-06-09: First Release Indexer: WordCo Indexing Services Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Rebecca Demarest The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Advanced Analytics with Spark, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. 978-1-491-97295-3 [LSI]
    www.allitebooks.com Table of Contents Foreword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix 1. Analyzing Big Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 The Challenges of Data Science 3 Introducing Apache Spark 4 About This Book 6 The Second Edition 7 2. Introduction to Data Analysis with Scala and Spark. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Scala for Data Scientists 10 The Spark Programming Model 11 Record Linkage 12 Getting Started: The Spark Shell and SparkContext 13 Bringing Data from the Cluster to the Client 19 Shipping Code from the Client to the Cluster 22 From RDDs to Data Frames 23 Analyzing Data with the DataFrame API 26 Fast Summary Statistics for DataFrames 32 Pivoting and Reshaping DataFrames 33 Joining DataFrames and Selecting Features 37 Preparing Models for Production Environments 38 Model Evaluation 40 Where to Go from Here 41 3. Recommending Music and the Audioscrobbler Data Set. . . . . . . . . . . . . . . . . . . . . . . . . . 43 Data Set 44 iii
    www.allitebooks.com The Alternating Least Squares Recommender Algorithm 45 Preparing the Data 48 Building a First Model 51 Spot Checking Recommendations 54 Evaluating Recommendation Quality 56 Computing AUC 58 Hyperparameter Selection 60 Making Recommendations 62 Where to Go from Here 64 4. Predicting Forest Cover with Decision Trees. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 Fast Forward to Regression 65 Vectors and Features 66 Training Examples 67 Decision Trees and Forests 68 Covtype Data Set 71 Preparing the Data 71 A First Decision Tree 74 Decision Tree Hyperparameters 80 Tuning Decision Trees 82 Categorical Features Revisited 86 Random Decision Forests 88 Making Predictions 91 Where to Go from Here 91 5. Anomaly Detection in Network Traffic with K-means Clustering. . . . . . . . . . . . . . . . . . . 93 Anomaly Detection 94 K-means Clustering 94 Network Intrusion 95 KDD Cup 1999 Data Set 96 A First Take on Clustering 97 Choosing k 99 Visualization with SparkR 102 Feature Normalization 106 Categorical Variables 108 Using Labels with Entropy 109 Clustering in Action 111 Where to Go from Here 112 6. Understanding Wikipedia with Latent Semantic Analysis. . . . . . . . . . . . . . . . . . . . . . . . 115 The Document-Term Matrix 116 Getting the Data 118 iv | Table of Contents
    www.allitebooks.com Parsing and Preparing the Data 118 Lemmatization 120 Computing the TF-IDFs 121 Singular Value Decomposition 123 Finding Important Concepts 125 Querying and Scoring with a Low-Dimensional Representation 129 Term-Term Relevance 130 Document-Document Relevance 132 Document-Term Relevance 133 Multiple-Term Queries 134 Where to Go from Here 136 7. Analyzing Co-Occurrence Networks with GraphX. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 The MEDLINE Citation Index: A Network Analysis 139 Getting the Data 140 Parsing XML Documents with Scala’s XML Library 142 Analyzing the MeSH Major Topics and Their Co-Occurrences 143 Constructing a Co-Occurrence Network with GraphX 146 Understanding the Structure of Networks 150 Connected Components 150 Degree Distribution 153 Filtering Out Noisy Edges 155 Processing EdgeTriplets 156 Analyzing the Filtered Graph 158 Small-World Networks 159 Cliques and Clustering Coefficients 160 Computing Average Path Length with Pregel 161 Where to Go from Here 166 8. Geospatial and Temporal Data Analysis on New York City Taxi Trip Data. . . . . . . . . . . 169 Getting the Data 170 Working with Third-Party Libraries in Spark 171 Geospatial Data with the Esri Geometry API and Spray 172 Exploring the Esri Geometry API 172 Intro to GeoJSON 174 Preparing the New York City Taxi Trip Data 176 Handling Invalid Records at Scale 178 Geospatial Analysis 182 Sessionization in Spark 185 Building Sessions: Secondary Sorts in Spark 186 Where to Go from Here 189 Table of Contents | v
    www.allitebooks.com 9. Estimating Financial Risk Through Monte Carlo Simulation. . . . . . . . . . . . . . . . . . . . . . 191 Terminology 192 Methods for Calculating VaR 193 Variance-Covariance 193 Historical Simulation 193 Monte Carlo Simulation 193 Our Model 194 Getting the Data 195 Preprocessing 195 Determining the Factor Weights 198 Sampling 201 The Multivariate Normal Distribution 204 Running the Trials 205 Visualizing the Distribution of Returns 208 Evaluating Our Results 209 Where to Go from Here 211 10. Analyzing Genomics Data and the BDG Project. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213 Decoupling Storage from Modeling 214 Ingesting Genomics Data with the ADAM CLI 217 Parquet Format and Columnar Storage 223 Predicting Transcription Factor Binding Sites from ENCODE Data 225 Querying Genotypes from the 1000 Genomes Project 232 Where to Go from Here 235 11. Analyzing Neuroimaging Data with PySpark and Thunder. . . . . . . . . . . . . . . . . . . . . . . 237 Overview of PySpark 238 PySpark Internals 239 Overview and Installation of the Thunder Library 241 Loading Data with Thunder 241 Thunder Core Data Types 248 Categorizing Neuron Types with Thunder 249 Where to Go from Here 254 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 vi | Table of Contents
    www.allitebooks.com Foreword Ever since we started the Spark project at Berkeley, I’ve been excited about not just building fast parallel systems, but helping more and more people make use of large- scale computing. This is why I’m very happy to see this book, written by four experts in data science, on advanced analytics with Spark. Sandy, Uri, Sean, and Josh have been working with Spark for a while, and have put together a great collection of con‐ tent with equal parts explanations and examples. The thing I like most about this book is its focus on examples, which are all drawn from real applications on real-world data sets. It’s hard to find one, let alone 10, examples that cover big data and that you can run on your laptop, but the authors have managed to create such a collection and set everything up so you can run them in Spark. Moreover, the authors cover not just the core algorithms, but the intricacies of data preparation and model tuning that are needed to really get good results. You should be able to take the concepts in these examples and directly apply them to your own problems. Big data processing is undoubtedly one of the most exciting areas in computing today, and remains an area of fast evolution and introduction of new ideas. I hope that this book helps you get started in this exciting new field. — Matei Zaharia, CTO at Databricks and Vice President, Apache Spark vii
    www.allitebooks.com
    分享到：
    赞 收藏

    相关推荐

    资料库

    大数据

    共收录841份资料，累计11个分类，关注成员有19位，主要包括：Matlab,统计分析,数据挖掘,Hbase,Hive,flink,kafka,spark,Storm,算法与数据结构,Hadoop

    热门标签

    Matlab 统计分析 数据挖掘 Hbase Hive flink kafka spark Storm 算法与数据结构 Hadoop

    最新资料