logo资料库

Building.Intelligent.Systems.A.Guide.to.Machine.Learning.Engineering..pdf

第1页 / 共346页
第2页 / 共346页
第3页 / 共346页
第4页 / 共346页
第5页 / 共346页
第6页 / 共346页
第7页 / 共346页
第8页 / 共346页
资料共346页,剩余部分请下载后查看
Table of Contents
About the Author
About the Technical Reviewer
Acknowledgments
Introduction
Part I: Approaching an Intelligent Systems Project
Chapter 1: Introducing Intelligent Systems
Elements of an Intelligent System
An Example Intelligent System
The Internet Toaster
Using Data to Toast
Sensors and Heuristic Intelligence
Toasting with Machine Learning
Making an Intelligent System
Summary
For Thought…
Chapter 2: Knowing When to Use Intelligent Systems
Types of Problems That Need Intelligent Systems
Big Problems
Open-Ended Problems
Time-Changing Problems
Intrinsically Hard Problems
Situations When Intelligent Systems Work
When a Partial System Is Viable and Interesting
When You Can Use Data from the System to Improve
When the System Can Interface with the Objective
When it is Cost Effective
When You Aren’t Sure You Need an Intelligent System
Summary
For Thought
Chapter 3: A Brief Refresher on Working with Data
Structured Data
Asking Simple Questions of Data
Working with Data Models
Conceptual Machine Learning
Common Pitfalls of Working with Data
Summary
For Thought
Chapter 4: Defining the Intelligent System’s Goals
Criteria for a Good Goal
An Example of Why Choosing Goals Is Hard
Types of Goals
Organizational Objectives
Leading Indicators
User Outcomes
Model Properties
Layering Goals
Ways to Measure Goals
Waiting for More Information
A/B Testing
Hand Labeling
Asking Users
Decoupling Goals
Keeping Goals Healthy
Summary
For Thought
Part II: Intelligent Experiences
Chapter 5: The Components of Intelligent Experiences
Presenting Intelligence to Users
An Example of Presenting Intelligence
Achieve the System’s Objectives
An Example of Achieving Objectives
Minimize Intelligence Flaws
Create Data to Grow the System
An Example of Collecting Data
Summary
For Thought…
Chapter 6: Why Creating Intelligent Experiences Is Hard
Intelligence Make Mistakes
Intelligence Makes Crazy Mistakes
Intelligence Makes Different Types of Mistakes
Intelligence Changes
The Human Factor
Summary
For Thought…
Chapter 7: Balancing Intelligent Experiences
Forcefulness
Frequency
Value of Success
Cost of Mistakes
Knowing There Is a Mistake
Recovering from a Mistake
Intelligence Quality
Summary
For Thought…
Chapter 8: Modes of Intelligent Interaction
Automate
Prompt
Organize
Annotate
Hybrid Experiences
Summary
For Thought…
Chapter 9: Getting Data from Experience
An Example: TeamMaker
Simple Interactions
Making It Fun
Connecting to Outcomes
Properties of Good Data
Context, Actions, and Outcomes
Good Coverage
Real Usage
Unbiased
Does Not Contain Feedback Loops
Scale
Ways to Understand Outcomes
Implicit Outcomes
Ratings
Reports
Escalations
User Classifications
Summary
For Thought…
Chapter 10: Verifying Intelligent Experiences
Getting Intended Experiences
Working with Context
Working with Intelligence
Bringing it Together
Achieving Goals
Continual Verification
Summary
For Thought…
Part III: Implementing Intelligence
Chapter 11: The Components of an Intelligence Implementation
An Example of Intelligence Implementation
Components of an Intelligence Implementation
The Intelligence Runtime
Intelligence Management
Intelligence Telemetry Pipeline
The Intelligence Creation Environment
Intelligence Orchestration
Summary
For Thought…
Chapter 12: The Intelligence Runtime
Context
Feature Extraction
Models
Execution
Results
Instability in Intelligence
Intelligence APIs
Summary
For Thought…
Chapter 13: Where Intelligence Lives
Considerations for Positioning Intelligence
Latency in Updating
Quality Is Evolving Quickly
Problem Changing Quickly
Risk of Costly Mistakes
Latency in Execution
Latency in Intelligent Experience
The Right Answer Changes Drastically
Cost of Operation
The Cost of Distributing Intelligence
The Cost of Executing Intelligence
Offline Operation
Places to Put Intelligence
Static Intelligence in the Product
Client-Side Intelligence
Server-Centric Intelligence
Back-End (Cached) Intelligence
Hybrid Intelligence
Summary
For Thought…
Chapter 14: Intelligence Management
Overview of Intelligence Management
Complexity in Intelligent Management
Frequency in Intelligence Management
Human Systems
Sanity-Checking Intelligence
Checking for Compatibility
Checking for Runtime Constraints
Checking for Obvious Mistakes
Lighting Up Intelligence
Single Deployment
Silent Intelligence
Controlled Rollout
Flighting
Turning Off Intelligence
Summary
For Thought…
Chapter 15: Intelligent Telemetry
Why Telemetry Is Needed
Make Sure Things Are Working
Understand Outcomes
Gather Data to Grow Intelligence
Properties of an Effective Telemetry System
Sampling
Summarizing
Flexible Targeting
Common Challenges
Bias
Rare Events
Indirect Value
Privacy
Summary
For Thought…
Part IV: Creating Intelligence
Chapter 16: Overview of Intelligence
An Example Intelligence
Contexts
Implemented at Runtime
Available for Intelligence Creation
Things Intelligence Can Predict
Classifications
Probability Estimates
Regressions
Rankings
Hybrids and Combinations
Summary
For Thought…
Chapter 17: Representing Intelligence
Criteria for Representing Intelligence
Representing Intelligence with Code
Representing Intelligence with Lookup Tables
Representing Intelligence with Models
Linear Models
Decision Trees
Neural Networks
Summary
For Thought…
Chapter 18: The Intelligence Creation Process
An Example of Intelligence Creation: Blinker
Understanding the Environment
Define Success
Get Data
Bootstrap Data
Data from Usage
Get Ready to Evaluate
Simple Heuristics
Machine Learning
Understanding the Tradeoffs
Assess and Iterate
Maturity in Intelligence Creation
Being Excellent at Intelligence Creation
Data Debugging
Verification-Based Approach
Intuition with the Toolbox
Math (?)
Summary
For Thought…
Chapter 19: Evaluating Intelligence
Evaluating Accuracy
Generalization
Types of Mistakes
Distribution of Mistakes
Evaluating Other Types of Predictions
Evaluating Regressions
Evaluating Probabilities
Evaluating Rankings
Using Data for Evaluation
Independent Evaluation Data
Independence in Practice
Evaluating for Sub-Populations
The Right Amount of Data
Comparing Intelligences
Operating Points
Curves
Subjective Evaluations
Exploring the Mistakes
Imagining the User Experience
Finding the Worst Thing
Summary
For Thought…
Chapter 20: Machine Learning Intelligence
How Machine Learning Works
The Pros and Cons of Complexity
Underfitting
Overfitting
Balancing Complexity
Feature Engineering
Converting Data to Useable Format
Helping your Model Use the Data
Normalizing
Exposing Hidden Information
Expanding the Context
Eliminating Misleading Things
Modeling
Complexity Parameters
Identifying Overfitting
Summary
For Thought…
Chapter 21: Organizing Intelligence
Reasons to Organize Intelligence
Properties of a Well-Organized Intelligence
Ways to Organize Intelligence
Decouple Feature Engineering
Multiple Model Searches
Chase Mistakes
Meta-Models
Model Sequencing
Partition Contexts
Overrides
Summary
For Thought…
Part V: Orchestrating Intelligent Systems
Chapter 22: Overview of Intelligence Orchestration
Properties of a Well-Orchestrated Intelligence
Why Orchestration Is Needed
Objective Changes
Users Change
Problem Changes
Intelligence Changes
Costs Change
Abuse
The Orchestration Team
Summary
For Thought…
Chapter 23: The Intelligence Orchestration Environment
Monitor the Success Criteria
Inspect Interactions
Balance the Experience
Override Intelligence
Create Intelligence
Summary
For Thought…
Chapter 24: Dealing with Mistakes
The Worst Thing That Could Happen
Ways Intelligence Can Break
System Outage
Model Outage
Intelligence Errors
Intelligence Degradation
Mitigating Mistakes
Invest in Intelligence
Balance the Experience
Adjust Intelligence Management Parameters
Implement Guardrails
Override Errors
Summary
For Thought…
Chapter 25: Adversaries and Abuse
Abuse Is a Business
Abuse Scales
Estimating Your Risk
What an Abuse Problem Looks Like
Ways to Combat Abuse
Add Costs
Becoming Less Interesting to Abusers
Machine Learning with an Adversary
Get the Abuser out of the Loop
Summary
For Thought…
Chapter 26: Approaching Your Own Intelligent System
An Intelligent System Checklist
Approach the Intelligent System Project
Plan for the Intelligent Experience
Plan the Intelligent System Implementation
Get Ready to Create Intelligence
Orchestrate Your Intelligent System
Summary
For Thought…
Index
分享到:
收藏