第1页 / 共392页
第2页 / 共392页
第3页 / 共392页
第4页 / 共392页
第5页 / 共392页
第6页 / 共392页
第7页 / 共392页
第8页 / 共392页
Designing with Data: Improving the User Experience with A/B Test....pdf
Designing with Data: Improving the User Experience with A/B Testing
Praise for Designing with Data
Foreword
Preface
Design and Data: A Perfect Synergy
Our Focus: A/B Testing
Some Orienting Principles
Who Is This Book For?
Scope
About Us
A Word from Rochelle
A Word from Elizabeth
A Word from Caitlin
How This Book Is Organized
How to Read This Book
Introducing our “Running a Camp” Metaphor
O’Reilly Safari
How to Contact Us
Acknowledgments
Rochelle
Elizabeth
Caitlin
1. Introducing a Data Mindset
Data as a Trend
Three Ways to Think About Data
What Does This Mean for You as a Designer?
Data Can Help to Align Design with Business
On Data Quality
With a Little Help from Your Friends...
Data Producers
Data Consumers
What If You Don’t Have Data Friends (Yet)?
Themes You’ll See in This Book
Summary
Questions to Ask Yourself
2. The ABCs of Using Data
The Diversity of Data
Many Dimensions of Data
Why are you collecting data?
When is the data collected?
How is the data collected?
How much data to collect?
Why Experiment?
Learning About Causality
Statistically Significant, not Anecdotal
Informed Opinions about what will happen in the Wild
Basics of Experimentation
Language and Concepts
Race to the Campsite!
Experimentation in the Internet Age
A/B Testing: Online Experiments
Sampling Your Users Online
Cohorts and segments
Demographic information
New users versus existing users
Metrics: The Dependent Variable of A/B Testing
Detecting a Difference in Your Groups
How big is the difference you want to measure?
A big enough sample to power your test
Significance level
Your Hypothesis and Why It Matters
Defining a Hypothesis or Hypotheses
Know What You Want to Learn
Running Creative A/B Tests
Data Triangulation: Strength in Mixed Methods
The Landscape of Design Activities
Exploring and evaluating Ideas
Thinking Global and Thinking Local
Summary
Questions to Ask Yourself
3. A Framework for Experimentation
Introducing Our Framework
Working with Data Should Feel Familiar...
Three Phases: Definition, Execution, and Analysis
The Definition Phase
The Execution Phase
The Analysis Phase
Examples: Data and Design in Action
Summary
Questions to Ask Yourself
4. The Definition Phase (How to Frame Your Experiments)
Getting Started: Defining Your Goal
Defining Your Metric of Interest
Metric sensitivity
Tracking multiple metrics
Getting the full picture
Your metrics may change over time
Competing metrics
Refining Your Goals with Data
Identifying the Problem You Are Solving
Remember Where You Are
Building Hypotheses for the Problem at Hand
Example: A Summer Camp Hypothesis
Example: Netflix—transitioning from DVD Rentals to Streaming
The Importance of Going Broad
Multiple Ways to Influence a Metric
Focus on New and Existing Users
Revisit the Scope of Your Problem
Example: Netflix on the PlayStation 3
Involve Your Team and Your Data Friends
Which Hypotheses to Choose?
Consider Potential Impact
Using What You Already Know
Using Other Methods to Evaluate Your Hypotheses
Consider the Reality of Your Test
How much measurable impact do you believe your hypothesis can make?
Can you draw all the conclusions you want to draw from your test?
Balancing learning and speed
Keep Your Old Hypotheses in Your Back Pocket
Summary
Questions to Ask Yourself
5. The Execution Phase (How to Put Your Experiments into Action)
Designing to Learn
Engaging Your Users in a Conversation
Having Quality Conversations
Designing to extremes to learn about your users
Revisiting the minimum detectable effect
Designing the Best Representation of Your Hypothesis
Understanding Your Variables
Not all variables are visible
Your Design Can Influence Your Data
Example: Netflix Wii
Revisiting the Space of Design Activities
Avoiding Local Maxima
Different problems for summer camp
Directional testing: “Painted door” tests
Picking the right level of granularity for your experiment
Example: Netflix on Playstation 3
Example: Spotify Navigation
Experiment 1: Defining the hypothesis to get early directional feedback
Experiment 1: Designing the hypotheses
Interlude: Quick explorations using prototypes and usability testing
Experiment 2: Refining the “tabbed” navigation
“Designing” your tests
Other Considerations When Designing to Learn
Polishing your design too much, too early
Edge cases and “worst-case” scenarios
Taking advantage of other opportunities to learn about your design
Identifying the Right Level of Testing for Different Stages of Experimentation
Running parallel experiments
Thinking about “Experiment 0”
Summary
Questions to Ask Yourself
6. The Analysis Phase (Getting Answers From Your Experiments)
Vetting Your Designs Ahead of Launch
Lab Studies: Interviews and Usability Testing
Surveys
Working with Your Peers in Data
Launching Your Design
Balancing Trade Offs to Power Your Test
Weighing sample size and significance level
Getting the sample that you need (rollout % versus test time)
Who are you including in your sample?
Practical Implementation Details
Is your experience “normal” right now?
Sanity check: Questions to ask yourself
Evaluating Your Results
Revisiting Statistical Significance
What Does the Data Say?
Expected (“Positive”) Results
Unexpected and Undesirable (“Negative”) Results
When the World is Flat
Errors
Replication
Using secondary metrics
Using multiple test cells
Rolling out to more users
Revisiting “thick” data
Getting Trustworthy Data
Novelty effect
Seasonality bias
Rolling Out Your Experience, or Not
What’s Next for Your Designs?
Were you exploring or evaluating?
Was your problem global or local?
Knowing when to stop
Ramp Up
Holdback Groups
Taking Communication into Account
Case Study: Netflix on PlayStation 3
Many Treatments of the Four Hypotheses
Evolving the Design Through Iterative Tests
What If You Still Believe?
Summary
Questions to Ask Yourself
7. Creating the Right Environment for Data-Aware Design
Principle 1: Shared Company Culture and Values
Depth: Communicating Across Levels
Breadth: Beyond Design and Product
The Importance of a Learning Culture
The rewards of taking risks: Redefining “failure”
The value of developing your customer instinct
Principle 2: Hiring and Growing the Right People
Establishing a Data-Aware Environment Through Your Peers
Hiring for Success
Building the team with data involved from the start
Principle 3: Processes to Support and Align
Establishing a Knowledge Baseline
Establishing a Common Vocabulary
Developing a Rhythm Around Data Collection and Sharing
Project review meetings
Spreading data across the organization
Creating a Presence in the Office
Learning from the Past
Summary
Questions to Ask Yourself
8. Conclusion
Ethical Considerations
Ethics in Online Experimentation
Design Experimentation Versus Social Experimentation
Two “Power of Suggestion” Experiments
Toward Ethical A/B Testing
Key Concepts
Asking Questions, Thinking Ethically
Last Words
A. Resources
Keywords
Chapter 1
Chapter 2
Chapter 3
Chapters 4, 5, and 6
Chapter 7
Chapter 8
Books
Online Articles, Papers, and Blogs
Courses
Tools
Professional Groups, Meetups, and Societies
B. About the Authors
About the Authors
Colophon
Index
Copyright
Designing with Data: Improving the
User Experience with A/B Testing
Rochelle King
Elizabeth F Churchill
Caitlin Tan
Beijing • Boston • Farnham • Sebastopol • Tokyo
Special Upgrade Offer
If you purchased this ebook directly from oreilly.com, you have the following
benefits:
DRM-free ebooks — use your ebooks across devices without restrictions or
limitations
Multiple formats — use on your laptop, tablet, or phone
Lifetime access, with free updates
Dropbox syncing — your files, anywhere
If you purchased this ebook from another retailer, you can upgrade your ebook to
take advantage of all these benefits for just $4.99. Click here to access your
ebook upgrade.
Please note that upgrade offers are not available from sample content.
Praise for Designing with Data
“A clear, approachable and common sense guide to mastering data-driven design — a skill set that is
becoming mandatory for the 21st century designer. This book is an invaluable contribution to our
profession.”
— KHOI VINH, PRINCIPAL DESIGNER, ADOBE, AND DESIGN WRITER,
SUBTRACTION.COM
“King, Churchill, and Tan have produced a magnificently accessible introduction to designing modern
software products with data. The authors sail through concepts in statistics that aren’t weighed down
in math, and instead are grounded in real-life examples from top companies like Netflix and Airbnb.
Building on a deceivingly simple model of designer/user behavior, this book will make a designer able
to wield key concepts that are now being integrated in the tech industry from the social sciences and
modern data science.”
— JOHN MAEDA, GLOBAL HEAD, COMPUTATIONAL DESIGN AND INCLUSION,
AUTOMATTIC
“A/B testing is becoming an essential component of digital product development, and it’s vital for
designers to embrace it as a tool of their trade. This book provides the perfect introduction to it and
gives practical advice for how to apply, execute, and analyze A/B testing successfully.”
— MARTIN CHARLIER, PRODUCT MANAGER, UNMADE, AND COAUTHOR OF
DESIGNING CONNECTED PRODUCTS
“As an elegant, informed blend of concrete case studies and rigorous knowledge about data analysis,
this book not only will help design practitioners familiarize themselves with A/B testing, but will also
provide an approach to integrating data and analytical methods into the overall design and
development process.”
— NALINI P. KOTAMRAJU, HEAD OF USER RESEARCH & ANALYTICS, SALESFORCE
Foreword
This book offers a simple but exciting promise for anyone involved with the
design of internet products:
If you compare what you think should happen (in theory) to what actually
happens (in reality), you can better understand how your work changes user
behavior.
This book is your training camp to getting started using data — design’s often
overlooked but essential companion — to improve your work. You can augment
this approach with your existing design practices to amplify the impact of each
design: from identifying the problem you are solving, to designing for
experimentation, to learning from the data.
What’s unique about this training camp are the field guides at hand: not only
your authors sharing experiences from the world’s leading data-driven
organizations, but also the contributions from many practitioners offering you
the advantage of their standard practices, case studies, and hard-fought lessons
from designing with data every single day.
COLIN MCFARLAND,
HEAD OF EXPERIMENTATION AT SKYSCANNER
Preface
Design and Data: A Perfect Synergy
THE MYTH OF THE “genius designer,” someone whose instincts and intuition lead
to great design decisions, is a popular one. This seductive myth can lead some to
conclude that design is never grounded in data, that design is never an empirical
discipline, and that design practice stands in opposition to data and data sciences.
We understand where this myth comes from. On the surface, data science and
design practice are not obviously compatible. Design philosophy and practice
emphasizes empathy with users, and the creation and crafting of artful user
experiences in the form of effective products. For some designers (and for many
outside the design world who valorize design as “inspired genius”), design is a
nonlinear, exploratory journey. It is a fundamentally human, fluid, and creative
process, unlike the procedures and rigors of “science.” Design is emotional.
Design cannot be articulated as a set of procedures or steps to be followed.
Design cannot be rationalized and constrained. For some, incorporating data into
the design process is a cause for concern.
Some concerns we have heard expressed include:
Data undermines and undervalues the designer’s intuition and experience
Data stifles creativity and removes the “art” from the design process
Data dehumanizes the design process, reducing human experience and design
evaluation to “just numbers”
Data overemphasizes minutiae and optimization of small variations in the
design
Data enslaves you — believing in data as way to evaluate designs takes the
power away from design as a practice
On the other side, proponents of design through experimentation and data
science value measurement. Some see data as rational, and numbers as
irrefutable. Data science reveals the truth — data science is a proceduralized,
scientific endeavor where rigor leads to irrefutable results and therefore to
certainty. Data science is a trustworthy and precise craft. This view is reinforced
by the increasing fascination with measures and metrics for business, commonly
referred to today as “big data.” An extreme view is that large-scale experiments
can be run to gather data from millions of users to answer all design questions
and that such analytics can, therefore, replace design. Under this view, font
types, colors, and sizes, and questions such as “Should we have a blue or a red
dialogue box?”, “Do people engage more with a list or a carousel?”, or “Does a
wizard help with new user onboarding flow?” fall under the purview of data
science and not design practice. This could be characterized as “Let the crowd
speak with their clicks, and what emerges will necessarily be the best design.”
We deliberately present these extreme positions to illustrate a point. We believe
that the extreme views we just outlined draw a false dichotomy between data and
design. In reality, data sciences and design practices are working toward the
same goal: understanding users and crafting elegant experiences for them.
Design is and always has been informed by data. The “data” may be an
accumulated set of experiences and informally gathered observations that
provide the basis for design genius and “craft knowledge.” The data may also be
derived from more systematic studies of users’ activities and opinions, such as
lab-based studies, field observations, and surveys. Design practice has always
been about different forms of data. In an ever-changing marketplace and industry
where new applications and new behaviors are constantly emerging, data can
play a big role in helping us learn and respond in a timely way to shifts in user
interests and needs. By harnessing and leveraging the power of data at scale —
that is, data in high volume, often arriving in streams from millions of users, and
which may be of disparate types — new ways to understand people, “users,” are
emerging. Data at all scales from individuals to millions and hundreds of
millions of users — systematically collected, analyzed, communicated, and
leveraged — can empower design.
We want to acknowledge that designers’ concerns about large-scale data
collection have some grain of truth. Personally, we have all experienced some
circumstances and work situations where these criticisms held — for example,
where data gathered at scale contradicts what we know or believe to be true
about the human experience. Personally, we believe that these potential
misalignments in belief and data reflections arise precisely because designers
have historically not been included in the experimental process, data collection,
and analysis that informs design. We believe that design intent and evaluation
are often poorly matched to the data capture and analysis because designers with
a desire to understand user experience have not been in effective dialogue with
data scientists and machine-learning experts. This is a two-way conversation:
design can bring deeper meaning to data. By developing an awareness of and an
affinity for data, such conversations will benefit both disciplines. Similarly,
design practice can be enhanced by data. When managed well, data science in
the form of large-scale experiments can demonstrate the worth of creativity in
design rather than stifle it.
In sum, we believe designers have to engage in the practice and business of
designing experiments and managing the data collection process, by being part
相关推荐
2023年江西萍乡中考道德与法治真题及答案.doc
2012年重庆南川中考生物真题及答案.doc
2013年江西师范大学地理学综合及文艺理论基础考研真题.doc
2020年四川甘孜小升初语文真题及答案I卷.doc
2020年注册岩土工程师专业基础考试真题及答案.doc
2023-2024学年福建省厦门市九年级上学期数学月考试题及答案.doc
2021-2022学年辽宁省沈阳市大东区九年级上学期语文期末试题及答案.doc
2022-2023学年北京东城区初三第一学期物理期末试卷及答案.doc
2018上半年江西教师资格初中地理学科知识与教学能力真题及答案.doc
2012年河北国家公务员申论考试真题及答案-省级.doc
2020-2021学年江苏省扬州市江都区邵樊片九年级上学期数学第一次质量检测试题及答案.doc
2022下半年黑龙江教师资格证中学综合素质真题及答案.doc
