Index
A
- abandonment metric, Metric sensitivity
- Abstract (company), Running parallel experiments
- A/B testing, How much data to collect?
- (see also experimentation)
- about, Design and Data: A Perfect Synergy, Data as a Trend
- analyzing, With a Little Help from Your Friends...
- drawing conclusions from, Summary
- establishing causality, How much data to collect?, Experimentation in the Internet Age
- identifying right level of, Thinking about “Experiment 0”
- informed opinions and, Informed Opinions about what will happen in the Wild
- metrics in, New users versus existing users
- running creative tests, Running Creative A/B Tests
- statistical power and, Metrics: The Dependent Variable of A/B Testing, Keep Your Old Hypotheses in Your Back Pocket
- statistical significance and, Learning About Causality, Metrics: The Dependent Variable of A/B Testing, Keep Your Old Hypotheses in Your Back Pocket
- Wiley on, On Data Quality
- acquiescence bias, Many Dimensions of Data
- Active User (AU) metric, Metrics: The Dependent Variable of A/B Testing
- “affect” data, Many Dimensions of Data
- Airbnb rental service
- competing metrics and, Refining Your Goals with Data
- creating the right design environment, Building the team with data involved from the start, Learning from the Past
- demographic information and, Cohorts and segments
- development process, Examples: Data and Design in Action
- effect of testing on customers, Case Study: Netflix on PlayStation 3
- international considerations, Cohorts and segments, Running parallel experiments
- key metrics and, Your metrics may change over time
- negative or confounding test results and, Questions to Ask Yourself
- worst-case scenarios and, Running parallel experiments
- Analysis Phase
- about, Examples: Data and Design in Action, Working with Your Peers in Data
- expected (positive) results, Unexpected and Undesirable (“Negative”) Results
- flat results, When the World is Flat
- launching designs, Getting the sample that you need (rollout % versus test time)
- Netflix case study, Case Study: Netflix on PlayStation 3
- rolling out experience, Was your problem global or local?
- trustworthy data, Rolling Out Your Experience, or Not
- unexpected (negative) results, Unexpected and Undesirable (“Negative”) Results
- vetting designs ahead of launch, Launching Your Design
- anecdotal evidence, Learning About Causality
- applied common sense, Using Other Methods to Evaluate Your Hypotheses
- attitudinal data, Data Producers, Many Dimensions of Data, How is the data collected?
B
- behavioral data
- about, Many Dimensions of Data, Running Creative A/B Tests
- Analysis Phase and, Practical Implementation Details, When the World is Flat
- Definition Phase and, Getting Started: Defining Your Goal, Tracking multiple metrics, Balancing learning and speed
- Execution Phase and, Revisiting the Space of Design Activities
- informed opinions about, Informed Opinions about what will happen in the Wild
- Maliwat on, Remember Where You Are
- McClain on, Remember Where You Are
- understanding underlying causes of, Learning About Causality
- user researchers and, Data Producers
- bias
- big data, Novelty effect
- brainstorming, Focus on New and Existing Users, Consider Potential Impact
- Brewer, Josh, Running parallel experiments
- broad hypotheses
- business managers, Data Consumers
- buyer’s remorse (eBay), Metric sensitivity
C
- call to action, Thinking Global and Thinking Local, Directional testing: “Painted door” tests
- card sort method, Balancing learning and speed
- causality and correlation, How much data to collect?, Experimentation in the Internet Age
- change as experiment building block, Informed Opinions about what will happen in the Wild
- Chen, Andrew, Data as a Trend
- Ciancutti, John
- clinical drug trials, randomized, Statistically Significant, not Anecdotal
- cohorts and segmentation
- collecting data (see data collection)
- Colson, Eric, Your metrics may change over time, Designing the Best Representation of Your Hypothesis, Who are you including in your sample?, Rolling out to more users
- common vocabulary, establishing, Developing a Rhythm Around Data Collection and Sharing
- communication considerations
- confidence level, A big enough sample to power your test, Who are you including in your sample?
- confirmation bias, Learning About Causality, Consider the Reality of Your Test
- conflict resolution, Depth: Communicating Across Levels
- confounds, Race to the Campsite!
- consumers of data, Data Consumers
- contextual data, Why are you collecting data?, Running Creative A/B Tests, Revisiting Statistical Significance
- control groups, Language and Concepts, Detecting a Difference in Your Groups, Who are you including in your sample?
- controlled variables, Race to the Campsite!
- conversations
- correlation and causality, How much data to collect?, Experimentation in the Internet Age
- Coursera learning platform
- customer instinct, developing, The value of developing your customer instinct
- customer journey, Metrics: The Dependent Variable of A/B Testing, Learning from the Past
- customers (see users)
D
- Daily Active Users (DAU) metric, Metrics: The Dependent Variable of A/B Testing
- data analysts, With a Little Help from Your Friends..., Using What You Already Know, Weighing sample size and significance level
- data-aware design
- about, Data as a Trend, Three Ways to Think About Data, Introducing Our Framework, Principle 1: Shared Company Culture and Values
- Analysis Phase and, Working with Your Peers in Data
- defining hypotheses and, Defining a Hypothesis or Hypotheses
- Definition Phase and, Getting Started: Defining Your Goal, The Importance of Going Broad
- Execution Phase and, Having Quality Conversations, Not all variables are visible
- hiring and growing the right people principle, Establishing a Data-Aware Environment Through Your Peers
- processes to support and align principle, Establishing a Knowledge Baseline
- shared company culture and values principle, Depth: Communicating Across Levels
- value of, Informed Opinions about what will happen in the Wild, A Framework for Experimentation
- data collection
- data consumers, Data Consumers
- data, defined, The Diversity of Data, Identifying the Problem You Are Solving
- data-driven design
- data friends, With a Little Help from Your Friends..., Tracking multiple metrics, Using What You Already Know, Hiring for Success
- data-informed design
- data literacy, Depth: Communicating Across Levels
- data mindset
- data producers, With a Little Help from Your Friends...
- data quality, Data Can Help to Align Design with Business, Identifying the Problem You Are Solving
- data scientists, Who Is This Book For?, Data as a Trend, With a Little Help from Your Friends..., Principle 3: Processes to Support and Align
- data triangulation, Running Creative A/B Tests
- Definition Phase
- demographic information, Cohorts and segments
- dependent variables, Race to the Campsite!, New users versus existing users
- designers
- designing with data
- ABCs of using data, The ABCs of Using Data
- aligning design with business, Data Can Help to Align Design with Business, Data Consumers
- Analysis Phase, Examples: Data and Design in Action, Working with Your Peers in Data
- creating the right environment, Principle 1: Shared Company Culture and Values
- Definition Phase, The Execution Phase, Getting Started: Defining Your Goal
- Execution Phase, The Analysis Phase, Engaging Your Users in a Conversation
- framework for experimentation, A Framework for Experimentation
- introducing a data mindset, Data as a Trend
- Dill, Katie
- dimensions of data, The Diversity of Data, Informed Opinions about what will happen in the Wild, Exploring and evaluating Ideas
- directional testing, Example: Netflix on Playstation 3
- diversity of data, The Diversity of Data
- Dodson, Fitzhugh, Getting Started: Defining Your Goal
- double diamond framework, Introducing Our Framework
- drug trials, randomized clinical, Statistically Significant, not Anecdotal
E
- eBay (company), Metric sensitivity
- edge cases, Running parallel experiments
- effect size, Detecting a Difference in Your Groups
- emotional data, Why are you collecting data?
- engineers, Data Consumers, Using What You Already Know
- errors in experiments
- Etsy (company)
- evaluation stage
- Execution Phase
- expected (positive) results, Unexpected and Undesirable (“Negative”) Results
- experience-driven perspective, Three Ways to Think About Data
- Experiment 0 thinking, Questions to Ask Yourself
- experimental groups, Language and Concepts, Experimentation in the Internet Age, Detecting a Difference in Your Groups
- experimentation
- about, How much data to collect?, A Framework for Experimentation, Getting Started: Defining Your Goal
- Analysis Phase (see Analysis Phase)
- causality and, How much data to collect?, Experimentation in the Internet Age
- Definition Phase (see Definition Phase)
- Execution Phase (see Execution Phase)
- framework for, A Framework for Experimentation
- goal of, Language and Concepts, A Framework for Experimentation
- granularity of, Example: Netflix on Playstation 3
- hypothesis in, Significance level
- identifying right level of testing for, Thinking about “Experiment 0”
- implementation details, Sanity check: Questions to ask yourself
- in Internet age, Experimentation in the Internet Age
- informed opinions and, Informed Opinions about what will happen in the Wild
- iteration in, Defining a Hypothesis or Hypotheses, Summary, The Execution Phase (How to Put Your Experiments into Action)
- language and concepts, Informed Opinions about what will happen in the Wild
- online experiments, A/B Testing: Online Experiments
- statistical power and, Metrics: The Dependent Variable of A/B Testing, Keep Your Old Hypotheses in Your Back Pocket
- statistical significance and, Learning About Causality, Metrics: The Dependent Variable of A/B Testing, Keep Your Old Hypotheses in Your Back Pocket
- working with data, Introducing Our Framework
- exploration stage
H
- hamburger menu, Experiment 1: Designing the hypotheses, Experiment 1: Designing the hypotheses, Polishing your design too much, too early
- Hastings, Reed, Depth: Communicating Across Levels
- healthcare-related decisions, Data Triangulation: Strength in Mixed Methods
- Henne, Randal, Depth: Communicating Across Levels
- hiring and growing the right people principle
- holdback groups, Taking Communication into Account
- Hunt, Neil, Depth: Communicating Across Levels
- hypotheses
- about, Significance level, Getting Started: Defining Your Goal
- broad, Example: Netflix—transitioning from DVD rentals to Streaming, The Importance of Going Broad
- building, Building Hypotheses for the Problem at Hand
- choosing, Using Other Methods to Evaluate Your Hypotheses
- defining, Significance level
- designing best representation of, Not all variables are visible
- identifying right level of testing for, Thinking about “Experiment 0”
- implementation details, Sanity check: Questions to ask yourself
- iterating experimental designs, Defining a Hypothesis or Hypotheses
- knowing what you want to learn, Defining a Hypothesis or Hypotheses
- narrow, Example: Netflix—transitioning from DVD rentals to Streaming
- prioritizing, Example: A Summer Camp Hypothesis, Using Other Methods to Evaluate Your Hypotheses, Balancing learning and speed, Running parallel experiments
- rejecting, Language and Concepts
- understanding variables, Your Design Can Influence Your Data
I
- IDEO (company)
- implementation of experiments, Sanity check: Questions to ask yourself
- independent variables, Race to the Campsite!
- informed opinions, Informed Opinions about what will happen in the Wild, Balancing learning and speed
- instinct-driven perspective, Three Ways to Think About Data
- Internet age, experimentation in, Experimentation in the Internet Age
- interviews, How is the data collected?, Balancing learning and speed, Weighing sample size and significance level
- isolation, collecting data in, Why are you collecting data?, Identifying the Problem You Are Solving
K
- key metrics
- about, Metrics: The Dependent Variable of A/B Testing
- changing over time, Your metrics may change over time
- Coursera example, Metrics: The Dependent Variable of A/B Testing, Metric sensitivity
- getting the full picture and, Your metrics may change over time
- Google example, Revisiting the Space of Design Activities
- holdback groups and, Case Study: Netflix on PlayStation 3
- impact of changes on, Using secondary metrics
- metric sensitivity and, Metric sensitivity
- proxy metrics and, Case Study: Netflix on PlayStation 3
- secondary metrics and, Using multiple test cells
- Stitch Fix example, Understanding Your Variables
- tracking multiple metrics, Tracking multiple metrics
- knowledge baseline, establishing, Establishing a Knowledge Baseline
- Kohavi, Ron, Questions to Ask Yourself, Getting Trustworthy Data, Rolling Out Your Experience, or Not
L
- large sample research, How much data to collect?
- launching designs
- learning, balancing with speed, The Execution Phase (How to Put Your Experiments into Action), Who are you including in your sample?
- learning culture
- learning effect, New users versus existing users
- lectures, The rewards of taking risks: Redefining “failure”
- lifelong learners, Cohorts and segments
- Lind, James, Experimentation in the Internet Age
- LinkedIn (company), Experimentation in the Internet Age
- local problems/solutions, Exploring and evaluating Ideas, Example: Netflix on the PlayStation 3, Avoiding Local Maxima, Ramp Up
- longitudinal data, Why are you collecting data?
- low-fidelity mocks, Can you draw all the conclusions you want to draw from your test?
- lunch and learns, The rewards of taking risks: Redefining “failure”
M
- Maliwat, Chris
- McClain, Arianna
- on behavioral data, Remember Where You Are
- on collaborative teams, Principle 3: Processes to Support and Align
- on healthcare-related project, Data Triangulation: Strength in Mixed Methods
- on making data compelling, Creating a Presence in the Office
- on statistical methods, Statistically Significant, not Anecdotal
- on subjectivity of measurement, Metrics: The Dependent Variable of A/B Testing
- on testing considerations, Ramp Up
- on thick data, Rolling Out Your Experience, or Not
- McCord, Patty
- McFarland, Colin, Informed Opinions about what will happen in the Wild, Exploring and evaluating Ideas, Understanding Your Variables, Holdback Groups
- McKinley, Dan
- metric of interest
- metrics
- about, New users versus existing users
- broad hypotheses and, Focus on New and Existing Users
- measuring if efforts worth it, Building Hypotheses for the Problem at Hand
- quantitative data and, How is the data collected?, Defining Your Metric of Interest
- statistical significance and, Statistically Significant, not Anecdotal, Metrics: The Dependent Variable of A/B Testing, Keep Your Old Hypotheses in Your Back Pocket
- subjectivity of measurement, Metrics: The Dependent Variable of A/B Testing
- Microsoft (company), Questions to Ask Yourself
- minimum detectable effect (MDE)
- mocks, low-fidelity, Can you draw all the conclusions you want to draw from your test?
- moderated data, How is the data collected?, Balancing learning and speed
- Monthly Active Users (MAU) metric, Metrics: The Dependent Variable of A/B Testing
- multiple metrics, tracking, Tracking multiple metrics
- multiple test cells, Revisiting “thick” data
N
- narrow hypotheses, Example: Netflix—transitioning from DVD rentals to Streaming
- negative (unexpected) results, Unexpected and Undesirable (“Negative”) Results, Were you exploring or evaluating?
- Netflix (company)
- Ciancutti and, On Data Quality, Depth: Communicating Across Levels
- data-aware design environment and, Depth: Communicating Across Levels, The Importance of a Learning Culture
- DVD to streaming example, Refining Your Goals with Data, Example: Netflix—transitioning from DVD rentals to Streaming
- Maliwat and, Defining Your Metric of Interest
- PS3 example, Involve Your Team and Your Data Friends, Example: Netflix on Playstation 3, Questions to Ask Yourself, Case Study: Netflix on PlayStation 3
- retention metric, Competing metrics
- testing hypotheses example, Your Design Can Influence Your Data, Sanity check: Questions to ask yourself
- Wii example, Avoiding Local Maxima
- net promoter score (NPS), Metric sensitivity
- Nielsen Norman Group, How much data to collect?
- novelty effect, Were you exploring or evaluating?
- null (flat) results, When the World is Flat, Were you exploring or evaluating?, Knowing when to stop
P
- “painted door” experiments, Example: Netflix on Playstation 3, Questions to Ask Yourself
- parallel experiments, Thinking about “Experiment 0”
- pilot groups, Hiring for Success
- pilot studies, How much measurable impact do you believe your hypothesis can make?, Hiring for Success
- polarizing style (Stitch Fit), Designing the Best Representation of Your Hypothesis
- positive (expected) results, Unexpected and Undesirable (“Negative”) Results
- power, statistical, Metrics: The Dependent Variable of A/B Testing, Keep Your Old Hypotheses in Your Back Pocket, Who are you including in your sample?
- principles (data-aware design)
- prioritizing hypotheses, Example: A Summer Camp Hypothesis, Using Other Methods to Evaluate Your Hypotheses, Balancing learning and speed, Running parallel experiments
- problem/opportunity areas
- processes to support and align principle
- product managers
- product marketers, Data Producers
- project review meetings, Project review meetings
- prototypes, Can you draw all the conclusions you want to draw from your test?, “Designing” your tests
- proxy metrics, Metrics: The Dependent Variable of A/B Testing, Metric sensitivity, Case Study: Netflix on PlayStation 3
- p-values, Statistically Significant, not Anecdotal, A big enough sample to power your test, Expected (“Positive”) Results, Using secondary metrics
Q
- qualitative data
- about, Data Producers, How is the data collected?, Learning About Causality
- building hypotheses and, Example: A Summer Camp Hypothesis, What If You Still Believe?
- identifying the problem with, Remember Where You Are
- metrics and, Metrics: The Dependent Variable of A/B Testing, Defining Your Metric of Interest
- miscommunicating, Creating a Presence in the Office
- self-reported data as, Can you draw all the conclusions you want to draw from your test?
- usability studies and, Balancing learning and speed
- quality of data, Data Can Help to Align Design with Business, Identifying the Problem You Are Solving
- Qualtrics (company), Data Consumers
- quantitative data
R
- ramp-up strategy, Holdback Groups
- randomized clinical drug trials, Statistically Significant, not Anecdotal
- Reichelt, Leisa, Project review meetings
- rejecting a hypothesis, Language and Concepts
- replication, Using multiple test cells
- resources (hiring and growing the right people principle)
- retention metric, Competing metrics, Understanding Your Variables
- return on investment (ROI), Establishing a Data-Aware Environment Through Your Peers
- rolling out experience
- Running Camp example
- building hypotheses, Example: A Summer Camp Hypothesis
- camp advertisements, How much data to collect?, Cohorts and segments
- data triangulation and, Data Triangulation: Strength in Mixed Methods
- defining hypotheses, Defining a Hypothesis or Hypotheses
- designing for different types of problems, Example: Netflix on Playstation 3
- engaging users in conversations, Revisiting the minimum detectable effect
- experimentation and, Language and Concepts, Cohorts and segments, Metrics: The Dependent Variable of A/B Testing, Three Phases: Definition, Execution, and Analysis
- focusing on users, Example: Netflix on the PlayStation 3
- identifying the problem, Identifying the Problem You Are Solving
- influencing metrics, Focus on New and Existing Users
- metrics and, Metrics: The Dependent Variable of A/B Testing
- new versus existing users, New users versus existing users
- sampling considerations, Who are you including in your sample?
S
- sampling populations
- about, Language and Concepts, A/B Testing: Online Experiments
- defining sample size, How is the data collected?, Who are you including in your sample?
- demographic information, Cohorts and segments
- detecting difference in groups, Metrics: The Dependent Variable of A/B Testing
- new users versus existing users, Demographic information
- rollout % versus test time, Who are you including in your sample?
- selecting users, Who are you including in your sample?
- statistical power and, How big is the difference you want to measure?
- scurvy experiment, Experimentation in the Internet Age
- seasoned professionals, Cohorts and segments
- secondary metrics
- segmentation (see cohorts and segmentation)
- self-report data, How is the data collected?
- self-selecting groups, Tracking multiple metrics
- sensitivity, metric, Metric sensitivity
- shared company culture and values principle
- 60dB (company), On Data Quality, The value of developing your customer instinct
- Skyscanner travel site
- small sample research, How is the data collected?
- snapshot data, Why are you collecting data?
- social desirability response, Many Dimensions of Data
- Sommerfield, Dan, Depth: Communicating Across Levels
- speed
- Spotify (company)
- statistical power, Metrics: The Dependent Variable of A/B Testing, Keep Your Old Hypotheses in Your Back Pocket, Who are you including in your sample?
- statistical significance, Learning About Causality, Metrics: The Dependent Variable of A/B Testing, Keep Your Old Hypotheses in Your Back Pocket, Expected (“Positive”) Results
- Stitch Fix
- SurveyMonkey (company), Data Consumers
- surveys, How is the data collected?, Can you draw all the conclusions you want to draw from your test?, Weighing sample size and significance level
T
- test cells
- A/B testing weaknesses, Running Creative A/B Tests
- about, A/B Testing: Online Experiments, The Analysis Phase, Engaging Your Users in a Conversation, Edge cases and “worst-case” scenarios
- designing hypotheses, Not all variables are visible, Identifying the Right Level of Testing for Different Stages of Experimentation, Weighing sample size and significance level
- experiment granularity and, Example: Netflix on Playstation 3
- multiple, Revisiting “thick” data
- test design, Polishing your design too much, too early
- thick data, Novelty effect
- throwing designs over the wall, Hiring for Success
- tracking multiple metrics, Tracking multiple metrics
- trustworthy data
- Tukey, John, With a Little Help from Your Friends...
- Twitter social media site, Metrics: The Dependent Variable of A/B Testing, Running parallel experiments
U
- undesirable (negative) results, Unexpected and Undesirable (“Negative”) Results, Were you exploring or evaluating?
- unexpected (negative) results, Unexpected and Undesirable (“Negative”) Results, Were you exploring or evaluating?
- unmoderated data, How is the data collected?, Running Creative A/B Tests
- unseasoned professionals, Cohorts and segments
- usability studies, Balancing learning and speed, “Designing” your tests
- usability testing, Balancing learning and speed, Weighing sample size and significance level, Getting Trustworthy Data
- user researchers
- users
- UserTesting.com (company), Data Consumers, Identifying the Problem You Are Solving
- UserZoom.com (company), Identifying the Problem You Are Solving
..................Content has been hidden....................
You can't read the all page of ebook, please click
here login for view all page.