Blueprint #5 Last Edited 4 days ago
Part I: Why are you conducting research? (goals)
What is the strategic importance and/or impact to the organization of rigorously studying this topic?
What do you want to learn in this particular study?
don’t forget to SAVE your work
Part II: What are you measuring? (variables)
What change(s) are you going to make, that you think will in turn change the outcome in the world that you care about? E.g., the words in an email subject line, the color of a website button, the greeting a salesperson says when a customer walks in the door, etc.
What outcome in the world do you care about, can measure, and want to try to influence with your idea? E.g., customers open our emails more frequently, customers purchase higher value products, employees stay with the company longer, etc.
Categorical: a frequency, probability, or percentage describing an action that is or isn’t taken, a threshold that is or isn’t met, or a label that is or isn’t given, e.g., the % of customer open a savings account in 2020, the % of customers save over 5k a year, the % of customers gave a satisfaction rating of 5 out of 5
Continuous: numerical data with a wide, continuous range of values like money, weight, or count, e.g., the average amount saved by customers, the average number of days an account stays open
Select a category of test subjects that you want to compare, for instance “reported the highest satisfaction rating”, or “clicked on the promotion”
What other outcomes in the world do you care about, can measure, and want to try to influence with your idea? These will be used in exploratory analyses, while the “primary outcome” will be the focus of this experiment blueprint
Through what medium are you running the experiment? E.g., email, call center, website, posters in a physical space, etc.
Whose behavior are you trying to change? E.g., all customers, customers in a certain region, customers of a certain product, all employees, employees on a certain team, etc.
don’t forget to SAVE your work
Part III: What do you predict and why? (hypothesis)
Your hypothesis should be a statement with an “if-then” logic (e.g., “if I change X then Y will happen”); a counterfactual or control (e.g., “versus if I do NOT change X”); specific (describe, X, Y, and the direction of the changes so that a colleague as well as a stranger on the street could understand); testable (you can make the changes to X that you desire, and you can measure the changes to Y that you expect); and falsifiable (there is at least some possibility that you could see different results than you predict)
What is the supporting rationale for your prediction? Your rationale could come from prior experiments (e.g., psychology studies in academia, other experiments you have run), prior non-experimental research (e.g., focus groups, interviews, surveys), user feedback (e.g., comments, complaints, anecdotes), external examples (e.g., competitor initiatives, ex-industry initiatives), or other sources. In any case, the source of your hypothesis should be clearly stated and go beyond mere gut feeling.
don’t forget to SAVE your work
Part IV: What is your experiment setup? (design)
If participants will only experience one version of the independent variable (one condition), you are comparing them with each other and running a between-subjects design. If participants will experience more than one version of the independent variable (2 or more conditions), you are comparing them with themselves and running a within-subjects design.
How many versions of the independent variable will you have? Will there be a “control” group that doesn’t get any version, or gets a “plain” version? (e.g., 1 control + 2 ‘ new’ versions to test = 3 groups)
What version of the independent variable will the control group experience? Nothing? Something ‘vanilla’? The status quo?
What version of the independent variable will test group 1 experience?
For each additional test group, what version of the independent variable will they experience?
This level could be customers, retail locations, training centers, accounts, etc. The lowest level (i.e., individuals or sessions) is best, but higher level groups may be necessary for operational reasons.
don’t forget to SAVE your work
Part V: Who is participating in the experiment? (sampling plan and treatment assignation)
Often we cannot run an experiment on the entire population we care about; we must pick a sub-set. Sometimes we cannot even run an experiment on anyone in the population we care about; we must pick a proxy. From where will you source your experiment participants (e.g., a sample of clients attending our annual conference, Amazon Mechanical Turk workers, customer emails)?
It may be important to filter out, or select for, different qualities in order to ensure you zero-in on the 'right' (representative) population.
It may be important to filter out, or select for, different qualities in order to ensure you zero-in on the 'right' (representative) population.
Participants can be assigned to groups based on a variety of methods, but the assignment must be random
Effective randomization is essential for you to be able to build any confidence in causality between your idea (independent variable) and your outcome (dependent variable). You can check for randomization in your experiment results by examining variables that, if randomization effectively occurred, would show no systematic pattern between the groups. E.g., if you are experimenting with emails, you can check to see if the bounce-back rate is similar between your groups; or you can measure whether participants fall out of the experiment (attrition) at similar rates across groups. If these rates are highly distinct (**how much is too much? by more than 1 standard deviation?**), you should re-examine your method for random assignment and appropriately caveat any findings from the experiment.
Ideally participants will not know they are part of an experiment to avoid confounding the interpretation of results. If there is a risk they will, plan ahead to avoid or mitigate.
When will you begin and end your experiment? Sometimes this timing is based on calendars; other times this timing is based on the expected length of time to sample enough participants. (“Enough” will be determined in the next part of the blueprint.)
don’t forget to SAVE your work
Part VI: How are you powering your experiment? (statistical & business risk)
proportions = binary outcomes | averages = continuous outcomes
proportions = binary outcomes | averages = continuous outcomes
Sensitivity power approach: start with your pre-set sample size, and compute a minimum detectible effect size.
| Inputs (record your values) | Rationale (how did you land on these?) | |
|---|---|---|
| Alpha level (risk of false positives) | ||
| Power (1-risk of false negatives) | ||
| Sample size per comparison group | ||
| Pooled standard deviation of DV | ||
| Control group expected average | ||
| Test group expected average | ||
| Minimum Detectable Effect (MDE) (test - control group) |
| Inputs (record your values) | Rationale (how did you land on these?) | |
|---|---|---|
| Alpha level (risk of false positives) | ||
| Power (1-risk of false negatives) | ||
| Sample size per comparison group | ||
| Control group expected proportion % | ||
| Test group expected proportion % | ||
| Minimum Detectable Effect (MDE) (test-control proportion) (test - control proportion) |
How many subjects do you need to complete the experiment for it to be adequately powered?
don’t forget to SAVE your work
Part VII: Who is reviewing, launching and running the experiment, and how? (operations)
From whom do you need informal or formal review and approvals? Who are they, what exactly do they need to engage on and in what step of the process, how long does their review or approval they typically take, and who will shepherd your experiment design through them? You may want to draw a process flow with sequential and / or parallel steps, gatekeepers, and approximate time duration.
Are there any communications, approvals, engineering investments, or other administrative needs that will occur to enable your experiment? What are they, how long do they typically take, and who will shepherd your experiment design through them? You may want to draw a process flow with sequential and / or parallel steps, gatekeepers, and approximate time duration.
Who will be pulling the result data, cleaning it (if necessary), analyzing and interpreting your primary outcome, exploring your secondary outcome(s) if any, and packaging that information together? Write a list of your expected primary and secondary analyses here. How long is the analysis phase expected to take, and is there an external deadline that needs to be met? If so, what is it?
Who needs to learn about the results of your experiment, in what level of detail, and in what sequence (if any)? What do those stakeholders stand to gain from the results of your experiment? How do you intend to effectively communicate your interpretation of the results (direction, magnitude, and confidence level)?
How will you ensure these stakeholders understand the limitations of your experiment? How will you ensure that null or otherwise unexpected or undesirable results are fully processed and learned from?
don’t forget to SAVE your work