Predict-Then-Test
Commit your prediction before you see the data, then update on the gap
- Difficulty
- Moderate
- Time to result
- ~ongoing to results
- Steps
- 5
- Confidence
- 80%
The reliable way to learn is to make a prediction before you look at the data, then check whether it held. Epstein describes how people, including scientists, often gather data first and retrospectively hunt for associations, which he likens to a sharpshooter firing randomly at a wall and then drawing a bullseye around a cluster. This is HARKing: hypothesizing after the results are known. It feels legitimate but, for statistical reasons, is equivalent to running an infinite number of tests, so it proves nothing. The fix is to record a specific prediction up front, gather the data, compare, and update your beliefs little by little. When a funding agency forced cardiovascular trials to pre-register predictions around the year 2000, a wave of previously positive results turned negative, revealing how much retrospective story-fitting had been going on. The same discipline applied to startups produced more pivots and more success.
Origin
Extracted from David Epstein's discussion of the replication crisis and pre-registration, drawing on his own training as a scientist and studies of scientific-method training for businesses.
Core principles
- 01Learning only happens through updating a prior belief.
- 02A prediction made after seeing the data is not a real test.
- 03Constraints on how you discern truth are what make the truth trustworthy.
- 04Data mined for whatever pops out is effectively an infinite number of tests and proves nothing.
- 05You should make far more explicit predictions before making decisions.
How to run it
- 1
State your theory
Before collecting anything, articulate your theory of the world, the drug, the product, or the value you think you are adding.
- 2
Record a specific prediction
Write down concretely what you expect to happen, so the prediction cannot drift after you see results.
Pro tip Make the prediction specific enough that it can be clearly right or wrong.
Watch out A vague or unrecorded prediction lets you unconsciously rewrite it to fit the data.
- 3
Gather the data or run the test
Collect the evidence or run the experiment exactly as designed, without adjusting the target as you go.
- 4
Compare outcome to prediction
Check whether the result matched what you predicted. A mismatch is the signal that your model of the world needs revising.
- 5
Update and pivot
Tweak your beliefs slowly according to the gap, and change course if the test showed your theory was wrong.
Pro tip Update little by little rather than swinging fully on a single result.
In the wild
For decades leading up to 2000, big trials of medications and supplements for cardiovascular health were mostly positive. Then a funding agency required researchers to record their predictions in advance. From 2000 on, almost all results turned negative, revealing that the earlier positives had come from retrospectively sifting the data for associations.
→ A single constraint, predicting first, exposed years of false positives.
Businesses were randomized into different market-research training. Those taught the scientific method formed specific hypotheses about how their product fit the market, built tests, and ran them. Most discovered something in their theory was wrong and pivoted; companies that never made strong predictions did not learn and did not adjust.
→ The predict-then-test group was much more likely to succeed and start making money.
Common mistakes
HARKing, hypothesizing after results are known
Making the prediction after seeing the data is like a sharpshooter drawing a bullseye around wherever the bullets landed; it looks like a hit but is meaningless.
Mining data for anything that pops
Sifting a dataset for whatever association appears is statistically equivalent to running infinite tests, so it can suggest what to test next but cannot support a true conclusion.
Is it for you?
Best for
Anyone testing a product, a habit, a treatment, or a hypothesis who wants to genuinely learn from results.
Not ideal for
Pure exploratory scouting where you are only generating hypotheses to test later, not drawing conclusions.
From the transcript
“It's like a sharpshooter firing randomly at a wall and then drawing a bullseye around some clump... that's what a lot of scientists have been…”
“Make a prediction for what you think... whether it'll work or not, and then test it. And then you tweak your beliefs slowly according to…”
From the episode
Why You Feel Overwhelmed All The Time (and how to fix it) - David Epstein - #1121
David Epstein