Privatizing the Apocalypse
Spot when a decision-maker pockets the upside while everyone shares the catastrophic downside
- Difficulty
- Moderate
- Time to result
- ~days to results
- Steps
- 5
- Confidence
- 68%
Reid's lens explains why brilliant, well-intentioned people take risks that could end civilization. Each actor silently runs a personal expected-value calculation: the upside (career glory, an IPO, a Nobel-tier paper) is large and personal, while the perceived personal risk feels tiny, so they proceed. What that calculation omits is that the true downside is not personal but shared by all of humanity, a cost they never price in because we were never trained to think 'if I mess up, everyone dies.' Reid calls this privatised gains and socialised losses, and 'democratizing the apocalypse' once the number of such actors grows. The danger compounds: a one-in-three-million risk taken by one careful person is bearable, but the same risk spread across thousands of private actors becomes a near-certain catastrophe. The antidote is alignment, keeping the decision a public good where deciders fully share the downside.
Origin
Coined and developed by Rob Reid across his existential-risk work and articulated here on Modern Wisdom, contrasting Cold War nuclear decisions (a public good) with modern privatised gain-of-function and AI research.
Core principles
- 01Individuals optimise their own utility curve, not humanity's
- 02A sliver-of-a-percent risk per actor becomes near-certain across thousands of actors
- 03Privatised gains plus socialised losses systematically underprice catastrophe
- 04When the decider fully shares the downside, decisions get far more careful
How to run it
- 1
Name the upside and its owner
Identify the concrete personal reward driving the decision-maker: money, an IPO, a paper in Science or Nature, celebrity, career momentum. Be specific about who pockets it.
- 2
Name the catastrophic downside and who bears it
Spell out the worst-case loss and who actually pays it. In the cases Reid worries about, the downside is borne by all of humanity, not the actor.
- 3
Test for alignment
Ask whether the decider fully shares the downside. Cold War leaders faced the same annihilation as everyone, so they were careful; a private actor who mostly captures gains is not.
Pro tip The Manhattan Project ran the atmosphere-ignition numbers up to the day before Trinity precisely because the deciders shared the risk.
Watch out Good intentions do not fix the misalignment; sincere people still under-price shared losses.
- 4
Count the actors holding the button
Establish how many independent parties can take the risky action. Two closely watched governments is a different world from thousands of private labs.
- 5
Aggregate the risk
Multiply the tiny per-actor probability by the number of actors over time. Arithmetic that looks safe for one person becomes untenable across a crowd.
Watch out One actor outside any sane safety window is enough for the whole system to fail.
In the wild
Reid imagines the Wisconsin researcher who made H5N1 airborne. His motivations are pure, but his utility curve rewards papers, grants, and celebrity, while the risk of a leak feels negligible after decades without one, like an expert driver sure he'll never crash. He faces the same death as everyone if the world ends, but pockets the entire career upside, so his personal expected-value calculation greenlights research that puts all of humanity in the risk pool.
→ Shows how sincere expertise plus private incentives produces globally reckless choices.
Reid uses the financial crisis as the pattern's clearest case: buy-side, sell-side, and fund actors took odious risks with the world economy because higher risk meant higher personal returns, inhaling money into super-yachts and Picassos. When it collapsed, the bill was socialised onto taxpayers worldwide. Privatised gains, socialised losses, executed at scale.
→ A concrete, non-existential illustration that the lens generalises beyond biosecurity.
Common mistakes
Assuming good intentions remove the hazard
The problem is structural, not moral. A sincere, brilliant actor still under-prices a catastrophic downside they personally will not bear, so intentions do not fix the misaligned incentive.
Evaluating one actor's tiny risk in isolation
A one-in-millions risk looks acceptable for a single careful person but becomes near-certain catastrophe once thousands of independent actors each take it.
Is it for you?
Best for
Evaluating dual-use technology, moral hazard, and any situation where a few actors can impose catastrophic risk on many.
Not ideal for
Purely personal decisions where the actor genuinely bears all the consequences of their own choice.
From the transcript
“you've got privatized gains but socialized losses”
“is that what you call democratizing the apocalypse yes that is exactly what i call democratizing the apocalypse”
“he's thinking for himself but he's got all of us in the risk curve and he's not calculating that expected value of what happens if…”
From the episode
How To Avoid Destroying Humanity - Rob Reid - #346
Rob Reid