MModern Wisdom
← All episodes
Rob Reid15 July 2021

How To Avoid Destroying Humanity - Rob Reid - #346

3Frameworks
14Insights

Listen

Frameworks in this episode

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Hot Take· 3

Hot Take29:00

COVID was remarkably benign compared to what's coming

Reid makes the unsettling case that COVID is striking for how mild it is. Its case fatality rate of roughly half a percent to one percent is far below SARS (10-20x deadlier), MERS (~30 percent), or H5N1 flu (~60 percent). A malicious actor with sophisticated enough tools would not release something that kills half a percent; they would dial lethality and transmissibility far higher, giving it civilization-toppling potential.

  • COVID's fatality rate is far lower than SARS, MERS, or H5N1
  • It could have been far worse on both lethality and transmissibility
  • A deliberate attacker would engineer something much more deadly
  • COVID should be read as a mild warning, not the worst case

it is remarkable on a number of levels how benign this this horrific thing is

Rob Reid · 29:00
#covid#biosecurity#lethality#bioterror
Hot Take82:30

Why we can't just pause technology until wisdom catches up

Asked whether we should curtail technological progress for a few thousand years until our wisdom catches up, Reid says no. First it is impossible: if Western democracies stop, China won't, and if China stops, others won't, an unsolvable coordination problem. Second, synthetic biology holds enormous promise, from eventually beating cancer to clean meat, so the path forward is heavy safeguards, not abandonment. And there's no guarantee millennia would deliver the wisdom we seek.

  • A global tech pause is defeated by the coordination problem
  • If any actor keeps going, everyone else's restraint is pointless
  • Synthetic biology could beat cancer and enable clean meat
  • The answer is safeguards and layered defense, not halting progress

if china stops russia's not going to stop and if china and russia stop north korea is not going to stop

Rob Reid · 83:30
#technology#coordination-problem#synthetic-biology#progress
Hot Take99:30

Should we build an air-gapped backup for civilization?

Williamson floats building a defended, self-sufficient, air-gapped community as a civilizational backup, and Reid is struck by the idea. They discuss the Long Now Foundation's library of ~1,500 books to reboot civilization, the failed early-90s Biosphere 2 experiment that slowly starved and asphyxiated its crew, and Neal Stephenson's Seveneves. Reid suggests a rotating national or international effort, arguing that if the choice is some people suffering versus humanity dying, the backup is worth it.

  • A defended, self-sufficient, air-gapped enclave as a species backup
  • Long Now's ~1,500-book library to reboot civilization
  • Biosphere 2 failed as agriculture and internal atmosphere collapsed
  • Reduce risk rather than demand perfection; rotate people through in shifts

if the choice is between some people suffering and humanity dying i think that

Chris Williamson · 102:30
#existential-risk#resilience#biosphere-2#civilization-backup

Explainer· 3

Explainer03:30

Why humanity's power to destroy itself is brand new

Reid explains why existential risk feels absent from daily life: as a quarter-million-year-old species we have only had the ability to wipe ourselves out for about 60 years. Before the mid-1950s there was no plausible step to end everyone; only once hydrogen bombs proliferated onto missiles and bombers on both sides did true civilizational destruction become possible. He argues serious attention to the topic is now 10-30x what it was 15 years ago.

  • Before the mid-1950s no single act could wipe out humanity
  • Two superpowers with proliferated H-bombs made annihilation possible only ~60 years ago
  • We evolved anxiety biases but not for species-level extinction
  • Attention to existential risk is growing rapidly but from a tiny base

as a quarter million year old species we've been facing this for 60 years

Rob Reid · 04:30
#existential-risk#nuclear#history#attention
Explainer07:00

Why climate change gets attention and existential risk doesn't

Reid argues climate change dominates the conversation because its awareness has been compounding since roughly 1970's first Earth Day, like an investment growing at 20 percent a year for fifty years. It also spawned huge industries and economic interest groups, from electric cars to climate academics to politicians, who are full-time committed to it. Existential-risk work has none of that scale yet, just a handful of academics, but far more brainpower than 15 years ago.

  • Awareness compounds like a long-running investment
  • Climate got a ~50-year head start and a founding base of committed people
  • Large industries and economic interests now sustain climate attention
  • Existential risk lacks any comparable interest group or funding base

it's like an investment that compounds at 20 per annum

Rob Reid · 08:00
#climate#attention#compounding#incentives
Explainer46:00

How scientists made H5N1 airborne, then only 'paused' the work

Reid recounts how in 2011 two teams, in Holland and Wisconsin, performed gain-of-function research that made H5N1 flu (roughly 60 percent lethal but barely transmissible) airborne in ferrets, in BSL-3 not even BSL-4 labs. After outcry, publication and US government funding were paused, not banned, for only a few years. The funding was later switched back on, so gain-of-function research is again active and never faced a real global ban.

  • Two 2011 teams made barely-transmissible H5N1 airborne in ferrets
  • The work was done in BSL-3, not BSL-4, labs
  • The US government paused funding for a few years but never banned it
  • Funding was later restored and the research resumed

the world that said oh my god this is scary let's hit the pause button has since said it's cool they hit the play button

Rob Reid · 53:00
#gain-of-function#biosecurity#h5n1#policy

Story· 6

Story00:30

The coolant choice that could have destroyed the ozone layer

Reid opens with a Reddit story about how the near-arbitrary decision to use CFCs instead of BFCs as refrigerants may have saved humanity. BFCs destroy ozone at over 100 times the rate of CFCs, so had that coin flipped the other way the ozone layer could have been obliterated before anyone understood what was happening. It sets the theme: many existential near-misses were pure luck.

  • BFCs destroy ozone at over 100 times the rate of CFCs
  • The choice between them was essentially arbitrary
  • CFCs moved slowly enough that the problem was fixable in time
  • A wrong coolant choice was floated as a possible 'great filter' for civilizations

bfcs destroyed the ozone at over 100 times the rate of cfcs

Chris Williamson · 00:30

maybe that was the great filter that all the other civilizations just chose the wrong coolant medium

Chris Williamson · 01:00
#existential-risk#ozone#near-miss#luck
Story10:30

The submarine officer who vetoed a nuclear war

Reid recounts one of the Cold War's closest calls: during the Cuban Missile Crisis, a Soviet submarine being harried by American practice depth charges came within one vote of firing a nuclear torpedo at the US fleet. Firing normally required two officers to agree, but on this sub a third man, a party political officer, also held a vote and said no. Had he been absent or said yes, a nuclear weapon would likely have been used, almost certainly escalating to doomsday.

  • Soviet subs off Cuba carried nuclear torpedoes and were being menaced by depth charges
  • Firing normally needed two officers to agree
  • A third officer's veto on one sub stopped a launch
  • A single nuclear detonation would have been almost impossible to de-escalate

these are people who are fully empowered to start a nuclear war

Rob Reid · 12:00
#nuclear#cold-war#near-miss#cuban-missile-crisis
Story13:00

The Soviet officer whose gut instinct stopped Armageddon

Reid tells the story of the Soviet early-warning officer who, in the early 1980s, saw American missiles appear on the equivalent of NORAD's screens. His instinct told him a real first strike would involve everything at once, not one or two missiles, so despite orders to launch everything he refused. His judgment that the system had malfunctioned proved correct. Reid stresses his job was to say yes to Moscow, not to think, which makes the call extraordinary.

  • Soviet systems falsely showed a small number of incoming US missiles
  • The officer reasoned a genuine first strike would launch everything
  • He refused a direct order to launch a full response
  • His role was designed for obedience, not independent judgment

your job was to say yes to moscow your job wasn't to think

Rob Reid · 14:30
#nuclear#cold-war#near-miss#false-alarm
Story37:00

The anthrax letters that prove every lab can leak

Reid uses the post-9/11 anthrax letters, which reached Senate Majority Leader Tom Daschle's office, to argue no lab is leak-proof. The military-milled spores almost certainly came from a US Army lab, released by a malicious insider despite the nation being on maximum alert. He adds that foot-and-mouth spores later escaped a UK BSL-4 lab by accident. If the most security-capable institutions leak, any mid-grade university lab can too.

  • Weaponised anthrax reached one of the most protected offices in the US government
  • The spores likely came from a US Army lab via a malicious insider
  • Foot-and-mouth escaped a UK BSL-4 lab by accident
  • Biosafety protocols mostly guard against accidents, not malicious actors

all labs leak all labs at every biosafety level can absolutely leak

Rob Reid · 45:00
#biosecurity#lab-leak#anthrax#insider-threat
Story57:30

The bomb test that might have set the atmosphere on fire

Reid describes how Manhattan Project scientists realised in 1941 that a nuclear detonation might ignite the entire atmosphere, and kept running the numbers up to the day before the Trinity test, estimating the odds at somewhere between one in three million and one in fourteen million. He uses it as the contrast case to privatised risk: the deciders shared the downside, faced a real chance Hitler would get the bomb first, and made the call as a careful public good.

  • Scientists estimated a tiny but non-zero chance of igniting the atmosphere
  • They ran the numbers right up to the day before Trinity
  • The alternative risk was Hitler getting a nuclear weapon first
  • The decision was made carefully as a public good, deciders sharing the risk

there was a non-zero risk that the entire atmosphere could be satellite permanently curtailing not only all human life but everything

Chris Williamson · 57:30
#nuclear#manhattan-project#risk#history
Story107:00

The physics experiment that could have destroyed the universe

Reid describes a 1990s Brookhaven experiment that carried a tiny chance of creating a 'strangelet,' a hypothetical form of matter akin to Vonnegut's ice-nine in Cat's Cradle, which could convert all of Earth, and possibly propagate to destroy the universe. As with Trinity, a cloistered elite deemed it unlikely and proceeded, and only that same elite was positioned to judge or veto it, leaving outsiders dismissed as unqualified nincompoops.

  • A Brookhaven experiment carried a tiny chance of creating a strangelet
  • A strangelet could convert Earth, and possibly the universe, like ice-nine
  • Only the specialist elite could evaluate or veto the risk
  • Outside critics were dismissed as unqualified to understand it

if the strange lit thing had happened it would have turned all of earth into strange lit matter

Rob Reid · 109:30
#existential-risk#physics#strangelet#expert-gatekeeping

Tool· 1

Tool87:30

Screening dangerous DNA orders, and why it must be mandatory

Reid explains the IGSC, an industry body whose members screen every DNA synthesis order against a database of dangerous sequences, flagging them red, yellow, or green at about $15 of work per order, and calling the FBI in the worst cases. The problem: it is voluntary, covers maybe 80 percent (surely less given China), so a bad actor just uses the other 20 percent. As synthesis moves to distributed benchtop printers like the BioXP, those safeguards must become mandatory and built into the machines.

  • IGSC members screen DNA orders against a dangerous-sequence database
  • Orders are rated red, yellow, or green, ~$15 of screening each
  • Screening is voluntary and incomplete, so bad actors route around it
  • Distributed DNA printers must have mandatory screening built in

it needs to be a hundred percent which means that it needs to be mandatory

Rob Reid · 91:30
#biosecurity#dna-synthesis#self-regulation#biotech

Takeaway· 1

Takeaway70:30

The $40 billion bet that could pandemic-proof humanity

Reid argues the single most obvious untaken step is funding pan-familial (universal) vaccines that attack the core of an entire virus family rather than chasing strains. Citing Harvey Fineberg, he says a universal flu vaccine might cost $200 million over ten years with a 75 percent chance of success, or $2 billion worst case. With flu costing the US $365 billion a year, spending ~$40 billion over a decade to target every human-threatening virus family is, he says, an obvious deal we are not taking.

  • Universal vaccines target a whole virus family's core, not individual strains
  • A universal flu vaccine may cost ~$200M and take 10 years, 75% success odds
  • Flu costs the US ~$365 billion a year in productivity and medical bills
  • ~$40 billion over 10 years could cover every human-threatening virus family

let's invest those 40 billion dollars over the next 10 years and get pan vaccines for every virus that infects and kills humans

Rob Reid · 74:00
#vaccines#pandemic-prevention#biosecurity#policy