Cognitive Biases Potentially Affecting Judgment of Global Risks

Previously discussed on July 16 2018

Introduction

  • Most people would prefer not to destroy the world
  • Even evildoers need a world to do evil in
  • As a result, if the world is destroyed, it will likely be the result of a mistake rather than an intentional act
  • In order to minimize these mistakes, we should study heuristics and biases in order to understand how we can be led into a situation where we inadvertently destroy the world or ourselves

Conclusion

  • We need to have an organized body of thinking about existential risks not because the risks are similar, but the way we think about those risks is similar
  • Skilled practitioners in a field cannot be relied upon to reliably estimate the level of existential risk in their field
  • Right now, most people stumble across the knowledge of biases and heuristics accidentally – a more organized body of knowledge would make this information more accessible to people outside of psychology and social science
  • Thinking about existential risk falls prey to the same cognitive distortions we use for all of our thinking, but the consequences of a mistake with existential risk are much more severe

How Likely Is An Existential Catastrophe

  • An existential risk is any future event that would either render humanity extinct or permanently reduce it to a Stone-Age civilization
  • Formalized by Nick Bostrom in 2002
  • The number of existential risk scenarios has risen in the past few decades and will only continue to rise in the coming century
    • Prior to the past few decades, all the existential risks that humanity faced were natural
      • Super-volcanoes
      • Asteroid impact
      • Global pandemic
    • Today, in addition to the natural existential risks, we have a number of man-made existential risks
      • Nuclear weapons
      • Climate change
      • Biodiversity loss
    • In addition, we have a number of new man-made existential risks on the horizon
      • Engineered bio-weapons
      • Nanotechnology
      • Geoengineering
      • Artificial intelligence
    • This means that there will be even more existential risks by the beginning of the 22nd century
  • The increasing number of existential risk scenarios suggests that the probability of existential risk has also risen
    • Survey at Future of Humanity Institute yielded a 19% chance of human extinction this century
    • Sir Martin Rees, co-founder of the Center for the Study of Existential Risk, estimates that civilization has only a 50-50 chance of making it through the 21st century
  • Putting risks in perspective
    • These estimates are really high
    • If FHI’s estimate of existential risk is correct, then it means that a person living today is 1500 times more likely to die in a human extinction event than they are to die in a plane crash
    • If Rees’ estimate of civilizational collapse is accurate, it means that a person living today is 50 times more likely to encounter the collapse of human civilization than they are to die in a motor vehicle accident
  • Objective and subjective estimates
    • Should we dismiss these estimates of the probability of existential risk because they are so high?
    • The probabilities of some existential risks can be estimated
      • Analysis of craters on the earth and moon indicates that an asteroid or comet capable of causing an existential catastrophe hits roughly once every 500,000 years
      • Geological records indicate that a super-volcanic eruption capable of causing a “volcanic winter” roughly happens once every 50,000 years
      • Climate scientists have detailed models which allow them to assess the risk posed by increasing levels of greenhouse gases
    • Other existential risks, however, require some subjective analysis
      • A subjective estimate does not have to be arbitrary or haphazard
      • Can use evidence from technological trends and the social sciences to inform estimates
  • The biological threat
    • The cost of genome sequencing has decreased at a rate faster than Moore’s Law
    • This decline is indicative of progress in biotechnology
    • This means that biotechnology and synthetic biology will become a significant risk later this century
  • The nanotechnology threat
    • A “nanofactory” is a hypothetical system that would be capable of manufacturing products with atomic precision for a fraction of the cost of current manufacturing
    • Only three resources are needed to operate a nanofactory:
      • Power
      • Instructions
      • Supply of feedstock molecules
    • Nanofactories would allow terrorists to manufacture vast amounts of conventional and potentially even nuclear weapons
    • Nanofactories could disrupt trade networks, making state level conflict more likely
    • Increased use of nanotechnology could result in releases of toxic nanoparticles
    • Increasing risk of nanotechnology means increasing risk of uncontrolled self-replicators (“gray goo”)
  • The superintelligence threat
    • An artificial superintelligence whose values are even slightly misaligned with ours could be disastrous
    • A superintelligence which is programmed to maximize some resource could very well destroy earth’s biosphere in order to do so
    • A 2014 survey of AI experts indicated a 50% probability of a human-level AI by 2040 and a 90% probability by 2075
  • The human threat
    • As powerful technologies become cheaper, they become more available to malicious actors
    • This means that the risk of a catastrophe occurring because of technology falling into the wrong hands is increasing
    • The number and membership of extremist organizations is increasing
  • Closer to midnight
    • We have never been closer to disaster than today in our 200,000 year history
    • While our ability to quantify emerging risks is far more subjective than our ability to quantify natural risks, there are reasons for concern about both
    • Yet people are still more concerned about “mundane” dangers like plane crashes than they are about existential risk
    • This is a problem because recognizing risks usually precedes solving them

Petrov Day Shenanigans

  • Seattle and Oxford rationality communities decided to hold a Petrov Day simulation
  • Created an application to simulate a “nuclear launch” that would cause the other side to end their party and destroy their cake
  • Application would notify the other side of launches, but would also generate false alarms
  • The server and the Seattle computer got slightly out of sync, so even though the game appeared over in Seattle, there was still a few seconds left to go
  • Seattle decided to press the button, since the game was “over”
  • This triggered a launch warning in Oxford
  • After verifying that the launch was real, and not a false alarm, Oxford decided to end their party and burn their cake, out the desire to remain within the spirit of the game
  • The experience serves as both a warning and a demonstration of the risks of accidental nuclear war

S-Risks: Why They Are The Worst Existential Risks and How To Prevent Them

  • S-risk is the risk of severe suffering on a cosmic scale, exceeding all suffering on Earth to this date
  • S-risk is a subtype of X-risk
  • S-risk are the outcomes that are “worse than extinction”
  • However, the fact that S-risks are “worse” than X-risk isn’t, in and of itself, a good reason to prioritize working on S-risk
  • Still need to prove that S-risks have non-negligible probability, are tractable to work on and are a neglected area
  • Probability
    • S-risk isn’t significantly more improbable than AI-related X-risk
    • Artificial sentience
      • The capacity for suffering isn’t necessarily limited to biological creatures
      • It’s possible that sufficiently advanced artificial intelligences could suffer too
    • It’s unlikely that anyone will intentionally create AIs that experience suffering
    • However, if our first AIs are unable to communicate via language, they may experience suffering and be unable to communicate that to us
    • Superintelligent AIs may create simulations of suffering or use subagents that suffer
    • Humans or AIs engaged in competition for a fixed set of resources may engage in behavior that causes suffering, even if no individual agent values suffering
  • Tractability
    • Some work on reducing S-risk overlaps with work on reducing X-risk
    • However, there are measure that we could take, such as building in time limits on AIs that would reduce S-risk without necessarily reducing X-risk
    • While S-risk is currently outside the Overton window of serious discourse, that’s not a reason to stop working on it – after all, AI X-risk was once outside that window as well
  • Neglectedness
    • X-risk seems to be conflated with extinction risk
    • We don’t think about “worse than death” outcomes
    • As a result, interventions that would reduce S-risk without reducing X-risk are understudied, and there may be low-hanging fruit there
  • Shaping the far future is not a binary choice between extinction and utopia