Cognitive Biases Potentially Affecting judgment of Global Risks


  • Most people would prefer not to destroy the world
  • Even evildoers need a world to do evil in
  • As a result, if the world is destroyed, it will likely be the result of a mistake rather than an intentional act
  • In order to minimize these mistakes, we should study heuristics and biases in order to understand how we can be led into a situation where we inadvertently destroy the world or ourselves


  • People estimate the risk of events based upon how easy it is to recall examples of the risk rather than how often the risk actually occurs
  • Examples:
    • People think there are more words that start with r than words with r as the third character because it’s a lot easier to recall words that start with a particular letter than words that have that letter as the third character
    • People think homicides kill more than stomach cancer
    • People evaluate flood risk by thinking of the worst flood that they’ve experienced rather than the worst flood that has occurred
  • Engaging in risk mitigation can actually increase our vulnerability to risk, if we reduce common but mild examples of risks with uncommon but extreme versions
    • Example: building dams can actually increase vulnerability to flooding, as the frequency of flooding decreases but the impact of each flooding event increases dramatically
  • Societies that are well-protected against minor hazards don’t protect against major hazards
  • Societies that have many examples of mild examples of a major hazard use those examples as the upper bound of what’s possible

Hindsight Bias

  • People routinely say that things are more predictable in hindsight than they actually are
  • In hindsight, people look at the cost of mitigating the risk that actually occurred, not the cost of mitigating all of the risks at that level of probability
  • Attempts to de-bias people by telling them to avoid hindsight bias are, in general, not successful

Black Swans

  • A “black swan” is a situation in which most of the variation comes from random, hard-to-forecast, low-probability-but-high-impact events
  • Many financial strategies fall victim to this phenomenon - generate steady returns in good years, but are then wiped out by a single low-probability event
  • Our vulnerability to black swans results from a combination of hindsight and availability bias
  • Availability bias predisposes us to place optimistically low upper bounds on how much damage can occur
  • Hindsight bias causes us to learn overly specific lessons, which leaves us vulnerable to the next black swan event
  • The prevention of black swan events is not easily seen or rewarded

The Conjunction Fallacy

  • The conjunction rule states that the probability of two events occurring together can never be higher than the probability of each event occurring separately
  • However, adding details to a story makes the story seem more true, even as each additional detail makes the probability of the story being true less
  • People use “representativeness” as a heuristic when thinking about probability – adding details can make a story seem more representative of a particular category of scenarios, even as it reduces the probability of that particular story being true
  • As a result, vivid, specific scenarios can distort our sense of security
    • Vivid scenarios can make low-probability events seem more probable
    • They also make us think that high-impact events have been “solved”, when in reality they have only been solved for a particular scenario
  • People tend to overestimate conjunctive fallacies and underestimate disjunctive fallacies
    • Overestimate the probability of 7 events, each with a probability of 90% all occurring
    • Underestimate the probability of at least 1 of 7 events, each with a probability of 10%, occurring

Confirmation Bias

  • People try to confirm hypotheses rather than disprove them – will look for evidence that shows that their hypothesis is true, rather than their hypothesis being false
  • Comes in two forms, cold and hot
    • Cold - emotionally neutral
    • Hot - emotionally charged (i.e. politics)
    • Unsurprisingly, the hot form of confirmation bias is harder to combat
  • Confirmation bias can result in two observers of the same stream of evidence updating in opposite directions, as they selectively choose what to believe and what to reject
  • People decide what they believe far more quickly than they realize – if you can guess what your answer to a question will be, chances are that will be your answer to the question

Anchoring, Adjustment and Contamination

  • People anchor estimates to data that they’ve just received, even when that data is completely irrelevant
  • People start estimates from an anchoring point, and then adjust upwards or downwards until they reach a figure that seems reasonable
  • Often, the “reasonable” figure is far higher or lower than the actual figure - people don’t adjust as much as they should
  • The generalized form of anchoring is contamination
  • Almost any information can contaminate a judgment
  • Contamination is hard to offset
  • People will say that they were not affected by anchoring or contaminating information even when the statistical evidence shows they clearly were

The Affect Heuristic

  • People’s subjective appraisals of the “goodness” or “badness” of a technology colors theirs appraisal of its risks and benefits
  • Providing information that increases perception of benefit decreases perception of risk and vice versa
  • This effect is magnified by sparse information, which is especially troubling for X-risks, since we don’t know very much about them
  • More powerful technologies can be rated as less risky if they also promise great benefits

Scope Neglect

  • The amount people are willing to pay to mitigate a risk has little to do with the magnitude of that risk
  • Possible explanations
    • Combination of affect heuristic and availability bias – people pay based upon whether they can remember the risk occurring and how they feel about the consequences of that risk
    • People are choosing to buy a certain amount of moral satisfaction – pay based upon how much of a warm glow they feel after paying
    • People pay based upon the cause area of the risk, regardless of the impact of the intervention
  • Scope neglect applies equally to human and non-human impacts – doesn’t seem to matter if humans or animals will benefit from risk mitigation

Calibration and Overconfidence

  • People are wildly overconfident in their estimates
  • The true value lies outside of people’s 98% confidence intervals about 42% of the time
  • Letting people know about calibration makes them better calibrated, but their calibration is still pretty bad
  • People don’t realize how wide a range they need in order to have a 98% or 99% confidence interval
  • The planning fallacy is a specific example of calibration error
    • Only 45% of students completed their honors thesis by the date they’d specified on their 99% confidence interval
  • Reality usually delivers results that are worse than the “worst case scenario”

Bystander Apathy

  • People are less likely to act in a group than they are to act on their own
  • Most situations are not unambiguous emergencies
  • In a situation that has some level of ambiguity, people look to others to judge how to react
  • As a result, nobody ends up doing anything because everyone is looking to someone else

A Final Caution

  • Every true idea that discomforts you will match at least one psychological error
  • We care about cognitive biases and distortions only insofar as they result in factual errors
  • If there are no factual errors, then what do you care about the psychology?


  • We need to have an organized body of thinking about existential risks not because the risks are similar, but the way we think about those risks is similar
  • Skilled practitioners in a field cannot be relied upon to reliably estimate the level of existential risk in their field
  • Right now, most people stumble across the knowledge of biases and heuristics accidentally – a more organized body of knowledge would make this information more accessible to people outside of psychology and social science
  • Thinking about existential risk falls prey to the same cognitive distortions we use for all of our thinking, but the consequences of a mistake with existential risk are much more severe