Ethical Injunctions

  • Ethical injunctions are things you shouldn’t do even if you believe they’re the right thing to do
  • Never try to deceive yourself or believes something other than what is probably true. It’s more likely that you’ve made a mistake than found the one case in which a false belief is more helpful than a true belief
  • Believing something false is a black-swan bet. Even if it works out most of the time, the one time it doesn’t work out can undo all the gains you got from winning the bet
  • Your brain is corrupted hardware - you will always believe that your calculations are correct, even if they’re way off
    • So why compound the errors by choosing to believe something that you know is probably not true?
  • Spreading falsehoods for the greater good can backfire really badly
    • Truth is always easier to make consistent than falsehood
  • When people “rationally” abandon ethical principles, it always seems to be that they go towards greater ethical leninence, rather than greater ethical strictness

Something To Protect

  • Rationality must have purpose, otherwise it collapses into infinite recursion
  • Rationality is about being more effective at achieving your goals
  • “Truth” is too abstract a goal - too easy to claim that you’re defending truth when you’re defending superstition
  • Requiring beliefs to yield predictions and then testing those predictions brings us much closer to truth than trying to reason out what’s true on our own
  • Rationality is not about choosing only methods that are guaranteed to work
    • Rationality is about embracing the hard math of probability and being willing to at least look at the conclusion of that math, even if it’s considered “morally repugnant”
  • Part of the aesthetic of rationality is subordinating rationality to a higher purpose - treat rationality as a means, rather than an end

When (Not) To Use Probabilities

  • If you don’t know the probability of something, don’t make up a number for the sake of having a number
  • Everything has a probability, but that probability may be too difficult to compute
  • In those case, we should avoid making up numbers to give ourselves a feeling of having more information than we actually do
  • If you find an inconsistency in your model of the world, patching your model to be consistent doesn’t necessarily make it more accurate
  • Example numbers may be helpful, but don’t confuse those example numbers with true probabilities
  • Counterpoint from Slate Star Codex: If It’s Worth Doing, It’s Worth Doing With Made Up Statistics

Newcomb’s Problem and the Regret of Rationality

  • Setup:
    • Box A: Transparent, contains $1000
    • Box B: Opaque, may contain $1,000,000 or nothing
    • Superintelligent being Omega offers you a choice:
      • Box B only (“one-box”)
      • Box A and Box B (“two-box”)
    • Omega chooses the boxes before you’re offered the choice - neither you nor Omega can alter the contents of the boxes
    • Everyone who has chosen the two-box strategy has found that Box B is empty
    • Everyone who has chosen the one-box strategy has found Box B to contain a million dollars
    • Which box should you choose?
  • Causal decision theory says that you should two-box, even though everyone who has taken the two-box strategy has had a worse outcome
  • “It’s not your fault that Omega chooses to reward irrationality”
  • However, rationality is about winning
  • Omega doesn’t care the reasoning by which you come to your decision
  • You don’t have to have a “good reason” for choosing Box B, you only need to choose Box B
  • Instead of starting with a “rational” decision theory and then saying that Omega rewards irrationality, we should start by looking at the outcomes and then come up with a decision theory that ensures that we’ll choose the best outcomes
  • Rationalists should never envy the choices that nonrationalists make - there is nothing stopping us from making the same choices

Tsuyoku Naritai! (I Want To Become Stronger)

  • The important thing is to make progress
  • Never take pride in your ignorance; take pride in the work that you do to remedy your ignorance