- In 2000, Eliezer still doesn’t fully appreciate the risk that AI poses
- The notion of inscribing morality into the AI is a “backup plan” against there being no objective morality out there for the AI to discover
- This leads to a process in which Eliezer slowly updates away from AI being automatically “safe”
- Eliezer thinks he was too slow in this process
- The moment he realized that AI was not automatically safe, he should have stopped and rethought his entire strategy
- It’s a lot more efficient to admit a single large mistake than it is to admit many small mistakes
- The moment you find out that one of your beliefs is false, you need to examine all of the plans and strategies based on that belief
- The moment at which Eliezer understands the folly of building an AI without morality is when he comprehends the notion of intelligence being a subcategory of an optimization process
- Human intelligence and natural selection belong to the same category of things, even though human intelligence has foresight and natural selection is a blind optimizer
- All forms of optimization seek to constrain the future
- When Eliezer realizes that “ordinary physical processes” like natural selection can constrain the future just as well as intelligence can, he realizes that intelligence doesn’t imply morality
- There is no guarantee that an alien intelligence will constrain the future into a form that we find morally valuable
- Reading E.T. Jaynes was the first time that Eliezer picked up a sense of formidability from mathematical arguments
- Eliezer aspires to be as formidable in AI morality as Jaynes was in probability theory
- However, that may never happen
- Part of the reason that Eliezer is spending so much time writing about friendly AI is that he hopes that someone more intelligent than him can come and pick up the pieces and advance the topic farther, faster than Eliezer could on his own
- Eliezer lost a lot of status when he admitted that he wasn’t ready to work on AI because there was no guarantee that the AI would be moral
- However, Eliezer also realized that there is no rule that says that Nature has to preserve humanity
- There is also no rule that says that humanity cannot self-destruct
- AI researchers find the possibility of being upstaged more emotionally visceral than the possibility of designing a system that leads to 7 billion dead
- Following all the rules and doing everything that you’re “supposed to” is no guarantee preventing you from building a system that wipes out humanity
- The root of Eliezer’s folly was his belief in the invulnerability of the future
- The problem is that this belief is equivalent to believing in God - that there is some force out there preventing humanity from coming to a bad end overall
- There is no rule that history has to make sense, has to have a positive trend, or that big effects need big causes
- There is no rule that says that the challenges that Nature sets in front of you are challenges that you’ll be able to overcome
- It is up to humanity to make the future better, and doing so is a collective choice, made up of all our individual choices