- Allegorical story about a shepherd who keeps track of sheep by putting stones in a bucket
- Model the number of sheep in the field by the number of stones in the bucket
- Model and reality do not automatically correspond – there has to be a mechanism to update the model to reflect reality
- Updating the model does not update reality
- Reality is that which exists even when you don’t believe in it
- “The map is not the territory” - coined by Alfred Korzybski
- Abstractions are distinct from the underlying reality that they represent
- Abstractions are necessary - a map that is 1:1 to the territory is useless
- However, we must understand the limits of our abstractions
- Limitations of maps
- Map may be incorrect in ways that we don’t realize
- Map loses information by virtue of being a map - this information may be important
- Maps require interpretation, and that interpretation may itself be a source of error
- We keep applying models that have worked for us in the past, even when we’ve gone beyond the domain the model was designed for
- Example: JC Penny’s redesign
- Ron Johnson tried to make JC Penny’s into a more upscale shopping experience
- Break up the space into multiple smaller “micro-stores”
- Discontinue discounts - “straightforward” pricing model
- Redesign was a disaster - alienated existing shoppers and failed to bring in new shoppers
- Ron Johnson was led astray by his experience at Apple - tried to apply a model which worked for Apple’s customers to JC Penny’s customers
- Example: Value At Risk
- Common risk model used in finance
- Uses historical data to compute how much money is at risk under various confidence intervals (95%, 99%, etc)
- The problem with VaR is that it ignores the probability of you choosing the wrong probability distribution
- In 1987, on one day, the stock market dropped by 22.7%
- The previous record had been a 12.82% decline
- According to VaR, the odds of a 22.7% decline in one day would have been infinitesimally small
- The other problem with VaR is that the error bars on the model are often larger than the value itself
- Using VaR is a problem because banks optimize to VaR - seek to maximize the amount of return for a given VaR
- However, because VaR mis-measures risk, these strategies are more risky than the banks think they are, which, in turn causes larger losses than banks have planned for
- The correct thing to do is to assume that there are no correct maps, and use heuristics that are informed by models
- Design organizations with margins of safety at every level
- The downside to this approach is that you have a lower short turn rate of return - trade this off for a higher chance of long-term survival
- The important thing is to never forget that you’re using a model, and that your model does not represent reality, especially in highly abstract fields like finance
Percentages, Frequencies and Waterfalls
- Example: disease screening
- A disease occurs with probability of 20% in a target population
- Of people with the disease, 90% register positive on the indicator
- Of people without the disease, 30% register positive on the indicator
- Given a positive indicator, what is the probability that someone has the disease?
- Odds of a person having the disease is 1:4
- Odds of a person with the disease registering positive on the indicator vs. someone without the disease registering positive is 3:1
- Odds of a person with a positive indication having the disease is (1:4)·(3:1) = 3:4
- So the probability of someone having the disease, given that they have a positive indicator is 3/7 ⋍ 43%
- Bayes rule is the general form of the process above:
- Start with a prior odds ratio
- Multiply by a likelihood, which you get from evidence
- Results in a posterior odds ratio
- One way of visualizing this is with a waterfall:
- Think of a waterfall with two streams, red and blue
- On each stream, a portion is diverted away, and the remainder is allowed to continue to the bottom
- The important thing isn’t the amount of water, but the ratios between the amounts of red and blue water, and the amounts that are diverted away
- The original ratio of red:blue water is the prior probability
- The ratio of red:blue that’s diverted away is the likelihood
- The ratio of red:blue in the purple water at the bottom is the posterior
- Test problem
- 90% of widgets from a particular assembly line are good, and 10% are bad
- If a widget is good, it has a 4% chance of producing sparks
- If a widget is bad, it has a 12% chance of producing sparks
- Given a widget that is producing sparks, what is the probability of it being bad?
Applications of Bayesian Reasoning
- Bayesian reasoning forms a framework that we can apply even to problems that are more qualitative
- Thinking in terms of priors, likelihoods, and posteriors forms a powerful framework that we can apply even to problems where we don’t have known odds ratios
- Example: OKCupid date
- OKCupid reports a 96% match
- From prior experience, the person knew that this indicated 2:5 odds of the relationship being worth pursuing
- Date canceled without further explanation
- Canceling on the first date has a 1:3 odds ratio of this being a relationship worth pursuing
- Calculating it out, we have (2:5)·(1:3) = 2:15 odds of this being a relationship worth pursuing
- The key thing here isn’t the explicit numbers, but rather knowing that canceling on the first date is strong evidence against the pursuing this relationship
- Example: Internment of Japanese Americans in World War 2:
- Earl Warren said that the lack of sabotage by Japanese-Americans was designed to lull the US into a false sense of security
- However, we know, even without explicit numbers, that the lack of sabotage can only make our posterior probability estimate go down
- Therefore it’s patently ridiculous to suggest that a lack of attacks was evidence for a conspiracy