Lessons · Probability

Conditional probability and Bayes' rule

Interviewers love questions where new information changes the odds, because intuition fails so reliably. The fix is a method, not cleverness: shrink the sample space, or count in natural frequencies.

Conditioning shrinks the world

The probability of A given B is the share of B's outcomes in which A also happens:

P(A | B) = P(A and B) / P(B)

Flip two fair coins and learn that at least one is heads. The possible outcomes are now HH, HT and TH, still equally likely, so P(both heads) = 1/3, not 1/2. If instead you learn that the first coin is heads, only HH and HT remain, and the answer is 1/2. What you're told, and exactly how you learn it, matters.

Bayes' rule

Bayes' rule turns P(evidence | hypothesis) into P(hypothesis | evidence):

P(H | E) = P(E | H) · P(H) / P(E)

The denominator adds up every way the evidence can happen: P(E) = P(E | H)P(H) + P(E | not H)P(not H).

Worked example: a positive test

A condition affects 1% of people. A test catches 99% of cases and falsely flags 1% of healthy people. You test positive. How worried should you be?

Count with 10,000 imaginary people. 100 have the condition, and the test catches 99 of them. Of the 9,900 healthy people, 1% (99) are falsely flagged. So there are 198 positives, and only 99 are real: a 50% chance, not 99%. When a condition is rare, false positives pile up from the large healthy group. This is called base-rate neglect, and natural frequencies are the cure.

Monty Hall, explained by counting

You pick one of three doors. The host, who knows where the prize is, opens an empty door you didn't pick and offers a switch. Your first pick is right 1/3 of the time and wrong 2/3 of the time. If it was wrong, the host's choice is forced, and switching wins. So switching wins 2/3 of the time. The host's knowledge is what makes the opened door informative.

Updating step by step

Evidence can be applied one piece at a time. One coin is fair and one has two heads; you pick one at random and see three heads. Each head is twice as likely under the two-headed coin, so the odds go from 1 : 1 to 8 : 1, and P(two-headed) = 8/9. Working in odds (multiply by the likelihood ratio for each observation) is often the fastest route.

Common traps

• Mixing up P(A | B) and P(B | A). P(positive | sick) = 99% is not P(sick | positive).
• Ignoring how information was revealed. "At least one is a girl" and "the older one is a girl" give different answers.
• Forgetting the base rate. Always ask how common the hypothesis was before the evidence.

Practice

Next: Random walks and gambler's ruin · Previous: Expected value