WEBVTT 00:00:00.046 --> 00:00:03.806 Suppose a medical test is described as ninety-nine percent accurate, 00:00:03.889 --> 00:00:05.841 and it comes back positive. 00:00:06.037 --> 00:00:08.046 That sounds almost conclusive. 00:00:08.150 --> 00:00:09.681 But if the disease is rare, 00:00:09.764 --> 00:00:13.306 the positive result can still be more likely wrong than right. 00:00:13.502 --> 00:00:18.264 We are going to see why by counting people before writing any probability formula. 00:00:19.916 --> 00:00:23.016 Here is the question in its most personal form. 00:00:23.213 --> 00:00:26.372 A positive result says you have a rare disease. 00:00:26.684 --> 00:00:28.461 Does ninety-nine percent accurate 00:00:28.461 --> 00:00:31.445 mean there is a ninety-nine percent chance you have it? 00:00:31.642 --> 00:00:36.415 No. That number describes how the test behaves inside known groups. 00:00:36.518 --> 00:00:40.780 It does not yet answer what group a positive person probably came from. 00:00:41.391 --> 00:00:43.412 Take ten thousand people. 00:00:43.515 --> 00:00:44.268 The large gray 00:00:44.351 --> 00:00:48.996 field represents the people in this cohort who do not have the disease. 00:00:49.308 --> 00:00:53.047 I have magnified the affected people so we can actually see them. 00:00:53.960 --> 00:00:56.980 Let the disease affect one person in a thousand. 00:00:57.083 --> 00:01:00.393 That is a prevalence of zero point one percent. 00:01:00.706 --> 00:01:05.014 In ten thousand people, only ten actually have the disease. 00:01:05.326 --> 00:01:09.170 The remaining nine thousand nine hundred ninety are healthy. 00:01:10.083 --> 00:01:13.822 Now we must say exactly what ninety-nine percent accurate means. 00:01:14.019 --> 00:01:16.608 For this lecture, it means two things. 00:01:16.921 --> 00:01:22.225 Sensitivity is ninety-nine percent, so among people who truly have the disease, 00:01:22.308 --> 00:01:25.420 the test is positive ninety-nine percent of the time. 00:01:25.872 --> 00:01:28.332 Specificity is also ninety-nine percent, 00:01:28.415 --> 00:01:32.978 so among healthy people, the test is negative ninety-nine percent of the time. 00:01:33.403 --> 00:01:36.480 Apply sensitivity to the ten affected people. 00:01:36.677 --> 00:01:39.731 Ninety-nine percent of ten is nine point nine. 00:01:39.928 --> 00:01:42.190 So across many cohorts like this one, 00:01:42.273 --> 00:01:46.268 we expect about nine point nine true positive results. 00:01:47.064 --> 00:01:49.491 Now turn to the healthy majority. 00:01:49.688 --> 00:01:54.635 Ninety-nine percent specificity leaves a one percent false-positive rate. 00:01:54.831 --> 00:01:59.871 One percent sounds tiny, but it acts on nine thousand nine hundred ninety people. 00:02:00.667 --> 00:02:06.032 One percent of that enormous healthy group is ninety-nine point nine false positives. 00:02:06.229 --> 00:02:11.686 The decimal counts are expected counts, averages over many equally sized cohorts. 00:02:11.883 --> 00:02:16.446 In one real cohort we would see whole people, very close to these values. 00:02:17.519 --> 00:02:20.805 The test has now run on all ten thousand people. 00:02:21.002 --> 00:02:26.088 But after a positive result, most of that original cohort is no longer relevant. 00:02:26.703 --> 00:02:30.790 We need a new reference group: everyone whose result was positive. 00:02:31.587 --> 00:02:34.571 The green pile contains the positive results from people 00:02:34.571 --> 00:02:36.627 who truly have the disease. 00:02:36.823 --> 00:02:39.575 Its expected size is nine point nine. 00:02:39.772 --> 00:02:41.735 These are the true positives. 00:02:42.439 --> 00:02:45.922 The yellow pile contains positive results from healthy people. 00:02:46.119 --> 00:02:49.324 Its expected size is ninety-nine point nine. 00:02:49.521 --> 00:02:53.875 These are false positives, contributed by the enormous healthy majority. 00:02:54.672 --> 00:02:56.101 Pause on the picture. 00:02:56.413 --> 00:02:59.201 The test is excellent inside either group. 00:02:59.397 --> 00:03:02.460 Yet the false-positive pile is about ten times taller, 00:03:02.543 --> 00:03:06.178 because the healthy group supplying it began nine hundred ninety-nine times 00:03:06.178 --> 00:03:07.978 larger than the disease group. 00:03:08.682 --> 00:03:10.456 Now gather the two piles. 00:03:10.539 --> 00:03:15.393 Write nine point nine true positives above ninety-nine point nine false positives. 00:03:15.706 --> 00:03:17.771 Rule beneath them and add. 00:03:17.854 --> 00:03:22.139 The positive-test group contains about one hundred nine point eight people. 00:03:22.936 --> 00:03:24.455 To answer our question, 00:03:24.538 --> 00:03:28.254 ask what fraction of that positive group came from the green pile. 00:03:28.450 --> 00:03:31.363 The numerator is nine point nine true positives. 00:03:31.446 --> 00:03:35.963 The denominator is every positive result, one hundred nine point eight. 00:03:36.759 --> 00:03:39.175 That fraction is about nine percent. 00:03:39.372 --> 00:03:41.729 So after one positive test, the chance 00:03:41.729 --> 00:03:46.048 of actually having the disease is only about nine percent under our assumptions. 00:03:46.961 --> 00:03:50.874 The complementary probability is about ninety-one percent. 00:03:51.071 --> 00:03:55.494 In other words, this positive result is probably wrong, even though the test 00:03:55.494 --> 00:04:00.058 has ninety-nine percent sensitivity and ninety-nine percent specificity. 00:04:00.971 --> 00:04:03.084 Nothing paradoxical happened. 00:04:03.281 --> 00:04:06.892 The test made errors on only one percent of healthy people. 00:04:07.089 --> 00:04:11.048 There were simply so many healthy people that their small error rate produced far 00:04:11.048 --> 00:04:13.684 more positive results than the rare disease did. 00:04:14.785 --> 00:04:16.687 Now that the populations are visible, 00:04:16.770 --> 00:04:19.906 we can compress the same reasoning into Bayes' theorem. 00:04:20.102 --> 00:04:25.688 Start with the rule we already used: true positives divided by all positive results. 00:04:26.484 --> 00:04:28.807 The vertical bar means given. 00:04:29.004 --> 00:04:31.614 P of D given positive asks: 00:04:31.697 --> 00:04:36.400 among people known to have a positive result, what fraction have disease? 00:04:36.713 --> 00:04:39.535 The phrase after the bar names the reference group. 00:04:40.331 --> 00:04:43.583 Before building the formula, name its ingredients. 00:04:43.780 --> 00:04:48.398 P of D is prevalence, the fraction who have the disease before testing. 00:04:48.481 --> 00:04:51.268 Here it is zero point zero zero one. 00:04:51.972 --> 00:04:57.569 P of positive given D is sensitivity, the positive rate inside the disease group. 00:04:57.672 --> 00:04:59.972 Here it is zero point nine nine. 00:05:00.513 --> 00:05:04.958 P of D complement is the healthy share, zero point nine nine nine. 00:05:05.041 --> 00:05:09.265 And P of positive given D complement is the false-positive rate, 00:05:09.348 --> 00:05:11.137 zero point zero one. 00:05:11.934 --> 00:05:15.464 Now replace each pile by the probability that creates it. 00:05:15.660 --> 00:05:16.904 Sensitivity 00:05:17.518 --> 00:05:20.619 times prevalence creates the true-positive share. 00:05:20.815 --> 00:05:22.720 False-positive rate times 00:05:23.033 --> 00:05:26.122 healthy share creates the false-positive share. 00:05:26.919 --> 00:05:29.543 The numerator keeps the true-positive route. 00:05:29.647 --> 00:05:33.084 The denominator adds both routes into the positive group. 00:05:33.188 --> 00:05:35.799 This is exactly what the two visible piles did, 00:05:35.882 --> 00:05:38.831 with the common population size canceled out. 00:05:39.883 --> 00:05:41.997 Substitute our values. 00:05:42.194 --> 00:05:47.524 The true-positive route is zero point nine nine times zero point zero zero one. 00:05:47.720 --> 00:05:53.700 The false-positive route is zero point zero one times zero point nine nine nine. 00:05:54.404 --> 00:05:59.095 The result is about zero point zero nine zero, or nine percent. 00:05:59.408 --> 00:06:02.390 Bayes' theorem has not introduced a new argument. 00:06:02.473 --> 00:06:06.688 It has merely named the count in a form that works for any cohort size. 00:06:07.485 --> 00:06:10.330 The common mistake is to reverse the condition. 00:06:10.527 --> 00:06:13.755 Ninety-nine percent sensitivity describes positive results among 00:06:13.755 --> 00:06:16.472 people already known to have disease. 00:06:16.785 --> 00:06:20.675 We wanted disease among people already known to have a positive result. 00:06:20.871 --> 00:06:22.591 Those are different questions. 00:06:23.640 --> 00:06:27.577 Bayes' formula lets us change one ingredient at a time. 00:06:27.774 --> 00:06:32.438 First keep the test fixed at ninety-nine percent sensitivity and specificity, 00:06:32.522 --> 00:06:35.089 and vary only the disease prevalence. 00:06:35.885 --> 00:06:39.230 The horizontal coordinate is prevalence as a percentage. 00:06:39.334 --> 00:06:43.607 The vertical coordinate is the chance of disease after a positive result. 00:06:43.804 --> 00:06:48.808 At zero point one percent prevalence, our yellow point reads about nine percent. 00:06:49.860 --> 00:06:52.218 Raise prevalence to one percent. 00:06:52.670 --> 00:06:56.745 Now one person in a hundred has the disease before testing. 00:06:56.942 --> 00:06:59.029 The point climbs to fifty percent, 00:06:59.113 --> 00:07:03.375 because the expected true-positive and false-positive piles are equal. 00:07:04.078 --> 00:07:06.413 Raise prevalence to five percent. 00:07:06.609 --> 00:07:08.441 The test has not improved at all, 00:07:08.525 --> 00:07:12.705 but the positive result now means about eighty-three point nine percent. 00:07:13.618 --> 00:07:15.136 At ten percent prevalence, 00:07:15.220 --> 00:07:18.680 the posterior reaches about ninety-one point seven percent. 00:07:18.993 --> 00:07:23.242 The same test result means something very different in a high-risk population than 00:07:23.242 --> 00:07:25.193 in a low-risk population. 00:07:26.245 --> 00:07:31.285 Prevalence is the starting information, sometimes called the prior probability. 00:07:31.482 --> 00:07:34.396 A positive test updates that starting point. 00:07:34.500 --> 00:07:36.161 It does not erase it. 00:07:37.674 --> 00:07:39.648 Now restore the very rare prevalence 00:07:39.648 --> 00:07:43.514 of zero point one percent and change the test itself. 00:07:43.711 --> 00:07:46.692 To keep the phrase accuracy unambiguous, 00:07:46.776 --> 00:07:50.085 a will mean both sensitivity and specificity. 00:07:50.882 --> 00:07:55.109 At ninety-nine percent accuracy, the point again sits near nine percent. 00:07:55.305 --> 00:07:59.904 Its label gives accuracy first and the posterior probability second. 00:08:00.817 --> 00:08:03.732 Drop accuracy to ninety-five percent. 00:08:03.928 --> 00:08:06.553 The posterior falls below two percent. 00:08:06.750 --> 00:08:08.677 A five percent false-positive rate 00:08:08.677 --> 00:08:13.821 applied to nearly ten thousand healthy people overwhelms the true-positive pile. 00:08:14.618 --> 00:08:19.030 Return to ninety-nine percent, and we recover about nine percent. 00:08:19.482 --> 00:08:22.780 Now push the accuracy to ninety-nine point nine percent. 00:08:23.093 --> 00:08:27.401 The false-positive rate falls from one percent to one tenth of one percent. 00:08:28.198 --> 00:08:33.121 That extra nine in the accuracy raises the posterior to about fifty percent. 00:08:33.434 --> 00:08:37.576 For an extremely rare disease, tiny changes in the false-positive rate 00:08:37.660 --> 00:08:41.945 can matter enormously because that rate acts on the healthy majority. 00:08:42.742 --> 00:08:47.363 So the phrase ninety-nine percent accurate is incomplete on its own. 00:08:47.560 --> 00:08:51.404 We need sensitivity, specificity, and prevalence. 00:08:51.507 --> 00:08:56.024 Change any one of them and the meaning of a positive result can swing dramatically. 00:08:57.111 --> 00:09:01.942 Suppose the same person is tested again and the second result is also positive. 00:09:02.255 --> 00:09:05.204 Begin with the group that survived the first test: 00:09:05.308 --> 00:09:10.707 about nine point nine true positives and ninety-nine point nine false positives. 00:09:11.504 --> 00:09:15.032 Assume the second test is conditionally independent of the first. 00:09:15.115 --> 00:09:19.829 Among the people who truly have disease, it again detects ninety-nine percent. 00:09:19.933 --> 00:09:24.322 Ninety-nine percent of nine point nine is nine point eight zero one. 00:09:25.119 --> 00:09:28.205 Among the healthy people who produced the first false positive, 00:09:28.288 --> 00:09:32.027 only one percent produce another false positive independently. 00:09:32.340 --> 00:09:36.811 One percent of ninety-nine point nine is zero point nine nine nine. 00:09:37.607 --> 00:09:39.976 Now read the two surviving piles. 00:09:40.173 --> 00:09:43.399 About nine point eight people are true positives twice, 00:09:43.482 --> 00:09:46.638 while about one person is falsely positive twice. 00:09:46.721 --> 00:09:50.100 The green pile is finally much larger than the yellow pile. 00:09:50.897 --> 00:09:54.623 The probability of disease after two positive results is the green 00:09:54.623 --> 00:09:57.608 count divided by the two surviving counts together. 00:09:58.520 --> 00:10:01.099 That is about ninety point eight percent. 00:10:01.411 --> 00:10:05.148 One independent repeat test has moved the answer from about nine percent 00:10:05.231 --> 00:10:08.877 to about ninety-one percent by filtering both piles again. 00:10:10.092 --> 00:10:13.100 There is a compact way to understand that jump. 00:10:13.297 --> 00:10:17.546 Start with disease odds of ten to nine thousand nine hundred ninety, which reduce 00:10:17.546 --> 00:10:20.124 to one to nine hundred ninety-nine. 00:10:20.921 --> 00:10:25.564 A positive result is ninety-nine times more likely when disease is present than 00:10:25.564 --> 00:10:27.016 when it is absent. 00:10:27.120 --> 00:10:32.078 That factor, ninety-nine, is called the positive likelihood ratio. 00:10:32.782 --> 00:10:36.788 The first positive result multiplies the prior odds by ninety-nine. 00:10:36.985 --> 00:10:40.677 That produces the same roughly nine percent probability we found 00:10:40.677 --> 00:10:42.477 from the first two piles. 00:10:43.274 --> 00:10:44.794 Under conditional independence, 00:10:44.794 --> 00:10:48.232 the second positive multiplies by the same factor again. 00:10:48.428 --> 00:10:53.281 Two positives contribute ninety-nine times ninety-nine, changing the odds by a factor 00:10:53.281 --> 00:10:55.638 of nine thousand eight hundred one. 00:10:56.690 --> 00:10:58.293 Convert those final odds back 00:10:58.293 --> 00:11:02.009 to a probability and we recover ninety point eight percent. 00:11:02.205 --> 00:11:06.328 The count method and the odds method are two views of the same update. 00:11:07.031 --> 00:11:09.447 The independence assumption matters. 00:11:09.551 --> 00:11:12.974 If both tests use the same sample, the same instrument, 00:11:13.057 --> 00:11:17.167 or the same biological signal, their errors may be correlated. 00:11:17.364 --> 00:11:21.298 A repeated error can then be more likely than this calculation assumes, 00:11:21.381 --> 00:11:24.342 so the second positive may add less evidence. 00:11:25.743 --> 00:11:28.727 The lesson is not to distrust accurate tests. 00:11:28.923 --> 00:11:31.292 It is to ask the complete question. 00:11:31.605 --> 00:11:33.475 How rare is the disease? 00:11:33.672 --> 00:11:36.668 What are the sensitivity and specificity? 00:11:36.864 --> 00:11:39.907 And is new evidence genuinely independent? 00:11:40.220 --> 00:11:45.143 With those facts, a surprising positive result becomes a count we can understand.