WEBVTT 00:00:00.058 --> 00:00:02.076 Suppose this sample is all we have. 00:00:02.159 --> 00:00:03.146 The study is finished, 00:00:03.146 --> 00:00:07.478 the subjects are gone, or another round of measurement is simply impossible. 00:00:07.930 --> 00:00:11.424 We can compute an estimate from these observations, but one number does not 00:00:11.424 --> 00:00:15.361 tell us how much that estimate would have wobbled under another sample. 00:00:16.158 --> 00:00:20.965 Here is the observed sample, drawn without any invented scientific story. 00:00:21.069 --> 00:00:22.508 The labels x one 00:00:22.508 --> 00:00:27.385 through x six stand for whatever values the research actually produced. 00:00:27.698 --> 00:00:30.334 Six is only a small display size. 00:00:30.531 --> 00:00:34.398 In practice the sample contains all n observed values. 00:00:35.194 --> 00:00:36.796 The bootstrap asks us to act 00:00:36.796 --> 00:00:40.733 as though this observed collection were a tiny stand-in population. 00:00:40.929 --> 00:00:45.667 For the first bootstrap resample, draw six indices from one through six. 00:00:45.864 --> 00:00:50.300 An index is returned after every draw, so it may be selected again. 00:00:51.096 --> 00:00:54.232 That phrase, with replacement, does real work. 00:00:54.428 --> 00:00:57.692 In this resample x two appears three times. 00:00:57.888 --> 00:01:02.068 Some observed values appear once, and others do not appear at all. 00:01:02.265 --> 00:01:06.108 The resample still has the same size as the original sample. 00:01:06.812 --> 00:01:09.202 Now run the statistic on the resample. 00:01:09.285 --> 00:01:13.347 T can be a mean, a median, a regression coefficient, 00:01:13.430 --> 00:01:16.205 a ratio, or another estimator. 00:01:16.402 --> 00:01:19.860 The star marks the result as a bootstrap version of the estimate, 00:01:19.943 --> 00:01:22.603 not as a new observation from the world. 00:01:23.515 --> 00:01:26.988 Then discard that temporary resample and draw another. 00:01:27.091 --> 00:01:29.402 This time other indices repeat. 00:01:29.599 --> 00:01:34.441 Recompute exactly the same statistic, store the result, and continue. 00:01:34.638 --> 00:01:38.923 The procedure varies the sample while keeping the analysis rule fixed. 00:01:40.319 --> 00:01:43.106 Here is the complete computational recipe. 00:01:43.419 --> 00:01:48.180 First, draw n indices with replacement from the n available indices. 00:01:48.493 --> 00:01:52.545 Drawing indices rather than values is usually the cleanest implementation 00:01:52.545 --> 00:01:57.143 because every row of the data travels with its corresponding observation. 00:01:57.940 --> 00:02:01.749 Second, use those indices to assemble one resample. 00:02:01.945 --> 00:02:06.416 Third, send that resample through the same statistic-producing code used 00:02:06.416 --> 00:02:07.960 on the original data. 00:02:08.157 --> 00:02:11.525 No special algebra is needed inside the estimator. 00:02:12.321 --> 00:02:17.024 Fourth, repeat the operation many times and retain the resulting estimates. 00:02:17.221 --> 00:02:20.043 Their collection is the bootstrap distribution. 00:02:20.240 --> 00:02:25.660 Its centre, spread, asymmetry, and tails describe how the estimator behaves 00:02:25.743 --> 00:02:29.389 when the empirical stand-in population is sampled repeatedly. 00:02:30.186 --> 00:02:31.508 Written mathematically, 00:02:31.591 --> 00:02:35.481 the stand-in population is the empirical distribution F hat. 00:02:35.677 --> 00:02:40.810 It places equal probability, one over n, on every observed value. 00:02:41.007 --> 00:02:44.836 Sampling from that discrete distribution is exactly the same operation 00:02:44.919 --> 00:02:47.161 as drawing rows with replacement. 00:02:47.957 --> 00:02:49.816 Replacement is essential. 00:02:50.012 --> 00:02:55.096 Sampling n rows without replacement would return every original row exactly once, 00:02:55.179 --> 00:02:58.709 merely shuffled, so most statistics would never change. 00:02:59.323 --> 00:03:03.748 Replacement creates the repetitions and omissions that make the estimates wobble. 00:03:04.825 --> 00:03:08.622 Start with the sample mean because it gives us a calibration case. 00:03:08.819 --> 00:03:12.021 We already know useful theory for its sampling variation, 00:03:12.105 --> 00:03:14.090 so the bootstrap has an independent answer 00:03:14.090 --> 00:03:19.246 to match rather than asking us to trust the resampling procedure on first contact. 00:03:20.159 --> 00:03:24.977 For bootstrap resample b, compute its mean in exactly the same way. 00:03:25.174 --> 00:03:27.691 One resample produces one starred mean. 00:03:27.775 --> 00:03:30.051 Another resample produces another. 00:03:30.248 --> 00:03:34.776 Each result becomes one brick at its position on this horizontal axis. 00:03:35.573 --> 00:03:37.234 Here come a few more. 00:03:37.430 --> 00:03:38.464 They do not all land 00:03:38.464 --> 00:03:43.561 at the original mean because the resamples repeat and omit different observations. 00:03:43.758 --> 00:03:48.600 Some fall to the left, some to the right, and some land close to the centre. 00:03:49.513 --> 00:03:52.114 Now let dozens of resamples run. 00:03:52.311 --> 00:03:57.571 Every repetition draws n rows, computes one mean, and adds one brick. 00:03:57.883 --> 00:03:59.091 The empty frame becomes 00:03:59.091 --> 00:04:03.654 a distribution built from estimates rather than from raw observations. 00:04:04.567 --> 00:04:06.112 Continue farther. 00:04:06.425 --> 00:04:11.243 The exact outline still jitters because a finite number of bootstrap runs has Monte 00:04:11.243 --> 00:04:14.375 Carlo noise, but its broad shape settles: 00:04:14.459 --> 00:04:17.943 many means near the middle and fewer out in either direction. 00:04:18.739 --> 00:04:22.130 The dashed line marks the mean of the observed sample. 00:04:22.234 --> 00:04:25.961 The bootstrap distribution is usually centred near that estimate. 00:04:26.158 --> 00:04:29.073 Its horizontal spread is the important quantity: 00:04:29.177 --> 00:04:33.089 it estimates how much the sample mean would vary across repeated samples 00:04:33.089 --> 00:04:34.611 from the population. 00:04:35.663 --> 00:04:39.889 Measure that spread by taking the standard deviation of the bootstrap means. 00:04:40.504 --> 00:04:43.129 This gives the bootstrap standard error. 00:04:43.325 --> 00:04:46.573 It is a standard deviation across replicated estimates, 00:04:46.657 --> 00:04:50.141 not the standard deviation of the original observations. 00:04:51.054 --> 00:04:55.997 For the mean, the familiar analytic check is s divided by square root n, 00:04:56.081 --> 00:04:59.588 assuming independent observations and finite variance. 00:04:59.901 --> 00:05:02.441 The bootstrap spread should be close to that formula 00:05:02.525 --> 00:05:05.173 when the sample is reasonably informative. 00:05:05.876 --> 00:05:08.223 The agreement is not mysterious. 00:05:08.326 --> 00:05:12.341 The empirical distribution has nearly the same measured spread as the sample, 00:05:12.425 --> 00:05:16.988 and every bootstrap mean averages n draws from that empirical distribution. 00:05:17.185 --> 00:05:20.309 The same square-root-n stabilization appears. 00:05:21.105 --> 00:05:23.823 There is a small finite-sample detail. 00:05:24.019 --> 00:05:29.139 If s uses its usual n minus one denominator, the basic empirical bootstrap 00:05:29.139 --> 00:05:34.875 spread for the mean is smaller by a factor square root of n minus one over n. 00:05:34.979 --> 00:05:38.927 That factor approaches one and is rarely the main uncertainty. 00:05:39.724 --> 00:05:40.894 Increasing the number B 00:05:40.978 --> 00:05:46.412 of resamples makes the histogram smoother and its measured spread more reproducible. 00:05:46.516 --> 00:05:48.765 It does not enlarge the original sample, 00:05:48.849 --> 00:05:53.993 correct selection bias, or manufacture information the sample never contained. 00:05:54.789 --> 00:05:57.228 So the mean has done its job. 00:05:57.425 --> 00:06:00.835 The resampling distribution behaves like a sampling distribution, 00:06:00.919 --> 00:06:03.405 and a known formula checks its scale. 00:06:03.601 --> 00:06:06.981 Now we can keep the procedure and change only the statistic. 00:06:08.103 --> 00:06:10.631 Now replace the mean by the sample median. 00:06:10.715 --> 00:06:15.835 The median is the middle observation after sorting, so it is often a sensible summary 00:06:15.835 --> 00:06:20.666 when the data are skewed or when a few large values would pull a mean around. 00:06:21.462 --> 00:06:24.424 Nothing in the resampling machinery changes. 00:06:24.621 --> 00:06:30.377 Draw n rows with replacement, form the same kind of bootstrap resample R b, 00:06:30.461 --> 00:06:35.233 and now ask the existing analysis code for its median rather than its mean. 00:06:36.030 --> 00:06:39.455 The next resample produces a second bootstrap median. 00:06:39.652 --> 00:06:40.918 Then a third. 00:06:41.115 --> 00:06:44.339 Duplicated observations can become middle observations, 00:06:44.423 --> 00:06:47.803 and omitted observations can move the middle elsewhere. 00:06:48.599 --> 00:06:50.667 Let the repetition continue. 00:06:51.119 --> 00:06:55.109 Because an empirical distribution contains only the observed values, 00:06:55.193 --> 00:06:57.644 bootstrap medians often repeat. 00:06:57.841 --> 00:07:02.903 The histogram can look stepped or lumpy, especially when n is small. 00:07:03.607 --> 00:07:07.160 With many more resamples the relative heights stabilize. 00:07:07.357 --> 00:07:11.293 The lumps are not automatically an error. They report the discrete set 00:07:11.293 --> 00:07:15.879 of middle values available under resampling from this observed sample. 00:07:16.931 --> 00:07:19.520 As before, take the standard deviation 00:07:19.520 --> 00:07:23.988 of all stored bootstrap medians to estimate the median's standard error. 00:07:24.072 --> 00:07:27.958 Quantiles of the same collection can also describe an interval, 00:07:28.042 --> 00:07:32.188 though interval construction brings extra choices that should be reported. 00:07:32.984 --> 00:07:37.699 There is analytic theory for a median, but it is already less friendly. 00:07:37.895 --> 00:07:42.481 Under smooth conditions and for a large sample, its standard error depends on one 00:07:42.481 --> 00:07:48.473 over twice the population density at the population median, times square root n. 00:07:49.385 --> 00:07:52.904 That density at the unknown median is itself unknown. 00:07:53.101 --> 00:07:55.431 Estimating it requires smoothing choices, 00:07:55.515 --> 00:07:59.104 and the approximation depends on regularity conditions. 00:07:59.301 --> 00:08:01.179 For more complicated estimators, 00:08:01.263 --> 00:08:04.955 the analytic derivation can become much less pleasant still. 00:08:05.868 --> 00:08:10.583 The bootstrap avoids deriving a fresh sampling formula for every statistic. 00:08:10.779 --> 00:08:14.670 We supply a resampling rule and a function T. 00:08:15.284 --> 00:08:18.881 The same loop that handled the mean now handles the median, 00:08:18.965 --> 00:08:22.831 while the estimator code remains the definition of what we want. 00:08:23.628 --> 00:08:26.392 That does not make the bootstrap exact. 00:08:26.496 --> 00:08:31.009 A tiny sample may provide a very coarse empirical distribution, 00:08:31.093 --> 00:08:35.169 so the bootstrap median can have only a few possible values. 00:08:35.621 --> 00:08:40.218 A strange-looking histogram is evidence to inspect, not something to smooth away 00:08:40.218 --> 00:08:41.496 without thought. 00:08:42.293 --> 00:08:44.651 This is the method's real advantage. 00:08:44.754 --> 00:08:49.086 The mean showed that resampling can reproduce a known scale. 00:08:49.282 --> 00:08:51.721 The median shows why we bother: 00:08:51.918 --> 00:08:53.288 we can estimate uncertainty 00:08:53.288 --> 00:08:57.027 without solving the estimator's sampling distribution by hand. 00:08:58.175 --> 00:09:02.181 We should now be precise about what this constructed distribution means. 00:09:02.377 --> 00:09:06.233 Imagine the experiment we truly want but cannot perform: 00:09:06.429 --> 00:09:10.236 repeatedly draw fresh samples of size n from the population F 00:09:10.319 --> 00:09:12.908 and recompute the estimate each time. 00:09:13.705 --> 00:09:18.118 The collection of those real repeated estimates is the sampling distribution. 00:09:18.314 --> 00:09:21.621 Its spread is the actual repeated-sampling uncertainty 00:09:21.704 --> 00:09:24.422 of the estimator under population F. 00:09:24.618 --> 00:09:29.217 But F is unknown, and we possess only one realized sample. 00:09:30.013 --> 00:09:34.739 The bootstrap replaces unknown F by the empirical distribution F hat. 00:09:34.936 --> 00:09:39.323 That distribution places mass one over n on each observed value, 00:09:39.406 --> 00:09:43.273 and it is completely determined once the observed sample is fixed. 00:09:44.069 --> 00:09:49.236 We repeatedly draw starred samples from F hat and compute starred estimates. 00:09:49.433 --> 00:09:53.091 Their conditional distribution is the bootstrap distribution. 00:09:53.287 --> 00:09:55.064 Conditional means that the original 00:09:55.064 --> 00:09:59.488 observed sample is now held fixed throughout the resampling experiment. 00:10:00.029 --> 00:10:02.724 The central approximation is now visible. 00:10:02.920 --> 00:10:07.193 The distribution of starred estimates under the empirical distribution is used 00:10:07.193 --> 00:10:08.702 to approximate the distribution 00:10:08.702 --> 00:10:12.302 of ordinary estimates under the unknown population. 00:10:13.098 --> 00:10:16.232 A bootstrap histogram therefore does not show new data 00:10:16.315 --> 00:10:18.266 that might have been observed. 00:10:18.462 --> 00:10:21.736 It shows the variability created by sampling from the empirical 00:10:21.736 --> 00:10:23.490 stand-in population. 00:10:23.687 --> 00:10:24.941 Its quality depends 00:10:24.941 --> 00:10:29.875 on how well that stand-in preserves the features relevant to the chosen statistic. 00:10:30.672 --> 00:10:34.724 Nor is this automatically a probability distribution for the unknown parameter 00:10:34.724 --> 00:10:36.431 after seeing the data. 00:10:36.535 --> 00:10:39.659 That would require a Bayesian model and a prior. 00:10:39.855 --> 00:10:43.420 The ordinary bootstrap is a repeated-sampling approximation 00:10:43.420 --> 00:10:45.766 conditional on the observations. 00:10:46.818 --> 00:10:49.373 The approximation carries assumptions. 00:10:49.570 --> 00:10:53.111 First, the original sample must be representative of the population 00:10:53.111 --> 00:10:56.014 to which the uncertainty statement will be applied. 00:10:56.211 --> 00:10:59.424 Resampling cannot repair a biased sampling frame, 00:10:59.507 --> 00:11:04.384 selective nonresponse, or a measurement process that misses part of the target. 00:11:05.181 --> 00:11:06.748 Second, the ordinary 00:11:06.748 --> 00:11:11.614 bootstrap treats rows as independent and identically distributed sampling units. 00:11:11.717 --> 00:11:16.244 If observations arrive in clusters, repeated measures, spatial neighborhoods, 00:11:16.327 --> 00:11:20.924 or a time series, resampling individual rows destroys the dependence 00:11:20.924 --> 00:11:22.992 that contributes to uncertainty. 00:11:23.788 --> 00:11:24.752 The remedy is not 00:11:24.752 --> 00:11:29.397 to ignore dependence but to resample at a scientifically defensible unit. 00:11:29.710 --> 00:11:30.917 Depending on the design, 00:11:30.917 --> 00:11:36.270 that might mean whole clusters, subjects, or blocks of consecutive observations. 00:11:36.467 --> 00:11:40.752 The resampling scheme must imitate the original sampling mechanism. 00:11:41.455 --> 00:11:43.743 Third, the empirical distribution 00:11:43.743 --> 00:11:46.681 must contain the features that govern the statistic. 00:11:46.878 --> 00:11:49.349 For means and many smooth estimators, 00:11:49.432 --> 00:11:52.753 the observed spread often supplies what is needed. 00:11:52.950 --> 00:11:55.537 For tail behavior or rare events, 00:11:55.620 --> 00:11:59.267 the missing part of the population may be exactly what matters. 00:12:00.063 --> 00:12:02.850 Fourth, the estimator should respond regularly 00:12:02.850 --> 00:12:05.370 to modest changes in the distribution. 00:12:05.566 --> 00:12:09.537 Statistics at boundaries, statistics defined by rare events, 00:12:09.537 --> 00:12:12.984 and some model-selection procedures can change abruptly. 00:12:13.067 --> 00:12:17.781 In those cases the ordinary bootstrap may approximate the wrong limiting behavior. 00:12:18.833 --> 00:12:22.036 These assumptions are not ceremonial fine print. 00:12:22.119 --> 00:12:25.495 They tell us what experiment the resampling is imitating, 00:12:25.578 --> 00:12:29.282 which object should be resampled, and whether the observed empirical 00:12:29.282 --> 00:12:32.986 distribution contains enough information for the target statistic. 00:12:34.145 --> 00:12:36.953 The classic failure is estimating a maximum. 00:12:37.036 --> 00:12:41.086 This question sounds like a natural extension of the mean and median examples, 00:12:41.169 --> 00:12:45.498 but the maximum depends on information just beyond the largest observation, 00:12:45.581 --> 00:12:48.949 precisely where the empirical distribution knows nothing. 00:12:49.745 --> 00:12:54.959 Order the observed values and mark the largest one, x sub n, in red. 00:12:55.411 --> 00:13:00.195 Every ordinary bootstrap resample draws only from these standing points. 00:13:00.392 --> 00:13:04.700 It may duplicate the red maximum, include it once, or omit it. 00:13:05.497 --> 00:13:10.466 If a resample contains x sub n, its maximum is x sub n. 00:13:10.663 --> 00:13:14.251 If it omits that observation, its maximum is smaller. 00:13:14.448 --> 00:13:19.151 No resample can produce a value larger than the largest value already present. 00:13:19.947 --> 00:13:21.538 Yet a genuinely new sample 00:13:21.538 --> 00:13:25.762 from the population may contain an unseen extreme beyond x sub n. 00:13:25.845 --> 00:13:30.232 That possibility contributes to the real sampling distribution of the maximum, 00:13:30.315 --> 00:13:34.194 but it is absent from the empirical resampling distribution. 00:13:34.897 --> 00:13:38.567 The problem is visible even before asymptotic theory. 00:13:38.764 --> 00:13:41.412 Suppose the observed maximum is unique. 00:13:41.608 --> 00:13:47.297 One bootstrap resample includes it unless all n draws select from the other n minus 00:13:47.297 --> 00:13:48.958 one observations. 00:13:49.755 --> 00:13:54.757 The chance of including it at least once is one minus one minus one over n 00:13:54.840 --> 00:13:56.350 to the nth power. 00:13:56.454 --> 00:14:02.434 For large n this approaches one minus e to the minus one, about zero point six three. 00:14:03.138 --> 00:14:04.775 So roughly sixty-three percent 00:14:04.775 --> 00:14:09.210 of bootstrap maxima pile up exactly at the observed maximum. 00:14:09.407 --> 00:14:11.184 The rest lie below it. 00:14:11.381 --> 00:14:14.665 The distribution has a hard ceiling created by the dataset, 00:14:14.748 --> 00:14:18.104 although the population itself need not have that ceiling. 00:14:19.016 --> 00:14:23.531 Running a million resamples only estimates this defective conditional distribution 00:14:23.614 --> 00:14:25.739 with great numerical precision. 00:14:25.843 --> 00:14:27.768 It cannot place probability beyond 00:14:27.851 --> 00:14:32.171 x sub n because the empirical distribution assigned none there. 00:14:33.084 --> 00:14:37.832 For endpoint and extreme-value questions, useful alternatives require additional 00:14:37.832 --> 00:14:43.510 structure: a justified tail model, a parametric resampling model, subsampling 00:14:43.510 --> 00:14:48.247 under appropriate theory, or methods developed specifically for extremes. 00:14:48.444 --> 00:14:51.916 The ordinary row bootstrap is not a universal default. 00:14:52.620 --> 00:14:56.904 A practical analysis starts by identifying the scientific sampling unit. 00:14:57.101 --> 00:14:59.560 Resample subjects if subjects were sampled, 00:14:59.643 --> 00:15:01.743 clusters if clusters were sampled, 00:15:01.826 --> 00:15:07.040 or valid dependence-preserving blocks when time or space links nearby observations. 00:15:07.836 --> 00:15:10.344 Then inspect the bootstrap distribution rather than 00:15:10.344 --> 00:15:12.783 reporting only one standard error. 00:15:13.096 --> 00:15:18.761 Strong discreteness, severe skew, a pile against a boundary, or large changes 00:15:18.761 --> 00:15:23.917 across reasonable analysis choices can reveal that the approximation is fragile. 00:15:24.714 --> 00:15:29.591 Very small samples deserve caution because their empirical distributions are coarse. 00:15:29.787 --> 00:15:35.499 Extreme quantiles, maxima, minima, and rare-event statistics deserve caution 00:15:35.499 --> 00:15:38.971 because the unobserved tail may control the answer. 00:15:39.768 --> 00:15:44.482 Dependence, censoring, selection, missingness, and complicated fitted 00:15:44.482 --> 00:15:48.778 procedures may require a design-aware or specialized bootstrap. 00:15:49.091 --> 00:15:53.712 The resampling scheme should reproduce the source of variability that the uncertainty 00:15:53.712 --> 00:15:55.721 statement is supposed to include. 00:15:56.773 --> 00:15:59.572 Finally, separate two sample sizes. 00:15:59.769 --> 00:16:03.786 The original n controls how much information came from the world. 00:16:03.983 --> 00:16:05.514 The bootstrap count B 00:16:05.597 --> 00:16:09.986 controls how accurately we simulate the conditional resampling distribution. 00:16:10.299 --> 00:16:14.920 Making B enormous cannot compensate for an uninformative n. 00:16:15.716 --> 00:16:18.573 The bootstrap is a disciplined thought experiment. 00:16:18.770 --> 00:16:20.627 It asks how the estimator varies 00:16:20.627 --> 00:16:23.902 when sampling is repeated from the empirical population. 00:16:24.099 --> 00:16:27.141 For the mean, a formula checked the spread. 00:16:27.338 --> 00:16:31.890 For the median, resampling avoided unpleasant analytic work. 00:16:32.086 --> 00:16:36.441 For the maximum, the missing tail exposed the boundary of the trick. 00:16:37.237 --> 00:16:39.465 So the practical habit is simple: 00:16:39.548 --> 00:16:44.575 resample the right units, recompute the entire statistic, watch the distribution 00:16:44.575 --> 00:16:48.163 form, and then ask whether the observed sample could genuinely stand 00:16:48.163 --> 00:16:52.041 in for the population feature that controls your uncertainty. 00:16:52.238 --> 00:16:56.790 The calculation is easy. That final judgment is the statistical work.