The Bootstrap: Measuring How Much Your Estimate Could Have Wobbled

Suppose this sample is all we have. The study is finished, the subjects are gone, or another round of measurement is simply impossible. We can compute an estimate from these observations, but one number does not tell us how much that estimate would have wobbled under another sample. Here is the observed sample, drawn without any invented scientific story. The labels x one through x six stand for whatever values the research actually produced. Six is only a small display size. In practice the sample contains all n observed values. The bootstrap asks us to act as though this observed collection were a tiny stand-in population. For the first bootstrap resample, draw six indices from one through six. An index is returned after every draw, so it may be selected again. That phrase, with replacement, does real work. In this resample x two appears three times. Some observed values appear once, and others do not appear at all. The resample still has the same size as the original sample. Now run the statistic on the resample. T can be a mean, a median, a regression coefficient, a ratio, or another estimator. The star marks the result as a bootstrap version of the estimate, not as a new observation from the world. Then discard that temporary resample and draw another. This time other indices repeat. Recompute exactly the same statistic, store the result, and continue. The procedure varies the sample while keeping the analysis rule fixed. Here is the complete computational recipe. First, draw n indices with replacement from the n available indices. Drawing indices rather than values is usually the cleanest implementation because every row of the data travels with its corresponding observation. Second, use those indices to assemble one resample. Third, send that resample through the same statistic-producing code used on the original data. No special algebra is needed inside the estimator. Fourth, repeat the operation many times and retain the resulting estimates. Their collection is the bootstrap distribution. Its centre, spread, asymmetry, and tails describe how the estimator behaves when the empirical stand-in population is sampled repeatedly. Written mathematically, the stand-in population is the empirical distribution F hat. It places equal probability, one over n, on every observed value. Sampling from that discrete distribution is exactly the same operation as drawing rows with replacement. Replacement is essential. Sampling n rows without replacement would return every original row exactly once, merely shuffled, so most statistics would never change. Replacement creates the repetitions and omissions that make the estimates wobble.

Start with the sample mean because it gives us a calibration case. We already know useful theory for its sampling variation, so the bootstrap has an independent answer to match rather than asking us to trust the resampling procedure on first contact. For bootstrap resample b, compute its mean in exactly the same way. One resample produces one starred mean. Another resample produces another. Each result becomes one brick at its position on this horizontal axis. Here come a few more. They do not all land at the original mean because the resamples repeat and omit different observations. Some fall to the left, some to the right, and some land close to the centre. Now let dozens of resamples run. Every repetition draws n rows, computes one mean, and adds one brick. The empty frame becomes a distribution built from estimates rather than from raw observations. Continue farther. The exact outline still jitters because a finite number of bootstrap runs has Monte Carlo noise, but its broad shape settles: many means near the middle and fewer out in either direction. The dashed line marks the mean of the observed sample. The bootstrap distribution is usually centred near that estimate. Its horizontal spread is the important quantity: it estimates how much the sample mean would vary across repeated samples from the population. Measure that spread by taking the standard deviation of the bootstrap means. This gives the bootstrap standard error. It is a standard deviation across replicated estimates, not the standard deviation of the original observations. For the mean, the familiar analytic check is s divided by square root n, assuming independent observations and finite variance. The bootstrap spread should be close to that formula when the sample is reasonably informative. The agreement is not mysterious. The empirical distribution has nearly the same measured spread as the sample, and every bootstrap mean averages n draws from that empirical distribution. The same square-root-n stabilization appears. There is a small finite-sample detail. If s uses its usual n minus one denominator, the basic empirical bootstrap spread for the mean is smaller by a factor square root of n minus one over n. That factor approaches one and is rarely the main uncertainty. Increasing the number B of resamples makes the histogram smoother and its measured spread more reproducible. It does not enlarge the original sample, correct selection bias, or manufacture information the sample never contained. So the mean has done its job. The resampling distribution behaves like a sampling distribution, and a known formula checks its scale. Now we can keep the procedure and change only the statistic.

Now replace the mean by the sample median. The median is the middle observation after sorting, so it is often a sensible summary when the data are skewed or when a few large values would pull a mean around. Nothing in the resampling machinery changes. Draw n rows with replacement, form the same kind of bootstrap resample R b, and now ask the existing analysis code for its median rather than its mean. The next resample produces a second bootstrap median. Then a third. Duplicated observations can become middle observations, and omitted observations can move the middle elsewhere. Let the repetition continue. Because an empirical distribution contains only the observed values, bootstrap medians often repeat. The histogram can look stepped or lumpy, especially when n is small. With many more resamples the relative heights stabilize. The lumps are not automatically an error. They report the discrete set of middle values available under resampling from this observed sample. As before, take the standard deviation of all stored bootstrap medians to estimate the median's standard error. Quantiles of the same collection can also describe an interval, though interval construction brings extra choices that should be reported. There is analytic theory for a median, but it is already less friendly. Under smooth conditions and for a large sample, its standard error depends on one over twice the population density at the population median, times square root n. That density at the unknown median is itself unknown. Estimating it requires smoothing choices, and the approximation depends on regularity conditions. For more complicated estimators, the analytic derivation can become much less pleasant still. The bootstrap avoids deriving a fresh sampling formula for every statistic. We supply a resampling rule and a function T. The same loop that handled the mean now handles the median, while the estimator code remains the definition of what we want. That does not make the bootstrap exact. A tiny sample may provide a very coarse empirical distribution, so the bootstrap median can have only a few possible values. A strange-looking histogram is evidence to inspect, not something to smooth away without thought. This is the method's real advantage. The mean showed that resampling can reproduce a known scale. The median shows why we bother: we can estimate uncertainty without solving the estimator's sampling distribution by hand.

We should now be precise about what this constructed distribution means. Imagine the experiment we truly want but cannot perform: repeatedly draw fresh samples of size n from the population F and recompute the estimate each time. The collection of those real repeated estimates is the sampling distribution. Its spread is the actual repeated-sampling uncertainty of the estimator under population F. But F is unknown, and we possess only one realized sample. The bootstrap replaces unknown F by the empirical distribution F hat. That distribution places mass one over n on each observed value, and it is completely determined once the observed sample is fixed. We repeatedly draw starred samples from F hat and compute starred estimates. Their conditional distribution is the bootstrap distribution. Conditional means that the original observed sample is now held fixed throughout the resampling experiment. The central approximation is now visible. The distribution of starred estimates under the empirical distribution is used to approximate the distribution of ordinary estimates under the unknown population. A bootstrap histogram therefore does not show new data that might have been observed. It shows the variability created by sampling from the empirical stand-in population. Its quality depends on how well that stand-in preserves the features relevant to the chosen statistic. Nor is this automatically a probability distribution for the unknown parameter after seeing the data. That would require a Bayesian model and a prior. The ordinary bootstrap is a repeated-sampling approximation conditional on the observations. The approximation carries assumptions. First, the original sample must be representative of the population to which the uncertainty statement will be applied. Resampling cannot repair a biased sampling frame, selective nonresponse, or a measurement process that misses part of the target. Second, the ordinary bootstrap treats rows as independent and identically distributed sampling units. If observations arrive in clusters, repeated measures, spatial neighborhoods, or a time series, resampling individual rows destroys the dependence that contributes to uncertainty. The remedy is not to ignore dependence but to resample at a scientifically defensible unit. Depending on the design, that might mean whole clusters, subjects, or blocks of consecutive observations. The resampling scheme must imitate the original sampling mechanism. Third, the empirical distribution must contain the features that govern the statistic. For means and many smooth estimators, the observed spread often supplies what is needed. For tail behavior or rare events, the missing part of the population may be exactly what matters. Fourth, the estimator should respond regularly to modest changes in the distribution. Statistics at boundaries, statistics defined by rare events, and some model-selection procedures can change abruptly. In those cases the ordinary bootstrap may approximate the wrong limiting behavior. These assumptions are not ceremonial fine print. They tell us what experiment the resampling is imitating, which object should be resampled, and whether the observed empirical distribution contains enough information for the target statistic.

The classic failure is estimating a maximum. This question sounds like a natural extension of the mean and median examples, but the maximum depends on information just beyond the largest observation, precisely where the empirical distribution knows nothing. Order the observed values and mark the largest one, x sub n, in red. Every ordinary bootstrap resample draws only from these standing points. It may duplicate the red maximum, include it once, or omit it. If a resample contains x sub n, its maximum is x sub n. If it omits that observation, its maximum is smaller. No resample can produce a value larger than the largest value already present. Yet a genuinely new sample from the population may contain an unseen extreme beyond x sub n. That possibility contributes to the real sampling distribution of the maximum, but it is absent from the empirical resampling distribution. The problem is visible even before asymptotic theory. Suppose the observed maximum is unique. One bootstrap resample includes it unless all n draws select from the other n minus one observations. The chance of including it at least once is one minus one minus one over n to the nth power. For large n this approaches one minus e to the minus one, about zero point six three. So roughly sixty-three percent of bootstrap maxima pile up exactly at the observed maximum. The rest lie below it. The distribution has a hard ceiling created by the dataset, although the population itself need not have that ceiling. Running a million resamples only estimates this defective conditional distribution with great numerical precision. It cannot place probability beyond x sub n because the empirical distribution assigned none there. For endpoint and extreme-value questions, useful alternatives require additional structure: a justified tail model, a parametric resampling model, subsampling under appropriate theory, or methods developed specifically for extremes. The ordinary row bootstrap is not a universal default. A practical analysis starts by identifying the scientific sampling unit. Resample subjects if subjects were sampled, clusters if clusters were sampled, or valid dependence-preserving blocks when time or space links nearby observations. Then inspect the bootstrap distribution rather than reporting only one standard error. Strong discreteness, severe skew, a pile against a boundary, or large changes across reasonable analysis choices can reveal that the approximation is fragile. Very small samples deserve caution because their empirical distributions are coarse. Extreme quantiles, maxima, minima, and rare-event statistics deserve caution because the unobserved tail may control the answer. Dependence, censoring, selection, missingness, and complicated fitted procedures may require a design-aware or specialized bootstrap. The resampling scheme should reproduce the source of variability that the uncertainty statement is supposed to include. Finally, separate two sample sizes. The original n controls how much information came from the world. The bootstrap count B controls how accurately we simulate the conditional resampling distribution. Making B enormous cannot compensate for an uninformative n. The bootstrap is a disciplined thought experiment. It asks how the estimator varies when sampling is repeated from the empirical population. For the mean, a formula checked the spread. For the median, resampling avoided unpleasant analytic work. For the maximum, the missing tail exposed the boundary of the trick. So the practical habit is simple: resample the right units, recompute the entire statistic, watch the distribution form, and then ask whether the observed sample could genuinely stand in for the population feature that controls your uncertainty. The calculation is easy. That final judgment is the statistical work.