{"version":1,"lectureId":"01M14TYNGGHB7PNCH61Q2K755C","attempt":0,"publication":{"slug":"decision-trees-and-why-forests-beat-them","title":"From One Decision Tree to a Random Forest","subject":"machine-learning","summary":"Grow a classification tree on a small labelled dataset by comparing impurity reductions, while its branches remain tied to the axis-aligned regions they create. Follow the tree to pure but fragile leaves, prune it with validation evidence, then construct a random forest from bootstrap samples and random feature subsets. The aggregate boundary, variance calculation, and practical workflow explain why averaging helps, why correlated tree errors survive, and how the underlying ideas map to familiar library controls.","metaDescription":"Grow, overfit, and prune one decision tree, then build a random forest and see why averaging decorrelated trees reduces prediction variance.","transcript":"Here is the entire training problem in miniature. Each dot is one labelled observation, blue or red, and each position contains two measured features. We want a rule that predicts the colour of a new point. A decision tree will not draw a diagonal or fit a smooth curve. At one node it chooses one feature, one threshold, and one yes-or-no question. That question cuts the current region with an axis-aligned line. But which question should it ask first? The usual classification answer is the split that reduces impurity most. Impurity is not model error. It measures how mixed the labels are inside a node. For Gini impurity, subtract the squared class proportions from one. At the root we have eight blue and eight red, so both proportions are one half. The root impurity is zero point five, its largest possible value for two classes. The fitting code now considers thresholds between observed values. Let us compare two representative candidates. A vertical cut at x one equals four leaves both children evenly mixed. Its weighted child impurity is still zero point five. Subtract that from the root impurity and the gain is zero. The cut changed the addresses of the points, but learned nothing about their labels. Now try x two less than four. Below the line are six blue and two red. Above it are two blue and six red. Each child has Gini impurity three eighths, and their weighted average is also three eighths. The reduction is zero point five minus zero point three seven five, which is zero point one two five. That is larger than the vertical candidate's zero, so this horizontal question wins. The first branch of the tree and the first pair of rectangles are the same decision written in two languages. The root asks x two less than four. The left child receives the lower rectangle, and the right child receives the upper one. Notice what has not happened. We have not classified everything correctly, and neither child is pure. We have only made the labels less mixed. Training a tree means repeating this exact competition inside each child. That is the basic fitting loop behind the library call. Enumerate legal feature thresholds, score their weighted impurity, choose the best gain, partition the observations, and repeat on the resulting nodes. Continue from the root split. The tree on the left and feature space on the right will grow together. A new internal node must always correspond to a new axis-aligned cut inside exactly one existing rectangle. Start in the lower rectangle. It contains six blue observations and two red ones. The best available gain first separates the far-right red point with x one less than six point one. The right child is pure, but the large left child is not. Within that child, a horizontal cut at x two equals two point five creates a pure blue strip below and a mixed strip above. That upper strip is mixed because of one red observation at two point two, two point eight. Another vertical threshold isolates it. The training algorithm is rewarded, because two new leaves become perfectly pure. Now do the same work above the root. Most points there are red, but two blue exceptions force the recursion to continue. The first upper split separates the far-right pair from the rest. The far-right pair still disagrees. A horizontal cut isolates the blue point at the top from the red point below. Again, training impurity falls to zero in both resulting leaves. The left upper region has its own blue exception. A split at x one equals two point eight narrows the search, and another horizontal split at x two equals five point one separates one red point. One final threshold at x one equals one point five isolates the remaining upper blue point. Every terminal region now contains only one class. Consequently every leaf has Gini impurity zero. On the training set, this looks perfect. Every observation is classified correctly. But look at the geometry required to achieve it: thin strips, short corridors, and thresholds whose only purpose is to rescue one exceptional dot. A new point can cross one of those arbitrary thresholds after an imperceptible change in a feature. Its predicted class then jumps, even though the training labels gave us almost no evidence that such a jump should exist. This is the characteristic strength and weakness of an unconstrained decision tree. It has low bias because it can represent complicated interactions. It also has high variance because a few observations can rearrange entire branches and rectangles. Pure leaves are therefore a training condition, not evidence of a useful model. The next question is whether every branch earns its complexity on data that did not choose the branch. The fully grown tree has zero training impurity, but it pays for that fit with nine leaves and several thresholds supported by a single unusual observation. Pruning asks whether those extra leaves earn their keep. Cost-complexity pruning gives the trade a precise form. R of T measures the fitted tree's error or impurity. The second term charges alpha for every terminal leaf. At alpha zero, extra leaves are free, so the pure tree wins. As alpha increases, a weak pair of leaves can cost more than the small reduction in training error that created it. Prune the weakest lower twigs first. Their narrow thresholds disappear from feature space at the same moment their branches disappear from the tree. The replacement leaf predicts the local majority class. Now prune the corresponding upper twigs. Again, the replacement is not pure on the training set. It deliberately accepts a few mistakes in exchange for a much larger and more stable prediction region. The remaining four-leaf tree still captures the broad interaction. The horizontal root separates low from high x two. Within each half, one x one threshold handles the main exception near the far-right edge. This smaller tree has higher training error by construction. The relevant question is whether it has lower error on observations that did not participate in choosing all those thresholds. A pruning path supplies a nested sequence of subtrees. Here the unpruned tree has nine leaves and no training mistakes, but its validation error is the worst entry in the table. A small penalty removes three leaves. Training error rises, validation error falls. A larger penalty leaves four terminal regions, and the validation error reaches its minimum. Push alpha farther and the tree collapses to two leaves. That model is now too simple for the data, so validation error rises again. Pruning is not a ritual of making trees small. It is model selection along a structured complexity path. Select alpha using cross-validation, a held-out set, or a nested procedure when tuning itself must be evaluated. Never choose the pruning strength by returning to the same training impurity that rewarded every twig. Libraries also offer pre-pruning controls such as maximum depth, minimum samples per leaf, minimum impurity decrease, and maximum leaf count. Those prevent growth. Cost-complexity pruning fits first and removes branches afterward. Either route trades some bias for less variance. A single pruned tree is often much easier to explain and more stable than the pure tree. The forest will take a different route: keep many unstable trees, then make their instability cancel. A forest does not begin by cloning one fitted tree. It creates many related training problems. For each tree, draw n observations from the original n with replacement. With replacement means an observation can appear more than once. In the first sample, indices one and five are repeated, while three and six are absent. The sample still contains n rows. A second bootstrap draw repeats different observations and omits different ones. A third does it again. Each tree therefore sees a perturbed empirical distribution, even though all draws came from the same dataset. Fitting deep trees to these samples is ordinary bootstrap aggregation, or bagging. It creates diversity because a marginal observation may be duplicated in one sample and unavailable in another. A random forest adds a second source of variation. At every node, it offers the split search only a random subset of the available features. Here is one tree in an eight-feature problem. The formula records a fresh feature subset at node j. At the root, features two, five, and eight are offered, and feature two supplies the best available impurity reduction. The left child gets a fresh draw, features one, four, and eight. The right child gets another draw, features three, five, and seven. Feature availability is local to a node. This restriction can force a tree to ignore the dominant predictor at a particular node. That sounds inefficient for one tree. Its purpose is to prevent every tree from making the same early decisions. Bootstrap sampling perturbs the observations. Feature sampling perturbs the available questions. Together they produce trees whose errors are less synchronized. These are three trees trained from the same original problem. Each diagram shows the axis-aligned boundaries produced by one bootstrap sample and one sequence of random feature offers. The first tree chooses a horizontal root and then several local vertical cuts. Its boundary is coherent within each rectangle, but jagged as a whole. The second tree begins vertically because its sample and offered features differ. It partitions the same feature plane into a visibly different collection of rectangles. The third tree returns to a horizontal root, but its thresholds and deeper branches are different again. None of these trees is intended to be the final boundary. For classification, each tree supplies class probabilities from its reached leaf, commonly the class proportions among that leaf's training samples. The forest averages those probabilities and then chooses a class. For regression, the same architecture averages numeric predictions. Bootstrap samples and random feature subsets still serve the same purpose: build individually flexible models whose mistakes are not identical. Return to the labelled plane. One deep tree supplies one jagged boundary. It reacts strongly to the particular observations and random choices that shaped its branches. For each query point x, tree b supplies a class probability. The forest averages those probabilities. Its displayed boundary is where the winning averaged class changes. Add trees until B equals ten. Each individual boundary is still made of rectangular steps, but their idiosyncratic notches occur at different places. Majority support removes many notches that only one tree wanted. Now average one hundred trees. The aggregate is still a staircase if we inspect it finely enough, because every contributor is a tree. At this scale, however, it follows the broad class structure rather than every isolated observation. Calling this boundary smoother does not mean that a forest fits a smooth analytic function. It means the averaged prediction varies more stably across nearby points and fewer decisions depend on one tree's narrow rectangular accident. Why should averaging unstable models work? Imagine each tree prediction as a useful signal plus a zero-mean fitting error. Averaging keeps the shared signal. Errors that point in different directions can cancel. If the tree errors were independent and each had variance sigma squared, the mean of B trees would have variance sigma squared over B. Double the number of trees and this variance contribution halves. Real tree errors are not independent. They use the same original dataset, and strong predictors can make them discover similar branches. Let rho represent their average pairwise error correlation. Here is the extreme problem. Five trees all make an error of plus one on the same case. Their average error is still plus one. Repetition did not remove a shared mistake. On the right, errors differ across trees. Positive and negative errors offset, and their average is zero in this small illustration. Diversity is useful when it concerns errors, not merely visual differences between diagrams. With equal variance and average correlation rho, the ensemble variance is sigma squared times rho plus one minus rho over B. The second part shrinks as trees are added. The correlated part does not. Take the number of trees toward infinity. The variance approaches rho sigma squared, not zero. If rho is close to one, a huge forest behaves like repeated copies of one unstable tree. This is why decorrelation is essential. Bootstrap samples alter which observations drive the branches. Random feature subsets prevent one dominant predictor from forcing the same root and early splits in every tree. There is a trade-off. Offering fewer features can weaken each individual tree by denying it useful predictors. But if that loss is modest and the correlation falls substantially, the average can generalize better. Adding trees mainly reduces Monte Carlo noise in the fitted ensemble. It does not repair severe bias, leakage, bad labels, a shifted deployment population, or a feature set that contains no useful signal. The forest's advantage is therefore specific. Deep trees provide flexible, high-variance base predictions. Resampling and feature randomness make their errors less alike. Averaging then removes the part of the variance that is not shared. Here is a practical sequence that preserves the logic we have developed. Start with the prediction problem, not the estimator. Fix the target, evaluation unit, split strategy, and metric before fitting. A random row split is wrong whenever rows from the same customer, patient, device, location, or future time can leak information across folds. Group and temporal structure belong in the validation design. Fit a small decision tree before the forest. Its splits expose coding mistakes, target proxies, implausible thresholds, and feature interactions that a large ensemble can conceal behind a good aggregate score. Then tune tree structure against validation performance. Maximum depth limits path length. Minimum samples per leaf demands evidence in each terminal region. Cost-complexity alpha removes weak fitted branches. For a forest, raise the number of estimators until the validation metric and predictions stabilize. More trees usually increase compute rather than overfitting in the familiar single-tree sense, but returns become negligible. Finally inspect more than one aggregate score. Check important subgroups, threshold-sensitive decisions, probability calibration, drift, and the cost of the errors the model actually makes. Maximum depth and maximum leaf nodes limit global tree size. They are coarse controls on how many successive rectangles the tree may create. Minimum samples per leaf directly attacks tiny regions. Minimum impurity decrease requires a split to earn enough local gain before it is allowed. Cost-complexity alpha names the post-pruning penalty we used earlier. Its numerical scale depends on the data, weights, impurity, and implementation, so tune it through the supplied pruning path rather than by folklore. Bootstrap turns observation resampling on or off. Max features controls the random candidate subset at each split. Lowering max features usually reduces correlation, but can also weaken individual trees. The number of estimators is the ensemble size B. Parallel-job settings change wall-clock cost, not the fitted statistical objective. Do not confuse faster execution with stronger regularization. Classification probabilities deserve separate validation. A forest averages leaf class proportions, which can rank cases very effectively while remaining overconfident or underconfident as probabilities. Out-of-bag predictions use, for each training row, only trees whose bootstrap samples omitted that row. They provide a convenient internal diagnostic, but they do not override grouped, temporal, or external validation requirements. Impurity-based feature importance summarizes how fitted splits used a feature. It can favour variables with many available thresholds and divide credit awkwardly among correlated predictors. It is not a causal effect. Permutation importance asks how predictive performance changes after one feature is disrupted. It is often closer to the operational question, but correlated features can substitute for one another and hide each other's importance. Class weights alter the fitting objective. They do not decide which deployment metric matters. A random seed makes the stochastic fit reproducible; it does not make sampling uncertainty disappear. The whole lecture can be compressed into three ingredients. Trees must be strong enough to capture useful structure. Their errors must be sufficiently decorrelated. Averaging then reduces the unshared variance. One tree turns impurity reduction into an interpretable hierarchy of rectangles. Pruning makes that hierarchy less fragile. A random forest keeps many flexible trees, makes their mistakes less alike, and averages what remains.","watch":{"version":1,"scenes":[{"title":"The First Split","start":0,"end":164.81004166666668,"objects":{"blue_points":"a Point [blue] drawn in plane (location=(0.8, 1.2))","blue_points_2":"a Point [blue] drawn in plane (location=(1.5, 2.0))","blue_points_3":"a Point [blue] drawn in plane (location=(2.8, 1.0))","blue_points_4":"a Point [blue] drawn in plane (location=(3.6, 2.8))","blue_points_5":"a Point [blue] drawn in plane (location=(4.5, 1.5))","blue_points_6":"a Point [blue] drawn in plane (location=(5.6, 2.7))","blue_points_7":"a Point [blue] drawn in plane (location=(2.4, 5.5))","blue_points_8":"a Point [blue] drawn in plane (location=(6.8, 6.4))","candidates":"a Table [text] that says \"Candidate Children Weighted Gini Gain $x_1 < 4$ $4/4$ and $4/4$ $0.500$ $0.000$ $x_2 < 4$ $6/2$ and $2/6$ $0.375$ $0.125$\" (rows=(('Candidate', 'Children', 'Weighted Gini', 'Gain'), ('$x_1 < 4…, header=True)","chosen":"a Math [text] that says \"$x_2 < 4 thin arrow.r thin upright(\"best gain\")$\"","heading":"a Heading that says \"Every Threshold Competes\"","high_box":"a Polygon [red] drawn in tree (vertices=((6.0, 0.8), (9.0, 0.8), (9.0, 1.7), (6.0, 1.7)), fill_opacity=0.12)","high_label":"a Math [red] that says \"$2 B, thin 6 R$\" drawn in tree","high_region":"a Polygon [red] drawn in plane (vertices=((0.0, 4.0), (8.0, 4.0), (8.0, 8.0), (0.0, 8.0)), fill_opacity=0.08)","horizontal_candidate":"a Line [yellow] drawn in plane (start=(0.0, 4.0), end=(8.0, 4.0), dashed=True)","left_edge":"a Line [gray] drawn in tree (start=(4.4, 2.9), end=(2.5, 1.7))","low_box":"a Polygon [blue] drawn in tree (vertices=((1.0, 0.8), (4.0, 0.8), (4.0, 1.7), (1.0, 1.7)), fill_opacity=0.12)","low_label":"a Math [blue] that says \"$6 B, thin 2 R$\" drawn in tree","low_region":"a Polygon [blue] drawn in plane (vertices=((0.0, 0.0), (8.0, 0.0), (8.0, 4.0), (0.0, 4.0)), fill_opacity=0.08)","plane":"an Axes (x_range=(0.0, 8.0), y_range=(0.0, 8.0), aspect=(1.0, 1.0))","question":"a Panel that says \"Given two measured features and two labelled classes, which threshold should become the root of the decision tree?\"","red_points":"a Point [red] drawn in plane (location=(0.9, 6.3))","red_points_2":"a Point [red] drawn in plane (location=(1.8, 4.8))","red_points_3":"a Point [red] drawn in plane (location=(3.3, 6.7))","red_points_4":"a Point [red] drawn in plane (location=(4.2, 5.2))","red_points_5":"a Point [red] drawn in plane (location=(5.4, 6.1))","red_points_6":"a Point [red] drawn in plane (location=(7.2, 4.8))","red_points_7":"a Point [red] drawn in plane (location=(2.2, 2.8))","red_points_8":"a Point [red] drawn in plane (location=(6.5, 1.2))","right_edge":"a Line [gray] drawn in tree (start=(5.6, 2.9), end=(7.5, 1.7))","root_box":"a Polygon [yellow] drawn in tree (vertices=((3.5, 2.9), (6.5, 2.9), (6.5, 3.8), (3.5, 3.8)), fill_opacity=0.12)","root_impurity":"a Math [text] that says \"$G_(upright(\"root\")) = 1 - 2 (frac(1,2))^2 = 0.50$\"","root_label":"a Math [yellow] that says \"$x_2 < 4?$\" drawn in tree","tree":"a Figure (x_range=(0.0, 10.0), y_range=(0.0, 4.0), aspect=(5.0, 2.0))","vertical_candidate":"a Line [gray] drawn in plane (start=(4.0, 0.0), end=(4.0, 8.0), dashed=True)"},"beats":[{"start":0,"say":"Here is the entire training problem in miniature. Each dot is one labelled observation, blue or red, and each position contains two measured features. We want a rule that predicts the colour of a new point.","live":[],"does":[[0,"question is shown on the screen, written out."],[3.727,"plane is shown on the screen, written out."],[6.072,"blue_points is shown on the screen, written out."],[6.152,"blue_points_2 is shown on the screen, written out."],[6.232,"blue_points_3 is shown on the screen, written out."],[6.312,"blue_points_4 is shown on the screen, written out."],[6.392,"blue_points_5 is shown on the screen, written out."],[6.472,"blue_points_6 is shown on the screen, written out."],[6.552,"blue_points_7 is shown on the screen, written out."],[6.632,"blue_points_8 is shown on the screen, written out."],[6.699,"red_points is shown on the screen, written out."],[6.779,"red_points_2 is shown on the screen, written out."],[6.859,"red_points_3 is shown on the screen, written out."],[6.939,"red_points_4 is shown on the screen, written out."],[7.019,"red_points_5 is shown on the screen, written out."],[7.099,"red_points_6 is shown on the screen, written out."],[7.179,"red_points_7 is shown on the screen, written out."],[7.259,"red_points_8 is shown on the screen, written out."]]},{"start":14.95,"say":"A decision tree will not draw a diagonal or fit a smooth curve. At one node it chooses one feature, one threshold, and one yes-or-no question. That question cuts the current region with an axis-aligned line.","live":["plane","question","blue_points","blue_points_2","blue_points_3","blue_points_4","blue_points_5","blue_points_6","blue_points_7","blue_points_8","red_points","red_points_2","red_points_3","red_points_4","red_points_5","red_points_6","red_points_7","red_points_8"],"does":[[22.020000000000003,"vertical_candidate is shown on the screen, written out."],[29.4275,"vertical_candidate is hidden from the screen."]]},{"start":30.027499999999996,"say":"But which question should it ask first? The usual classification answer is the split that reduces impurity most. Impurity is not model error. It measures how mixed the labels are inside a node.","live":null,"does":[[30.607999999999997,"question (the \"which threshold\" part) is emphasized."],[36.552,"question (the \"which threshold\" part) is no longer emphasized."]]},{"start":44.61749999999999,"say":"For Gini impurity, subtract the squared class proportions from one. At the root we have eight blue and eight red, so both proportions are one half. The root impurity is zero point five, its largest possible value for two classes.","live":null,"does":[[45.24399999999999,"plane moves to a new place on the board."],[45.24399999999999,"root_impurity is shown on the screen, written out."],[53.97499999999999,"root_impurity (the \"frac(1,2)\" part) is indicated — a transient flash."],[56.42499999999999,"root_impurity (the \"0.50\" part) is indicated — a transient flash."],[60.76649999999999,"plane moves to a new place on the board."],[60.76649999999999,"question is hidden from the screen — left the board."],[60.76649999999999,"root_impurity is hidden from the screen — left the board."]]},{"start":61.366499999999995,"say":"The fitting code now considers thresholds between observed values. Let us compare two representative candidates. A vertical cut at x one equals four leaves both children evenly mixed.","live":["plane","blue_points","blue_points_2","blue_points_3","blue_points_4","blue_points_5","blue_points_6","blue_points_7","blue_points_8","red_points","red_points_2","red_points_3","red_points_4","red_points_5","red_points_6","red_points_7","red_points_8"],"does":[[61.366499999999995,"heading is shown on the screen, written out."],[66.649,"candidates is shown on the screen, written out."],[69.459,"vertical_candidate is shown on the screen, written out."],[69.459,"candidates is shown on the screen, written out."],[73.348,"candidates (the \"$0.500$\" part) is indicated — a transient flash."]]},{"start":74.761,"say":"Its weighted child impurity is still zero point five. Subtract that from the root impurity and the gain is zero. The cut changed the addresses of the points, but learned nothing about their labels.","live":["plane","blue_points","blue_points_2","blue_points_3","blue_points_4","blue_points_5","blue_points_6","blue_points_7","blue_points_8","red_points","red_points_2","red_points_3","red_points_4","red_points_5","red_points_6","red_points_7","red_points_8","vertical_candidate","heading"],"does":[[77.501,"candidates (the \"$0.000$\" part) is indicated — a transient flash."],[86.93900000000001,"vertical_candidate is hidden from the screen."]]},{"start":89.664,"say":"Now try x two less than four. Below the line are six blue and two red. Above it are two blue and six red. Each child has Gini impurity three eighths, and their weighted average is also three eighths.","live":["plane","blue_points","blue_points_2","blue_points_3","blue_points_4","blue_points_5","blue_points_6","blue_points_7","blue_points_8","red_points","red_points_2","red_points_3","red_points_4","red_points_5","red_points_6","red_points_7","red_points_8","heading"],"does":[[90.76700000000001,"candidates is shown on the screen, written out."],[92.85700000000001,"low_region is shown on the screen, faded in."],[93.36800000000001,"horizontal_candidate is shown on the screen, written out."],[93.89000000000001,"candidates (the \"$6/2$ and $2/6$\" part) is indicated — a transient flash."],[96.09600000000002,"high_region is shown on the screen, faded in."]]},{"start":105.39150000000001,"say":"The reduction is zero point five minus zero point three seven five, which is zero point one two five. That is larger than the vertical candidate's zero, so this horizontal question wins.","live":["plane","blue_points","blue_points_2","blue_points_3","blue_points_4","blue_points_5","blue_points_6","blue_points_7","blue_points_8","red_points","red_points_2","red_points_3","red_points_4","red_points_5","red_points_6","red_points_7","red_points_8","heading","horizontal_candidate","low_region","high_region"],"does":[[110.50000000000001,"candidates (the \"$0.125$\" part) is indicated — a transient flash."],[117.17500000000001,"chosen is shown on the screen, written out."]]},{"start":118.64600000000002,"say":"The first branch of the tree and the first pair of rectangles are the same decision written in two languages. The root asks x two less than four. The left child receives the lower rectangle, and the right child receives the upper one.","live":["plane","blue_points","blue_points_2","blue_points_3","blue_points_4","blue_points_5","blue_points_6","blue_points_7","blue_points_8","red_points","red_points_2","red_points_3","red_points_4","red_points_5","red_points_6","red_points_7","red_points_8","chosen","heading","horizontal_candidate","low_region","high_region"],"does":[[120.06300000000002,"tree is shown on the screen, written out."],[125.64700000000002,"root_box is shown on the screen, written out."],[125.92600000000002,"root_label is shown on the screen, written out."],[129.15300000000002,"left_edge is shown on the screen, written out."],[130.512,"low_box is shown on the screen, written out."],[130.512,"low_label is shown on the screen, written out."],[132.13600000000002,"right_edge is shown on the screen, written out."],[133.17000000000002,"high_box is shown on the screen, written out."],[133.17000000000002,"high_label is shown on the screen, written out."]]},{"start":134.5995,"say":"Notice what has not happened. We have not classified everything correctly, and neither child is pure. We have only made the labels less mixed. Training a tree means repeating this exact competition inside each child.","live":["plane","blue_points","blue_points_2","blue_points_3","blue_points_4","blue_points_5","blue_points_6","blue_points_7","blue_points_8","red_points","red_points_2","red_points_3","red_points_4","red_points_5","red_points_6","red_points_7","red_points_8","chosen","tree","heading","horizontal_candidate","low_region","high_region","root_box","root_label","left_edge","right_edge","low_box","low_label","high_box","high_label"],"does":[[139.49300000000002,"low_label is indicated — a transient flash."],[140.27100000000002,"high_label is indicated — a transient flash."]]},{"start":149.73000000000002,"say":"That is the basic fitting loop behind the library call. Enumerate legal feature thresholds, score their weighted impurity, choose the best gain, partition the observations, and repeat on the resulting nodes.","live":null,"does":[[158.36800000000002,"A box is drawn around chosen."],[163.76837500000002,"candidates is hidden from the screen — left the board."],[163.76837500000002,"chosen is hidden from the screen — left the board."],[163.76837500000002,"heading is hidden from the screen — left the board."],[163.76837500000002,"plane is hidden from the screen — left the board."],[163.76837500000002,"blue_points is hidden from the screen — plane left the board."],[163.76837500000002,"blue_points_2 is hidden from the screen — plane left the board."],[163.76837500000002,"blue_points_3 is hidden from the screen — plane left the board."],[163.76837500000002,"blue_points_4 is hidden from the screen — plane left the board."],[163.76837500000002,"blue_points_5 is hidden from the screen — plane left the board."],[163.76837500000002,"blue_points_6 is hidden from the screen — plane left the board."],[163.76837500000002,"blue_points_7 is hidden from the screen — plane left the board."],[163.76837500000002,"blue_points_8 is hidden from the screen — plane left the board."],[163.76837500000002,"red_points is hidden from the screen — plane left the board."],[163.76837500000002,"red_points_2 is hidden from the screen — plane left the board."],[163.76837500000002,"red_points_3 is hidden from the screen — plane left the board."],[163.76837500000002,"red_points_4 is hidden from the screen — plane left the board."],[163.76837500000002,"red_points_5 is hidden from the screen — plane left the board."],[163.76837500000002,"red_points_6 is hidden from the screen — plane left the board."],[163.76837500000002,"red_points_7 is hidden from the screen — plane left the board."],[163.76837500000002,"red_points_8 is hidden from the screen — plane left the board."],[163.76837500000002,"horizontal_candidate is hidden from the screen — plane left the board."],[163.76837500000002,"low_region is hidden from the screen — plane left the board."],[163.76837500000002,"high_region is hidden from the screen — plane left the board."],[163.76837500000002,"tree is hidden from the screen — left the board."],[163.76837500000002,"root_box is hidden from the screen — tree left the board."],[163.76837500000002,"root_label is hidden from the screen — tree left the board."],[163.76837500000002,"left_edge is hidden from the screen — tree left the board."],[163.76837500000002,"right_edge is hidden from the screen — tree left the board."],[163.76837500000002,"low_box is hidden from the screen — tree left the board."],[163.76837500000002,"low_label is hidden from the screen — tree left the board."],[163.76837500000002,"high_box is hidden from the screen — tree left the board."],[163.76837500000002,"high_label is hidden from the screen — tree left the board."]]}]},{"title":"Growing to Purity","start":164.81004166666668,"end":330.5074166666667,"objects":{"blue_points":"a Point [blue] drawn in plane (location=(0.8, 1.2))","blue_points_2":"a Point [blue] drawn in plane (location=(1.5, 2.0))","blue_points_3":"a Point [blue] drawn in plane (location=(2.8, 1.0))","blue_points_4":"a Point [blue] drawn in plane (location=(3.6, 2.8))","blue_points_5":"a Point [blue] drawn in plane (location=(4.5, 1.5))","blue_points_6":"a Point [blue] drawn in plane (location=(5.6, 2.7))","blue_points_7":"a Point [blue] drawn in plane (location=(2.4, 5.5))","blue_points_8":"a Point [blue] drawn in plane (location=(6.8, 6.4))","edge_high":"a Line [gray] drawn in tree (start=(6.7, 6.0), end=(8.8, 5.0))","edge_high_a":"a Line [gray] drawn in tree (start=(8.3, 4.2), end=(7.2, 3.2))","edge_high_b":"a Line [gray] drawn in tree (start=(9.3, 4.2), end=(10.4, 3.2))","edge_high_c":"a Line [gray] drawn in tree (start=(6.7, 2.4), end=(5.8, 1.4))","edge_high_d":"a Line [gray] drawn in tree (start=(7.7, 2.4), end=(8.2, 1.4))","edge_low":"a Line [gray] drawn in tree (start=(5.3, 6.0), end=(3.2, 5.0))","edge_low_a":"a Line [gray] drawn in tree (start=(2.7, 4.2), end=(1.6, 3.2))","edge_low_b":"a Line [gray] drawn in tree (start=(3.7, 4.2), end=(4.8, 3.2))","edge_low_c":"a Line [gray] drawn in tree (start=(1.2, 2.4), end=(0.8, 1.4))","edge_low_d":"a Line [gray] drawn in tree (start=(2.0, 2.4), end=(2.8, 1.4))","heading":"a Heading that says \"Repeat Until Every Leaf Is Pure\"","high_a":"a Polygon [magenta] drawn in tree (vertices=((6.0, 2.4), (8.4, 2.4), (8.4, 3.2), (6.0, 3.2)), fill_opacity=0.1)","high_a_text":"a Math [magenta] that says \"$x_1 < 2.8?$\" drawn in tree","high_b":"a Polygon [magenta] drawn in tree (vertices=((9.2, 2.4), (11.6, 2.4), (11.6, 3.2), (9.2, 3.2)), fill_opacity=0.1)","high_b_text":"a Math [magenta] that says \"$x_2 < 5.6?$\" drawn in tree","high_c":"a Polygon [yellow] drawn in tree (vertices=((5.0, 0.6), (6.6, 0.6), (6.6, 1.4), (5.0, 1.4)), fill_opacity=0.1)","high_c_text":"a Math [yellow] that says \"$x_2 < 5.1?$\" drawn in tree","high_cut_five":"a Line [magenta] drawn in plane (start=(1.5, 5.1), end=(1.5, 8.0))","high_cut_four":"a Line [magenta] drawn in plane (start=(0.0, 5.1), end=(2.8, 5.1))","high_cut_one":"a Line [magenta] drawn in plane (start=(6.2, 4.0), end=(6.2, 8.0))","high_cut_three":"a Line [magenta] drawn in plane (start=(2.8, 4.0), end=(2.8, 8.0))","high_cut_two":"a Line [magenta] drawn in plane (start=(6.2, 5.6), end=(8.0, 5.6))","high_d":"a Polygon [red] drawn in tree (vertices=((7.3, 0.6), (9.1, 0.6), (9.1, 1.4), (7.3, 1.4)), fill_opacity=0.12)","high_d_text":"a Math [red] that says \"$R$\" drawn in tree","high_node":"a Polygon [magenta] drawn in tree (vertices=((7.6, 4.2), (10.0, 4.2), (10.0, 5.0), (7.6, 5.0)), fill_opacity=0.1)","high_text":"a Math [magenta] that says \"$x_1 < 6.2?$\" drawn in tree","leaf_count":"a Math [text] that says \"$upright(\"leaves\") = 2$\"","low_a":"a Polygon [green] drawn in tree (vertices=((0.5, 2.4), (2.7, 2.4), (2.7, 3.2), (0.5, 3.2)), fill_opacity=0.1)","low_a_text":"a Math [green] that says \"$x_2 < 2.5?$\" drawn in tree","low_b":"a Polygon [red] drawn in tree (vertices=((3.7, 2.4), (5.9, 2.4), (5.9, 3.2), (3.7, 3.2)), fill_opacity=0.12)","low_b_text":"a Math [red] that says \"$R$\" drawn in tree","low_c":"a Polygon [blue] drawn in tree (vertices=((0.1, 0.6), (1.5, 0.6), (1.5, 1.4), (0.1, 1.4)), fill_opacity=0.12)","low_c_text":"a Math [blue] that says \"$B$\" drawn in tree","low_cut_one":"a Line [green] drawn in plane (start=(6.1, 0.0), end=(6.1, 4.0))","low_cut_three":"a Line [green] drawn in plane (start=(2.5, 2.5), end=(2.5, 4.0))","low_cut_two":"a Line [green] drawn in plane (start=(0.0, 2.5), end=(6.1, 2.5))","low_d":"a Polygon [yellow] drawn in tree (vertices=((2.0, 0.6), (3.6, 0.6), (3.6, 1.4), (2.0, 1.4)), fill_opacity=0.1)","low_d_text":"a Math [yellow] that says \"$x_1 < 2.5?$\" drawn in tree","low_node":"a Polygon [green] drawn in tree (vertices=((2.0, 4.2), (4.4, 4.2), (4.4, 5.0), (2.0, 5.0)), fill_opacity=0.1)","low_text":"a Math [green] that says \"$x_1 < 6.1?$\" drawn in tree","plane":"an Axes (x_range=(0.0, 8.0), y_range=(0.0, 8.0), aspect=(1.0, 1.0))","point":"a Point [yellow] drawn in plane (location=(2.45, 5.4))","point_2":"a Point [yellow] drawn in plane (location=(2.9, 5.4))","purity":"a Math [text] that says \"$G_(upright(\"every leaf\")) = 0$\"","red_points":"a Point [red] drawn in plane (location=(0.9, 6.3))","red_points_2":"a Point [red] drawn in plane (location=(1.8, 4.8))","red_points_3":"a Point [red] drawn in plane (location=(3.3, 6.7))","red_points_4":"a Point [red] drawn in plane (location=(4.2, 5.2))","red_points_5":"a Point [red] drawn in plane (location=(5.4, 6.1))","red_points_6":"a Point [red] drawn in plane (location=(7.2, 4.8))","red_points_7":"a Point [red] drawn in plane (location=(2.2, 2.8))","red_points_8":"a Point [red] drawn in plane (location=(6.5, 1.2))","root_box":"a Polygon [yellow] drawn in tree (vertices=((4.7, 6.0), (7.3, 6.0), (7.3, 6.8), (4.7, 6.8)), fill_opacity=0.12)","root_cut":"a Line [yellow] drawn in plane (start=(0.0, 4.0), end=(8.0, 4.0))","root_text":"a Math [yellow] that says \"$x_2 < 4?$\" drawn in tree","tree":"a Figure (x_range=(0.0, 12.0), y_range=(0.0, 7.0), aspect=(12.0, 7.0))"},"beats":[{"start":164.81004166666668,"say":"Continue from the root split. The tree on the left and feature space on the right will grow together. A new internal node must always correspond to a new axis-aligned cut inside exactly one existing rectangle.","live":[],"does":[[164.81004166666668,"heading is shown on the screen, written out."],[164.81004166666668,"tree is shown on the screen, written out."],[164.81004166666668,"plane is shown on the screen, written out."],[164.81004166666668,"blue_points is shown on the screen, written out."],[164.8600416666667,"blue_points_2 is shown on the screen, written out."],[164.91004166666667,"blue_points_3 is shown on the screen, written out."],[164.96004166666668,"blue_points_4 is shown on the screen, written out."],[165.01004166666667,"blue_points_5 is shown on the screen, written out."],[165.06004166666668,"blue_points_6 is shown on the screen, written out."],[165.1100416666667,"blue_points_7 is shown on the screen, written out."],[165.16004166666667,"blue_points_8 is shown on the screen, written out."],[165.21004166666668,"red_points is shown on the screen, written out."],[165.26004166666667,"red_points_2 is shown on the screen, written out."],[165.31004166666668,"red_points_3 is shown on the screen, written out."],[165.3600416666667,"red_points_4 is shown on the screen, written out."],[165.41004166666667,"red_points_5 is shown on the screen, written out."],[165.46004166666668,"red_points_6 is shown on the screen, written out."],[165.51004166666667,"red_points_7 is shown on the screen, written out."],[165.56004166666668,"red_points_8 is shown on the screen, written out."],[165.6580416666667,"root_box is shown on the screen, written out."],[165.6580416666667,"root_text is shown on the screen, written out."],[167.58504166666668,"edge_low is shown on the screen, written out."],[167.58504166666668,"low_node is shown on the screen, written out."],[167.58504166666668,"low_text is shown on the screen, written out."],[169.00104166666668,"edge_high is shown on the screen, written out."],[169.00104166666668,"high_node is shown on the screen, written out."],[169.00104166666668,"high_text is shown on the screen, written out."],[169.4310416666667,"leaf_count is shown on the screen, written out."],[174.51604166666667,"root_cut is shown on the screen, written out."]]},{"start":178.48304166666668,"say":"Start in the lower rectangle. It contains six blue observations and two red ones. The best available gain first separates the far-right red point with x one less than six point one.","live":["tree","leaf_count","plane","heading","blue_points","blue_points_2","blue_points_3","blue_points_4","blue_points_5","blue_points_6","blue_points_7","blue_points_8","red_points","red_points_2","red_points_3","red_points_4","red_points_5","red_points_6","red_points_7","red_points_8","root_box","root_text","edge_low","edge_high","low_node","low_text","high_node","high_text","root_cut"],"does":[[183.46304166666667,"low_b is shown on the screen, written out."],[183.46304166666667,"low_b_text is shown on the screen, written out."],[184.8910416666667,"low_a is shown on the screen, written out."],[184.8910416666667,"low_a_text is shown on the screen, written out."],[186.1220416666667,"edge_low_a is shown on the screen, written out."],[186.1220416666667,"edge_low_b is shown on the screen, written out."],[186.71404166666667,"red_points_8 is indicated — a transient flash."],[189.17604166666666,"low_cut_one is shown on the screen, written out."],[190.44104166666668,"leaf_count becomes \"$upright(\"leaves\") = 4$\"."]]},{"start":191.04104166666667,"say":"The right child is pure, but the large left child is not. Within that child, a horizontal cut at x two equals two point five creates a pure blue strip below and a mixed strip above.","live":["tree","leaf_count","plane","heading","blue_points","blue_points_2","blue_points_3","blue_points_4","blue_points_5","blue_points_6","blue_points_7","blue_points_8","red_points","red_points_2","red_points_3","red_points_4","red_points_5","red_points_6","red_points_7","red_points_8","root_box","root_text","edge_low","edge_high","low_node","low_text","high_node","high_text","root_cut","low_cut_one","edge_low_a","edge_low_b","low_a","low_a_text","low_b","low_b_text"],"does":[[197.1360416666667,"low_cut_two is shown on the screen, written out."],[200.90904166666667,"edge_low_c is shown on the screen, written out."],[200.90904166666667,"low_c is shown on the screen, written out."],[200.90904166666667,"low_c_text is shown on the screen, written out."],[202.3020416666667,"edge_low_d is shown on the screen, written out."],[202.3020416666667,"low_d is shown on the screen, written out."],[202.3020416666667,"low_d_text is shown on the screen, written out."],[203.66104166666668,"leaf_count becomes \"$upright(\"leaves\") = 6$\"."]]},{"start":204.26104166666667,"say":"That upper strip is mixed because of one red observation at two point two, two point eight. Another vertical threshold isolates it. The training algorithm is rewarded, because two new leaves become perfectly pure.","live":["tree","leaf_count","plane","heading","blue_points","blue_points_2","blue_points_3","blue_points_4","blue_points_5","blue_points_6","blue_points_7","blue_points_8","red_points","red_points_2","red_points_3","red_points_4","red_points_5","red_points_6","red_points_7","red_points_8","root_box","root_text","edge_low","edge_high","low_node","low_text","high_node","high_text","root_cut","low_cut_one","edge_low_a","edge_low_b","low_a","low_a_text","low_b","low_b_text","low_cut_two","edge_low_c","edge_low_d","low_c","low_c_text","low_d","low_d_text"],"does":[[206.61804166666667,"red_points_7 is indicated — a transient flash."],[210.11204166666667,"low_cut_three is shown on the screen, written out."],[216.54404166666666,"low_d_text is indicated — a transient flash."]]},{"start":217.94554166666668,"say":"Now do the same work above the root. Most points there are red, but two blue exceptions force the recursion to continue. The first upper split separates the far-right pair from the rest.","live":["tree","leaf_count","plane","heading","blue_points","blue_points_2","blue_points_3","blue_points_4","blue_points_5","blue_points_6","blue_points_7","blue_points_8","red_points","red_points_2","red_points_3","red_points_4","red_points_5","red_points_6","red_points_7","red_points_8","root_box","root_text","edge_low","edge_high","low_node","low_text","high_node","high_text","root_cut","low_cut_one","edge_low_a","edge_low_b","low_a","low_a_text","low_b","low_b_text","low_cut_two","edge_low_c","edge_low_d","low_c","low_c_text","low_d","low_d_text","low_cut_three"],"does":[[223.1700416666667,"blue_points_7 is indicated — a transient flash."],[223.1700416666667,"blue_points_8 is indicated — a transient flash."],[226.45504166666666,"high_cut_one is shown on the screen, written out."],[227.16404166666666,"edge_high_a is shown on the screen, written out."],[227.16404166666666,"edge_high_b is shown on the screen, written out."],[228.41704166666668,"high_b is shown on the screen, written out."],[228.41704166666668,"high_b_text is shown on the screen, written out."],[229.5320416666667,"high_a is shown on the screen, written out."],[229.5320416666667,"high_a_text is shown on the screen, written out."]]},{"start":230.99104166666666,"say":"The far-right pair still disagrees. A horizontal cut isolates the blue point at the top from the red point below. Again, training impurity falls to zero in both resulting leaves.","live":["tree","leaf_count","plane","heading","blue_points","blue_points_2","blue_points_3","blue_points_4","blue_points_5","blue_points_6","blue_points_7","blue_points_8","red_points","red_points_2","red_points_3","red_points_4","red_points_5","red_points_6","red_points_7","red_points_8","root_box","root_text","edge_low","edge_high","low_node","low_text","high_node","high_text","root_cut","low_cut_one","edge_low_a","edge_low_b","low_a","low_a_text","low_b","low_b_text","low_cut_two","edge_low_c","edge_low_d","low_c","low_c_text","low_d","low_d_text","low_cut_three","high_cut_one","edge_high_a","edge_high_b","high_a","high_a_text","high_b","high_b_text"],"does":[[234.38104166666668,"high_cut_two is shown on the screen, written out."],[236.01804166666668,"blue_points_8 is indicated — a transient flash."],[241.2660416666667,"high_b_text is indicated — a transient flash."]]},{"start":244.40854166666668,"say":"The left upper region has its own blue exception. A split at x one equals two point eight narrows the search, and another horizontal split at x two equals five point one separates one red point.","live":["tree","leaf_count","plane","heading","blue_points","blue_points_2","blue_points_3","blue_points_4","blue_points_5","blue_points_6","blue_points_7","blue_points_8","red_points","red_points_2","red_points_3","red_points_4","red_points_5","red_points_6","red_points_7","red_points_8","root_box","root_text","edge_low","edge_high","low_node","low_text","high_node","high_text","root_cut","low_cut_one","edge_low_a","edge_low_b","low_a","low_a_text","low_b","low_b_text","low_cut_two","edge_low_c","edge_low_d","low_c","low_c_text","low_d","low_d_text","low_cut_three","high_cut_one","edge_high_a","edge_high_b","high_a","high_a_text","high_b","high_b_text","high_cut_two"],"does":[[246.62604166666668,"high_c is shown on the screen, written out."],[246.62604166666668,"high_c_text is shown on the screen, written out."],[249.93504166666668,"high_cut_three is shown on the screen, written out."],[250.91004166666667,"edge_high_c is shown on the screen, written out."],[250.91004166666667,"edge_high_d is shown on the screen, written out."],[252.8840416666667,"high_cut_four is shown on the screen, written out."],[257.12104166666666,"high_d is shown on the screen, written out."],[257.12104166666666,"high_d_text is shown on the screen, written out."],[258.07354166666664,"leaf_count becomes \"$upright(\"leaves\") = 9$\"."]]},{"start":258.67354166666667,"say":"One final threshold at x one equals one point five isolates the remaining upper blue point. Every terminal region now contains only one class. Consequently every leaf has Gini impurity zero.","live":["tree","leaf_count","plane","heading","blue_points","blue_points_2","blue_points_3","blue_points_4","blue_points_5","blue_points_6","blue_points_7","blue_points_8","red_points","red_points_2","red_points_3","red_points_4","red_points_5","red_points_6","red_points_7","red_points_8","root_box","root_text","edge_low","edge_high","low_node","low_text","high_node","high_text","root_cut","low_cut_one","edge_low_a","edge_low_b","low_a","low_a_text","low_b","low_b_text","low_cut_two","edge_low_c","edge_low_d","low_c","low_c_text","low_d","low_d_text","low_cut_three","high_cut_one","edge_high_a","edge_high_b","high_a","high_a_text","high_b","high_b_text","high_cut_two","high_cut_three","high_cut_four","edge_high_c","edge_high_d","high_c","high_c_text","high_d","high_d_text"],"does":[[259.3120416666667,"high_cut_five is shown on the screen, written out."],[264.4200416666667,"blue_points_7 is indicated — a transient flash."],[271.3630416666667,"purity is shown on the screen, written out."]]},{"start":272.81054166666667,"say":"On the training set, this looks perfect. Every observation is classified correctly. But look at the geometry required to achieve it: thin strips, short corridors, and thresholds whose only purpose is to rescue one exceptional dot.","live":["tree","leaf_count","purity","plane","heading","blue_points","blue_points_2","blue_points_3","blue_points_4","blue_points_5","blue_points_6","blue_points_7","blue_points_8","red_points","red_points_2","red_points_3","red_points_4","red_points_5","red_points_6","red_points_7","red_points_8","root_box","root_text","edge_low","edge_high","low_node","low_text","high_node","high_text","root_cut","low_cut_one","edge_low_a","edge_low_b","low_a","low_a_text","low_b","low_b_text","low_cut_two","edge_low_c","edge_low_d","low_c","low_c_text","low_d","low_d_text","low_cut_three","high_cut_one","edge_high_a","edge_high_b","high_a","high_a_text","high_b","high_b_text","high_cut_two","high_cut_three","high_cut_four","edge_high_c","edge_high_d","high_c","high_c_text","high_d","high_d_text","high_cut_five"],"does":[[281.89004166666666,"low_cut_three is indicated — a transient flash."],[282.9350416666667,"high_cut_four is indicated — a transient flash."],[286.6960416666667,"high_cut_five is indicated — a transient flash."]]},{"start":289.0610416666667,"say":"A new point can cross one of those arbitrary thresholds after an imperceptible change in a feature. Its predicted class then jumps, even though the training labels gave us almost no evidence that such a jump should exist.","live":null,"does":[[289.5720416666667,"point is shown on the screen, grown."],[291.5720416666667,"point is hidden from the screen."],[296.8510416666667,"point_2 is shown on the screen, grown."],[298.8510416666667,"point_2 is hidden from the screen."]]},{"start":302.7920416666667,"say":"This is the characteristic strength and weakness of an unconstrained decision tree. It has low bias because it can represent complicated interactions. It also has high variance because a few observations can rearrange entire branches and rectangles.","live":null,"does":[[306.9950416666667,"root_text is indicated — a transient flash."],[314.19304166666666,"low_d_text is indicated — a transient flash."],[314.19304166666666,"high_c_text is indicated — a transient flash."]]},{"start":318.3455416666667,"say":"Pure leaves are therefore a training condition, not evidence of a useful model. The next question is whether every branch earns its complexity on data that did not choose the branch.","live":null,"does":[[320.2030416666667,"A box is drawn around purity."],[329.46575,"heading is hidden from the screen — left the board."],[329.46575,"leaf_count is hidden from the screen — left the board."],[329.46575,"plane is hidden from the screen — left the board."],[329.46575,"blue_points is hidden from the screen — plane left the board."],[329.46575,"blue_points_2 is hidden from the screen — plane left the board."],[329.46575,"blue_points_3 is hidden from the screen — plane left the board."],[329.46575,"blue_points_4 is hidden from the screen — plane left the board."],[329.46575,"blue_points_5 is hidden from the screen — plane left the board."],[329.46575,"blue_points_6 is hidden from the screen — plane left the board."],[329.46575,"blue_points_7 is hidden from the screen — plane left the board."],[329.46575,"blue_points_8 is hidden from the screen — plane left the board."],[329.46575,"red_points is hidden from the screen — plane left the board."],[329.46575,"red_points_2 is hidden from the screen — plane left the board."],[329.46575,"red_points_3 is hidden from the screen — plane left the board."],[329.46575,"red_points_4 is hidden from the screen — plane left the board."],[329.46575,"red_points_5 is hidden from the screen — plane left the board."],[329.46575,"red_points_6 is hidden from the screen — plane left the board."],[329.46575,"red_points_7 is hidden from the screen — plane left the board."],[329.46575,"red_points_8 is hidden from the screen — plane left the board."],[329.46575,"root_cut is hidden from the screen — plane left the board."],[329.46575,"low_cut_one is hidden from the screen — plane left the board."],[329.46575,"low_cut_two is hidden from the screen — plane left the board."],[329.46575,"low_cut_three is hidden from the screen — plane left the board."],[329.46575,"high_cut_one is hidden from the screen — plane left the board."],[329.46575,"high_cut_two is hidden from the screen — plane left the board."],[329.46575,"high_cut_three is hidden from the screen — plane left the board."],[329.46575,"high_cut_four is hidden from the screen — plane left the board."],[329.46575,"high_cut_five is hidden from the screen — plane left the board."],[329.46575,"purity is hidden from the screen — left the board."],[329.46575,"tree is hidden from the screen — left the board."],[329.46575,"root_box is hidden from the screen — tree left the board."],[329.46575,"root_text is hidden from the screen — tree left the board."],[329.46575,"edge_low is hidden from the screen — tree left the board."],[329.46575,"edge_high is hidden from the screen — tree left the board."],[329.46575,"low_node is hidden from the screen — tree left the board."],[329.46575,"low_text is hidden from the screen — tree left the board."],[329.46575,"high_node is hidden from the screen — tree left the board."],[329.46575,"high_text is hidden from the screen — tree left the board."],[329.46575,"edge_low_a is hidden from the screen — tree left the board."],[329.46575,"edge_low_b is hidden from the screen — tree left the board."],[329.46575,"low_a is hidden from the screen — tree left the board."],[329.46575,"low_a_text is hidden from the screen — tree left the board."],[329.46575,"low_b is hidden from the screen — tree left the board."],[329.46575,"low_b_text is hidden from the screen — tree left the board."],[329.46575,"edge_low_c is hidden from the screen — tree left the board."],[329.46575,"edge_low_d is hidden from the screen — tree left the board."],[329.46575,"low_c is hidden from the screen — tree left the board."],[329.46575,"low_c_text is hidden from the screen — tree left the board."],[329.46575,"low_d is hidden from the screen — tree left the board."],[329.46575,"low_d_text is hidden from the screen — tree left the board."],[329.46575,"edge_high_a is hidden from the screen — tree left the board."],[329.46575,"edge_high_b is hidden from the screen — tree left the board."],[329.46575,"high_a is hidden from the screen — tree left the board."],[329.46575,"high_a_text is hidden from the screen — tree left the board."],[329.46575,"high_b is hidden from the screen — tree left the board."],[329.46575,"high_b_text is hidden from the screen — tree left the board."],[329.46575,"edge_high_c is hidden from the screen — tree left the board."],[329.46575,"edge_high_d is hidden from the screen — tree left the board."],[329.46575,"high_c is hidden from the screen — tree left the board."],[329.46575,"high_c_text is hidden from the screen — tree left the board."],[329.46575,"high_d is hidden from the screen — tree left the board."],[329.46575,"high_d_text is hidden from the screen — tree left the board."]]}]},{"title":"Pruning the Tree","start":330.5074166666667,"end":509.9435833333333,"objects":{"blue_points":"a Point [blue] drawn in plane (location=(0.8, 1.2))","blue_points_2":"a Point [blue] drawn in plane (location=(1.5, 2.0))","blue_points_3":"a Point [blue] drawn in plane (location=(2.8, 1.0))","blue_points_4":"a Point [blue] drawn in plane (location=(3.6, 2.8))","blue_points_5":"a Point [blue] drawn in plane (location=(4.5, 1.5))","blue_points_6":"a Point [blue] drawn in plane (location=(5.6, 2.7))","blue_points_7":"a Point [blue] drawn in plane (location=(2.4, 5.5))","blue_points_8":"a Point [blue] drawn in plane (location=(6.8, 6.4))","branch_left":"a Line [gray] drawn in tree (start=(4.3, 5.1), end=(2.5, 4.0))","branch_right":"a Line [gray] drawn in tree (start=(5.7, 5.1), end=(7.5, 4.0))","criterion":"a Math [text] that says \"$R_alpha(T) = R(T) + alpha thin |upright(\"leaves\")(T)|$\"","heading":"a Heading that says \"Trade Fit for a Smaller Tree\"","heading_path":"a Heading that says \"Choose Complexity Outside the Training Fit\"","high_detail_four":"a Line [gray] drawn in plane (start=(1.5, 5.1), end=(1.5, 8.0))","high_detail_one":"a Line [gray] drawn in plane (start=(6.2, 5.6), end=(8.0, 5.6))","high_detail_three":"a Line [gray] drawn in plane (start=(0.0, 5.1), end=(2.8, 5.1))","high_detail_two":"a Line [gray] drawn in plane (start=(2.8, 4.0), end=(2.8, 8.0))","high_main":"a Line [magenta] drawn in plane (start=(6.2, 4.0), end=(6.2, 8.0))","left_node":"a Polygon [green] drawn in tree (vertices=((1.3, 3.3), (3.7, 3.3), (3.7, 4.0), (1.3, 4.0)), fill_opacity=0.1)","left_text":"a Math [green] that says \"$x_1 < 6.1?$\" drawn in tree","left_twigs":"a Line [gray] drawn in tree (start=(2.0, 3.3), end=(1.1, 2.2))","left_twigs_2":"a Line [gray] drawn in tree (start=(3.0, 3.3), end=(3.9, 2.2))","left_twigs_3":"a Line [gray] drawn in tree (start=(1.1, 2.2), end=(0.6, 1.0))","left_twigs_4":"a Line [gray] drawn in tree (start=(1.1, 2.2), end=(1.8, 1.0))","left_twigs_5":"a Line [gray] drawn in tree (start=(3.9, 2.2), end=(3.3, 1.0))","left_twigs_6":"a Line [gray] drawn in tree (start=(3.9, 2.2), end=(4.6, 1.0))","low_detail_one":"a Line [gray] drawn in plane (start=(0.0, 2.5), end=(6.1, 2.5))","low_detail_two":"a Line [gray] drawn in plane (start=(2.5, 2.5), end=(2.5, 4.0))","low_main":"a Line [green] drawn in plane (start=(6.1, 0.0), end=(6.1, 4.0))","note":"a Panel that says \"Training impurity chooses splits. Validation evidence chooses how much of the fitted tree to keep.\"","path":"a Table [text] that says \"$alpha$ Leaves Train error Validation error $0.000$ $9$ $0.000$ $0.250$ $0.015$ $6$ $0.063$ $0.188$ $0.040$ $4$ $0.125$ $0.125$ $0.090$ $2$ $0.250$ $0.188$\" (rows=(('$alpha$', 'Leaves', 'Train error', 'Validation error'), ('$0…, header=True)","plane":"an Axes (x_range=(0.0, 8.0), y_range=(0.0, 8.0), aspect=(1.0, 1.0))","pruned_left_leaf":"a Polygon [blue] drawn in tree (vertices=((1.2, 1.7), (3.8, 1.7), (3.8, 2.5), (1.2, 2.5)), fill_opacity=0.14)","pruned_left_text":"a Math [blue] that says \"$upright(\"mostly B\")$\" drawn in tree","pruned_right_leaf":"a Polygon [red] drawn in tree (vertices=((6.2, 1.7), (8.8, 1.7), (8.8, 2.5), (6.2, 2.5)), fill_opacity=0.14)","pruned_right_text":"a Math [red] that says \"$upright(\"mostly R\")$\" drawn in tree","red_points":"a Point [red] drawn in plane (location=(0.9, 6.3))","red_points_2":"a Point [red] drawn in plane (location=(1.8, 4.8))","red_points_3":"a Point [red] drawn in plane (location=(3.3, 6.7))","red_points_4":"a Point [red] drawn in plane (location=(4.2, 5.2))","red_points_5":"a Point [red] drawn in plane (location=(5.4, 6.1))","red_points_6":"a Point [red] drawn in plane (location=(7.2, 4.8))","red_points_7":"a Point [red] drawn in plane (location=(2.2, 2.8))","red_points_8":"a Point [red] drawn in plane (location=(6.5, 1.2))","right_node":"a Polygon [magenta] drawn in tree (vertices=((6.3, 3.3), (8.7, 3.3), (8.7, 4.0), (6.3, 4.0)), fill_opacity=0.1)","right_text":"a Math [magenta] that says \"$x_1 < 6.2?$\" drawn in tree","right_twigs":"a Line [gray] drawn in tree (start=(7.0, 3.3), end=(6.1, 2.2))","right_twigs_2":"a Line [gray] drawn in tree (start=(8.0, 3.3), end=(8.9, 2.2))","right_twigs_3":"a Line [gray] drawn in tree (start=(6.1, 2.2), end=(5.4, 1.0))","right_twigs_4":"a Line [gray] drawn in tree (start=(6.1, 2.2), end=(6.8, 1.0))","right_twigs_5":"a Line [gray] drawn in tree (start=(8.9, 2.2), end=(8.2, 1.0))","right_twigs_6":"a Line [gray] drawn in tree (start=(8.9, 2.2), end=(9.6, 1.0))","root":"a Polygon [yellow] drawn in tree (vertices=((3.7, 5.1), (6.3, 5.1), (6.3, 5.8), (3.7, 5.8)), fill_opacity=0.12)","root_cut":"a Line [yellow] drawn in plane (start=(0.0, 4.0), end=(8.0, 4.0))","root_text":"a Math [yellow] that says \"$x_2 < 4?$\" drawn in tree","selection":"a Math [text] that says \"$alpha = 0.040 thin arrow.r thin 4 thin upright(\"leaves\")$\"","tree":"a Figure (x_range=(0.0, 10.0), y_range=(0.0, 6.0), aspect=(5.0, 3.0))","twig_nodes":"a Point [gray] drawn in tree (location=(1.1, 2.2))","twig_nodes_10":"a Point [blue] drawn in tree (location=(6.8, 1.0))","twig_nodes_11":"a Point [red] drawn in tree (location=(8.2, 1.0))","twig_nodes_12":"a Point [blue] drawn in tree (location=(9.6, 1.0))","twig_nodes_2":"a Point [gray] drawn in tree (location=(3.9, 2.2))","twig_nodes_3":"a Point [gray] drawn in tree (location=(6.1, 2.2))","twig_nodes_4":"a Point [gray] drawn in tree (location=(8.9, 2.2))","twig_nodes_5":"a Point [blue] drawn in tree (location=(0.6, 1.0))","twig_nodes_6":"a Point [red] drawn in tree (location=(1.8, 1.0))","twig_nodes_7":"a Point [blue] drawn in tree (location=(3.3, 1.0))","twig_nodes_8":"a Point [red] drawn in tree (location=(4.6, 1.0))","twig_nodes_9":"a Point [red] drawn in tree (location=(5.4, 1.0))"},"beats":[{"start":330.5074166666667,"say":"The fully grown tree has zero training impurity, but it pays for that fit with nine leaves and several thresholds supported by a single unusual observation. Pruning asks whether those extra leaves earn their keep.","live":[],"does":[[330.5074166666667,"heading is shown on the screen, written out."],[330.5074166666667,"tree is shown on the screen, written out."],[330.5074166666667,"plane is shown on the screen, written out."],[330.5074166666667,"blue_points is shown on the screen, written out."],[330.5074166666667,"root is shown on the screen, written out."],[330.5074166666667,"root_text is shown on the screen, written out."],[330.5074166666667,"branch_left is shown on the screen, written out."],[330.5074166666667,"branch_right is shown on the screen, written out."],[330.5074166666667,"left_node is shown on the screen, written out."],[330.5074166666667,"right_node is shown on the screen, written out."],[330.5074166666667,"left_text is shown on the screen, written out."],[330.5074166666667,"right_text is shown on the screen, written out."],[330.5074166666667,"root_cut is shown on the screen, written out."],[330.5074166666667,"low_main is shown on the screen, written out."],[330.5074166666667,"high_main is shown on the screen, written out."],[330.5074166666667,"low_detail_one is shown on the screen, written out."],[330.5074166666667,"low_detail_two is shown on the screen, written out."],[330.5074166666667,"high_detail_one is shown on the screen, written out."],[330.5074166666667,"high_detail_two is shown on the screen, written out."],[330.5074166666667,"high_detail_three is shown on the screen, written out."],[330.5074166666667,"high_detail_four is shown on the screen, written out."],[330.5474166666667,"blue_points_2 is shown on the screen, written out."],[330.5874166666667,"blue_points_3 is shown on the screen, written out."],[330.6274166666667,"blue_points_4 is shown on the screen, written out."],[330.6674166666667,"blue_points_5 is shown on the screen, written out."],[330.7074166666667,"blue_points_6 is shown on the screen, written out."],[330.7474166666667,"blue_points_7 is shown on the screen, written out."],[330.7874166666667,"blue_points_8 is shown on the screen, written out."],[330.8274166666667,"red_points is shown on the screen, written out."],[330.8674166666667,"red_points_2 is shown on the screen, written out."],[330.9074166666667,"red_points_3 is shown on the screen, written out."],[330.9474166666667,"red_points_4 is shown on the screen, written out."],[330.9874166666667,"red_points_5 is shown on the screen, written out."],[331.0274166666667,"red_points_6 is shown on the screen, written out."],[331.0674166666667,"red_points_7 is shown on the screen, written out."],[331.1074166666667,"red_points_8 is shown on the screen, written out."],[331.3074166666667,"left_twigs is shown on the screen, written out."],[331.3574166666667,"left_twigs_2 is shown on the screen, written out."],[331.4074166666667,"left_twigs_3 is shown on the screen, written out."],[331.4574166666667,"left_twigs_4 is shown on the screen, written out."],[331.5074166666667,"left_twigs_5 is shown on the screen, written out."],[331.5574166666667,"left_twigs_6 is shown on the screen, written out."],[331.6074166666667,"right_twigs is shown on the screen, written out."],[331.6274166666667,"twig_nodes is shown on the screen, written out."],[331.6574166666667,"right_twigs_2 is shown on the screen, written out."],[331.6674166666667,"twig_nodes_2 is shown on the screen, written out."],[331.7074166666667,"twig_nodes_3 is shown on the screen, written out."],[331.7074166666667,"right_twigs_3 is shown on the screen, written out."],[331.7474166666667,"twig_nodes_4 is shown on the screen, written out."],[331.7574166666667,"right_twigs_4 is shown on the screen, written out."],[331.7874166666667,"twig_nodes_5 is shown on the screen, written out."],[331.8074166666667,"right_twigs_5 is shown on the screen, written out."],[331.8274166666667,"twig_nodes_6 is shown on the screen, written out."],[331.8574166666667,"right_twigs_6 is shown on the screen, written out."],[331.8674166666667,"twig_nodes_7 is shown on the screen, written out."],[331.9074166666667,"twig_nodes_8 is shown on the screen, written out."],[331.9474166666667,"twig_nodes_9 is shown on the screen, written out."],[331.9874166666667,"twig_nodes_10 is shown on the screen, written out."],[332.0274166666667,"twig_nodes_11 is shown on the screen, written out."],[332.0674166666667,"twig_nodes_12 is shown on the screen, written out."],[340.0974166666667,"criterion is shown on the screen, written out."]]},{"start":343.8549166666667,"say":"Cost-complexity pruning gives the trade a precise form. R of T measures the fitted tree's error or impurity. The second term charges alpha for every terminal leaf.","live":["criterion","tree","plane","heading","blue_points","blue_points_2","blue_points_3","blue_points_4","blue_points_5","blue_points_6","blue_points_7","blue_points_8","red_points","red_points_2","red_points_3","red_points_4","red_points_5","red_points_6","red_points_7","red_points_8","root","root_text","branch_left","branch_right","left_node","right_node","left_text","right_text","left_twigs","left_twigs_2","left_twigs_3","left_twigs_4","left_twigs_5","left_twigs_6","right_twigs","right_twigs_2","right_twigs_3","right_twigs_4","right_twigs_5","right_twigs_6","twig_nodes","twig_nodes_2","twig_nodes_3","twig_nodes_4","twig_nodes_5","twig_nodes_6","twig_nodes_7","twig_nodes_8","twig_nodes_9","twig_nodes_10","twig_nodes_11","twig_nodes_12","root_cut","low_main","high_main","low_detail_one","low_detail_two","high_detail_one","high_detail_two","high_detail_three","high_detail_four"],"does":[[350.3794166666667,"criterion (the \"R(T)\" part) is emphasized."],[353.0614166666667,"criterion (the \"R(T)\" part) is no longer emphasized."],[353.0614166666667,"criterion (the \"alpha thin |upright(\"leaves\")(T)|\" part) is emphasized."],[355.5574166666667,"criterion (the \"alpha thin |upright(\"leaves\")(T)|\" part) is no longer emphasized."]]},{"start":356.1574166666667,"say":"At alpha zero, extra leaves are free, so the pure tree wins. As alpha increases, a weak pair of leaves can cost more than the small reduction in training error that created it.","live":null,"does":[[356.7264166666667,"criterion (the \"alpha\" part) is indicated — a transient flash."],[362.8214166666667,"low_detail_two is indicated — a transient flash."],[362.8214166666667,"high_detail_four is indicated — a transient flash."]]},{"start":368.2509166666667,"say":"Prune the weakest lower twigs first. Their narrow thresholds disappear from feature space at the same moment their branches disappear from the tree. The replacement leaf predicts the local majority class.","live":null,"does":[[372.5584166666667,"low_detail_one is hidden from the screen."],[372.5584166666667,"low_detail_two is hidden from the screen."],[375.1824166666667,"left_twigs is hidden from the screen."],[375.1824166666667,"left_twigs_2 is hidden from the screen."],[375.1824166666667,"left_twigs_3 is hidden from the screen."],[375.1824166666667,"left_twigs_4 is hidden from the screen."],[375.1824166666667,"left_twigs_5 is hidden from the screen."],[375.1824166666667,"left_twigs_6 is hidden from the screen."],[377.6554166666667,"pruned_left_leaf is shown on the screen, written out."],[379.29241666666667,"pruned_left_text is shown on the screen, written out."]]},{"start":381.2269166666667,"say":"Now prune the corresponding upper twigs. Again, the replacement is not pure on the training set. It deliberately accepts a few mistakes in exchange for a much larger and more stable prediction region.","live":["criterion","tree","plane","heading","blue_points","blue_points_2","blue_points_3","blue_points_4","blue_points_5","blue_points_6","blue_points_7","blue_points_8","red_points","red_points_2","red_points_3","red_points_4","red_points_5","red_points_6","red_points_7","red_points_8","root","root_text","branch_left","branch_right","left_node","right_node","left_text","right_text","right_twigs","right_twigs_2","right_twigs_3","right_twigs_4","right_twigs_5","right_twigs_6","twig_nodes","twig_nodes_2","twig_nodes_3","twig_nodes_4","twig_nodes_5","twig_nodes_6","twig_nodes_7","twig_nodes_8","twig_nodes_9","twig_nodes_10","twig_nodes_11","twig_nodes_12","root_cut","low_main","high_main","high_detail_one","high_detail_two","high_detail_three","high_detail_four","pruned_left_leaf","pruned_left_text"],"does":[[381.9474166666667,"high_detail_two is hidden from the screen."],[381.9474166666667,"high_detail_three is hidden from the screen."],[381.9474166666667,"high_detail_four is hidden from the screen."],[383.3284166666667,"right_twigs is hidden from the screen."],[383.3284166666667,"right_twigs_2 is hidden from the screen."],[383.3284166666667,"right_twigs_3 is hidden from the screen."],[383.3284166666667,"right_twigs_4 is hidden from the screen."],[383.3284166666667,"right_twigs_5 is hidden from the screen."],[383.3284166666667,"right_twigs_6 is hidden from the screen."],[385.3254166666667,"pruned_right_leaf is shown on the screen, written out."],[385.3254166666667,"pruned_right_text is shown on the screen, written out."]]},{"start":394.2499166666667,"say":"The remaining four-leaf tree still captures the broad interaction. The horizontal root separates low from high x two. Within each half, one x one threshold handles the main exception near the far-right edge.","live":["criterion","tree","plane","heading","blue_points","blue_points_2","blue_points_3","blue_points_4","blue_points_5","blue_points_6","blue_points_7","blue_points_8","red_points","red_points_2","red_points_3","red_points_4","red_points_5","red_points_6","red_points_7","red_points_8","root","root_text","branch_left","branch_right","left_node","right_node","left_text","right_text","twig_nodes","twig_nodes_2","twig_nodes_3","twig_nodes_4","twig_nodes_5","twig_nodes_6","twig_nodes_7","twig_nodes_8","twig_nodes_9","twig_nodes_10","twig_nodes_11","twig_nodes_12","root_cut","low_main","high_main","high_detail_one","pruned_left_leaf","pruned_left_text","pruned_right_leaf","pruned_right_text"],"does":[[398.7664166666667,"root_cut is indicated — a transient flash."],[403.5614166666667,"low_main is indicated — a transient flash."],[403.5614166666667,"high_main is indicated — a transient flash."]]},{"start":408.4449166666667,"say":"This smaller tree has higher training error by construction. The relevant question is whether it has lower error on observations that did not participate in choosing all those thresholds.","live":null,"does":[[419.3344166666667,"criterion is hidden from the screen — left the board."],[419.3344166666667,"heading is hidden from the screen — left the board."],[419.3344166666667,"plane is hidden from the screen — left the board."],[419.3344166666667,"blue_points is hidden from the screen — plane left the board."],[419.3344166666667,"blue_points_2 is hidden from the screen — plane left the board."],[419.3344166666667,"blue_points_3 is hidden from the screen — plane left the board."],[419.3344166666667,"blue_points_4 is hidden from the screen — plane left the board."],[419.3344166666667,"blue_points_5 is hidden from the screen — plane left the board."],[419.3344166666667,"blue_points_6 is hidden from the screen — plane left the board."],[419.3344166666667,"blue_points_7 is hidden from the screen — plane left the board."],[419.3344166666667,"blue_points_8 is hidden from the screen — plane left the board."],[419.3344166666667,"red_points is hidden from the screen — plane left the board."],[419.3344166666667,"red_points_2 is hidden from the screen — plane left the board."],[419.3344166666667,"red_points_3 is hidden from the screen — plane left the board."],[419.3344166666667,"red_points_4 is hidden from the screen — plane left the board."],[419.3344166666667,"red_points_5 is hidden from the screen — plane left the board."],[419.3344166666667,"red_points_6 is hidden from the screen — plane left the board."],[419.3344166666667,"red_points_7 is hidden from the screen — plane left the board."],[419.3344166666667,"red_points_8 is hidden from the screen — plane left the board."],[419.3344166666667,"root_cut is hidden from the screen — plane left the board."],[419.3344166666667,"low_main is hidden from the screen — plane left the board."],[419.3344166666667,"high_main is hidden from the screen — plane left the board."],[419.3344166666667,"high_detail_one is hidden from the screen — plane left the board."],[419.3344166666667,"tree is hidden from the screen — left the board."],[419.3344166666667,"root is hidden from the screen — tree left the board."],[419.3344166666667,"root_text is hidden from the screen — tree left the board."],[419.3344166666667,"branch_left is hidden from the screen — tree left the board."],[419.3344166666667,"branch_right is hidden from the screen — tree left the board."],[419.3344166666667,"left_node is hidden from the screen — tree left the board."],[419.3344166666667,"right_node is hidden from the screen — tree left the board."],[419.3344166666667,"left_text is hidden from the screen — tree left the board."],[419.3344166666667,"right_text is hidden from the screen — tree left the board."],[419.3344166666667,"twig_nodes is hidden from the screen — tree left the board."],[419.3344166666667,"twig_nodes_2 is hidden from the screen — tree left the board."],[419.3344166666667,"twig_nodes_3 is hidden from the screen — tree left the board."],[419.3344166666667,"twig_nodes_4 is hidden from the screen — tree left the board."],[419.3344166666667,"twig_nodes_5 is hidden from the screen — tree left the board."],[419.3344166666667,"twig_nodes_6 is hidden from the screen — tree left the board."],[419.3344166666667,"twig_nodes_7 is hidden from the screen — tree left the board."],[419.3344166666667,"twig_nodes_8 is hidden from the screen — tree left the board."],[419.3344166666667,"twig_nodes_9 is hidden from the screen — tree left the board."],[419.3344166666667,"twig_nodes_10 is hidden from the screen — tree left the board."],[419.3344166666667,"twig_nodes_11 is hidden from the screen — tree left the board."],[419.3344166666667,"twig_nodes_12 is hidden from the screen — tree left the board."],[419.3344166666667,"pruned_left_leaf is hidden from the screen — tree left the board."],[419.3344166666667,"pruned_left_text is hidden from the screen — tree left the board."],[419.3344166666667,"pruned_right_leaf is hidden from the screen — tree left the board."],[419.3344166666667,"pruned_right_text is hidden from the screen — tree left the board."]]},{"start":420.5344166666667,"say":"A pruning path supplies a nested sequence of subtrees. Here the unpruned tree has nine leaves and no training mistakes, but its validation error is the worst entry in the table.","live":[],"does":[[420.5344166666667,"heading_path is shown on the screen, written out."],[421.4984166666667,"path is shown on the screen, written out."],[426.2344166666667,"path is shown on the screen, written out."],[429.9964166666667,"path (the \"$0.250$\" part) is indicated — a transient flash."]]},{"start":432.2219166666667,"say":"A small penalty removes three leaves. Training error rises, validation error falls. A larger penalty leaves four terminal regions, and the validation error reaches its minimum.","live":["heading_path"],"does":[[432.7794166666667,"path is shown on the screen, written out."],[440.3604166666667,"path is shown on the screen, written out."],[443.6224166666667,"path (the \"$0.125$\" part) is indicated — a transient flash."]]},{"start":445.0934166666667,"say":"Push alpha farther and the tree collapses to two leaves. That model is now too simple for the data, so validation error rises again. Pruning is not a ritual of making trees small. It is model selection along a structured complexity path.","live":null,"does":[[446.1614166666667,"path is shown on the screen, written out."],[452.7794166666667,"path (the \"$0.188$\" part) is indicated — a transient flash."]]},{"start":461.8079166666667,"say":"Select alpha using cross-validation, a held-out set, or a nested procedure when tuning itself must be evaluated. Never choose the pruning strength by returning to the same training impurity that rewarded every twig.","live":null,"does":[[462.1564166666667,"selection is shown on the screen, written out."],[463.3174166666667,"path is indicated — a transient flash."],[469.9234166666667,"note is shown on the screen, written out."]]},{"start":475.84091666666666,"say":"Libraries also offer pre-pruning controls such as maximum depth, minimum samples per leaf, minimum impurity decrease, and maximum leaf count. Those prevent growth. Cost-complexity pruning fits first and removes branches afterward.","live":["selection","note","heading_path"],"does":[[482.6554166666667,"note (the \"Training impurity\" part) is indicated — a transient flash."],[491.6764166666667,"A box is drawn around selection."]]},{"start":493.2984166666667,"say":"Either route trades some bias for less variance. A single pruned tree is often much easier to explain and more stable than the pure tree. The forest will take a different route: keep many unstable trees, then make their instability cancel.","live":null,"does":[[508.9019166666667,"heading_path is hidden from the screen — left the board."],[508.9019166666667,"note is hidden from the screen — left the board."],[508.9019166666667,"path is hidden from the screen — left the board."],[508.9019166666667,"selection is hidden from the screen — left the board."]]}]},{"title":"Planting Many Trees","start":509.9435833333333,"end":706.4061875,"objects":{"bootstrap":"a Table [text] that says \"Tree Bootstrap indices Omitted $T_1$ $1,1,2,4,5,5,7,8$ $3,6$ $T_2$ $2,3,3,4,6,7,7,8$ $1,5$ $T_3$ $1,2,4,4,5,6,8,8$ $3,7$\" (rows=(('Tree', 'Bootstrap indices', 'Omitted'), ('$T_1$', '$1,1,2,4,…, header=True)","caption_one":"a Math [text] that says \"$T_1$\"","caption_three":"a Math [text] that says \"$T_3$\"","caption_two":"a Math [text] that says \"$T_2$\"","feature_law":"a Math [text] that says \"$S_(b,j) subset {1, dots, p}, quad |S_(b,j)| = m$\"","features":"a Table [text] that says \"Node Offered features Winning split Root $x_2, x_5, x_8$ $x_2 < 4.1$ Left child $x_1, x_4, x_8$ $x_1 < 6.0$ Right child $x_3, x_5, x_7$ $x_5 < 2.7$\" (rows=(('Node', 'Offered features', 'Winning split'), ('Root', '$x_2,…, header=True)","heading_features":"a Heading that says \"A Random Subset at Every Split\"","heading_samples":"a Heading that says \"Same Dataset, Different Training Samples\"","heading_trees":"a Heading that says \"Three Plausible Trees\"","lines_one":"a Line [red] drawn in tree_one (start=(0.0, 4.0), end=(8.0, 4.0))","lines_one_2":"a Line [red] drawn in tree_one (start=(5.8, 0.0), end=(5.8, 4.0))","lines_one_3":"a Line [red] drawn in tree_one (start=(2.3, 4.0), end=(2.3, 8.0))","lines_one_4":"a Line [red] drawn in tree_one (start=(2.3, 5.4), end=(6.6, 5.4))","lines_one_5":"a Line [red] drawn in tree_one (start=(6.6, 4.0), end=(6.6, 8.0))","lines_three":"a Line [blue] drawn in tree_three (start=(0.0, 4.5), end=(8.0, 4.5))","lines_three_2":"a Line [blue] drawn in tree_three (start=(6.2, 0.0), end=(6.2, 4.5))","lines_three_3":"a Line [blue] drawn in tree_three (start=(1.8, 4.5), end=(1.8, 8.0))","lines_three_4":"a Line [blue] drawn in tree_three (start=(1.8, 6.0), end=(5.0, 6.0))","lines_three_5":"a Line [blue] drawn in tree_three (start=(5.0, 4.5), end=(5.0, 8.0))","lines_two":"a Line [green] drawn in tree_two (start=(3.0, 0.0), end=(3.0, 8.0))","lines_two_2":"a Line [green] drawn in tree_two (start=(0.0, 3.4), end=(3.0, 3.4))","lines_two_3":"a Line [green] drawn in tree_two (start=(3.0, 4.6), end=(8.0, 4.6))","lines_two_4":"a Line [green] drawn in tree_two (start=(5.2, 0.0), end=(5.2, 4.6))","lines_two_5":"a Line [green] drawn in tree_two (start=(6.7, 4.6), end=(6.7, 8.0))","sample_law":"a Math [text] that says \"$D_b = upright(\"sample with replacement\")(D, thin n)$\"","tree_one":"an Axes (x_range=(0.0, 8.0), y_range=(0.0, 8.0), aspect=(1.0, 1.0))","tree_three":"an Axes (x_range=(0.0, 8.0), y_range=(0.0, 8.0), aspect=(1.0, 1.0))","tree_two":"an Axes (x_range=(0.0, 8.0), y_range=(0.0, 8.0), aspect=(1.0, 1.0))"},"beats":[{"start":509.9435833333333,"say":"A forest does not begin by cloning one fitted tree. It creates many related training problems. For each tree, draw n observations from the original n with replacement.","live":[],"does":[[509.9435833333333,"heading_samples is shown on the screen, written out."],[517.5365833333333,"sample_law is shown on the screen, written out."]]},{"start":522.2695833333333,"say":"With replacement means an observation can appear more than once. In the first sample, indices one and five are repeated, while three and six are absent. The sample still contains n rows.","live":["sample_law","heading_samples"],"does":[[527.7495833333334,"bootstrap is shown on the screen, written out."],[528.9685833333333,"bootstrap is shown on the screen, written out."],[530.7105833333334,"bootstrap (the \"$1,1,2,4,5,5,7,8$\" part) is emphasized."],[536.6895833333333,"bootstrap (the \"$1,1,2,4,5,5,7,8$\" part) is no longer emphasized."]]},{"start":537.2895833333333,"say":"A second bootstrap draw repeats different observations and omits different ones. A third does it again. Each tree therefore sees a perturbed empirical distribution, even though all draws came from the same dataset.","live":null,"does":[[537.7885833333333,"bootstrap is shown on the screen, written out."],[543.2575833333333,"bootstrap is shown on the screen, written out."]]},{"start":552.8665833333333,"say":"Fitting deep trees to these samples is ordinary bootstrap aggregation, or bagging. It creates diversity because a marginal observation may be duplicated in one sample and unavailable in another.","live":null,"does":[]},{"start":566.4580833333333,"say":"A random forest adds a second source of variation. At every node, it offers the split search only a random subset of the available features.","live":null,"does":[[576.5245833333333,"bootstrap is hidden from the screen — left the board."],[576.5245833333333,"heading_samples is hidden from the screen — left the board."],[576.5245833333333,"sample_law is hidden from the screen — left the board."]]},{"start":577.7245833333333,"say":"Here is one tree in an eight-feature problem. The formula records a fresh feature subset at node j. At the root, features two, five, and eight are offered, and feature two supplies the best available impurity reduction.","live":[],"does":[[577.7245833333333,"heading_features is shown on the screen, written out."],[579.6165833333333,"features is shown on the screen, written out."],[581.8225833333333,"feature_law is shown on the screen, written out."],[585.7705833333333,"features is shown on the screen, written out."],[589.8335833333333,"features (the \"$x_2 < 4.1$\" part) is indicated — a transient flash."]]},{"start":594.0790833333333,"say":"The left child gets a fresh draw, features one, four, and eight. The right child gets another draw, features three, five, and seven. Feature availability is local to a node.","live":["feature_law","heading_features"],"does":[[594.6825833333334,"features is shown on the screen, written out."],[595.5305833333333,"features (the \"column=2\" part) is emphasized."],[599.1765833333334,"features is shown on the screen, written out."],[606.2235833333333,"features (the \"column=2\" part) is no longer emphasized."]]},{"start":606.8235833333333,"say":"This restriction can force a tree to ignore the dominant predictor at a particular node. That sounds inefficient for one tree. Its purpose is to prevent every tree from making the same early decisions.","live":null,"does":[[607.3925833333333,"feature_law (the \"S_(b,j)\" part) is indicated — a transient flash."]]},{"start":620.3685833333333,"say":"Bootstrap sampling perturbs the observations. Feature sampling perturbs the available questions. Together they produce trees whose errors are less synchronized.","live":null,"does":[[630.9570833333333,"feature_law is hidden from the screen — left the board."],[630.9570833333333,"features is hidden from the screen — left the board."],[630.9570833333333,"heading_features is hidden from the screen — left the board."]]},{"start":632.1570833333333,"say":"These are three trees trained from the same original problem. Each diagram shows the axis-aligned boundaries produced by one bootstrap sample and one sequence of random feature offers.","live":[],"does":[[632.1570833333333,"heading_trees is shown on the screen, written out."],[632.8765833333333,"tree_one is shown on the screen, written out."],[632.8765833333333,"tree_two is shown on the screen, written out."],[632.8765833333333,"tree_three is shown on the screen, written out."],[639.4365833333334,"caption_one is shown on the screen, written out."],[639.6915833333334,"caption_two is shown on the screen, written out."],[640.2375833333333,"caption_three is shown on the screen, written out."]]},{"start":644.5405833333333,"say":"The first tree chooses a horizontal root and then several local vertical cuts. Its boundary is coherent within each rectangle, but jagged as a whole.","live":["tree_one","caption_one","tree_two","caption_two","tree_three","caption_three","heading_trees"],"does":[[645.2725833333334,"lines_one is shown on the screen, written out."],[645.3925833333333,"lines_one_2 is shown on the screen, written out."],[645.5125833333334,"lines_one_3 is shown on the screen, written out."],[645.6325833333333,"lines_one_4 is shown on the screen, written out."],[645.7525833333333,"lines_one_5 is shown on the screen, written out."]]},{"start":655.4850833333334,"say":"The second tree begins vertically because its sample and offered features differ. It partitions the same feature plane into a visibly different collection of rectangles.","live":["tree_one","caption_one","tree_two","caption_two","tree_three","caption_three","heading_trees","lines_one","lines_one_2","lines_one_3","lines_one_4","lines_one_5"],"does":[[656.0885833333334,"lines_two is shown on the screen, written out."],[656.2085833333333,"lines_two_2 is shown on the screen, written out."],[656.3285833333334,"lines_two_3 is shown on the screen, written out."],[656.4485833333333,"lines_two_4 is shown on the screen, written out."],[656.5685833333333,"lines_two_5 is shown on the screen, written out."]]},{"start":666.8015833333334,"say":"The third tree returns to a horizontal root, but its thresholds and deeper branches are different again. None of these trees is intended to be the final boundary.","live":["tree_one","caption_one","tree_two","caption_two","tree_three","caption_three","heading_trees","lines_one","lines_one_2","lines_one_3","lines_one_4","lines_one_5","lines_two","lines_two_2","lines_two_3","lines_two_4","lines_two_5"],"does":[[667.3585833333334,"lines_three is shown on the screen, written out."],[667.4785833333333,"lines_three_2 is shown on the screen, written out."],[667.5985833333334,"lines_three_3 is shown on the screen, written out."],[667.7185833333333,"lines_three_4 is shown on the screen, written out."],[667.8385833333333,"lines_three_5 is shown on the screen, written out."]]},{"start":677.1540833333333,"say":"For classification, each tree supplies class probabilities from its reached leaf, commonly the class proportions among that leaf's training samples. The forest averages those probabilities and then chooses a class.","live":["tree_one","caption_one","tree_two","caption_two","tree_three","caption_three","heading_trees","lines_one","lines_one_2","lines_one_3","lines_one_4","lines_one_5","lines_two","lines_two_2","lines_two_3","lines_two_4","lines_two_5","lines_three","lines_three_2","lines_three_3","lines_three_4","lines_three_5"],"does":[]},{"start":691.0005833333333,"say":"For regression, the same architecture averages numeric predictions. Bootstrap samples and random feature subsets still serve the same purpose: build individually flexible models whose mistakes are not identical.","live":null,"does":[[705.3645208333332,"caption_one is hidden from the screen — left the board."],[705.3645208333332,"caption_three is hidden from the screen — left the board."],[705.3645208333332,"caption_two is hidden from the screen — left the board."],[705.3645208333332,"heading_trees is hidden from the screen — left the board."],[705.3645208333332,"tree_one is hidden from the screen — left the board."],[705.3645208333332,"lines_one is hidden from the screen — tree_one left the board."],[705.3645208333332,"lines_one_2 is hidden from the screen — tree_one left the board."],[705.3645208333332,"lines_one_3 is hidden from the screen — tree_one left the board."],[705.3645208333332,"lines_one_4 is hidden from the screen — tree_one left the board."],[705.3645208333332,"lines_one_5 is hidden from the screen — tree_one left the board."],[705.3645208333332,"tree_three is hidden from the screen — left the board."],[705.3645208333332,"lines_three is hidden from the screen — tree_three left the board."],[705.3645208333332,"lines_three_2 is hidden from the screen — tree_three left the board."],[705.3645208333332,"lines_three_3 is hidden from the screen — tree_three left the board."],[705.3645208333332,"lines_three_4 is hidden from the screen — tree_three left the board."],[705.3645208333332,"lines_three_5 is hidden from the screen — tree_three left the board."],[705.3645208333332,"tree_two is hidden from the screen — left the board."],[705.3645208333332,"lines_two is hidden from the screen — tree_two left the board."],[705.3645208333332,"lines_two_2 is hidden from the screen — tree_two left the board."],[705.3645208333332,"lines_two_3 is hidden from the screen — tree_two left the board."],[705.3645208333332,"lines_two_4 is hidden from the screen — tree_two left the board."],[705.3645208333332,"lines_two_5 is hidden from the screen — tree_two left the board."]]}]},{"title":"Averaging High-Variance Trees","start":706.4061875,"end":944.6647083333332,"objects":{"blue_points":"a Point [blue] drawn in plane (location=(0.8, 1.2))","blue_points_2":"a Point [blue] drawn in plane (location=(1.5, 2.0))","blue_points_3":"a Point [blue] drawn in plane (location=(2.8, 1.0))","blue_points_4":"a Point [blue] drawn in plane (location=(3.6, 2.8))","blue_points_5":"a Point [blue] drawn in plane (location=(4.5, 1.5))","blue_points_6":"a Point [blue] drawn in plane (location=(5.6, 2.7))","blue_points_7":"a Point [blue] drawn in plane (location=(2.4, 5.5))","blue_points_8":"a Point [blue] drawn in plane (location=(6.8, 6.4))","correlated":"a Math [text] that says \"$upright(\"Var\")(overline(T)) = sigma^2 (rho + frac(1-rho, B))$\"","count":"a Math [text] that says \"$B = 1$\"","decision":"a Math [text] that says \"$hat(y)(x) = upright(\"arg max\")_k thin hat(p)_(B,k)(x)$\"","heading_boundary":"a Heading that says \"Average the Votes\"","heading_variance":"a Heading that says \"What Averaging Can and Cannot Remove\"","hundred_trees":"a Line [green] drawn in plane (start=(0.0, 4.7), end=(0.8, 4.7))","hundred_trees_10":"a Line [green] drawn in plane (start=(4.0, 4.5), end=(4.0, 4.4))","hundred_trees_11":"a Line [green] drawn in plane (start=(4.0, 4.4), end=(4.8, 4.4))","hundred_trees_12":"a Line [green] drawn in plane (start=(4.8, 4.4), end=(4.8, 4.3))","hundred_trees_13":"a Line [green] drawn in plane (start=(4.8, 4.3), end=(5.6, 4.3))","hundred_trees_14":"a Line [green] drawn in plane (start=(5.6, 4.3), end=(5.6, 4.2))","hundred_trees_15":"a Line [green] drawn in plane (start=(5.6, 4.2), end=(6.4, 4.2))","hundred_trees_16":"a Line [green] drawn in plane (start=(6.4, 4.2), end=(6.4, 4.3))","hundred_trees_17":"a Line [green] drawn in plane (start=(6.4, 4.3), end=(7.2, 4.3))","hundred_trees_18":"a Line [green] drawn in plane (start=(7.2, 4.3), end=(7.2, 4.5))","hundred_trees_19":"a Line [green] drawn in plane (start=(7.2, 4.5), end=(8.0, 4.5))","hundred_trees_2":"a Line [green] drawn in plane (start=(0.8, 4.7), end=(0.8, 4.9))","hundred_trees_3":"a Line [green] drawn in plane (start=(0.8, 4.9), end=(1.6, 4.9))","hundred_trees_4":"a Line [green] drawn in plane (start=(1.6, 4.9), end=(1.6, 5.0))","hundred_trees_5":"a Line [green] drawn in plane (start=(1.6, 5.0), end=(2.4, 5.0))","hundred_trees_6":"a Line [green] drawn in plane (start=(2.4, 5.0), end=(2.4, 4.8))","hundred_trees_7":"a Line [green] drawn in plane (start=(2.4, 4.8), end=(3.2, 4.8))","hundred_trees_8":"a Line [green] drawn in plane (start=(3.2, 4.8), end=(3.2, 4.5))","hundred_trees_9":"a Line [green] drawn in plane (start=(3.2, 4.5), end=(4.0, 4.5))","independent":"a Math [text] that says \"$upright(\"Var\")(overline(T)) = frac(sigma^2, B)$\"","limit":"a Math [text] that says \"$B arrow.r infinity: thin upright(\"Var\")(overline(T)) arrow.r rho sigma^2$\"","mixed_errors":"a Table [text] that says \"Tree $T_1$ $T_2$ $T_3$ $T_4$ $T_5$ Error $+1$ $-1$ $+1$ $-1$ $0$\" (rows=(('Tree', '$T_1$', '$T_2$', '$T_3$', '$T_4$', '$T_5$'), ('Error…)","mixed_label":"a Tex [text] that says \"Decorrelated errors\"","mixed_mean":"a Math [text] that says \"$upright(\"mean error\") = 0$\"","one_tree":"a Line [gray] drawn in plane (start=(0.0, 4.8), end=(1.2, 4.8))","one_tree_10":"a Line [gray] drawn in plane (start=(6.5, 3.6), end=(6.5, 5.2))","one_tree_11":"a Line [gray] drawn in plane (start=(6.5, 5.2), end=(8.0, 5.2))","one_tree_2":"a Line [gray] drawn in plane (start=(1.2, 4.8), end=(1.2, 6.2))","one_tree_3":"a Line [gray] drawn in plane (start=(1.2, 6.2), end=(2.3, 6.2))","one_tree_4":"a Line [gray] drawn in plane (start=(2.3, 6.2), end=(2.3, 3.3))","one_tree_5":"a Line [gray] drawn in plane (start=(2.3, 3.3), end=(3.6, 3.3))","one_tree_6":"a Line [gray] drawn in plane (start=(3.6, 3.3), end=(3.6, 5.7))","one_tree_7":"a Line [gray] drawn in plane (start=(3.6, 5.7), end=(5.1, 5.7))","one_tree_8":"a Line [gray] drawn in plane (start=(5.1, 5.7), end=(5.1, 3.6))","one_tree_9":"a Line [gray] drawn in plane (start=(5.1, 3.6), end=(6.5, 3.6))","plane":"an Axes (x_range=(0.0, 8.0), y_range=(0.0, 8.0), aspect=(1.0, 1.0))","point":"a Point [yellow] drawn in plane (location=(2.5, 4.8))","point_2":"a Point [yellow] drawn in plane (location=(2.7, 4.8))","red_points":"a Point [red] drawn in plane (location=(0.9, 6.3))","red_points_2":"a Point [red] drawn in plane (location=(1.8, 4.8))","red_points_3":"a Point [red] drawn in plane (location=(3.3, 6.7))","red_points_4":"a Point [red] drawn in plane (location=(4.2, 5.2))","red_points_5":"a Point [red] drawn in plane (location=(5.4, 6.1))","red_points_6":"a Point [red] drawn in plane (location=(7.2, 4.8))","red_points_7":"a Point [red] drawn in plane (location=(2.2, 2.8))","red_points_8":"a Point [red] drawn in plane (location=(6.5, 1.2))","same_errors":"a Table [text] that says \"Tree $T_1$ $T_2$ $T_3$ $T_4$ $T_5$ Error $+1$ $+1$ $+1$ $+1$ $+1$\" (rows=(('Tree', '$T_1$', '$T_2$', '$T_3$', '$T_4$', '$T_5$'), ('Error…)","same_label":"a Tex [text] that says \"Highly correlated errors\"","same_mean":"a Math [text] that says \"$upright(\"mean error\") = +1$\"","ten_trees":"a Line [yellow] drawn in plane (start=(0.0, 4.9), end=(1.5, 4.9))","ten_trees_10":"a Line [yellow] drawn in plane (start=(6.8, 4.1), end=(6.8, 4.6))","ten_trees_11":"a Line [yellow] drawn in plane (start=(6.8, 4.6), end=(8.0, 4.6))","ten_trees_2":"a Line [yellow] drawn in plane (start=(1.5, 4.9), end=(1.5, 5.3))","ten_trees_3":"a Line [yellow] drawn in plane (start=(1.5, 5.3), end=(2.7, 5.3))","ten_trees_4":"a Line [yellow] drawn in plane (start=(2.7, 5.3), end=(2.7, 4.2))","ten_trees_5":"a Line [yellow] drawn in plane (start=(2.7, 4.2), end=(4.1, 4.2))","ten_trees_6":"a Line [yellow] drawn in plane (start=(4.1, 4.2), end=(4.1, 4.7))","ten_trees_7":"a Line [yellow] drawn in plane (start=(4.1, 4.7), end=(5.5, 4.7))","ten_trees_8":"a Line [yellow] drawn in plane (start=(5.5, 4.7), end=(5.5, 4.1))","ten_trees_9":"a Line [yellow] drawn in plane (start=(5.5, 4.1), end=(6.8, 4.1))","vote":"a Math [text] that says \"$hat(p)_B(x) = frac(1,B) sum_(b=1)^B hat(p)_b(x)$\""},"beats":[{"start":706.4061875,"say":"Return to the labelled plane. One deep tree supplies one jagged boundary. It reacts strongly to the particular observations and random choices that shaped its branches.","live":[],"does":[[706.4061875,"heading_boundary is shown on the screen, written out."],[706.4061875,"plane is shown on the screen, written out."],[706.4061875,"blue_points is shown on the screen, written out."],[706.4461875,"blue_points_2 is shown on the screen, written out."],[706.4861875,"blue_points_3 is shown on the screen, written out."],[706.5261875,"blue_points_4 is shown on the screen, written out."],[706.5661875,"blue_points_5 is shown on the screen, written out."],[706.6061875,"blue_points_6 is shown on the screen, written out."],[706.6461875,"blue_points_7 is shown on the screen, written out."],[706.6861875,"blue_points_8 is shown on the screen, written out."],[706.7261875,"red_points is shown on the screen, written out."],[706.7661875,"red_points_2 is shown on the screen, written out."],[706.8061875,"red_points_3 is shown on the screen, written out."],[706.8461875,"red_points_4 is shown on the screen, written out."],[706.8861875,"red_points_5 is shown on the screen, written out."],[706.9261875,"red_points_6 is shown on the screen, written out."],[706.9661874999999,"red_points_7 is shown on the screen, written out."],[707.0061875,"red_points_8 is shown on the screen, written out."],[708.6821874999999,"plane moves to a new place on the board."],[708.6821874999999,"vote is shown on the screen, written out."],[708.6821874999999,"count is shown on the screen, written out."],[710.7371875,"one_tree is shown on the screen, written out."],[710.7971875,"one_tree_2 is shown on the screen, written out."],[710.8571875,"one_tree_3 is shown on the screen, written out."],[710.9171875,"one_tree_4 is shown on the screen, written out."],[710.9771875,"one_tree_5 is shown on the screen, written out."],[711.0371875,"one_tree_6 is shown on the screen, written out."],[711.0971875,"one_tree_7 is shown on the screen, written out."],[711.1311875,"decision is shown on the screen, written out."],[711.1571875,"one_tree_8 is shown on the screen, written out."],[711.2171875,"one_tree_9 is shown on the screen, written out."],[711.2771875,"one_tree_10 is shown on the screen, written out."],[711.3371875,"one_tree_11 is shown on the screen, written out."]]},{"start":718.2911875,"say":"For each query point x, tree b supplies a class probability. The forest averages those probabilities. Its displayed boundary is where the winning averaged class changes.","live":["vote","decision","count","plane","heading_boundary","blue_points","blue_points_2","blue_points_3","blue_points_4","blue_points_5","blue_points_6","blue_points_7","blue_points_8","red_points","red_points_2","red_points_3","red_points_4","red_points_5","red_points_6","red_points_7","red_points_8","one_tree","one_tree_2","one_tree_3","one_tree_4","one_tree_5","one_tree_6","one_tree_7","one_tree_8","one_tree_9","one_tree_10","one_tree_11"],"does":[[720.5321875,"vote (the \"hat(p)_b(x)\" part) is emphasized."],[724.2351874999999,"vote (the \"frac(1,B) sum_(b=1)^B\" part) is emphasized."],[724.2351874999999,"vote (the \"hat(p)_b(x)\" part) is no longer emphasized."],[727.4511875,"vote (the \"frac(1,B) sum_(b=1)^B\" part) is no longer emphasized."]]},{"start":731.1971874999999,"say":"Add trees until B equals ten. Each individual boundary is still made of rectangular steps, but their idiosyncratic notches occur at different places. Majority support removes many notches that only one tree wanted.","live":null,"does":[[731.5451875,"one_tree is hidden from the screen."],[731.5451875,"one_tree_2 is hidden from the screen."],[731.5451875,"one_tree_3 is hidden from the screen."],[731.5451875,"one_tree_4 is hidden from the screen."],[731.5451875,"one_tree_5 is hidden from the screen."],[731.5451875,"one_tree_6 is hidden from the screen."],[731.5451875,"one_tree_7 is hidden from the screen."],[731.5451875,"one_tree_8 is hidden from the screen."],[731.5451875,"one_tree_9 is hidden from the screen."],[731.5451875,"one_tree_10 is hidden from the screen."],[731.5451875,"one_tree_11 is hidden from the screen."],[733.3101875,"count becomes \"$B = 10$\"."],[733.3101875,"ten_trees is shown on the screen, written out."],[733.3701874999999,"ten_trees_2 is shown on the screen, written out."],[733.4301875,"ten_trees_3 is shown on the screen, written out."],[733.4901874999999,"ten_trees_4 is shown on the screen, written out."],[733.5501875,"ten_trees_5 is shown on the screen, written out."],[733.6101874999999,"ten_trees_6 is shown on the screen, written out."],[733.6701875,"ten_trees_7 is shown on the screen, written out."],[733.7301874999999,"ten_trees_8 is shown on the screen, written out."],[733.7901875,"ten_trees_9 is shown on the screen, written out."],[733.8501875,"ten_trees_10 is shown on the screen, written out."],[733.9101875,"ten_trees_11 is shown on the screen, written out."]]},{"start":746.8091875,"say":"Now average one hundred trees. The aggregate is still a staircase if we inspect it finely enough, because every contributor is a tree. At this scale, however, it follows the broad class structure rather than every isolated observation.","live":["vote","decision","count","plane","heading_boundary","blue_points","blue_points_2","blue_points_3","blue_points_4","blue_points_5","blue_points_6","blue_points_7","blue_points_8","red_points","red_points_2","red_points_3","red_points_4","red_points_5","red_points_6","red_points_7","red_points_8","ten_trees","ten_trees_2","ten_trees_3","ten_trees_4","ten_trees_5","ten_trees_6","ten_trees_7","ten_trees_8","ten_trees_9","ten_trees_10","ten_trees_11"],"does":[[748.5161875,"count becomes \"$B = 100$\"."],[748.5161875,"ten_trees is hidden from the screen."],[748.5161875,"ten_trees_2 is hidden from the screen."],[748.5161875,"ten_trees_3 is hidden from the screen."],[748.5161875,"ten_trees_4 is hidden from the screen."],[748.5161875,"ten_trees_5 is hidden from the screen."],[748.5161875,"ten_trees_6 is hidden from the screen."],[748.5161875,"ten_trees_7 is hidden from the screen."],[748.5161875,"ten_trees_8 is hidden from the screen."],[748.5161875,"ten_trees_9 is hidden from the screen."],[748.5161875,"ten_trees_10 is hidden from the screen."],[748.5161875,"ten_trees_11 is hidden from the screen."],[748.5161875,"hundred_trees is shown on the screen, written out."],[748.5561875,"hundred_trees_2 is shown on the screen, written out."],[748.5961875,"hundred_trees_3 is shown on the screen, written out."],[748.6361875,"hundred_trees_4 is shown on the screen, written out."],[748.6761875,"hundred_trees_5 is shown on the screen, written out."],[748.7161874999999,"hundred_trees_6 is shown on the screen, written out."],[748.7561875,"hundred_trees_7 is shown on the screen, written out."],[748.7961875,"hundred_trees_8 is shown on the screen, written out."],[748.8361874999999,"hundred_trees_9 is shown on the screen, written out."],[748.8761875,"hundred_trees_10 is shown on the screen, written out."],[748.9161875,"hundred_trees_11 is shown on the screen, written out."],[748.9561874999999,"hundred_trees_12 is shown on the screen, written out."],[748.9961875,"hundred_trees_13 is shown on the screen, written out."],[749.0361875,"hundred_trees_14 is shown on the screen, written out."],[749.0761875,"hundred_trees_15 is shown on the screen, written out."],[749.1161875,"hundred_trees_16 is shown on the screen, written out."],[749.1561875,"hundred_trees_17 is shown on the screen, written out."],[749.1961875,"hundred_trees_18 is shown on the screen, written out."],[749.2361875,"hundred_trees_19 is shown on the screen, written out."]]},{"start":762.5596875,"say":"Calling this boundary smoother does not mean that a forest fits a smooth analytic function. It means the averaged prediction varies more stably across nearby points and fewer decisions depend on one tree's narrow rectangular accident.","live":["vote","decision","count","plane","heading_boundary","blue_points","blue_points_2","blue_points_3","blue_points_4","blue_points_5","blue_points_6","blue_points_7","blue_points_8","red_points","red_points_2","red_points_3","red_points_4","red_points_5","red_points_6","red_points_7","red_points_8","hundred_trees","hundred_trees_2","hundred_trees_3","hundred_trees_4","hundred_trees_5","hundred_trees_6","hundred_trees_7","hundred_trees_8","hundred_trees_9","hundred_trees_10","hundred_trees_11","hundred_trees_12","hundred_trees_13","hundred_trees_14","hundred_trees_15","hundred_trees_16","hundred_trees_17","hundred_trees_18","hundred_trees_19"],"does":[[770.9531875,"point is shown on the screen, grown."],[770.9531875,"point_2 is shown on the screen, grown."],[772.9531875,"point is hidden from the screen."],[772.9531875,"point_2 is hidden from the screen."]]},{"start":776.8821875,"say":"Why should averaging unstable models work? Imagine each tree prediction as a useful signal plus a zero-mean fitting error. Averaging keeps the shared signal. Errors that point in different directions can cancel.","live":null,"does":[[777.8921875,"vote (the \"frac(1,B)\" part) is indicated — a transient flash."]]},{"start":792.3196875,"say":"If the tree errors were independent and each had variance sigma squared, the mean of B trees would have variance sigma squared over B. Double the number of trees and this variance contribution halves.","live":null,"does":[[794.0731875,"independent is shown on the screen, written out."],[799.6571875,"independent (the \"frac(sigma^2, B)\" part) is indicated — a transient flash."]]},{"start":806.3636875,"say":"Real tree errors are not independent. They use the same original dataset, and strong predictors can make them discover similar branches. Let rho represent their average pairwise error correlation.","live":["vote","decision","count","independent","plane","heading_boundary","blue_points","blue_points_2","blue_points_3","blue_points_4","blue_points_5","blue_points_6","blue_points_7","blue_points_8","red_points","red_points_2","red_points_3","red_points_4","red_points_5","red_points_6","red_points_7","red_points_8","hundred_trees","hundred_trees_2","hundred_trees_3","hundred_trees_4","hundred_trees_5","hundred_trees_6","hundred_trees_7","hundred_trees_8","hundred_trees_9","hundred_trees_10","hundred_trees_11","hundred_trees_12","hundred_trees_13","hundred_trees_14","hundred_trees_15","hundred_trees_16","hundred_trees_17","hundred_trees_18","hundred_trees_19"],"does":[[819.4826875,"count is hidden from the screen — left the board."],[819.4826875,"decision is hidden from the screen — left the board."],[819.4826875,"heading_boundary is hidden from the screen — left the board."],[819.4826875,"independent is hidden from the screen — left the board."],[819.4826875,"plane is hidden from the screen — left the board."],[819.4826875,"blue_points is hidden from the screen — plane left the board."],[819.4826875,"blue_points_2 is hidden from the screen — plane left the board."],[819.4826875,"blue_points_3 is hidden from the screen — plane left the board."],[819.4826875,"blue_points_4 is hidden from the screen — plane left the board."],[819.4826875,"blue_points_5 is hidden from the screen — plane left the board."],[819.4826875,"blue_points_6 is hidden from the screen — plane left the board."],[819.4826875,"blue_points_7 is hidden from the screen — plane left the board."],[819.4826875,"blue_points_8 is hidden from the screen — plane left the board."],[819.4826875,"red_points is hidden from the screen — plane left the board."],[819.4826875,"red_points_2 is hidden from the screen — plane left the board."],[819.4826875,"red_points_3 is hidden from the screen — plane left the board."],[819.4826875,"red_points_4 is hidden from the screen — plane left the board."],[819.4826875,"red_points_5 is hidden from the screen — plane left the board."],[819.4826875,"red_points_6 is hidden from the screen — plane left the board."],[819.4826875,"red_points_7 is hidden from the screen — plane left the board."],[819.4826875,"red_points_8 is hidden from the screen — plane left the board."],[819.4826875,"hundred_trees is hidden from the screen — plane left the board."],[819.4826875,"hundred_trees_2 is hidden from the screen — plane left the board."],[819.4826875,"hundred_trees_3 is hidden from the screen — plane left the board."],[819.4826875,"hundred_trees_4 is hidden from the screen — plane left the board."],[819.4826875,"hundred_trees_5 is hidden from the screen — plane left the board."],[819.4826875,"hundred_trees_6 is hidden from the screen — plane left the board."],[819.4826875,"hundred_trees_7 is hidden from the screen — plane left the board."],[819.4826875,"hundred_trees_8 is hidden from the screen — plane left the board."],[819.4826875,"hundred_trees_9 is hidden from the screen — plane left the board."],[819.4826875,"hundred_trees_10 is hidden from the screen — plane left the board."],[819.4826875,"hundred_trees_11 is hidden from the screen — plane left the board."],[819.4826875,"hundred_trees_12 is hidden from the screen — plane left the board."],[819.4826875,"hundred_trees_13 is hidden from the screen — plane left the board."],[819.4826875,"hundred_trees_14 is hidden from the screen — plane left the board."],[819.4826875,"hundred_trees_15 is hidden from the screen — plane left the board."],[819.4826875,"hundred_trees_16 is hidden from the screen — plane left the board."],[819.4826875,"hundred_trees_17 is hidden from the screen — plane left the board."],[819.4826875,"hundred_trees_18 is hidden from the screen — plane left the board."],[819.4826875,"hundred_trees_19 is hidden from the screen — plane left the board."],[819.4826875,"vote is hidden from the screen — left the board."]]},{"start":820.6826874999999,"say":"Here is the extreme problem. Five trees all make an error of plus one on the same case. Their average error is still plus one. Repetition did not remove a shared mistake.","live":[],"does":[[820.6826874999999,"heading_variance is shown on the screen, written out."],[823.2601875,"same_label is shown on the screen, written out."],[823.6311875,"same_errors is shown on the screen, written out."],[825.5241874999999,"same_errors is shown on the screen, written out."],[828.2521875,"same_mean is shown on the screen, written out."]]},{"start":834.3786875,"say":"On the right, errors differ across trees. Positive and negative errors offset, and their average is zero in this small illustration. Diversity is useful when it concerns errors, not merely visual differences between diagrams.","live":["same_label","same_mean","heading_variance"],"does":[[835.0871875,"mixed_label is shown on the screen, written out."],[837.0371875,"mixed_errors is shown on the screen, written out."],[838.3141875,"mixed_errors is shown on the screen, written out."],[842.1111874999999,"mixed_mean is shown on the screen, written out."],[851.1431875,"mixed_errors is hidden from the screen — left the board."],[851.1431875,"mixed_label is hidden from the screen — left the board."],[851.1431875,"mixed_mean is hidden from the screen — left the board."],[851.1431875,"same_errors is hidden from the screen — left the board."],[851.1431875,"same_label is hidden from the screen — left the board."],[851.1431875,"same_mean is hidden from the screen — left the board."],[851.1431875,"independent is shown on the screen, faded in — cast on this board again."],[851.1431875,"vote is shown on the screen, faded in — cast on this board again."]]},{"start":851.7431875,"say":"With equal variance and average correlation rho, the ensemble variance is sigma squared times rho plus one minus rho over B. The second part shrinks as trees are added. The correlated part does not.","live":["vote","independent","heading_variance"],"does":[[853.8211875,"correlated is shown on the screen, written out."],[863.5501875,"correlated (the \"frac(1-rho, B)\" part) is emphasized."],[866.0581875,"correlated (the \"frac(1-rho, B)\" part) is no longer emphasized."],[866.0581875,"correlated (the \"rho\" part) is emphasized."],[867.9391875,"correlated (the \"rho\" part) is no longer emphasized."]]},{"start":868.5391875,"say":"Take the number of trees toward infinity. The variance approaches rho sigma squared, not zero. If rho is close to one, a huge forest behaves like repeated copies of one unstable tree.","live":["vote","independent","heading_variance","correlated"],"does":[[870.3851875,"limit is shown on the screen, written out."],[873.0671875,"limit (the \"rho sigma^2\" part) is indicated — a transient flash."]]},{"start":882.8621875,"say":"This is why decorrelation is essential. Bootstrap samples alter which observations drive the branches. Random feature subsets prevent one dominant predictor from forcing the same root and early splits in every tree.","live":["vote","independent","heading_variance","correlated","limit"],"does":[[883.7441875,"correlated (the \"rho\" part) is indicated — a transient flash."]]},{"start":897.9396875,"say":"There is a trade-off. Offering fewer features can weaken each individual tree by denying it useful predictors. But if that loss is modest and the correlation falls substantially, the average can generalize better.","live":null,"does":[[902.0151874999999,"vote (the \"hat(p)_b(x)\" part) is indicated — a transient flash."]]},{"start":912.7266875,"say":"Adding trees mainly reduces Monte Carlo noise in the fitted ensemble. It does not repair severe bias, leakage, bad labels, a shifted deployment population, or a feature set that contains no useful signal.","live":null,"does":[[913.2841874999999,"independent (the \"B\" part) is indicated — a transient flash."]]},{"start":927.2471875,"say":"The forest's advantage is therefore specific. Deep trees provide flexible, high-variance base predictions. Resampling and feature randomness make their errors less alike. Averaging then removes the part of the variance that is not shared.","live":null,"does":[[942.3511874999999,"A box is drawn around correlated."],[943.6230416666666,"correlated is hidden from the screen — left the board."],[943.6230416666666,"heading_variance is hidden from the screen — left the board."],[943.6230416666666,"independent is hidden from the screen — left the board."],[943.6230416666666,"limit is hidden from the screen — left the board."],[943.6230416666666,"vote is hidden from the screen — left the board."]]}]},{"title":"From the Idea to the Library","start":944.6647083333332,"end":1200.1914375,"objects":{"diagnostics":"a Table [text] that says \"Term Practical reading Out-of-bag score Predictions from trees that omitted each row Feature importance A summary to investigate, not a causal effect Permutation importance Loss after disrupting one feature Class weight A change to fitting…\" (rows=(('Term', 'Practical reading'), ('Out-of-bag score', 'Predictio…, header=True)","final_law":"a Math [text] that says \"$upright(\"strong trees\") + upright(\"low error correlation\") + upright(\"averaging\")$\"","forest_controls":"a Table [text] that says \"Forest control Effect `bootstrap` Observation resampling `max_features` Correlation and tree strength `n_estimators` Monte Carlo stability `n_jobs` Compute, not statistics\" (rows=(('Forest control', 'Effect'), ('`bootstrap`', 'Observation res…, header=True)","heading_close":"a Heading that says \"What to Carry Away\"","heading_controls":"a Heading that says \"Translate the Picture into Parameters\"","heading_diagnostics":"a Heading that says \"Diagnostics with Precise Meanings\"","heading_workflow":"a Heading that says \"A Practical Fitting Sequence\"","plane":"an Axes (x_range=(0.0, 8.0), y_range=(0.0, 8.0), aspect=(1.0, 1.0))","point":"a Point [yellow] drawn in plane (location=(5.4, 4.5))","probability_note":"a Panel that says \"A forest probability is an average of leaf-level estimates. It may rank well without being calibrated well.\"","simple_boundary":"a Line [green] drawn in plane (start=(0.0, 4.4), end=(3.0, 4.4))","simple_boundary_2":"a Line [green] drawn in plane (start=(3.0, 4.4), end=(3.0, 4.7))","simple_boundary_3":"a Line [green] drawn in plane (start=(3.0, 4.7), end=(5.5, 4.7))","simple_boundary_4":"a Line [green] drawn in plane (start=(5.5, 4.7), end=(5.5, 4.2))","simple_boundary_5":"a Line [green] drawn in plane (start=(5.5, 4.2), end=(8.0, 4.2))","steps":"a Block [text] that says \"Define the prediction target, split strategy, and evaluation metric before fitting. Fit a small tree first; inspect depth, leaf sizes, impurity gains, and obvious leakage. Tune structural regularization with validation data, not training p…\"","train_points":"a Point [blue] drawn in plane (location=(1.0, 1.2))","train_points_10":"a Point [red] drawn in plane (location=(7.0, 6.2))","train_points_2":"a Point [blue] drawn in plane (location=(2.0, 2.4))","train_points_3":"a Point [blue] drawn in plane (location=(3.4, 1.7))","train_points_4":"a Point [blue] drawn in plane (location=(4.8, 2.6))","train_points_5":"a Point [blue] drawn in plane (location=(6.2, 1.8))","train_points_6":"a Point [red] drawn in plane (location=(1.2, 6.1))","train_points_7":"a Point [red] drawn in plane (location=(2.5, 5.4))","train_points_8":"a Point [red] drawn in plane (location=(4.1, 6.5))","train_points_9":"a Point [red] drawn in plane (location=(5.8, 5.2))","tree_controls":"a Table [text] that says \"Tree control Effect `max_depth`, `max_leaf_nodes` Global complexity `min_samples_leaf` Small-region stability `min_impurity_decrease` Required local gain `ccp_alpha` Cost-complexity subtree\" (rows=(('Tree control', 'Effect'), ('`max_depth`, `max_leaf_nodes`', …, header=True)"},"beats":[{"start":944.6647083333332,"say":"Here is a practical sequence that preserves the logic we have developed. Start with the prediction problem, not the estimator. Fix the target, evaluation unit, split strategy, and metric before fitting.","live":[],"does":[[944.6647083333332,"heading_workflow is shown on the screen, written out."],[944.6647083333332,"plane is shown on the screen, written out."],[944.6647083333332,"train_points is shown on the screen, written out."],[944.7147083333332,"train_points_2 is shown on the screen, written out."],[944.7647083333333,"train_points_3 is shown on the screen, written out."],[944.8147083333332,"train_points_4 is shown on the screen, written out."],[944.8647083333333,"train_points_5 is shown on the screen, written out."],[944.9147083333332,"train_points_6 is shown on the screen, written out."],[944.9647083333332,"train_points_7 is shown on the screen, written out."],[945.0147083333333,"train_points_8 is shown on the screen, written out."],[945.0647083333332,"train_points_9 is shown on the screen, written out."],[945.1147083333333,"train_points_10 is shown on the screen, written out."],[945.7217083333333,"steps is shown on the screen, written out."],[953.4417083333333,"steps (the \"Define the prediction target\" part) is emphasized."],[958.4342083333332,"steps (the \"Define the prediction target\" part) is no longer emphasized."]]},{"start":959.0342083333333,"say":"A random row split is wrong whenever rows from the same customer, patient, device, location, or future time can leak information across folds. Group and temporal structure belong in the validation design.","live":["steps","plane","heading_workflow","train_points","train_points_2","train_points_3","train_points_4","train_points_5","train_points_6","train_points_7","train_points_8","train_points_9","train_points_10"],"does":[[960.3807083333332,"steps (the \"split strategy\" part) is emphasized."],[973.1987083333332,"steps (the \"split strategy\" part) is no longer emphasized."]]},{"start":973.7987083333333,"say":"Fit a small decision tree before the forest. Its splits expose coding mistakes, target proxies, implausible thresholds, and feature interactions that a large ensemble can conceal behind a good aggregate score.","live":null,"does":[[974.4257083333332,"steps (the \"Fit a small tree first\" part) is emphasized."],[977.3867083333332,"simple_boundary is shown on the screen, written out."],[977.4667083333333,"simple_boundary_2 is shown on the screen, written out."],[977.5467083333332,"simple_boundary_3 is shown on the screen, written out."],[977.6267083333332,"simple_boundary_4 is shown on the screen, written out."],[977.7067083333333,"simple_boundary_5 is shown on the screen, written out."],[987.3712083333332,"steps (the \"Fit a small tree first\" part) is no longer emphasized."]]},{"start":987.9712083333333,"say":"Then tune tree structure against validation performance. Maximum depth limits path length. Minimum samples per leaf demands evidence in each terminal region. Cost-complexity alpha removes weak fitted branches.","live":["steps","plane","heading_workflow","train_points","train_points_2","train_points_3","train_points_4","train_points_5","train_points_6","train_points_7","train_points_8","train_points_9","train_points_10","simple_boundary","simple_boundary_2","simple_boundary_3","simple_boundary_4","simple_boundary_5"],"does":[[988.7607083333332,"steps (the \"Tune structural regularization\" part) is emphasized."],[998.6867083333333,"point is shown on the screen, grown."],[1000.6867083333333,"point is hidden from the screen."],[1003.9812083333333,"steps (the \"Tune structural regularization\" part) is no longer emphasized."]]},{"start":1004.5812083333333,"say":"For a forest, raise the number of estimators until the validation metric and predictions stabilize. More trees usually increase compute rather than overfitting in the familiar single-tree sense, but returns become negligible.","live":null,"does":[[1005.9977083333332,"steps (the \"increase tree count\" part) is emphasized."],[1018.4897083333333,"steps (the \"increase tree count\" part) is no longer emphasized."]]},{"start":1019.0897083333332,"say":"Finally inspect more than one aggregate score. Check important subgroups, threshold-sensitive decisions, probability calibration, drift, and the cost of the errors the model actually makes.","live":null,"does":[[1020.1347083333333,"steps (the \"Inspect subgroup errors\" part) is emphasized."],[1031.7562083333332,"heading_workflow is hidden from the screen — left the board."],[1031.7562083333332,"plane is hidden from the screen — left the board."],[1031.7562083333332,"train_points is hidden from the screen — plane left the board."],[1031.7562083333332,"train_points_2 is hidden from the screen — plane left the board."],[1031.7562083333332,"train_points_3 is hidden from the screen — plane left the board."],[1031.7562083333332,"train_points_4 is hidden from the screen — plane left the board."],[1031.7562083333332,"train_points_5 is hidden from the screen — plane left the board."],[1031.7562083333332,"train_points_6 is hidden from the screen — plane left the board."],[1031.7562083333332,"train_points_7 is hidden from the screen — plane left the board."],[1031.7562083333332,"train_points_8 is hidden from the screen — plane left the board."],[1031.7562083333332,"train_points_9 is hidden from the screen — plane left the board."],[1031.7562083333332,"train_points_10 is hidden from the screen — plane left the board."],[1031.7562083333332,"simple_boundary is hidden from the screen — plane left the board."],[1031.7562083333332,"simple_boundary_2 is hidden from the screen — plane left the board."],[1031.7562083333332,"simple_boundary_3 is hidden from the screen — plane left the board."],[1031.7562083333332,"simple_boundary_4 is hidden from the screen — plane left the board."],[1031.7562083333332,"simple_boundary_5 is hidden from the screen — plane left the board."],[1031.7562083333332,"steps is hidden from the screen — left the board."],[1031.7562083333332,"steps (the \"Inspect subgroup errors\" part) is no longer emphasized."]]},{"start":1032.3562083333331,"say":"Maximum depth and maximum leaf nodes limit global tree size. They are coarse controls on how many successive rectangles the tree may create.","live":[],"does":[[1032.3562083333331,"heading_controls is shown on the screen, written out."],[1032.7047083333332,"tree_controls is shown on the screen, written out."],[1032.7047083333332,"tree_controls is shown on the screen, written out."],[1032.7047083333332,"tree_controls (the \"`max_depth`, `max_leaf_nodes`\" part) is indicated — a transient flash."]]},{"start":1042.2557083333331,"say":"Minimum samples per leaf directly attacks tiny regions. Minimum impurity decrease requires a split to earn enough local gain before it is allowed.","live":["heading_controls"],"does":[[1042.6037083333333,"tree_controls is shown on the screen, written out."],[1044.9147083333332,"tree_controls (the \"`min_samples_leaf`\" part) is indicated — a transient flash."],[1046.8297083333332,"tree_controls is shown on the screen, written out."]]},{"start":1052.7472083333332,"say":"Cost-complexity alpha names the post-pruning penalty we used earlier. Its numerical scale depends on the data, weights, impurity, and implementation, so tune it through the supplied pruning path rather than by folklore.","live":null,"does":[[1054.1867083333332,"tree_controls is shown on the screen, written out."],[1056.0097083333333,"tree_controls (the \"`ccp_alpha`\" part) is indicated — a transient flash."]]},{"start":1068.4172083333333,"say":"Bootstrap turns observation resampling on or off. Max features controls the random candidate subset at each split. Lowering max features usually reduces correlation, but can also weaken individual trees.","live":null,"does":[[1068.7657083333331,"forest_controls is shown on the screen, written out."],[1068.7657083333331,"forest_controls is shown on the screen, written out."],[1072.8867083333332,"forest_controls is shown on the screen, written out."],[1079.6317083333333,"forest_controls (the \"`max_features`\" part) is indicated — a transient flash."]]},{"start":1083.6452083333334,"say":"The number of estimators is the ensemble size B. Parallel-job settings change wall-clock cost, not the fitted statistical objective. Do not confuse faster execution with stronger regularization.","live":null,"does":[[1084.1557083333332,"forest_controls is shown on the screen, written out."],[1087.5697083333332,"forest_controls is shown on the screen, written out."],[1090.6577083333332,"forest_controls (the \"Compute, not statistics\" part) is indicated — a transient flash."],[1097.5422083333333,"forest_controls is hidden from the screen — left the board."],[1097.5422083333333,"heading_controls is hidden from the screen — left the board."],[1097.5422083333333,"tree_controls is hidden from the screen — left the board."]]},{"start":1098.1422083333332,"say":"Classification probabilities deserve separate validation. A forest averages leaf class proportions, which can rank cases very effectively while remaining overconfident or underconfident as probabilities.","live":[],"does":[[1098.1422083333332,"heading_diagnostics is shown on the screen, written out."],[1099.3147083333333,"probability_note is shown on the screen, written out."],[1107.6037083333333,"probability_note (the \"calibrated\" part) is indicated — a transient flash."]]},{"start":1111.8032083333333,"say":"Out-of-bag predictions use, for each training row, only trees whose bootstrap samples omitted that row. They provide a convenient internal diagnostic, but they do not override grouped, temporal, or external validation requirements.","live":["probability_note","heading_diagnostics"],"does":[[1112.2097083333333,"diagnostics is shown on the screen, written out."],[1112.2097083333333,"diagnostics is shown on the screen, written out."],[1117.1667083333332,"diagnostics (the \"omitted each row\" part) is indicated — a transient flash."]]},{"start":1127.3917083333333,"say":"Impurity-based feature importance summarizes how fitted splits used a feature. It can favour variables with many available thresholds and divide credit awkwardly among correlated predictors. It is not a causal effect.","live":null,"does":[[1128.7147083333334,"diagnostics is shown on the screen, written out."],[1140.1047083333333,"diagnostics (the \"not a causal effect\" part) is indicated — a transient flash."]]},{"start":1142.2837083333334,"say":"Permutation importance asks how predictive performance changes after one feature is disrupted. It is often closer to the operational question, but correlated features can substitute for one another and hide each other's importance.","live":null,"does":[[1142.6897083333333,"diagnostics is shown on the screen, written out."],[1144.8607083333332,"diagnostics (the \"Loss\" part) is indicated — a transient flash."]]},{"start":1156.8737083333333,"say":"Class weights alter the fitting objective. They do not decide which deployment metric matters. A random seed makes the stochastic fit reproducible; it does not make sampling uncertainty disappear.","live":null,"does":[[1157.1757083333332,"diagnostics is shown on the screen, written out."],[1162.8877083333332,"diagnostics is shown on the screen, written out."],[1169.0292083333334,"diagnostics is hidden from the screen — left the board."],[1169.0292083333334,"heading_diagnostics is hidden from the screen — left the board."],[1169.0292083333334,"probability_note is hidden from the screen — left the board."]]},{"start":1170.2292083333332,"say":"The whole lecture can be compressed into three ingredients. Trees must be strong enough to capture useful structure. Their errors must be sufficiently decorrelated. Averaging then reduces the unshared variance.","live":[],"does":[[1170.2292083333332,"heading_close is shown on the screen, written out."],[1172.3657083333333,"final_law is shown on the screen, written out."],[1174.7107083333333,"final_law (the \"upright(\"strong trees\")\" part) is indicated — a transient flash."],[1178.5887083333332,"final_law (the \"upright(\"low error correlation\")\" part) is indicated — a transient flash."],[1180.0397083333332,"final_law (the \"upright(\"averaging\")\" part) is indicated — a transient flash."]]},{"start":1183.7277083333333,"say":"One tree turns impurity reduction into an interpretable hierarchy of rectangles. Pruning makes that hierarchy less fragile. A random forest keeps many flexible trees, makes their mistakes less alike, and averages what remains.","live":["final_law","heading_close"],"does":[[1197.1027083333333,"A box is drawn around final_law."],[1199.1497708333334,"final_law is hidden from the screen — left the board."],[1199.1497708333334,"heading_close is hidden from the screen — left the board."]]}]}]},"durationSeconds":1200,"chapters":[{"title":"The First Split","startSeconds":0,"narration":"Here is the entire training problem in miniature. Each dot is one labelled observation, blue or red, and each position contains two measured features. We want a rule that predicts the colour of a new point. A decision tree will not draw a diagonal or fit a smooth curve. At one node it chooses one feature, one threshold, and one yes-or-no question. That question cuts the current region with an axis-aligned line. But which question should it ask first? The usual classification answer is the split that reduces impurity most. Impurity is not model error. It measures how mixed the labels are inside a node. For Gini impurity, subtract the squared class proportions from one. At the root we have eight blue and eight red, so both proportions are one half. The root impurity is zero point five, its largest possible value for two classes. The fitting code now considers thresholds between observed values. Let us compare two representative candidates. A vertical cut at x one equals four leaves both children evenly mixed. Its weighted child impurity is still zero point five. Subtract that from the root impurity and the gain is zero. The cut changed the addresses of the points, but learned nothing about their labels. Now try x two less than four. Below the line are six blue and two red. Above it are two blue and six red. Each child has Gini impurity three eighths, and their weighted average is also three eighths. The reduction is zero point five minus zero point three seven five, which is zero point one two five. That is larger than the vertical candidate's zero, so this horizontal question wins. The first branch of the tree and the first pair of rectangles are the same decision written in two languages. The root asks x two less than four. The left child receives the lower rectangle, and the right child receives the upper one. Notice what has not happened. We have not classified everything correctly, and neither child is pure. We have only made the labels less mixed. Training a tree means repeating this exact competition inside each child. That is the basic fitting loop behind the library call. Enumerate legal feature thresholds, score their weighted impurity, choose the best gain, partition the observations, and repeat on the resulting nodes."},{"title":"Growing to Purity","startSeconds":164.81004166666668,"narration":"Continue from the root split. The tree on the left and feature space on the right will grow together. A new internal node must always correspond to a new axis-aligned cut inside exactly one existing rectangle. Start in the lower rectangle. It contains six blue observations and two red ones. The best available gain first separates the far-right red point with x one less than six point one. The right child is pure, but the large left child is not. Within that child, a horizontal cut at x two equals two point five creates a pure blue strip below and a mixed strip above. That upper strip is mixed because of one red observation at two point two, two point eight. Another vertical threshold isolates it. The training algorithm is rewarded, because two new leaves become perfectly pure. Now do the same work above the root. Most points there are red, but two blue exceptions force the recursion to continue. The first upper split separates the far-right pair from the rest. The far-right pair still disagrees. A horizontal cut isolates the blue point at the top from the red point below. Again, training impurity falls to zero in both resulting leaves. The left upper region has its own blue exception. A split at x one equals two point eight narrows the search, and another horizontal split at x two equals five point one separates one red point. One final threshold at x one equals one point five isolates the remaining upper blue point. Every terminal region now contains only one class. Consequently every leaf has Gini impurity zero. On the training set, this looks perfect. Every observation is classified correctly. But look at the geometry required to achieve it: thin strips, short corridors, and thresholds whose only purpose is to rescue one exceptional dot. A new point can cross one of those arbitrary thresholds after an imperceptible change in a feature. Its predicted class then jumps, even though the training labels gave us almost no evidence that such a jump should exist. This is the characteristic strength and weakness of an unconstrained decision tree. It has low bias because it can represent complicated interactions. It also has high variance because a few observations can rearrange entire branches and rectangles. Pure leaves are therefore a training condition, not evidence of a useful model. The next question is whether every branch earns its complexity on data that did not choose the branch."},{"title":"Pruning the Tree","startSeconds":330.5074166666667,"narration":"The fully grown tree has zero training impurity, but it pays for that fit with nine leaves and several thresholds supported by a single unusual observation. Pruning asks whether those extra leaves earn their keep. Cost-complexity pruning gives the trade a precise form. R of T measures the fitted tree's error or impurity. The second term charges alpha for every terminal leaf. At alpha zero, extra leaves are free, so the pure tree wins. As alpha increases, a weak pair of leaves can cost more than the small reduction in training error that created it. Prune the weakest lower twigs first. Their narrow thresholds disappear from feature space at the same moment their branches disappear from the tree. The replacement leaf predicts the local majority class. Now prune the corresponding upper twigs. Again, the replacement is not pure on the training set. It deliberately accepts a few mistakes in exchange for a much larger and more stable prediction region. The remaining four-leaf tree still captures the broad interaction. The horizontal root separates low from high x two. Within each half, one x one threshold handles the main exception near the far-right edge. This smaller tree has higher training error by construction. The relevant question is whether it has lower error on observations that did not participate in choosing all those thresholds. A pruning path supplies a nested sequence of subtrees. Here the unpruned tree has nine leaves and no training mistakes, but its validation error is the worst entry in the table. A small penalty removes three leaves. Training error rises, validation error falls. A larger penalty leaves four terminal regions, and the validation error reaches its minimum. Push alpha farther and the tree collapses to two leaves. That model is now too simple for the data, so validation error rises again. Pruning is not a ritual of making trees small. It is model selection along a structured complexity path. Select alpha using cross-validation, a held-out set, or a nested procedure when tuning itself must be evaluated. Never choose the pruning strength by returning to the same training impurity that rewarded every twig. Libraries also offer pre-pruning controls such as maximum depth, minimum samples per leaf, minimum impurity decrease, and maximum leaf count. Those prevent growth. Cost-complexity pruning fits first and removes branches afterward. Either route trades some bias for less variance. A single pruned tree is often much easier to explain and more stable than the pure tree. The forest will take a different route: keep many unstable trees, then make their instability cancel."},{"title":"Planting Many Trees","startSeconds":509.9435833333333,"narration":"A forest does not begin by cloning one fitted tree. It creates many related training problems. For each tree, draw n observations from the original n with replacement. With replacement means an observation can appear more than once. In the first sample, indices one and five are repeated, while three and six are absent. The sample still contains n rows. A second bootstrap draw repeats different observations and omits different ones. A third does it again. Each tree therefore sees a perturbed empirical distribution, even though all draws came from the same dataset. Fitting deep trees to these samples is ordinary bootstrap aggregation, or bagging. It creates diversity because a marginal observation may be duplicated in one sample and unavailable in another. A random forest adds a second source of variation. At every node, it offers the split search only a random subset of the available features. Here is one tree in an eight-feature problem. The formula records a fresh feature subset at node j. At the root, features two, five, and eight are offered, and feature two supplies the best available impurity reduction. The left child gets a fresh draw, features one, four, and eight. The right child gets another draw, features three, five, and seven. Feature availability is local to a node. This restriction can force a tree to ignore the dominant predictor at a particular node. That sounds inefficient for one tree. Its purpose is to prevent every tree from making the same early decisions. Bootstrap sampling perturbs the observations. Feature sampling perturbs the available questions. Together they produce trees whose errors are less synchronized. These are three trees trained from the same original problem. Each diagram shows the axis-aligned boundaries produced by one bootstrap sample and one sequence of random feature offers. The first tree chooses a horizontal root and then several local vertical cuts. Its boundary is coherent within each rectangle, but jagged as a whole. The second tree begins vertically because its sample and offered features differ. It partitions the same feature plane into a visibly different collection of rectangles. The third tree returns to a horizontal root, but its thresholds and deeper branches are different again. None of these trees is intended to be the final boundary. For classification, each tree supplies class probabilities from its reached leaf, commonly the class proportions among that leaf's training samples. The forest averages those probabilities and then chooses a class. For regression, the same architecture averages numeric predictions. Bootstrap samples and random feature subsets still serve the same purpose: build individually flexible models whose mistakes are not identical."},{"title":"Averaging High-Variance Trees","startSeconds":706.4061875,"narration":"Return to the labelled plane. One deep tree supplies one jagged boundary. It reacts strongly to the particular observations and random choices that shaped its branches. For each query point x, tree b supplies a class probability. The forest averages those probabilities. Its displayed boundary is where the winning averaged class changes. Add trees until B equals ten. Each individual boundary is still made of rectangular steps, but their idiosyncratic notches occur at different places. Majority support removes many notches that only one tree wanted. Now average one hundred trees. The aggregate is still a staircase if we inspect it finely enough, because every contributor is a tree. At this scale, however, it follows the broad class structure rather than every isolated observation. Calling this boundary smoother does not mean that a forest fits a smooth analytic function. It means the averaged prediction varies more stably across nearby points and fewer decisions depend on one tree's narrow rectangular accident. Why should averaging unstable models work? Imagine each tree prediction as a useful signal plus a zero-mean fitting error. Averaging keeps the shared signal. Errors that point in different directions can cancel. If the tree errors were independent and each had variance sigma squared, the mean of B trees would have variance sigma squared over B. Double the number of trees and this variance contribution halves. Real tree errors are not independent. They use the same original dataset, and strong predictors can make them discover similar branches. Let rho represent their average pairwise error correlation. Here is the extreme problem. Five trees all make an error of plus one on the same case. Their average error is still plus one. Repetition did not remove a shared mistake. On the right, errors differ across trees. Positive and negative errors offset, and their average is zero in this small illustration. Diversity is useful when it concerns errors, not merely visual differences between diagrams. With equal variance and average correlation rho, the ensemble variance is sigma squared times rho plus one minus rho over B. The second part shrinks as trees are added. The correlated part does not. Take the number of trees toward infinity. The variance approaches rho sigma squared, not zero. If rho is close to one, a huge forest behaves like repeated copies of one unstable tree. This is why decorrelation is essential. Bootstrap samples alter which observations drive the branches. Random feature subsets prevent one dominant predictor from forcing the same root and early splits in every tree. There is a trade-off. Offering fewer features can weaken each individual tree by denying it useful predictors. But if that loss is modest and the correlation falls substantially, the average can generalize better. Adding trees mainly reduces Monte Carlo noise in the fitted ensemble. It does not repair severe bias, leakage, bad labels, a shifted deployment population, or a feature set that contains no useful signal. The forest's advantage is therefore specific. Deep trees provide flexible, high-variance base predictions. Resampling and feature randomness make their errors less alike. Averaging then removes the part of the variance that is not shared."},{"title":"From the Idea to the Library","startSeconds":944.6647083333332,"narration":"Here is a practical sequence that preserves the logic we have developed. Start with the prediction problem, not the estimator. Fix the target, evaluation unit, split strategy, and metric before fitting. A random row split is wrong whenever rows from the same customer, patient, device, location, or future time can leak information across folds. Group and temporal structure belong in the validation design. Fit a small decision tree before the forest. Its splits expose coding mistakes, target proxies, implausible thresholds, and feature interactions that a large ensemble can conceal behind a good aggregate score. Then tune tree structure against validation performance. Maximum depth limits path length. Minimum samples per leaf demands evidence in each terminal region. Cost-complexity alpha removes weak fitted branches. For a forest, raise the number of estimators until the validation metric and predictions stabilize. More trees usually increase compute rather than overfitting in the familiar single-tree sense, but returns become negligible. Finally inspect more than one aggregate score. Check important subgroups, threshold-sensitive decisions, probability calibration, drift, and the cost of the errors the model actually makes. Maximum depth and maximum leaf nodes limit global tree size. They are coarse controls on how many successive rectangles the tree may create. Minimum samples per leaf directly attacks tiny regions. Minimum impurity decrease requires a split to earn enough local gain before it is allowed. Cost-complexity alpha names the post-pruning penalty we used earlier. Its numerical scale depends on the data, weights, impurity, and implementation, so tune it through the supplied pruning path rather than by folklore. Bootstrap turns observation resampling on or off. Max features controls the random candidate subset at each split. Lowering max features usually reduces correlation, but can also weaken individual trees. The number of estimators is the ensemble size B. Parallel-job settings change wall-clock cost, not the fitted statistical objective. Do not confuse faster execution with stronger regularization. Classification probabilities deserve separate validation. A forest averages leaf class proportions, which can rank cases very effectively while remaining overconfident or underconfident as probabilities. Out-of-bag predictions use, for each training row, only trees whose bootstrap samples omitted that row. They provide a convenient internal diagnostic, but they do not override grouped, temporal, or external validation requirements. Impurity-based feature importance summarizes how fitted splits used a feature. It can favour variables with many available thresholds and divide credit awkwardly among correlated predictors. It is not a causal effect. Permutation importance asks how predictive performance changes after one feature is disrupted. It is often closer to the operational question, but correlated features can substitute for one another and hide each other's importance. Class weights alter the fitting objective. They do not decide which deployment metric matters. A random seed makes the stochastic fit reproducible; it does not make sampling uncertainty disappear. The whole lecture can be compressed into three ingredients. Trees must be strong enough to capture useful structure. Their errors must be sufficiently decorrelated. Averaging then reduces the unshared variance. One tree turns impurity reduction into an interpretable hierarchy of rectangles. Pruning makes that hierarchy less fragile. A random forest keeps many flexible trees, makes their mistakes less alike, and averages what remains."}]}}