# From One Decision Tree to a Random Forest > Grow a classification tree on a small labelled dataset by comparing impurity reductions, while its branches remain tied to the axis-aligned regions they create. Follow the tree to pure but fragile leaves, prune it with validation evidence, then construct a random forest from bootstrap samples and random feature subsets. The aggregate boundary, variance calculation, and practical workflow explain why averaging helps, why correlated tree errors survive, and how the underlying ideas map to familiar library controls. - Canonical watch page: [From One Decision Tree to a Random Forest](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them) - Publisher: [Academa, Inc.](https://academa.ai) - Subject: Machine Learning - Published: 2026-08-28T22:51:24.000Z - Updated: 2026-08-28T22:51:24.000Z - Duration: PT1200S (20 minutes) - Chapters: 6 - Views: 0 - Language: en-US - Access: Free - Video stream: [HLS content](https://academa.ai/media/l/01M14TYNGGHB7PNCH61Q2K755C/0/dark/master.m3u8) - Audiovisual record: [Semantic JSON](https://academa.ai/media/l/01M14TYNGGHB7PNCH61Q2K755C/0/semantic.json) - Thumbnail: [Image](https://academa.ai/media/l/01M14TYNGGHB7PNCH61Q2K755C/0/dark/poster.jpg) ## Description Grow, overfit, and prune one decision tree, then build a random forest and see why averaging decorrelated trees reduces prediction variance. ## Chapters - [00:00–02:44.81 · The First Split](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=0) - [02:44.81–05:30.507 · Growing to Purity](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=164.81004166666668) - [05:30.507–08:29.944 · Pruning the Tree](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5074166666667) - [08:29.944–11:46.406 · Planting Many Trees](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=509.9435833333333) - [11:46.406–15:44.665 · Averaging High-Variance Trees](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=706.4061875) - [15:44.665–20:00 · From the Idea to the Library](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=944.6647083333332) ## Transcript ### [00:00 · The First Split](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=0) Here is the entire training problem in miniature. Each dot is one labelled observation, blue or red, and each position contains two measured features. We want a rule that predicts the colour of a new point. A decision tree will not draw a diagonal or fit a smooth curve. At one node it chooses one feature, one threshold, and one yes-or-no question. That question cuts the current region with an axis-aligned line. But which question should it ask first? The usual classification answer is the split that reduces impurity most. Impurity is not model error. It measures how mixed the labels are inside a node. For Gini impurity, subtract the squared class proportions from one. At the root we have eight blue and eight red, so both proportions are one half. The root impurity is zero point five, its largest possible value for two classes. The fitting code now considers thresholds between observed values. Let us compare two representative candidates. A vertical cut at x one equals four leaves both children evenly mixed. Its weighted child impurity is still zero point five. Subtract that from the root impurity and the gain is zero. The cut changed the addresses of the points, but learned nothing about their labels. Now try x two less than four. Below the line are six blue and two red. Above it are two blue and six red. Each child has Gini impurity three eighths, and their weighted average is also three eighths. The reduction is zero point five minus zero point three seven five, which is zero point one two five. That is larger than the vertical candidate's zero, so this horizontal question wins. The first branch of the tree and the first pair of rectangles are the same decision written in two languages. The root asks x two less than four. The left child receives the lower rectangle, and the right child receives the upper one. Notice what has not happened. We have not classified everything correctly, and neither child is pure. We have only made the labels less mixed. Training a tree means repeating this exact competition inside each child. That is the basic fitting loop behind the library call. Enumerate legal feature thresholds, score their weighted impurity, choose the best gain, partition the observations, and repeat on the resulting nodes. ### [02:44.81 · Growing to Purity](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=164.81004166666668) Continue from the root split. The tree on the left and feature space on the right will grow together. A new internal node must always correspond to a new axis-aligned cut inside exactly one existing rectangle. Start in the lower rectangle. It contains six blue observations and two red ones. The best available gain first separates the far-right red point with x one less than six point one. The right child is pure, but the large left child is not. Within that child, a horizontal cut at x two equals two point five creates a pure blue strip below and a mixed strip above. That upper strip is mixed because of one red observation at two point two, two point eight. Another vertical threshold isolates it. The training algorithm is rewarded, because two new leaves become perfectly pure. Now do the same work above the root. Most points there are red, but two blue exceptions force the recursion to continue. The first upper split separates the far-right pair from the rest. The far-right pair still disagrees. A horizontal cut isolates the blue point at the top from the red point below. Again, training impurity falls to zero in both resulting leaves. The left upper region has its own blue exception. A split at x one equals two point eight narrows the search, and another horizontal split at x two equals five point one separates one red point. One final threshold at x one equals one point five isolates the remaining upper blue point. Every terminal region now contains only one class. Consequently every leaf has Gini impurity zero. On the training set, this looks perfect. Every observation is classified correctly. But look at the geometry required to achieve it: thin strips, short corridors, and thresholds whose only purpose is to rescue one exceptional dot. A new point can cross one of those arbitrary thresholds after an imperceptible change in a feature. Its predicted class then jumps, even though the training labels gave us almost no evidence that such a jump should exist. This is the characteristic strength and weakness of an unconstrained decision tree. It has low bias because it can represent complicated interactions. It also has high variance because a few observations can rearrange entire branches and rectangles. Pure leaves are therefore a training condition, not evidence of a useful model. The next question is whether every branch earns its complexity on data that did not choose the branch. ### [05:30.507 · Pruning the Tree](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5074166666667) The fully grown tree has zero training impurity, but it pays for that fit with nine leaves and several thresholds supported by a single unusual observation. Pruning asks whether those extra leaves earn their keep. Cost-complexity pruning gives the trade a precise form. R of T measures the fitted tree's error or impurity. The second term charges alpha for every terminal leaf. At alpha zero, extra leaves are free, so the pure tree wins. As alpha increases, a weak pair of leaves can cost more than the small reduction in training error that created it. Prune the weakest lower twigs first. Their narrow thresholds disappear from feature space at the same moment their branches disappear from the tree. The replacement leaf predicts the local majority class. Now prune the corresponding upper twigs. Again, the replacement is not pure on the training set. It deliberately accepts a few mistakes in exchange for a much larger and more stable prediction region. The remaining four-leaf tree still captures the broad interaction. The horizontal root separates low from high x two. Within each half, one x one threshold handles the main exception near the far-right edge. This smaller tree has higher training error by construction. The relevant question is whether it has lower error on observations that did not participate in choosing all those thresholds. A pruning path supplies a nested sequence of subtrees. Here the unpruned tree has nine leaves and no training mistakes, but its validation error is the worst entry in the table. A small penalty removes three leaves. Training error rises, validation error falls. A larger penalty leaves four terminal regions, and the validation error reaches its minimum. Push alpha farther and the tree collapses to two leaves. That model is now too simple for the data, so validation error rises again. Pruning is not a ritual of making trees small. It is model selection along a structured complexity path. Select alpha using cross-validation, a held-out set, or a nested procedure when tuning itself must be evaluated. Never choose the pruning strength by returning to the same training impurity that rewarded every twig. Libraries also offer pre-pruning controls such as maximum depth, minimum samples per leaf, minimum impurity decrease, and maximum leaf count. Those prevent growth. Cost-complexity pruning fits first and removes branches afterward. Either route trades some bias for less variance. A single pruned tree is often much easier to explain and more stable than the pure tree. The forest will take a different route: keep many unstable trees, then make their instability cancel. ### [08:29.944 · Planting Many Trees](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=509.9435833333333) A forest does not begin by cloning one fitted tree. It creates many related training problems. For each tree, draw n observations from the original n with replacement. With replacement means an observation can appear more than once. In the first sample, indices one and five are repeated, while three and six are absent. The sample still contains n rows. A second bootstrap draw repeats different observations and omits different ones. A third does it again. Each tree therefore sees a perturbed empirical distribution, even though all draws came from the same dataset. Fitting deep trees to these samples is ordinary bootstrap aggregation, or bagging. It creates diversity because a marginal observation may be duplicated in one sample and unavailable in another. A random forest adds a second source of variation. At every node, it offers the split search only a random subset of the available features. Here is one tree in an eight-feature problem. The formula records a fresh feature subset at node j. At the root, features two, five, and eight are offered, and feature two supplies the best available impurity reduction. The left child gets a fresh draw, features one, four, and eight. The right child gets another draw, features three, five, and seven. Feature availability is local to a node. This restriction can force a tree to ignore the dominant predictor at a particular node. That sounds inefficient for one tree. Its purpose is to prevent every tree from making the same early decisions. Bootstrap sampling perturbs the observations. Feature sampling perturbs the available questions. Together they produce trees whose errors are less synchronized. These are three trees trained from the same original problem. Each diagram shows the axis-aligned boundaries produced by one bootstrap sample and one sequence of random feature offers. The first tree chooses a horizontal root and then several local vertical cuts. Its boundary is coherent within each rectangle, but jagged as a whole. The second tree begins vertically because its sample and offered features differ. It partitions the same feature plane into a visibly different collection of rectangles. The third tree returns to a horizontal root, but its thresholds and deeper branches are different again. None of these trees is intended to be the final boundary. For classification, each tree supplies class probabilities from its reached leaf, commonly the class proportions among that leaf's training samples. The forest averages those probabilities and then chooses a class. For regression, the same architecture averages numeric predictions. Bootstrap samples and random feature subsets still serve the same purpose: build individually flexible models whose mistakes are not identical. ### [11:46.406 · Averaging High-Variance Trees](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=706.4061875) Return to the labelled plane. One deep tree supplies one jagged boundary. It reacts strongly to the particular observations and random choices that shaped its branches. For each query point x, tree b supplies a class probability. The forest averages those probabilities. Its displayed boundary is where the winning averaged class changes. Add trees until B equals ten. Each individual boundary is still made of rectangular steps, but their idiosyncratic notches occur at different places. Majority support removes many notches that only one tree wanted. Now average one hundred trees. The aggregate is still a staircase if we inspect it finely enough, because every contributor is a tree. At this scale, however, it follows the broad class structure rather than every isolated observation. Calling this boundary smoother does not mean that a forest fits a smooth analytic function. It means the averaged prediction varies more stably across nearby points and fewer decisions depend on one tree's narrow rectangular accident. Why should averaging unstable models work? Imagine each tree prediction as a useful signal plus a zero-mean fitting error. Averaging keeps the shared signal. Errors that point in different directions can cancel. If the tree errors were independent and each had variance sigma squared, the mean of B trees would have variance sigma squared over B. Double the number of trees and this variance contribution halves. Real tree errors are not independent. They use the same original dataset, and strong predictors can make them discover similar branches. Let rho represent their average pairwise error correlation. Here is the extreme problem. Five trees all make an error of plus one on the same case. Their average error is still plus one. Repetition did not remove a shared mistake. On the right, errors differ across trees. Positive and negative errors offset, and their average is zero in this small illustration. Diversity is useful when it concerns errors, not merely visual differences between diagrams. With equal variance and average correlation rho, the ensemble variance is sigma squared times rho plus one minus rho over B. The second part shrinks as trees are added. The correlated part does not. Take the number of trees toward infinity. The variance approaches rho sigma squared, not zero. If rho is close to one, a huge forest behaves like repeated copies of one unstable tree. This is why decorrelation is essential. Bootstrap samples alter which observations drive the branches. Random feature subsets prevent one dominant predictor from forcing the same root and early splits in every tree. There is a trade-off. Offering fewer features can weaken each individual tree by denying it useful predictors. But if that loss is modest and the correlation falls substantially, the average can generalize better. Adding trees mainly reduces Monte Carlo noise in the fitted ensemble. It does not repair severe bias, leakage, bad labels, a shifted deployment population, or a feature set that contains no useful signal. The forest's advantage is therefore specific. Deep trees provide flexible, high-variance base predictions. Resampling and feature randomness make their errors less alike. Averaging then removes the part of the variance that is not shared. ### [15:44.665 · From the Idea to the Library](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=944.6647083333332) Here is a practical sequence that preserves the logic we have developed. Start with the prediction problem, not the estimator. Fix the target, evaluation unit, split strategy, and metric before fitting. A random row split is wrong whenever rows from the same customer, patient, device, location, or future time can leak information across folds. Group and temporal structure belong in the validation design. Fit a small decision tree before the forest. Its splits expose coding mistakes, target proxies, implausible thresholds, and feature interactions that a large ensemble can conceal behind a good aggregate score. Then tune tree structure against validation performance. Maximum depth limits path length. Minimum samples per leaf demands evidence in each terminal region. Cost-complexity alpha removes weak fitted branches. For a forest, raise the number of estimators until the validation metric and predictions stabilize. More trees usually increase compute rather than overfitting in the familiar single-tree sense, but returns become negligible. Finally inspect more than one aggregate score. Check important subgroups, threshold-sensitive decisions, probability calibration, drift, and the cost of the errors the model actually makes. Maximum depth and maximum leaf nodes limit global tree size. They are coarse controls on how many successive rectangles the tree may create. Minimum samples per leaf directly attacks tiny regions. Minimum impurity decrease requires a split to earn enough local gain before it is allowed. Cost-complexity alpha names the post-pruning penalty we used earlier. Its numerical scale depends on the data, weights, impurity, and implementation, so tune it through the supplied pruning path rather than by folklore. Bootstrap turns observation resampling on or off. Max features controls the random candidate subset at each split. Lowering max features usually reduces correlation, but can also weaken individual trees. The number of estimators is the ensemble size B. Parallel-job settings change wall-clock cost, not the fitted statistical objective. Do not confuse faster execution with stronger regularization. Classification probabilities deserve separate validation. A forest averages leaf class proportions, which can rank cases very effectively while remaining overconfident or underconfident as probabilities. Out-of-bag predictions use, for each training row, only trees whose bootstrap samples omitted that row. They provide a convenient internal diagnostic, but they do not override grouped, temporal, or external validation requirements. Impurity-based feature importance summarizes how fitted splits used a feature. It can favour variables with many available thresholds and divide credit awkwardly among correlated predictors. It is not a causal effect. Permutation importance asks how predictive performance changes after one feature is disrupted. It is often closer to the operational question, but correlated features can substitute for one another and hide each other's importance. Class weights alter the fitting objective. They do not decide which deployment metric matters. A random seed makes the stochastic fit reproducible; it does not make sampling uncertainty disappear. The whole lecture can be compressed into three ingredients. Trees must be strong enough to capture useful structure. Their errors must be sufficiently decorrelated. Averaging then reduces the unshared variance. One tree turns impurity reduction into an interpretable hierarchy of rectangles. Pruning makes that hierarchy less fragile. A random forest keeps many flexible trees, makes their mistakes less alike, and averages what remains. ## About Academa, Inc. Academa makes technical knowledge easier to understand through visual lectures and lets learners request new lecture videos on the topics they need. ## Complete audiovisual record Immutable source: [semantic.json](https://academa.ai/media/l/01M14TYNGGHB7PNCH61Q2K755C/0/semantic.json) Record version: 1. Render attempt: 0. ### How to read this timeline Each scene owns its object identifiers. A beat's board is the complete board when listed, empty when marked empty, and unchanged from the nearest earlier listed board in the same scene when marked unchanged. Action times are absolute positions in the published video. ### Scene 1: [The First Split](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=0) Span: 00:00–02:44.81 (0s–164.81004166666668s). #### Objects - blue\_points: a Point \[blue\] drawn in plane (location=(0.8, 1.2)) - blue\_points\_2: a Point \[blue\] drawn in plane (location=(1.5, 2.0)) - blue\_points\_3: a Point \[blue\] drawn in plane (location=(2.8, 1.0)) - blue\_points\_4: a Point \[blue\] drawn in plane (location=(3.6, 2.8)) - blue\_points\_5: a Point \[blue\] drawn in plane (location=(4.5, 1.5)) - blue\_points\_6: a Point \[blue\] drawn in plane (location=(5.6, 2.7)) - blue\_points\_7: a Point \[blue\] drawn in plane (location=(2.4, 5.5)) - blue\_points\_8: a Point \[blue\] drawn in plane (location=(6.8, 6.4)) - candidates: a Table \[text\] that says "Candidate Children Weighted Gini Gain $x\_1 \< 4$ $4/4$ and $4/4$ $0.500$ $0.000$ $x\_2 \< 4$ $6/2$ and $2/6$ $0.375$ $0.125$" (rows=(('Candidate', 'Children', 'Weighted Gini', 'Gain'), ('$x\_1 \< 4…, header=True) - chosen: a Math \[text\] that says "$x\_2 \< 4 thin arrow.r thin upright("best gain")$" - heading: a Heading that says "Every Threshold Competes" - high\_box: a Polygon \[red\] drawn in tree (vertices=((6.0, 0.8), (9.0, 0.8), (9.0, 1.7), (6.0, 1.7)), fill\_opacity=0.12) - high\_label: a Math \[red\] that says "$2 B, thin 6 R$" drawn in tree - high\_region: a Polygon \[red\] drawn in plane (vertices=((0.0, 4.0), (8.0, 4.0), (8.0, 8.0), (0.0, 8.0)), fill\_opacity=0.08) - horizontal\_candidate: a Line \[yellow\] drawn in plane (start=(0.0, 4.0), end=(8.0, 4.0), dashed=True) - left\_edge: a Line \[gray\] drawn in tree (start=(4.4, 2.9), end=(2.5, 1.7)) - low\_box: a Polygon \[blue\] drawn in tree (vertices=((1.0, 0.8), (4.0, 0.8), (4.0, 1.7), (1.0, 1.7)), fill\_opacity=0.12) - low\_label: a Math \[blue\] that says "$6 B, thin 2 R$" drawn in tree - low\_region: a Polygon \[blue\] drawn in plane (vertices=((0.0, 0.0), (8.0, 0.0), (8.0, 4.0), (0.0, 4.0)), fill\_opacity=0.08) - plane: an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)) - question: a Panel that says "Given two measured features and two labelled classes, which threshold should become the root of the decision tree?" - red\_points: a Point \[red\] drawn in plane (location=(0.9, 6.3)) - red\_points\_2: a Point \[red\] drawn in plane (location=(1.8, 4.8)) - red\_points\_3: a Point \[red\] drawn in plane (location=(3.3, 6.7)) - red\_points\_4: a Point \[red\] drawn in plane (location=(4.2, 5.2)) - red\_points\_5: a Point \[red\] drawn in plane (location=(5.4, 6.1)) - red\_points\_6: a Point \[red\] drawn in plane (location=(7.2, 4.8)) - red\_points\_7: a Point \[red\] drawn in plane (location=(2.2, 2.8)) - red\_points\_8: a Point \[red\] drawn in plane (location=(6.5, 1.2)) - right\_edge: a Line \[gray\] drawn in tree (start=(5.6, 2.9), end=(7.5, 1.7)) - root\_box: a Polygon \[yellow\] drawn in tree (vertices=((3.5, 2.9), (6.5, 2.9), (6.5, 3.8), (3.5, 3.8)), fill\_opacity=0.12) - root\_impurity: a Math \[text\] that says "$G\_(upright("root")) = 1 - 2 (frac(1,2))^2 = 0.50$" - root\_label: a Math \[yellow\] that says "$x\_2 \< 4?$" drawn in tree - tree: a Figure (x\_range=(0.0, 10.0), y\_range=(0.0, 4.0), aspect=(5.0, 2.0)) - vertical\_candidate: a Line \[gray\] drawn in plane (start=(4.0, 0.0), end=(4.0, 8.0), dashed=True) #### Beats ##### [00:00](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=0) Narration: Here is the entire training problem in miniature. Each dot is one labelled observation, blue or red, and each position contains two measured features. We want a rule that predicts the colour of a new point. Board: Empty. Actions: - [00:00](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=0): question is shown on the screen, written out. - [00:3.727](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=3.727): plane is shown on the screen, written out. - [00:6.072](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=6.072): blue\_points is shown on the screen, written out. - [00:6.152](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=6.152): blue\_points\_2 is shown on the screen, written out. - [00:6.232](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=6.232): blue\_points\_3 is shown on the screen, written out. - [00:6.312](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=6.312): blue\_points\_4 is shown on the screen, written out. - [00:6.392](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=6.392): blue\_points\_5 is shown on the screen, written out. - [00:6.472](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=6.472): blue\_points\_6 is shown on the screen, written out. - [00:6.552](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=6.552): blue\_points\_7 is shown on the screen, written out. - [00:6.632](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=6.632): blue\_points\_8 is shown on the screen, written out. - [00:6.699](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=6.699): red\_points is shown on the screen, written out. - [00:6.779](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=6.779): red\_points\_2 is shown on the screen, written out. - [00:6.859](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=6.859): red\_points\_3 is shown on the screen, written out. - [00:6.939](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=6.939): red\_points\_4 is shown on the screen, written out. - [00:7.019](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=7.019): red\_points\_5 is shown on the screen, written out. - [00:7.099](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=7.099): red\_points\_6 is shown on the screen, written out. - [00:7.179](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=7.179): red\_points\_7 is shown on the screen, written out. - [00:7.259](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=7.259): red\_points\_8 is shown on the screen, written out. ##### [00:14.95](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=14.95) Narration: A decision tree will not draw a diagonal or fit a smooth curve. At one node it chooses one feature, one threshold, and one yes-or-no question. That question cuts the current region with an axis-aligned line. Board: plane — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); question — a Panel that says "Given two measured features and two labelled classes, which threshold should become the root of the decision tree?"; blue\_points — a Point \[blue\] drawn in plane (location=(0.8, 1.2)); blue\_points\_2 — a Point \[blue\] drawn in plane (location=(1.5, 2.0)); blue\_points\_3 — a Point \[blue\] drawn in plane (location=(2.8, 1.0)); blue\_points\_4 — a Point \[blue\] drawn in plane (location=(3.6, 2.8)); blue\_points\_5 — a Point \[blue\] drawn in plane (location=(4.5, 1.5)); blue\_points\_6 — a Point \[blue\] drawn in plane (location=(5.6, 2.7)); blue\_points\_7 — a Point \[blue\] drawn in plane (location=(2.4, 5.5)); blue\_points\_8 — a Point \[blue\] drawn in plane (location=(6.8, 6.4)); red\_points — a Point \[red\] drawn in plane (location=(0.9, 6.3)); red\_points\_2 — a Point \[red\] drawn in plane (location=(1.8, 4.8)); red\_points\_3 — a Point \[red\] drawn in plane (location=(3.3, 6.7)); red\_points\_4 — a Point \[red\] drawn in plane (location=(4.2, 5.2)); red\_points\_5 — a Point \[red\] drawn in plane (location=(5.4, 6.1)); red\_points\_6 — a Point \[red\] drawn in plane (location=(7.2, 4.8)); red\_points\_7 — a Point \[red\] drawn in plane (location=(2.2, 2.8)); red\_points\_8 — a Point \[red\] drawn in plane (location=(6.5, 1.2)) Actions: - [00:22.02](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=22.020000000000003): vertical\_candidate is shown on the screen, written out. - [00:29.427](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=29.4275): vertical\_candidate is hidden from the screen. ##### [00:30.027](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=30.027499999999996) Narration: But which question should it ask first? The usual classification answer is the split that reduces impurity most. Impurity is not model error. It measures how mixed the labels are inside a node. Board: Unchanged from the preceding beat in this scene. Actions: - [00:30.608](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=30.607999999999997): question (the "which threshold" part) is emphasized. - [00:36.552](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=36.552): question (the "which threshold" part) is no longer emphasized. ##### [00:44.617](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=44.61749999999999) Narration: For Gini impurity, subtract the squared class proportions from one. At the root we have eight blue and eight red, so both proportions are one half. The root impurity is zero point five, its largest possible value for two classes. Board: Unchanged from the preceding beat in this scene. Actions: - [00:45.244](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=45.24399999999999): plane moves to a new place on the board. - [00:45.244](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=45.24399999999999): root\_impurity is shown on the screen, written out. - [00:53.975](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=53.97499999999999): root\_impurity (the "frac(1,2)" part) is indicated — a transient flash. - [00:56.425](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=56.42499999999999): root\_impurity (the "0.50" part) is indicated — a transient flash. - [01:0.766](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=60.76649999999999): plane moves to a new place on the board. - [01:0.766](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=60.76649999999999): question is hidden from the screen — left the board. - [01:0.766](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=60.76649999999999): root\_impurity is hidden from the screen — left the board. ##### [01:1.366](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=61.366499999999995) Narration: The fitting code now considers thresholds between observed values. Let us compare two representative candidates. A vertical cut at x one equals four leaves both children evenly mixed. Board: plane — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); blue\_points — a Point \[blue\] drawn in plane (location=(0.8, 1.2)); blue\_points\_2 — a Point \[blue\] drawn in plane (location=(1.5, 2.0)); blue\_points\_3 — a Point \[blue\] drawn in plane (location=(2.8, 1.0)); blue\_points\_4 — a Point \[blue\] drawn in plane (location=(3.6, 2.8)); blue\_points\_5 — a Point \[blue\] drawn in plane (location=(4.5, 1.5)); blue\_points\_6 — a Point \[blue\] drawn in plane (location=(5.6, 2.7)); blue\_points\_7 — a Point \[blue\] drawn in plane (location=(2.4, 5.5)); blue\_points\_8 — a Point \[blue\] drawn in plane (location=(6.8, 6.4)); red\_points — a Point \[red\] drawn in plane (location=(0.9, 6.3)); red\_points\_2 — a Point \[red\] drawn in plane (location=(1.8, 4.8)); red\_points\_3 — a Point \[red\] drawn in plane (location=(3.3, 6.7)); red\_points\_4 — a Point \[red\] drawn in plane (location=(4.2, 5.2)); red\_points\_5 — a Point \[red\] drawn in plane (location=(5.4, 6.1)); red\_points\_6 — a Point \[red\] drawn in plane (location=(7.2, 4.8)); red\_points\_7 — a Point \[red\] drawn in plane (location=(2.2, 2.8)); red\_points\_8 — a Point \[red\] drawn in plane (location=(6.5, 1.2)) Actions: - [01:1.366](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=61.366499999999995): heading is shown on the screen, written out. - [01:6.649](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=66.649): candidates is shown on the screen, written out. - [01:9.459](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=69.459): vertical\_candidate is shown on the screen, written out. - [01:9.459](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=69.459): candidates is shown on the screen, written out. - [01:13.348](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=73.348): candidates (the "$0.500$" part) is indicated — a transient flash. ##### [01:14.761](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=74.761) Narration: Its weighted child impurity is still zero point five. Subtract that from the root impurity and the gain is zero. The cut changed the addresses of the points, but learned nothing about their labels. Board: plane — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); blue\_points — a Point \[blue\] drawn in plane (location=(0.8, 1.2)); blue\_points\_2 — a Point \[blue\] drawn in plane (location=(1.5, 2.0)); blue\_points\_3 — a Point \[blue\] drawn in plane (location=(2.8, 1.0)); blue\_points\_4 — a Point \[blue\] drawn in plane (location=(3.6, 2.8)); blue\_points\_5 — a Point \[blue\] drawn in plane (location=(4.5, 1.5)); blue\_points\_6 — a Point \[blue\] drawn in plane (location=(5.6, 2.7)); blue\_points\_7 — a Point \[blue\] drawn in plane (location=(2.4, 5.5)); blue\_points\_8 — a Point \[blue\] drawn in plane (location=(6.8, 6.4)); red\_points — a Point \[red\] drawn in plane (location=(0.9, 6.3)); red\_points\_2 — a Point \[red\] drawn in plane (location=(1.8, 4.8)); red\_points\_3 — a Point \[red\] drawn in plane (location=(3.3, 6.7)); red\_points\_4 — a Point \[red\] drawn in plane (location=(4.2, 5.2)); red\_points\_5 — a Point \[red\] drawn in plane (location=(5.4, 6.1)); red\_points\_6 — a Point \[red\] drawn in plane (location=(7.2, 4.8)); red\_points\_7 — a Point \[red\] drawn in plane (location=(2.2, 2.8)); red\_points\_8 — a Point \[red\] drawn in plane (location=(6.5, 1.2)); vertical\_candidate — a Line \[gray\] drawn in plane (start=(4.0, 0.0), end=(4.0, 8.0), dashed=True); heading — a Heading that says "Every Threshold Competes" Actions: - [01:17.501](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=77.501): candidates (the "$0.000$" part) is indicated — a transient flash. - [01:26.939](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=86.93900000000001): vertical\_candidate is hidden from the screen. ##### [01:29.664](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=89.664) Narration: Now try x two less than four. Below the line are six blue and two red. Above it are two blue and six red. Each child has Gini impurity three eighths, and their weighted average is also three eighths. Board: plane — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); blue\_points — a Point \[blue\] drawn in plane (location=(0.8, 1.2)); blue\_points\_2 — a Point \[blue\] drawn in plane (location=(1.5, 2.0)); blue\_points\_3 — a Point \[blue\] drawn in plane (location=(2.8, 1.0)); blue\_points\_4 — a Point \[blue\] drawn in plane (location=(3.6, 2.8)); blue\_points\_5 — a Point \[blue\] drawn in plane (location=(4.5, 1.5)); blue\_points\_6 — a Point \[blue\] drawn in plane (location=(5.6, 2.7)); blue\_points\_7 — a Point \[blue\] drawn in plane (location=(2.4, 5.5)); blue\_points\_8 — a Point \[blue\] drawn in plane (location=(6.8, 6.4)); red\_points — a Point \[red\] drawn in plane (location=(0.9, 6.3)); red\_points\_2 — a Point \[red\] drawn in plane (location=(1.8, 4.8)); red\_points\_3 — a Point \[red\] drawn in plane (location=(3.3, 6.7)); red\_points\_4 — a Point \[red\] drawn in plane (location=(4.2, 5.2)); red\_points\_5 — a Point \[red\] drawn in plane (location=(5.4, 6.1)); red\_points\_6 — a Point \[red\] drawn in plane (location=(7.2, 4.8)); red\_points\_7 — a Point \[red\] drawn in plane (location=(2.2, 2.8)); red\_points\_8 — a Point \[red\] drawn in plane (location=(6.5, 1.2)); heading — a Heading that says "Every Threshold Competes" Actions: - [01:30.767](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=90.76700000000001): candidates is shown on the screen, written out. - [01:32.857](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=92.85700000000001): low\_region is shown on the screen, faded in. - [01:33.368](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=93.36800000000001): horizontal\_candidate is shown on the screen, written out. - [01:33.89](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=93.89000000000001): candidates (the "$6/2$ and $2/6$" part) is indicated — a transient flash. - [01:36.096](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=96.09600000000002): high\_region is shown on the screen, faded in. ##### [01:45.392](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=105.39150000000001) Narration: The reduction is zero point five minus zero point three seven five, which is zero point one two five. That is larger than the vertical candidate's zero, so this horizontal question wins. Board: plane — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); blue\_points — a Point \[blue\] drawn in plane (location=(0.8, 1.2)); blue\_points\_2 — a Point \[blue\] drawn in plane (location=(1.5, 2.0)); blue\_points\_3 — a Point \[blue\] drawn in plane (location=(2.8, 1.0)); blue\_points\_4 — a Point \[blue\] drawn in plane (location=(3.6, 2.8)); blue\_points\_5 — a Point \[blue\] drawn in plane (location=(4.5, 1.5)); blue\_points\_6 — a Point \[blue\] drawn in plane (location=(5.6, 2.7)); blue\_points\_7 — a Point \[blue\] drawn in plane (location=(2.4, 5.5)); blue\_points\_8 — a Point \[blue\] drawn in plane (location=(6.8, 6.4)); red\_points — a Point \[red\] drawn in plane (location=(0.9, 6.3)); red\_points\_2 — a Point \[red\] drawn in plane (location=(1.8, 4.8)); red\_points\_3 — a Point \[red\] drawn in plane (location=(3.3, 6.7)); red\_points\_4 — a Point \[red\] drawn in plane (location=(4.2, 5.2)); red\_points\_5 — a Point \[red\] drawn in plane (location=(5.4, 6.1)); red\_points\_6 — a Point \[red\] drawn in plane (location=(7.2, 4.8)); red\_points\_7 — a Point \[red\] drawn in plane (location=(2.2, 2.8)); red\_points\_8 — a Point \[red\] drawn in plane (location=(6.5, 1.2)); heading — a Heading that says "Every Threshold Competes"; horizontal\_candidate — a Line \[yellow\] drawn in plane (start=(0.0, 4.0), end=(8.0, 4.0), dashed=True); low\_region — a Polygon \[blue\] drawn in plane (vertices=((0.0, 0.0), (8.0, 0.0), (8.0, 4.0), (0.0, 4.0)), fill\_opacity=0.08); high\_region — a Polygon \[red\] drawn in plane (vertices=((0.0, 4.0), (8.0, 4.0), (8.0, 8.0), (0.0, 8.0)), fill\_opacity=0.08) Actions: - [01:50.5](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=110.50000000000001): candidates (the "$0.125$" part) is indicated — a transient flash. - [01:57.175](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=117.17500000000001): chosen is shown on the screen, written out. ##### [01:58.646](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=118.64600000000002) Narration: The first branch of the tree and the first pair of rectangles are the same decision written in two languages. The root asks x two less than four. The left child receives the lower rectangle, and the right child receives the upper one. Board: plane — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); blue\_points — a Point \[blue\] drawn in plane (location=(0.8, 1.2)); blue\_points\_2 — a Point \[blue\] drawn in plane (location=(1.5, 2.0)); blue\_points\_3 — a Point \[blue\] drawn in plane (location=(2.8, 1.0)); blue\_points\_4 — a Point \[blue\] drawn in plane (location=(3.6, 2.8)); blue\_points\_5 — a Point \[blue\] drawn in plane (location=(4.5, 1.5)); blue\_points\_6 — a Point \[blue\] drawn in plane (location=(5.6, 2.7)); blue\_points\_7 — a Point \[blue\] drawn in plane (location=(2.4, 5.5)); blue\_points\_8 — a Point \[blue\] drawn in plane (location=(6.8, 6.4)); red\_points — a Point \[red\] drawn in plane (location=(0.9, 6.3)); red\_points\_2 — a Point \[red\] drawn in plane (location=(1.8, 4.8)); red\_points\_3 — a Point \[red\] drawn in plane (location=(3.3, 6.7)); red\_points\_4 — a Point \[red\] drawn in plane (location=(4.2, 5.2)); red\_points\_5 — a Point \[red\] drawn in plane (location=(5.4, 6.1)); red\_points\_6 — a Point \[red\] drawn in plane (location=(7.2, 4.8)); red\_points\_7 — a Point \[red\] drawn in plane (location=(2.2, 2.8)); red\_points\_8 — a Point \[red\] drawn in plane (location=(6.5, 1.2)); chosen — a Math \[text\] that says "$x\_2 \< 4 thin arrow.r thin upright("best gain")$"; heading — a Heading that says "Every Threshold Competes"; horizontal\_candidate — a Line \[yellow\] drawn in plane (start=(0.0, 4.0), end=(8.0, 4.0), dashed=True); low\_region — a Polygon \[blue\] drawn in plane (vertices=((0.0, 0.0), (8.0, 0.0), (8.0, 4.0), (0.0, 4.0)), fill\_opacity=0.08); high\_region — a Polygon \[red\] drawn in plane (vertices=((0.0, 4.0), (8.0, 4.0), (8.0, 8.0), (0.0, 8.0)), fill\_opacity=0.08) Actions: - [02:0.063](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=120.06300000000002): tree is shown on the screen, written out. - [02:5.647](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=125.64700000000002): root\_box is shown on the screen, written out. - [02:5.926](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=125.92600000000002): root\_label is shown on the screen, written out. - [02:9.153](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=129.15300000000002): left\_edge is shown on the screen, written out. - [02:10.512](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=130.512): low\_box is shown on the screen, written out. - [02:10.512](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=130.512): low\_label is shown on the screen, written out. - [02:12.136](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=132.13600000000002): right\_edge is shown on the screen, written out. - [02:13.17](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=133.17000000000002): high\_box is shown on the screen, written out. - [02:13.17](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=133.17000000000002): high\_label is shown on the screen, written out. ##### [02:14.6](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=134.5995) Narration: Notice what has not happened. We have not classified everything correctly, and neither child is pure. We have only made the labels less mixed. Training a tree means repeating this exact competition inside each child. Board: plane — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); blue\_points — a Point \[blue\] drawn in plane (location=(0.8, 1.2)); blue\_points\_2 — a Point \[blue\] drawn in plane (location=(1.5, 2.0)); blue\_points\_3 — a Point \[blue\] drawn in plane (location=(2.8, 1.0)); blue\_points\_4 — a Point \[blue\] drawn in plane (location=(3.6, 2.8)); blue\_points\_5 — a Point \[blue\] drawn in plane (location=(4.5, 1.5)); blue\_points\_6 — a Point \[blue\] drawn in plane (location=(5.6, 2.7)); blue\_points\_7 — a Point \[blue\] drawn in plane (location=(2.4, 5.5)); blue\_points\_8 — a Point \[blue\] drawn in plane (location=(6.8, 6.4)); red\_points — a Point \[red\] drawn in plane (location=(0.9, 6.3)); red\_points\_2 — a Point \[red\] drawn in plane (location=(1.8, 4.8)); red\_points\_3 — a Point \[red\] drawn in plane (location=(3.3, 6.7)); red\_points\_4 — a Point \[red\] drawn in plane (location=(4.2, 5.2)); red\_points\_5 — a Point \[red\] drawn in plane (location=(5.4, 6.1)); red\_points\_6 — a Point \[red\] drawn in plane (location=(7.2, 4.8)); red\_points\_7 — a Point \[red\] drawn in plane (location=(2.2, 2.8)); red\_points\_8 — a Point \[red\] drawn in plane (location=(6.5, 1.2)); chosen — a Math \[text\] that says "$x\_2 \< 4 thin arrow.r thin upright("best gain")$"; tree — a Figure (x\_range=(0.0, 10.0), y\_range=(0.0, 4.0), aspect=(5.0, 2.0)); heading — a Heading that says "Every Threshold Competes"; horizontal\_candidate — a Line \[yellow\] drawn in plane (start=(0.0, 4.0), end=(8.0, 4.0), dashed=True); low\_region — a Polygon \[blue\] drawn in plane (vertices=((0.0, 0.0), (8.0, 0.0), (8.0, 4.0), (0.0, 4.0)), fill\_opacity=0.08); high\_region — a Polygon \[red\] drawn in plane (vertices=((0.0, 4.0), (8.0, 4.0), (8.0, 8.0), (0.0, 8.0)), fill\_opacity=0.08); root\_box — a Polygon \[yellow\] drawn in tree (vertices=((3.5, 2.9), (6.5, 2.9), (6.5, 3.8), (3.5, 3.8)), fill\_opacity=0.12); root\_label — a Math \[yellow\] that says "$x\_2 \< 4?$" drawn in tree; left\_edge — a Line \[gray\] drawn in tree (start=(4.4, 2.9), end=(2.5, 1.7)); right\_edge — a Line \[gray\] drawn in tree (start=(5.6, 2.9), end=(7.5, 1.7)); low\_box — a Polygon \[blue\] drawn in tree (vertices=((1.0, 0.8), (4.0, 0.8), (4.0, 1.7), (1.0, 1.7)), fill\_opacity=0.12); low\_label — a Math \[blue\] that says "$6 B, thin 2 R$" drawn in tree; high\_box — a Polygon \[red\] drawn in tree (vertices=((6.0, 0.8), (9.0, 0.8), (9.0, 1.7), (6.0, 1.7)), fill\_opacity=0.12); high\_label — a Math \[red\] that says "$2 B, thin 6 R$" drawn in tree Actions: - [02:19.493](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=139.49300000000002): low\_label is indicated — a transient flash. - [02:20.271](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=140.27100000000002): high\_label is indicated — a transient flash. ##### [02:29.73](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=149.73000000000002) Narration: That is the basic fitting loop behind the library call. Enumerate legal feature thresholds, score their weighted impurity, choose the best gain, partition the observations, and repeat on the resulting nodes. Board: Unchanged from the preceding beat in this scene. Actions: - [02:38.368](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=158.36800000000002): A box is drawn around chosen. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): candidates is hidden from the screen — left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): chosen is hidden from the screen — left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): heading is hidden from the screen — left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): plane is hidden from the screen — left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): blue\_points is hidden from the screen — plane left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): blue\_points\_2 is hidden from the screen — plane left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): blue\_points\_3 is hidden from the screen — plane left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): blue\_points\_4 is hidden from the screen — plane left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): blue\_points\_5 is hidden from the screen — plane left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): blue\_points\_6 is hidden from the screen — plane left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): blue\_points\_7 is hidden from the screen — plane left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): blue\_points\_8 is hidden from the screen — plane left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): red\_points is hidden from the screen — plane left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): red\_points\_2 is hidden from the screen — plane left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): red\_points\_3 is hidden from the screen — plane left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): red\_points\_4 is hidden from the screen — plane left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): red\_points\_5 is hidden from the screen — plane left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): red\_points\_6 is hidden from the screen — plane left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): red\_points\_7 is hidden from the screen — plane left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): red\_points\_8 is hidden from the screen — plane left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): horizontal\_candidate is hidden from the screen — plane left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): low\_region is hidden from the screen — plane left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): high\_region is hidden from the screen — plane left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): tree is hidden from the screen — left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): root\_box is hidden from the screen — tree left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): root\_label is hidden from the screen — tree left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): left\_edge is hidden from the screen — tree left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): right\_edge is hidden from the screen — tree left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): low\_box is hidden from the screen — tree left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): low\_label is hidden from the screen — tree left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): high\_box is hidden from the screen — tree left the board. - [02:43.768](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=163.76837500000002): high\_label is hidden from the screen — tree left the board. ### Scene 2: [Growing to Purity](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=164.81004166666668) Span: 02:44.81–05:30.507 (164.81004166666668s–330.5074166666667s). #### Objects - blue\_points: a Point \[blue\] drawn in plane (location=(0.8, 1.2)) - blue\_points\_2: a Point \[blue\] drawn in plane (location=(1.5, 2.0)) - blue\_points\_3: a Point \[blue\] drawn in plane (location=(2.8, 1.0)) - blue\_points\_4: a Point \[blue\] drawn in plane (location=(3.6, 2.8)) - blue\_points\_5: a Point \[blue\] drawn in plane (location=(4.5, 1.5)) - blue\_points\_6: a Point \[blue\] drawn in plane (location=(5.6, 2.7)) - blue\_points\_7: a Point \[blue\] drawn in plane (location=(2.4, 5.5)) - blue\_points\_8: a Point \[blue\] drawn in plane (location=(6.8, 6.4)) - edge\_high: a Line \[gray\] drawn in tree (start=(6.7, 6.0), end=(8.8, 5.0)) - edge\_high\_a: a Line \[gray\] drawn in tree (start=(8.3, 4.2), end=(7.2, 3.2)) - edge\_high\_b: a Line \[gray\] drawn in tree (start=(9.3, 4.2), end=(10.4, 3.2)) - edge\_high\_c: a Line \[gray\] drawn in tree (start=(6.7, 2.4), end=(5.8, 1.4)) - edge\_high\_d: a Line \[gray\] drawn in tree (start=(7.7, 2.4), end=(8.2, 1.4)) - edge\_low: a Line \[gray\] drawn in tree (start=(5.3, 6.0), end=(3.2, 5.0)) - edge\_low\_a: a Line \[gray\] drawn in tree (start=(2.7, 4.2), end=(1.6, 3.2)) - edge\_low\_b: a Line \[gray\] drawn in tree (start=(3.7, 4.2), end=(4.8, 3.2)) - edge\_low\_c: a Line \[gray\] drawn in tree (start=(1.2, 2.4), end=(0.8, 1.4)) - edge\_low\_d: a Line \[gray\] drawn in tree (start=(2.0, 2.4), end=(2.8, 1.4)) - heading: a Heading that says "Repeat Until Every Leaf Is Pure" - high\_a: a Polygon \[magenta\] drawn in tree (vertices=((6.0, 2.4), (8.4, 2.4), (8.4, 3.2), (6.0, 3.2)), fill\_opacity=0.1) - high\_a\_text: a Math \[magenta\] that says "$x\_1 \< 2.8?$" drawn in tree - high\_b: a Polygon \[magenta\] drawn in tree (vertices=((9.2, 2.4), (11.6, 2.4), (11.6, 3.2), (9.2, 3.2)), fill\_opacity=0.1) - high\_b\_text: a Math \[magenta\] that says "$x\_2 \< 5.6?$" drawn in tree - high\_c: a Polygon \[yellow\] drawn in tree (vertices=((5.0, 0.6), (6.6, 0.6), (6.6, 1.4), (5.0, 1.4)), fill\_opacity=0.1) - high\_c\_text: a Math \[yellow\] that says "$x\_2 \< 5.1?$" drawn in tree - high\_cut\_five: a Line \[magenta\] drawn in plane (start=(1.5, 5.1), end=(1.5, 8.0)) - high\_cut\_four: a Line \[magenta\] drawn in plane (start=(0.0, 5.1), end=(2.8, 5.1)) - high\_cut\_one: a Line \[magenta\] drawn in plane (start=(6.2, 4.0), end=(6.2, 8.0)) - high\_cut\_three: a Line \[magenta\] drawn in plane (start=(2.8, 4.0), end=(2.8, 8.0)) - high\_cut\_two: a Line \[magenta\] drawn in plane (start=(6.2, 5.6), end=(8.0, 5.6)) - high\_d: a Polygon \[red\] drawn in tree (vertices=((7.3, 0.6), (9.1, 0.6), (9.1, 1.4), (7.3, 1.4)), fill\_opacity=0.12) - high\_d\_text: a Math \[red\] that says "$R$" drawn in tree - high\_node: a Polygon \[magenta\] drawn in tree (vertices=((7.6, 4.2), (10.0, 4.2), (10.0, 5.0), (7.6, 5.0)), fill\_opacity=0.1) - high\_text: a Math \[magenta\] that says "$x\_1 \< 6.2?$" drawn in tree - leaf\_count: a Math \[text\] that says "$upright("leaves") = 2$" - low\_a: a Polygon \[green\] drawn in tree (vertices=((0.5, 2.4), (2.7, 2.4), (2.7, 3.2), (0.5, 3.2)), fill\_opacity=0.1) - low\_a\_text: a Math \[green\] that says "$x\_2 \< 2.5?$" drawn in tree - low\_b: a Polygon \[red\] drawn in tree (vertices=((3.7, 2.4), (5.9, 2.4), (5.9, 3.2), (3.7, 3.2)), fill\_opacity=0.12) - low\_b\_text: a Math \[red\] that says "$R$" drawn in tree - low\_c: a Polygon \[blue\] drawn in tree (vertices=((0.1, 0.6), (1.5, 0.6), (1.5, 1.4), (0.1, 1.4)), fill\_opacity=0.12) - low\_c\_text: a Math \[blue\] that says "$B$" drawn in tree - low\_cut\_one: a Line \[green\] drawn in plane (start=(6.1, 0.0), end=(6.1, 4.0)) - low\_cut\_three: a Line \[green\] drawn in plane (start=(2.5, 2.5), end=(2.5, 4.0)) - low\_cut\_two: a Line \[green\] drawn in plane (start=(0.0, 2.5), end=(6.1, 2.5)) - low\_d: a Polygon \[yellow\] drawn in tree (vertices=((2.0, 0.6), (3.6, 0.6), (3.6, 1.4), (2.0, 1.4)), fill\_opacity=0.1) - low\_d\_text: a Math \[yellow\] that says "$x\_1 \< 2.5?$" drawn in tree - low\_node: a Polygon \[green\] drawn in tree (vertices=((2.0, 4.2), (4.4, 4.2), (4.4, 5.0), (2.0, 5.0)), fill\_opacity=0.1) - low\_text: a Math \[green\] that says "$x\_1 \< 6.1?$" drawn in tree - plane: an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)) - point: a Point \[yellow\] drawn in plane (location=(2.45, 5.4)) - point\_2: a Point \[yellow\] drawn in plane (location=(2.9, 5.4)) - purity: a Math \[text\] that says "$G\_(upright("every leaf")) = 0$" - red\_points: a Point \[red\] drawn in plane (location=(0.9, 6.3)) - red\_points\_2: a Point \[red\] drawn in plane (location=(1.8, 4.8)) - red\_points\_3: a Point \[red\] drawn in plane (location=(3.3, 6.7)) - red\_points\_4: a Point \[red\] drawn in plane (location=(4.2, 5.2)) - red\_points\_5: a Point \[red\] drawn in plane (location=(5.4, 6.1)) - red\_points\_6: a Point \[red\] drawn in plane (location=(7.2, 4.8)) - red\_points\_7: a Point \[red\] drawn in plane (location=(2.2, 2.8)) - red\_points\_8: a Point \[red\] drawn in plane (location=(6.5, 1.2)) - root\_box: a Polygon \[yellow\] drawn in tree (vertices=((4.7, 6.0), (7.3, 6.0), (7.3, 6.8), (4.7, 6.8)), fill\_opacity=0.12) - root\_cut: a Line \[yellow\] drawn in plane (start=(0.0, 4.0), end=(8.0, 4.0)) - root\_text: a Math \[yellow\] that says "$x\_2 \< 4?$" drawn in tree - tree: a Figure (x\_range=(0.0, 12.0), y\_range=(0.0, 7.0), aspect=(12.0, 7.0)) #### Beats ##### [02:44.81](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=164.81004166666668) Narration: Continue from the root split. The tree on the left and feature space on the right will grow together. A new internal node must always correspond to a new axis-aligned cut inside exactly one existing rectangle. Board: Empty. Actions: - [02:44.81](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=164.81004166666668): heading is shown on the screen, written out. - [02:44.81](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=164.81004166666668): tree is shown on the screen, written out. - [02:44.81](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=164.81004166666668): plane is shown on the screen, written out. - [02:44.81](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=164.81004166666668): blue\_points is shown on the screen, written out. - [02:44.86](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=164.8600416666667): blue\_points\_2 is shown on the screen, written out. - [02:44.91](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=164.91004166666667): blue\_points\_3 is shown on the screen, written out. - [02:44.96](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=164.96004166666668): blue\_points\_4 is shown on the screen, written out. - [02:45.01](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=165.01004166666667): blue\_points\_5 is shown on the screen, written out. - [02:45.06](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=165.06004166666668): blue\_points\_6 is shown on the screen, written out. - [02:45.11](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=165.1100416666667): blue\_points\_7 is shown on the screen, written out. - [02:45.16](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=165.16004166666667): blue\_points\_8 is shown on the screen, written out. - [02:45.21](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=165.21004166666668): red\_points is shown on the screen, written out. - [02:45.26](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=165.26004166666667): red\_points\_2 is shown on the screen, written out. - [02:45.31](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=165.31004166666668): red\_points\_3 is shown on the screen, written out. - [02:45.36](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=165.3600416666667): red\_points\_4 is shown on the screen, written out. - [02:45.41](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=165.41004166666667): red\_points\_5 is shown on the screen, written out. - [02:45.46](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=165.46004166666668): red\_points\_6 is shown on the screen, written out. - [02:45.51](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=165.51004166666667): red\_points\_7 is shown on the screen, written out. - [02:45.56](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=165.56004166666668): red\_points\_8 is shown on the screen, written out. - [02:45.658](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=165.6580416666667): root\_box is shown on the screen, written out. - [02:45.658](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=165.6580416666667): root\_text is shown on the screen, written out. - [02:47.585](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=167.58504166666668): edge\_low is shown on the screen, written out. - [02:47.585](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=167.58504166666668): low\_node is shown on the screen, written out. - [02:47.585](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=167.58504166666668): low\_text is shown on the screen, written out. - [02:49.001](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=169.00104166666668): edge\_high is shown on the screen, written out. - [02:49.001](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=169.00104166666668): high\_node is shown on the screen, written out. - [02:49.001](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=169.00104166666668): high\_text is shown on the screen, written out. - [02:49.431](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=169.4310416666667): leaf\_count is shown on the screen, written out. - [02:54.516](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=174.51604166666667): root\_cut is shown on the screen, written out. ##### [02:58.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=178.48304166666668) Narration: Start in the lower rectangle. It contains six blue observations and two red ones. The best available gain first separates the far-right red point with x one less than six point one. Board: tree — a Figure (x\_range=(0.0, 12.0), y\_range=(0.0, 7.0), aspect=(12.0, 7.0)); leaf\_count — a Math \[text\] that says "$upright("leaves") = 2$"; plane — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); heading — a Heading that says "Repeat Until Every Leaf Is Pure"; blue\_points — a Point \[blue\] drawn in plane (location=(0.8, 1.2)); blue\_points\_2 — a Point \[blue\] drawn in plane (location=(1.5, 2.0)); blue\_points\_3 — a Point \[blue\] drawn in plane (location=(2.8, 1.0)); blue\_points\_4 — a Point \[blue\] drawn in plane (location=(3.6, 2.8)); blue\_points\_5 — a Point \[blue\] drawn in plane (location=(4.5, 1.5)); blue\_points\_6 — a Point \[blue\] drawn in plane (location=(5.6, 2.7)); blue\_points\_7 — a Point \[blue\] drawn in plane (location=(2.4, 5.5)); blue\_points\_8 — a Point \[blue\] drawn in plane (location=(6.8, 6.4)); red\_points — a Point \[red\] drawn in plane (location=(0.9, 6.3)); red\_points\_2 — a Point \[red\] drawn in plane (location=(1.8, 4.8)); red\_points\_3 — a Point \[red\] drawn in plane (location=(3.3, 6.7)); red\_points\_4 — a Point \[red\] drawn in plane (location=(4.2, 5.2)); red\_points\_5 — a Point \[red\] drawn in plane (location=(5.4, 6.1)); red\_points\_6 — a Point \[red\] drawn in plane (location=(7.2, 4.8)); red\_points\_7 — a Point \[red\] drawn in plane (location=(2.2, 2.8)); red\_points\_8 — a Point \[red\] drawn in plane (location=(6.5, 1.2)); root\_box — a Polygon \[yellow\] drawn in tree (vertices=((4.7, 6.0), (7.3, 6.0), (7.3, 6.8), (4.7, 6.8)), fill\_opacity=0.12); root\_text — a Math \[yellow\] that says "$x\_2 \< 4?$" drawn in tree; edge\_low — a Line \[gray\] drawn in tree (start=(5.3, 6.0), end=(3.2, 5.0)); edge\_high — a Line \[gray\] drawn in tree (start=(6.7, 6.0), end=(8.8, 5.0)); low\_node — a Polygon \[green\] drawn in tree (vertices=((2.0, 4.2), (4.4, 4.2), (4.4, 5.0), (2.0, 5.0)), fill\_opacity=0.1); low\_text — a Math \[green\] that says "$x\_1 \< 6.1?$" drawn in tree; high\_node — a Polygon \[magenta\] drawn in tree (vertices=((7.6, 4.2), (10.0, 4.2), (10.0, 5.0), (7.6, 5.0)), fill\_opacity=0.1); high\_text — a Math \[magenta\] that says "$x\_1 \< 6.2?$" drawn in tree; root\_cut — a Line \[yellow\] drawn in plane (start=(0.0, 4.0), end=(8.0, 4.0)) Actions: - [03:3.463](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=183.46304166666667): low\_b is shown on the screen, written out. - [03:3.463](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=183.46304166666667): low\_b\_text is shown on the screen, written out. - [03:4.891](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=184.8910416666667): low\_a is shown on the screen, written out. - [03:4.891](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=184.8910416666667): low\_a\_text is shown on the screen, written out. - [03:6.122](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=186.1220416666667): edge\_low\_a is shown on the screen, written out. - [03:6.122](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=186.1220416666667): edge\_low\_b is shown on the screen, written out. - [03:6.714](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=186.71404166666667): red\_points\_8 is indicated — a transient flash. - [03:9.176](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=189.17604166666666): low\_cut\_one is shown on the screen, written out. - [03:10.441](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=190.44104166666668): leaf\_count becomes "$upright("leaves") = 4$". ##### [03:11.041](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=191.04104166666667) Narration: The right child is pure, but the large left child is not. Within that child, a horizontal cut at x two equals two point five creates a pure blue strip below and a mixed strip above. Board: tree — a Figure (x\_range=(0.0, 12.0), y\_range=(0.0, 7.0), aspect=(12.0, 7.0)); leaf\_count — a Math \[text\] that says "$upright("leaves") = 2$"; plane — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); heading — a Heading that says "Repeat Until Every Leaf Is Pure"; blue\_points — a Point \[blue\] drawn in plane (location=(0.8, 1.2)); blue\_points\_2 — a Point \[blue\] drawn in plane (location=(1.5, 2.0)); blue\_points\_3 — a Point \[blue\] drawn in plane (location=(2.8, 1.0)); blue\_points\_4 — a Point \[blue\] drawn in plane (location=(3.6, 2.8)); blue\_points\_5 — a Point \[blue\] drawn in plane (location=(4.5, 1.5)); blue\_points\_6 — a Point \[blue\] drawn in plane (location=(5.6, 2.7)); blue\_points\_7 — a Point \[blue\] drawn in plane (location=(2.4, 5.5)); blue\_points\_8 — a Point \[blue\] drawn in plane (location=(6.8, 6.4)); red\_points — a Point \[red\] drawn in plane (location=(0.9, 6.3)); red\_points\_2 — a Point \[red\] drawn in plane (location=(1.8, 4.8)); red\_points\_3 — a Point \[red\] drawn in plane (location=(3.3, 6.7)); red\_points\_4 — a Point \[red\] drawn in plane (location=(4.2, 5.2)); red\_points\_5 — a Point \[red\] drawn in plane (location=(5.4, 6.1)); red\_points\_6 — a Point \[red\] drawn in plane (location=(7.2, 4.8)); red\_points\_7 — a Point \[red\] drawn in plane (location=(2.2, 2.8)); red\_points\_8 — a Point \[red\] drawn in plane (location=(6.5, 1.2)); root\_box — a Polygon \[yellow\] drawn in tree (vertices=((4.7, 6.0), (7.3, 6.0), (7.3, 6.8), (4.7, 6.8)), fill\_opacity=0.12); root\_text — a Math \[yellow\] that says "$x\_2 \< 4?$" drawn in tree; edge\_low — a Line \[gray\] drawn in tree (start=(5.3, 6.0), end=(3.2, 5.0)); edge\_high — a Line \[gray\] drawn in tree (start=(6.7, 6.0), end=(8.8, 5.0)); low\_node — a Polygon \[green\] drawn in tree (vertices=((2.0, 4.2), (4.4, 4.2), (4.4, 5.0), (2.0, 5.0)), fill\_opacity=0.1); low\_text — a Math \[green\] that says "$x\_1 \< 6.1?$" drawn in tree; high\_node — a Polygon \[magenta\] drawn in tree (vertices=((7.6, 4.2), (10.0, 4.2), (10.0, 5.0), (7.6, 5.0)), fill\_opacity=0.1); high\_text — a Math \[magenta\] that says "$x\_1 \< 6.2?$" drawn in tree; root\_cut — a Line \[yellow\] drawn in plane (start=(0.0, 4.0), end=(8.0, 4.0)); low\_cut\_one — a Line \[green\] drawn in plane (start=(6.1, 0.0), end=(6.1, 4.0)); edge\_low\_a — a Line \[gray\] drawn in tree (start=(2.7, 4.2), end=(1.6, 3.2)); edge\_low\_b — a Line \[gray\] drawn in tree (start=(3.7, 4.2), end=(4.8, 3.2)); low\_a — a Polygon \[green\] drawn in tree (vertices=((0.5, 2.4), (2.7, 2.4), (2.7, 3.2), (0.5, 3.2)), fill\_opacity=0.1); low\_a\_text — a Math \[green\] that says "$x\_2 \< 2.5?$" drawn in tree; low\_b — a Polygon \[red\] drawn in tree (vertices=((3.7, 2.4), (5.9, 2.4), (5.9, 3.2), (3.7, 3.2)), fill\_opacity=0.12); low\_b\_text — a Math \[red\] that says "$R$" drawn in tree Actions: - [03:17.136](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=197.1360416666667): low\_cut\_two is shown on the screen, written out. - [03:20.909](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=200.90904166666667): edge\_low\_c is shown on the screen, written out. - [03:20.909](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=200.90904166666667): low\_c is shown on the screen, written out. - [03:20.909](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=200.90904166666667): low\_c\_text is shown on the screen, written out. - [03:22.302](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=202.3020416666667): edge\_low\_d is shown on the screen, written out. - [03:22.302](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=202.3020416666667): low\_d is shown on the screen, written out. - [03:22.302](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=202.3020416666667): low\_d\_text is shown on the screen, written out. - [03:23.661](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=203.66104166666668): leaf\_count becomes "$upright("leaves") = 6$". ##### [03:24.261](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=204.26104166666667) Narration: That upper strip is mixed because of one red observation at two point two, two point eight. Another vertical threshold isolates it. The training algorithm is rewarded, because two new leaves become perfectly pure. Board: tree — a Figure (x\_range=(0.0, 12.0), y\_range=(0.0, 7.0), aspect=(12.0, 7.0)); leaf\_count — a Math \[text\] that says "$upright("leaves") = 2$"; plane — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); heading — a Heading that says "Repeat Until Every Leaf Is Pure"; blue\_points — a Point \[blue\] drawn in plane (location=(0.8, 1.2)); blue\_points\_2 — a Point \[blue\] drawn in plane (location=(1.5, 2.0)); blue\_points\_3 — a Point \[blue\] drawn in plane (location=(2.8, 1.0)); blue\_points\_4 — a Point \[blue\] drawn in plane (location=(3.6, 2.8)); blue\_points\_5 — a Point \[blue\] drawn in plane (location=(4.5, 1.5)); blue\_points\_6 — a Point \[blue\] drawn in plane (location=(5.6, 2.7)); blue\_points\_7 — a Point \[blue\] drawn in plane (location=(2.4, 5.5)); blue\_points\_8 — a Point \[blue\] drawn in plane (location=(6.8, 6.4)); red\_points — a Point \[red\] drawn in plane (location=(0.9, 6.3)); red\_points\_2 — a Point \[red\] drawn in plane (location=(1.8, 4.8)); red\_points\_3 — a Point \[red\] drawn in plane (location=(3.3, 6.7)); red\_points\_4 — a Point \[red\] drawn in plane (location=(4.2, 5.2)); red\_points\_5 — a Point \[red\] drawn in plane (location=(5.4, 6.1)); red\_points\_6 — a Point \[red\] drawn in plane (location=(7.2, 4.8)); red\_points\_7 — a Point \[red\] drawn in plane (location=(2.2, 2.8)); red\_points\_8 — a Point \[red\] drawn in plane (location=(6.5, 1.2)); root\_box — a Polygon \[yellow\] drawn in tree (vertices=((4.7, 6.0), (7.3, 6.0), (7.3, 6.8), (4.7, 6.8)), fill\_opacity=0.12); root\_text — a Math \[yellow\] that says "$x\_2 \< 4?$" drawn in tree; edge\_low — a Line \[gray\] drawn in tree (start=(5.3, 6.0), end=(3.2, 5.0)); edge\_high — a Line \[gray\] drawn in tree (start=(6.7, 6.0), end=(8.8, 5.0)); low\_node — a Polygon \[green\] drawn in tree (vertices=((2.0, 4.2), (4.4, 4.2), (4.4, 5.0), (2.0, 5.0)), fill\_opacity=0.1); low\_text — a Math \[green\] that says "$x\_1 \< 6.1?$" drawn in tree; high\_node — a Polygon \[magenta\] drawn in tree (vertices=((7.6, 4.2), (10.0, 4.2), (10.0, 5.0), (7.6, 5.0)), fill\_opacity=0.1); high\_text — a Math \[magenta\] that says "$x\_1 \< 6.2?$" drawn in tree; root\_cut — a Line \[yellow\] drawn in plane (start=(0.0, 4.0), end=(8.0, 4.0)); low\_cut\_one — a Line \[green\] drawn in plane (start=(6.1, 0.0), end=(6.1, 4.0)); edge\_low\_a — a Line \[gray\] drawn in tree (start=(2.7, 4.2), end=(1.6, 3.2)); edge\_low\_b — a Line \[gray\] drawn in tree (start=(3.7, 4.2), end=(4.8, 3.2)); low\_a — a Polygon \[green\] drawn in tree (vertices=((0.5, 2.4), (2.7, 2.4), (2.7, 3.2), (0.5, 3.2)), fill\_opacity=0.1); low\_a\_text — a Math \[green\] that says "$x\_2 \< 2.5?$" drawn in tree; low\_b — a Polygon \[red\] drawn in tree (vertices=((3.7, 2.4), (5.9, 2.4), (5.9, 3.2), (3.7, 3.2)), fill\_opacity=0.12); low\_b\_text — a Math \[red\] that says "$R$" drawn in tree; low\_cut\_two — a Line \[green\] drawn in plane (start=(0.0, 2.5), end=(6.1, 2.5)); edge\_low\_c — a Line \[gray\] drawn in tree (start=(1.2, 2.4), end=(0.8, 1.4)); edge\_low\_d — a Line \[gray\] drawn in tree (start=(2.0, 2.4), end=(2.8, 1.4)); low\_c — a Polygon \[blue\] drawn in tree (vertices=((0.1, 0.6), (1.5, 0.6), (1.5, 1.4), (0.1, 1.4)), fill\_opacity=0.12); low\_c\_text — a Math \[blue\] that says "$B$" drawn in tree; low\_d — a Polygon \[yellow\] drawn in tree (vertices=((2.0, 0.6), (3.6, 0.6), (3.6, 1.4), (2.0, 1.4)), fill\_opacity=0.1); low\_d\_text — a Math \[yellow\] that says "$x\_1 \< 2.5?$" drawn in tree Actions: - [03:26.618](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=206.61804166666667): red\_points\_7 is indicated — a transient flash. - [03:30.112](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=210.11204166666667): low\_cut\_three is shown on the screen, written out. - [03:36.544](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=216.54404166666666): low\_d\_text is indicated — a transient flash. ##### [03:37.946](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=217.94554166666668) Narration: Now do the same work above the root. Most points there are red, but two blue exceptions force the recursion to continue. The first upper split separates the far-right pair from the rest. Board: tree — a Figure (x\_range=(0.0, 12.0), y\_range=(0.0, 7.0), aspect=(12.0, 7.0)); leaf\_count — a Math \[text\] that says "$upright("leaves") = 2$"; plane — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); heading — a Heading that says "Repeat Until Every Leaf Is Pure"; blue\_points — a Point \[blue\] drawn in plane (location=(0.8, 1.2)); blue\_points\_2 — a Point \[blue\] drawn in plane (location=(1.5, 2.0)); blue\_points\_3 — a Point \[blue\] drawn in plane (location=(2.8, 1.0)); blue\_points\_4 — a Point \[blue\] drawn in plane (location=(3.6, 2.8)); blue\_points\_5 — a Point \[blue\] drawn in plane (location=(4.5, 1.5)); blue\_points\_6 — a Point \[blue\] drawn in plane (location=(5.6, 2.7)); blue\_points\_7 — a Point \[blue\] drawn in plane (location=(2.4, 5.5)); blue\_points\_8 — a Point \[blue\] drawn in plane (location=(6.8, 6.4)); red\_points — a Point \[red\] drawn in plane (location=(0.9, 6.3)); red\_points\_2 — a Point \[red\] drawn in plane (location=(1.8, 4.8)); red\_points\_3 — a Point \[red\] drawn in plane (location=(3.3, 6.7)); red\_points\_4 — a Point \[red\] drawn in plane (location=(4.2, 5.2)); red\_points\_5 — a Point \[red\] drawn in plane (location=(5.4, 6.1)); red\_points\_6 — a Point \[red\] drawn in plane (location=(7.2, 4.8)); red\_points\_7 — a Point \[red\] drawn in plane (location=(2.2, 2.8)); red\_points\_8 — a Point \[red\] drawn in plane (location=(6.5, 1.2)); root\_box — a Polygon \[yellow\] drawn in tree (vertices=((4.7, 6.0), (7.3, 6.0), (7.3, 6.8), (4.7, 6.8)), fill\_opacity=0.12); root\_text — a Math \[yellow\] that says "$x\_2 \< 4?$" drawn in tree; edge\_low — a Line \[gray\] drawn in tree (start=(5.3, 6.0), end=(3.2, 5.0)); edge\_high — a Line \[gray\] drawn in tree (start=(6.7, 6.0), end=(8.8, 5.0)); low\_node — a Polygon \[green\] drawn in tree (vertices=((2.0, 4.2), (4.4, 4.2), (4.4, 5.0), (2.0, 5.0)), fill\_opacity=0.1); low\_text — a Math \[green\] that says "$x\_1 \< 6.1?$" drawn in tree; high\_node — a Polygon \[magenta\] drawn in tree (vertices=((7.6, 4.2), (10.0, 4.2), (10.0, 5.0), (7.6, 5.0)), fill\_opacity=0.1); high\_text — a Math \[magenta\] that says "$x\_1 \< 6.2?$" drawn in tree; root\_cut — a Line \[yellow\] drawn in plane (start=(0.0, 4.0), end=(8.0, 4.0)); low\_cut\_one — a Line \[green\] drawn in plane (start=(6.1, 0.0), end=(6.1, 4.0)); edge\_low\_a — a Line \[gray\] drawn in tree (start=(2.7, 4.2), end=(1.6, 3.2)); edge\_low\_b — a Line \[gray\] drawn in tree (start=(3.7, 4.2), end=(4.8, 3.2)); low\_a — a Polygon \[green\] drawn in tree (vertices=((0.5, 2.4), (2.7, 2.4), (2.7, 3.2), (0.5, 3.2)), fill\_opacity=0.1); low\_a\_text — a Math \[green\] that says "$x\_2 \< 2.5?$" drawn in tree; low\_b — a Polygon \[red\] drawn in tree (vertices=((3.7, 2.4), (5.9, 2.4), (5.9, 3.2), (3.7, 3.2)), fill\_opacity=0.12); low\_b\_text — a Math \[red\] that says "$R$" drawn in tree; low\_cut\_two — a Line \[green\] drawn in plane (start=(0.0, 2.5), end=(6.1, 2.5)); edge\_low\_c — a Line \[gray\] drawn in tree (start=(1.2, 2.4), end=(0.8, 1.4)); edge\_low\_d — a Line \[gray\] drawn in tree (start=(2.0, 2.4), end=(2.8, 1.4)); low\_c — a Polygon \[blue\] drawn in tree (vertices=((0.1, 0.6), (1.5, 0.6), (1.5, 1.4), (0.1, 1.4)), fill\_opacity=0.12); low\_c\_text — a Math \[blue\] that says "$B$" drawn in tree; low\_d — a Polygon \[yellow\] drawn in tree (vertices=((2.0, 0.6), (3.6, 0.6), (3.6, 1.4), (2.0, 1.4)), fill\_opacity=0.1); low\_d\_text — a Math \[yellow\] that says "$x\_1 \< 2.5?$" drawn in tree; low\_cut\_three — a Line \[green\] drawn in plane (start=(2.5, 2.5), end=(2.5, 4.0)) Actions: - [03:43.17](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=223.1700416666667): blue\_points\_7 is indicated — a transient flash. - [03:43.17](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=223.1700416666667): blue\_points\_8 is indicated — a transient flash. - [03:46.455](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=226.45504166666666): high\_cut\_one is shown on the screen, written out. - [03:47.164](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=227.16404166666666): edge\_high\_a is shown on the screen, written out. - [03:47.164](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=227.16404166666666): edge\_high\_b is shown on the screen, written out. - [03:48.417](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=228.41704166666668): high\_b is shown on the screen, written out. - [03:48.417](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=228.41704166666668): high\_b\_text is shown on the screen, written out. - [03:49.532](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=229.5320416666667): high\_a is shown on the screen, written out. - [03:49.532](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=229.5320416666667): high\_a\_text is shown on the screen, written out. ##### [03:50.991](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=230.99104166666666) Narration: The far-right pair still disagrees. A horizontal cut isolates the blue point at the top from the red point below. Again, training impurity falls to zero in both resulting leaves. Board: tree — a Figure (x\_range=(0.0, 12.0), y\_range=(0.0, 7.0), aspect=(12.0, 7.0)); leaf\_count — a Math \[text\] that says "$upright("leaves") = 2$"; plane — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); heading — a Heading that says "Repeat Until Every Leaf Is Pure"; blue\_points — a Point \[blue\] drawn in plane (location=(0.8, 1.2)); blue\_points\_2 — a Point \[blue\] drawn in plane (location=(1.5, 2.0)); blue\_points\_3 — a Point \[blue\] drawn in plane (location=(2.8, 1.0)); blue\_points\_4 — a Point \[blue\] drawn in plane (location=(3.6, 2.8)); blue\_points\_5 — a Point \[blue\] drawn in plane (location=(4.5, 1.5)); blue\_points\_6 — a Point \[blue\] drawn in plane (location=(5.6, 2.7)); blue\_points\_7 — a Point \[blue\] drawn in plane (location=(2.4, 5.5)); blue\_points\_8 — a Point \[blue\] drawn in plane (location=(6.8, 6.4)); red\_points — a Point \[red\] drawn in plane (location=(0.9, 6.3)); red\_points\_2 — a Point \[red\] drawn in plane (location=(1.8, 4.8)); red\_points\_3 — a Point \[red\] drawn in plane (location=(3.3, 6.7)); red\_points\_4 — a Point \[red\] drawn in plane (location=(4.2, 5.2)); red\_points\_5 — a Point \[red\] drawn in plane (location=(5.4, 6.1)); red\_points\_6 — a Point \[red\] drawn in plane (location=(7.2, 4.8)); red\_points\_7 — a Point \[red\] drawn in plane (location=(2.2, 2.8)); red\_points\_8 — a Point \[red\] drawn in plane (location=(6.5, 1.2)); root\_box — a Polygon \[yellow\] drawn in tree (vertices=((4.7, 6.0), (7.3, 6.0), (7.3, 6.8), (4.7, 6.8)), fill\_opacity=0.12); root\_text — a Math \[yellow\] that says "$x\_2 \< 4?$" drawn in tree; edge\_low — a Line \[gray\] drawn in tree (start=(5.3, 6.0), end=(3.2, 5.0)); edge\_high — a Line \[gray\] drawn in tree (start=(6.7, 6.0), end=(8.8, 5.0)); low\_node — a Polygon \[green\] drawn in tree (vertices=((2.0, 4.2), (4.4, 4.2), (4.4, 5.0), (2.0, 5.0)), fill\_opacity=0.1); low\_text — a Math \[green\] that says "$x\_1 \< 6.1?$" drawn in tree; high\_node — a Polygon \[magenta\] drawn in tree (vertices=((7.6, 4.2), (10.0, 4.2), (10.0, 5.0), (7.6, 5.0)), fill\_opacity=0.1); high\_text — a Math \[magenta\] that says "$x\_1 \< 6.2?$" drawn in tree; root\_cut — a Line \[yellow\] drawn in plane (start=(0.0, 4.0), end=(8.0, 4.0)); low\_cut\_one — a Line \[green\] drawn in plane (start=(6.1, 0.0), end=(6.1, 4.0)); edge\_low\_a — a Line \[gray\] drawn in tree (start=(2.7, 4.2), end=(1.6, 3.2)); edge\_low\_b — a Line \[gray\] drawn in tree (start=(3.7, 4.2), end=(4.8, 3.2)); low\_a — a Polygon \[green\] drawn in tree (vertices=((0.5, 2.4), (2.7, 2.4), (2.7, 3.2), (0.5, 3.2)), fill\_opacity=0.1); low\_a\_text — a Math \[green\] that says "$x\_2 \< 2.5?$" drawn in tree; low\_b — a Polygon \[red\] drawn in tree (vertices=((3.7, 2.4), (5.9, 2.4), (5.9, 3.2), (3.7, 3.2)), fill\_opacity=0.12); low\_b\_text — a Math \[red\] that says "$R$" drawn in tree; low\_cut\_two — a Line \[green\] drawn in plane (start=(0.0, 2.5), end=(6.1, 2.5)); edge\_low\_c — a Line \[gray\] drawn in tree (start=(1.2, 2.4), end=(0.8, 1.4)); edge\_low\_d — a Line \[gray\] drawn in tree (start=(2.0, 2.4), end=(2.8, 1.4)); low\_c — a Polygon \[blue\] drawn in tree (vertices=((0.1, 0.6), (1.5, 0.6), (1.5, 1.4), (0.1, 1.4)), fill\_opacity=0.12); low\_c\_text — a Math \[blue\] that says "$B$" drawn in tree; low\_d — a Polygon \[yellow\] drawn in tree (vertices=((2.0, 0.6), (3.6, 0.6), (3.6, 1.4), (2.0, 1.4)), fill\_opacity=0.1); low\_d\_text — a Math \[yellow\] that says "$x\_1 \< 2.5?$" drawn in tree; low\_cut\_three — a Line \[green\] drawn in plane (start=(2.5, 2.5), end=(2.5, 4.0)); high\_cut\_one — a Line \[magenta\] drawn in plane (start=(6.2, 4.0), end=(6.2, 8.0)); edge\_high\_a — a Line \[gray\] drawn in tree (start=(8.3, 4.2), end=(7.2, 3.2)); edge\_high\_b — a Line \[gray\] drawn in tree (start=(9.3, 4.2), end=(10.4, 3.2)); high\_a — a Polygon \[magenta\] drawn in tree (vertices=((6.0, 2.4), (8.4, 2.4), (8.4, 3.2), (6.0, 3.2)), fill\_opacity=0.1); high\_a\_text — a Math \[magenta\] that says "$x\_1 \< 2.8?$" drawn in tree; high\_b — a Polygon \[magenta\] drawn in tree (vertices=((9.2, 2.4), (11.6, 2.4), (11.6, 3.2), (9.2, 3.2)), fill\_opacity=0.1); high\_b\_text — a Math \[magenta\] that says "$x\_2 \< 5.6?$" drawn in tree Actions: - [03:54.381](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=234.38104166666668): high\_cut\_two is shown on the screen, written out. - [03:56.018](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=236.01804166666668): blue\_points\_8 is indicated — a transient flash. - [04:1.266](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=241.2660416666667): high\_b\_text is indicated — a transient flash. ##### [04:4.409](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=244.40854166666668) Narration: The left upper region has its own blue exception. A split at x one equals two point eight narrows the search, and another horizontal split at x two equals five point one separates one red point. Board: tree — a Figure (x\_range=(0.0, 12.0), y\_range=(0.0, 7.0), aspect=(12.0, 7.0)); leaf\_count — a Math \[text\] that says "$upright("leaves") = 2$"; plane — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); heading — a Heading that says "Repeat Until Every Leaf Is Pure"; blue\_points — a Point \[blue\] drawn in plane (location=(0.8, 1.2)); blue\_points\_2 — a Point \[blue\] drawn in plane (location=(1.5, 2.0)); blue\_points\_3 — a Point \[blue\] drawn in plane (location=(2.8, 1.0)); blue\_points\_4 — a Point \[blue\] drawn in plane (location=(3.6, 2.8)); blue\_points\_5 — a Point \[blue\] drawn in plane (location=(4.5, 1.5)); blue\_points\_6 — a Point \[blue\] drawn in plane (location=(5.6, 2.7)); blue\_points\_7 — a Point \[blue\] drawn in plane (location=(2.4, 5.5)); blue\_points\_8 — a Point \[blue\] drawn in plane (location=(6.8, 6.4)); red\_points — a Point \[red\] drawn in plane (location=(0.9, 6.3)); red\_points\_2 — a Point \[red\] drawn in plane (location=(1.8, 4.8)); red\_points\_3 — a Point \[red\] drawn in plane (location=(3.3, 6.7)); red\_points\_4 — a Point \[red\] drawn in plane (location=(4.2, 5.2)); red\_points\_5 — a Point \[red\] drawn in plane (location=(5.4, 6.1)); red\_points\_6 — a Point \[red\] drawn in plane (location=(7.2, 4.8)); red\_points\_7 — a Point \[red\] drawn in plane (location=(2.2, 2.8)); red\_points\_8 — a Point \[red\] drawn in plane (location=(6.5, 1.2)); root\_box — a Polygon \[yellow\] drawn in tree (vertices=((4.7, 6.0), (7.3, 6.0), (7.3, 6.8), (4.7, 6.8)), fill\_opacity=0.12); root\_text — a Math \[yellow\] that says "$x\_2 \< 4?$" drawn in tree; edge\_low — a Line \[gray\] drawn in tree (start=(5.3, 6.0), end=(3.2, 5.0)); edge\_high — a Line \[gray\] drawn in tree (start=(6.7, 6.0), end=(8.8, 5.0)); low\_node — a Polygon \[green\] drawn in tree (vertices=((2.0, 4.2), (4.4, 4.2), (4.4, 5.0), (2.0, 5.0)), fill\_opacity=0.1); low\_text — a Math \[green\] that says "$x\_1 \< 6.1?$" drawn in tree; high\_node — a Polygon \[magenta\] drawn in tree (vertices=((7.6, 4.2), (10.0, 4.2), (10.0, 5.0), (7.6, 5.0)), fill\_opacity=0.1); high\_text — a Math \[magenta\] that says "$x\_1 \< 6.2?$" drawn in tree; root\_cut — a Line \[yellow\] drawn in plane (start=(0.0, 4.0), end=(8.0, 4.0)); low\_cut\_one — a Line \[green\] drawn in plane (start=(6.1, 0.0), end=(6.1, 4.0)); edge\_low\_a — a Line \[gray\] drawn in tree (start=(2.7, 4.2), end=(1.6, 3.2)); edge\_low\_b — a Line \[gray\] drawn in tree (start=(3.7, 4.2), end=(4.8, 3.2)); low\_a — a Polygon \[green\] drawn in tree (vertices=((0.5, 2.4), (2.7, 2.4), (2.7, 3.2), (0.5, 3.2)), fill\_opacity=0.1); low\_a\_text — a Math \[green\] that says "$x\_2 \< 2.5?$" drawn in tree; low\_b — a Polygon \[red\] drawn in tree (vertices=((3.7, 2.4), (5.9, 2.4), (5.9, 3.2), (3.7, 3.2)), fill\_opacity=0.12); low\_b\_text — a Math \[red\] that says "$R$" drawn in tree; low\_cut\_two — a Line \[green\] drawn in plane (start=(0.0, 2.5), end=(6.1, 2.5)); edge\_low\_c — a Line \[gray\] drawn in tree (start=(1.2, 2.4), end=(0.8, 1.4)); edge\_low\_d — a Line \[gray\] drawn in tree (start=(2.0, 2.4), end=(2.8, 1.4)); low\_c — a Polygon \[blue\] drawn in tree (vertices=((0.1, 0.6), (1.5, 0.6), (1.5, 1.4), (0.1, 1.4)), fill\_opacity=0.12); low\_c\_text — a Math \[blue\] that says "$B$" drawn in tree; low\_d — a Polygon \[yellow\] drawn in tree (vertices=((2.0, 0.6), (3.6, 0.6), (3.6, 1.4), (2.0, 1.4)), fill\_opacity=0.1); low\_d\_text — a Math \[yellow\] that says "$x\_1 \< 2.5?$" drawn in tree; low\_cut\_three — a Line \[green\] drawn in plane (start=(2.5, 2.5), end=(2.5, 4.0)); high\_cut\_one — a Line \[magenta\] drawn in plane (start=(6.2, 4.0), end=(6.2, 8.0)); edge\_high\_a — a Line \[gray\] drawn in tree (start=(8.3, 4.2), end=(7.2, 3.2)); edge\_high\_b — a Line \[gray\] drawn in tree (start=(9.3, 4.2), end=(10.4, 3.2)); high\_a — a Polygon \[magenta\] drawn in tree (vertices=((6.0, 2.4), (8.4, 2.4), (8.4, 3.2), (6.0, 3.2)), fill\_opacity=0.1); high\_a\_text — a Math \[magenta\] that says "$x\_1 \< 2.8?$" drawn in tree; high\_b — a Polygon \[magenta\] drawn in tree (vertices=((9.2, 2.4), (11.6, 2.4), (11.6, 3.2), (9.2, 3.2)), fill\_opacity=0.1); high\_b\_text — a Math \[magenta\] that says "$x\_2 \< 5.6?$" drawn in tree; high\_cut\_two — a Line \[magenta\] drawn in plane (start=(6.2, 5.6), end=(8.0, 5.6)) Actions: - [04:6.626](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=246.62604166666668): high\_c is shown on the screen, written out. - [04:6.626](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=246.62604166666668): high\_c\_text is shown on the screen, written out. - [04:9.935](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=249.93504166666668): high\_cut\_three is shown on the screen, written out. - [04:10.91](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=250.91004166666667): edge\_high\_c is shown on the screen, written out. - [04:10.91](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=250.91004166666667): edge\_high\_d is shown on the screen, written out. - [04:12.884](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=252.8840416666667): high\_cut\_four is shown on the screen, written out. - [04:17.121](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=257.12104166666666): high\_d is shown on the screen, written out. - [04:17.121](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=257.12104166666666): high\_d\_text is shown on the screen, written out. - [04:18.074](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=258.07354166666664): leaf\_count becomes "$upright("leaves") = 9$". ##### [04:18.674](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=258.67354166666667) Narration: One final threshold at x one equals one point five isolates the remaining upper blue point. Every terminal region now contains only one class. Consequently every leaf has Gini impurity zero. Board: tree — a Figure (x\_range=(0.0, 12.0), y\_range=(0.0, 7.0), aspect=(12.0, 7.0)); leaf\_count — a Math \[text\] that says "$upright("leaves") = 2$"; plane — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); heading — a Heading that says "Repeat Until Every Leaf Is Pure"; blue\_points — a Point \[blue\] drawn in plane (location=(0.8, 1.2)); blue\_points\_2 — a Point \[blue\] drawn in plane (location=(1.5, 2.0)); blue\_points\_3 — a Point \[blue\] drawn in plane (location=(2.8, 1.0)); blue\_points\_4 — a Point \[blue\] drawn in plane (location=(3.6, 2.8)); blue\_points\_5 — a Point \[blue\] drawn in plane (location=(4.5, 1.5)); blue\_points\_6 — a Point \[blue\] drawn in plane (location=(5.6, 2.7)); blue\_points\_7 — a Point \[blue\] drawn in plane (location=(2.4, 5.5)); blue\_points\_8 — a Point \[blue\] drawn in plane (location=(6.8, 6.4)); red\_points — a Point \[red\] drawn in plane (location=(0.9, 6.3)); red\_points\_2 — a Point \[red\] drawn in plane (location=(1.8, 4.8)); red\_points\_3 — a Point \[red\] drawn in plane (location=(3.3, 6.7)); red\_points\_4 — a Point \[red\] drawn in plane (location=(4.2, 5.2)); red\_points\_5 — a Point \[red\] drawn in plane (location=(5.4, 6.1)); red\_points\_6 — a Point \[red\] drawn in plane (location=(7.2, 4.8)); red\_points\_7 — a Point \[red\] drawn in plane (location=(2.2, 2.8)); red\_points\_8 — a Point \[red\] drawn in plane (location=(6.5, 1.2)); root\_box — a Polygon \[yellow\] drawn in tree (vertices=((4.7, 6.0), (7.3, 6.0), (7.3, 6.8), (4.7, 6.8)), fill\_opacity=0.12); root\_text — a Math \[yellow\] that says "$x\_2 \< 4?$" drawn in tree; edge\_low — a Line \[gray\] drawn in tree (start=(5.3, 6.0), end=(3.2, 5.0)); edge\_high — a Line \[gray\] drawn in tree (start=(6.7, 6.0), end=(8.8, 5.0)); low\_node — a Polygon \[green\] drawn in tree (vertices=((2.0, 4.2), (4.4, 4.2), (4.4, 5.0), (2.0, 5.0)), fill\_opacity=0.1); low\_text — a Math \[green\] that says "$x\_1 \< 6.1?$" drawn in tree; high\_node — a Polygon \[magenta\] drawn in tree (vertices=((7.6, 4.2), (10.0, 4.2), (10.0, 5.0), (7.6, 5.0)), fill\_opacity=0.1); high\_text — a Math \[magenta\] that says "$x\_1 \< 6.2?$" drawn in tree; root\_cut — a Line \[yellow\] drawn in plane (start=(0.0, 4.0), end=(8.0, 4.0)); low\_cut\_one — a Line \[green\] drawn in plane (start=(6.1, 0.0), end=(6.1, 4.0)); edge\_low\_a — a Line \[gray\] drawn in tree (start=(2.7, 4.2), end=(1.6, 3.2)); edge\_low\_b — a Line \[gray\] drawn in tree (start=(3.7, 4.2), end=(4.8, 3.2)); low\_a — a Polygon \[green\] drawn in tree (vertices=((0.5, 2.4), (2.7, 2.4), (2.7, 3.2), (0.5, 3.2)), fill\_opacity=0.1); low\_a\_text — a Math \[green\] that says "$x\_2 \< 2.5?$" drawn in tree; low\_b — a Polygon \[red\] drawn in tree (vertices=((3.7, 2.4), (5.9, 2.4), (5.9, 3.2), (3.7, 3.2)), fill\_opacity=0.12); low\_b\_text — a Math \[red\] that says "$R$" drawn in tree; low\_cut\_two — a Line \[green\] drawn in plane (start=(0.0, 2.5), end=(6.1, 2.5)); edge\_low\_c — a Line \[gray\] drawn in tree (start=(1.2, 2.4), end=(0.8, 1.4)); edge\_low\_d — a Line \[gray\] drawn in tree (start=(2.0, 2.4), end=(2.8, 1.4)); low\_c — a Polygon \[blue\] drawn in tree (vertices=((0.1, 0.6), (1.5, 0.6), (1.5, 1.4), (0.1, 1.4)), fill\_opacity=0.12); low\_c\_text — a Math \[blue\] that says "$B$" drawn in tree; low\_d — a Polygon \[yellow\] drawn in tree (vertices=((2.0, 0.6), (3.6, 0.6), (3.6, 1.4), (2.0, 1.4)), fill\_opacity=0.1); low\_d\_text — a Math \[yellow\] that says "$x\_1 \< 2.5?$" drawn in tree; low\_cut\_three — a Line \[green\] drawn in plane (start=(2.5, 2.5), end=(2.5, 4.0)); high\_cut\_one — a Line \[magenta\] drawn in plane (start=(6.2, 4.0), end=(6.2, 8.0)); edge\_high\_a — a Line \[gray\] drawn in tree (start=(8.3, 4.2), end=(7.2, 3.2)); edge\_high\_b — a Line \[gray\] drawn in tree (start=(9.3, 4.2), end=(10.4, 3.2)); high\_a — a Polygon \[magenta\] drawn in tree (vertices=((6.0, 2.4), (8.4, 2.4), (8.4, 3.2), (6.0, 3.2)), fill\_opacity=0.1); high\_a\_text — a Math \[magenta\] that says "$x\_1 \< 2.8?$" drawn in tree; high\_b — a Polygon \[magenta\] drawn in tree (vertices=((9.2, 2.4), (11.6, 2.4), (11.6, 3.2), (9.2, 3.2)), fill\_opacity=0.1); high\_b\_text — a Math \[magenta\] that says "$x\_2 \< 5.6?$" drawn in tree; high\_cut\_two — a Line \[magenta\] drawn in plane (start=(6.2, 5.6), end=(8.0, 5.6)); high\_cut\_three — a Line \[magenta\] drawn in plane (start=(2.8, 4.0), end=(2.8, 8.0)); high\_cut\_four — a Line \[magenta\] drawn in plane (start=(0.0, 5.1), end=(2.8, 5.1)); edge\_high\_c — a Line \[gray\] drawn in tree (start=(6.7, 2.4), end=(5.8, 1.4)); edge\_high\_d — a Line \[gray\] drawn in tree (start=(7.7, 2.4), end=(8.2, 1.4)); high\_c — a Polygon \[yellow\] drawn in tree (vertices=((5.0, 0.6), (6.6, 0.6), (6.6, 1.4), (5.0, 1.4)), fill\_opacity=0.1); high\_c\_text — a Math \[yellow\] that says "$x\_2 \< 5.1?$" drawn in tree; high\_d — a Polygon \[red\] drawn in tree (vertices=((7.3, 0.6), (9.1, 0.6), (9.1, 1.4), (7.3, 1.4)), fill\_opacity=0.12); high\_d\_text — a Math \[red\] that says "$R$" drawn in tree Actions: - [04:19.312](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=259.3120416666667): high\_cut\_five is shown on the screen, written out. - [04:24.42](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=264.4200416666667): blue\_points\_7 is indicated — a transient flash. - [04:31.363](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=271.3630416666667): purity is shown on the screen, written out. ##### [04:32.811](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=272.81054166666667) Narration: On the training set, this looks perfect. Every observation is classified correctly. But look at the geometry required to achieve it: thin strips, short corridors, and thresholds whose only purpose is to rescue one exceptional dot. Board: tree — a Figure (x\_range=(0.0, 12.0), y\_range=(0.0, 7.0), aspect=(12.0, 7.0)); leaf\_count — a Math \[text\] that says "$upright("leaves") = 2$"; purity — a Math \[text\] that says "$G\_(upright("every leaf")) = 0$"; plane — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); heading — a Heading that says "Repeat Until Every Leaf Is Pure"; blue\_points — a Point \[blue\] drawn in plane (location=(0.8, 1.2)); blue\_points\_2 — a Point \[blue\] drawn in plane (location=(1.5, 2.0)); blue\_points\_3 — a Point \[blue\] drawn in plane (location=(2.8, 1.0)); blue\_points\_4 — a Point \[blue\] drawn in plane (location=(3.6, 2.8)); blue\_points\_5 — a Point \[blue\] drawn in plane (location=(4.5, 1.5)); blue\_points\_6 — a Point \[blue\] drawn in plane (location=(5.6, 2.7)); blue\_points\_7 — a Point \[blue\] drawn in plane (location=(2.4, 5.5)); blue\_points\_8 — a Point \[blue\] drawn in plane (location=(6.8, 6.4)); red\_points — a Point \[red\] drawn in plane (location=(0.9, 6.3)); red\_points\_2 — a Point \[red\] drawn in plane (location=(1.8, 4.8)); red\_points\_3 — a Point \[red\] drawn in plane (location=(3.3, 6.7)); red\_points\_4 — a Point \[red\] drawn in plane (location=(4.2, 5.2)); red\_points\_5 — a Point \[red\] drawn in plane (location=(5.4, 6.1)); red\_points\_6 — a Point \[red\] drawn in plane (location=(7.2, 4.8)); red\_points\_7 — a Point \[red\] drawn in plane (location=(2.2, 2.8)); red\_points\_8 — a Point \[red\] drawn in plane (location=(6.5, 1.2)); root\_box — a Polygon \[yellow\] drawn in tree (vertices=((4.7, 6.0), (7.3, 6.0), (7.3, 6.8), (4.7, 6.8)), fill\_opacity=0.12); root\_text — a Math \[yellow\] that says "$x\_2 \< 4?$" drawn in tree; edge\_low — a Line \[gray\] drawn in tree (start=(5.3, 6.0), end=(3.2, 5.0)); edge\_high — a Line \[gray\] drawn in tree (start=(6.7, 6.0), end=(8.8, 5.0)); low\_node — a Polygon \[green\] drawn in tree (vertices=((2.0, 4.2), (4.4, 4.2), (4.4, 5.0), (2.0, 5.0)), fill\_opacity=0.1); low\_text — a Math \[green\] that says "$x\_1 \< 6.1?$" drawn in tree; high\_node — a Polygon \[magenta\] drawn in tree (vertices=((7.6, 4.2), (10.0, 4.2), (10.0, 5.0), (7.6, 5.0)), fill\_opacity=0.1); high\_text — a Math \[magenta\] that says "$x\_1 \< 6.2?$" drawn in tree; root\_cut — a Line \[yellow\] drawn in plane (start=(0.0, 4.0), end=(8.0, 4.0)); low\_cut\_one — a Line \[green\] drawn in plane (start=(6.1, 0.0), end=(6.1, 4.0)); edge\_low\_a — a Line \[gray\] drawn in tree (start=(2.7, 4.2), end=(1.6, 3.2)); edge\_low\_b — a Line \[gray\] drawn in tree (start=(3.7, 4.2), end=(4.8, 3.2)); low\_a — a Polygon \[green\] drawn in tree (vertices=((0.5, 2.4), (2.7, 2.4), (2.7, 3.2), (0.5, 3.2)), fill\_opacity=0.1); low\_a\_text — a Math \[green\] that says "$x\_2 \< 2.5?$" drawn in tree; low\_b — a Polygon \[red\] drawn in tree (vertices=((3.7, 2.4), (5.9, 2.4), (5.9, 3.2), (3.7, 3.2)), fill\_opacity=0.12); low\_b\_text — a Math \[red\] that says "$R$" drawn in tree; low\_cut\_two — a Line \[green\] drawn in plane (start=(0.0, 2.5), end=(6.1, 2.5)); edge\_low\_c — a Line \[gray\] drawn in tree (start=(1.2, 2.4), end=(0.8, 1.4)); edge\_low\_d — a Line \[gray\] drawn in tree (start=(2.0, 2.4), end=(2.8, 1.4)); low\_c — a Polygon \[blue\] drawn in tree (vertices=((0.1, 0.6), (1.5, 0.6), (1.5, 1.4), (0.1, 1.4)), fill\_opacity=0.12); low\_c\_text — a Math \[blue\] that says "$B$" drawn in tree; low\_d — a Polygon \[yellow\] drawn in tree (vertices=((2.0, 0.6), (3.6, 0.6), (3.6, 1.4), (2.0, 1.4)), fill\_opacity=0.1); low\_d\_text — a Math \[yellow\] that says "$x\_1 \< 2.5?$" drawn in tree; low\_cut\_three — a Line \[green\] drawn in plane (start=(2.5, 2.5), end=(2.5, 4.0)); high\_cut\_one — a Line \[magenta\] drawn in plane (start=(6.2, 4.0), end=(6.2, 8.0)); edge\_high\_a — a Line \[gray\] drawn in tree (start=(8.3, 4.2), end=(7.2, 3.2)); edge\_high\_b — a Line \[gray\] drawn in tree (start=(9.3, 4.2), end=(10.4, 3.2)); high\_a — a Polygon \[magenta\] drawn in tree (vertices=((6.0, 2.4), (8.4, 2.4), (8.4, 3.2), (6.0, 3.2)), fill\_opacity=0.1); high\_a\_text — a Math \[magenta\] that says "$x\_1 \< 2.8?$" drawn in tree; high\_b — a Polygon \[magenta\] drawn in tree (vertices=((9.2, 2.4), (11.6, 2.4), (11.6, 3.2), (9.2, 3.2)), fill\_opacity=0.1); high\_b\_text — a Math \[magenta\] that says "$x\_2 \< 5.6?$" drawn in tree; high\_cut\_two — a Line \[magenta\] drawn in plane (start=(6.2, 5.6), end=(8.0, 5.6)); high\_cut\_three — a Line \[magenta\] drawn in plane (start=(2.8, 4.0), end=(2.8, 8.0)); high\_cut\_four — a Line \[magenta\] drawn in plane (start=(0.0, 5.1), end=(2.8, 5.1)); edge\_high\_c — a Line \[gray\] drawn in tree (start=(6.7, 2.4), end=(5.8, 1.4)); edge\_high\_d — a Line \[gray\] drawn in tree (start=(7.7, 2.4), end=(8.2, 1.4)); high\_c — a Polygon \[yellow\] drawn in tree (vertices=((5.0, 0.6), (6.6, 0.6), (6.6, 1.4), (5.0, 1.4)), fill\_opacity=0.1); high\_c\_text — a Math \[yellow\] that says "$x\_2 \< 5.1?$" drawn in tree; high\_d — a Polygon \[red\] drawn in tree (vertices=((7.3, 0.6), (9.1, 0.6), (9.1, 1.4), (7.3, 1.4)), fill\_opacity=0.12); high\_d\_text — a Math \[red\] that says "$R$" drawn in tree; high\_cut\_five — a Line \[magenta\] drawn in plane (start=(1.5, 5.1), end=(1.5, 8.0)) Actions: - [04:41.89](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=281.89004166666666): low\_cut\_three is indicated — a transient flash. - [04:42.935](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=282.9350416666667): high\_cut\_four is indicated — a transient flash. - [04:46.696](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=286.6960416666667): high\_cut\_five is indicated — a transient flash. ##### [04:49.061](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=289.0610416666667) Narration: A new point can cross one of those arbitrary thresholds after an imperceptible change in a feature. Its predicted class then jumps, even though the training labels gave us almost no evidence that such a jump should exist. Board: Unchanged from the preceding beat in this scene. Actions: - [04:49.572](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=289.5720416666667): point is shown on the screen, grown. - [04:51.572](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=291.5720416666667): point is hidden from the screen. - [04:56.851](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=296.8510416666667): point\_2 is shown on the screen, grown. - [04:58.851](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=298.8510416666667): point\_2 is hidden from the screen. ##### [05:2.792](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=302.7920416666667) Narration: This is the characteristic strength and weakness of an unconstrained decision tree. It has low bias because it can represent complicated interactions. It also has high variance because a few observations can rearrange entire branches and rectangles. Board: Unchanged from the preceding beat in this scene. Actions: - [05:6.995](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=306.9950416666667): root\_text is indicated — a transient flash. - [05:14.193](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=314.19304166666666): low\_d\_text is indicated — a transient flash. - [05:14.193](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=314.19304166666666): high\_c\_text is indicated — a transient flash. ##### [05:18.346](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=318.3455416666667) Narration: Pure leaves are therefore a training condition, not evidence of a useful model. The next question is whether every branch earns its complexity on data that did not choose the branch. Board: Unchanged from the preceding beat in this scene. Actions: - [05:20.203](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=320.2030416666667): A box is drawn around purity. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): heading is hidden from the screen — left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): leaf\_count is hidden from the screen — left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): plane is hidden from the screen — left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): blue\_points is hidden from the screen — plane left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): blue\_points\_2 is hidden from the screen — plane left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): blue\_points\_3 is hidden from the screen — plane left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): blue\_points\_4 is hidden from the screen — plane left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): blue\_points\_5 is hidden from the screen — plane left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): blue\_points\_6 is hidden from the screen — plane left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): blue\_points\_7 is hidden from the screen — plane left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): blue\_points\_8 is hidden from the screen — plane left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): red\_points is hidden from the screen — plane left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): red\_points\_2 is hidden from the screen — plane left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): red\_points\_3 is hidden from the screen — plane left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): red\_points\_4 is hidden from the screen — plane left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): red\_points\_5 is hidden from the screen — plane left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): red\_points\_6 is hidden from the screen — plane left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): red\_points\_7 is hidden from the screen — plane left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): red\_points\_8 is hidden from the screen — plane left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): root\_cut is hidden from the screen — plane left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): low\_cut\_one is hidden from the screen — plane left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): low\_cut\_two is hidden from the screen — plane left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): low\_cut\_three is hidden from the screen — plane left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): high\_cut\_one is hidden from the screen — plane left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): high\_cut\_two is hidden from the screen — plane left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): high\_cut\_three is hidden from the screen — plane left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): high\_cut\_four is hidden from the screen — plane left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): high\_cut\_five is hidden from the screen — plane left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): purity is hidden from the screen — left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): tree is hidden from the screen — left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): root\_box is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): root\_text is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): edge\_low is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): edge\_high is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): low\_node is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): low\_text is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): high\_node is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): high\_text is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): edge\_low\_a is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): edge\_low\_b is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): low\_a is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): low\_a\_text is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): low\_b is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): low\_b\_text is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): edge\_low\_c is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): edge\_low\_d is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): low\_c is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): low\_c\_text is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): low\_d is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): low\_d\_text is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): edge\_high\_a is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): edge\_high\_b is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): high\_a is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): high\_a\_text is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): high\_b is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): high\_b\_text is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): edge\_high\_c is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): edge\_high\_d is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): high\_c is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): high\_c\_text is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): high\_d is hidden from the screen — tree left the board. - [05:29.466](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=329.46575): high\_d\_text is hidden from the screen — tree left the board. ### Scene 3: [Pruning the Tree](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5074166666667) Span: 05:30.507–08:29.944 (330.5074166666667s–509.9435833333333s). #### Objects - blue\_points: a Point \[blue\] drawn in plane (location=(0.8, 1.2)) - blue\_points\_2: a Point \[blue\] drawn in plane (location=(1.5, 2.0)) - blue\_points\_3: a Point \[blue\] drawn in plane (location=(2.8, 1.0)) - blue\_points\_4: a Point \[blue\] drawn in plane (location=(3.6, 2.8)) - blue\_points\_5: a Point \[blue\] drawn in plane (location=(4.5, 1.5)) - blue\_points\_6: a Point \[blue\] drawn in plane (location=(5.6, 2.7)) - blue\_points\_7: a Point \[blue\] drawn in plane (location=(2.4, 5.5)) - blue\_points\_8: a Point \[blue\] drawn in plane (location=(6.8, 6.4)) - branch\_left: a Line \[gray\] drawn in tree (start=(4.3, 5.1), end=(2.5, 4.0)) - branch\_right: a Line \[gray\] drawn in tree (start=(5.7, 5.1), end=(7.5, 4.0)) - criterion: a Math \[text\] that says "$R\_alpha(T) = R(T) + alpha thin \|upright("leaves")(T)\|$" - heading: a Heading that says "Trade Fit for a Smaller Tree" - heading\_path: a Heading that says "Choose Complexity Outside the Training Fit" - high\_detail\_four: a Line \[gray\] drawn in plane (start=(1.5, 5.1), end=(1.5, 8.0)) - high\_detail\_one: a Line \[gray\] drawn in plane (start=(6.2, 5.6), end=(8.0, 5.6)) - high\_detail\_three: a Line \[gray\] drawn in plane (start=(0.0, 5.1), end=(2.8, 5.1)) - high\_detail\_two: a Line \[gray\] drawn in plane (start=(2.8, 4.0), end=(2.8, 8.0)) - high\_main: a Line \[magenta\] drawn in plane (start=(6.2, 4.0), end=(6.2, 8.0)) - left\_node: a Polygon \[green\] drawn in tree (vertices=((1.3, 3.3), (3.7, 3.3), (3.7, 4.0), (1.3, 4.0)), fill\_opacity=0.1) - left\_text: a Math \[green\] that says "$x\_1 \< 6.1?$" drawn in tree - left\_twigs: a Line \[gray\] drawn in tree (start=(2.0, 3.3), end=(1.1, 2.2)) - left\_twigs\_2: a Line \[gray\] drawn in tree (start=(3.0, 3.3), end=(3.9, 2.2)) - left\_twigs\_3: a Line \[gray\] drawn in tree (start=(1.1, 2.2), end=(0.6, 1.0)) - left\_twigs\_4: a Line \[gray\] drawn in tree (start=(1.1, 2.2), end=(1.8, 1.0)) - left\_twigs\_5: a Line \[gray\] drawn in tree (start=(3.9, 2.2), end=(3.3, 1.0)) - left\_twigs\_6: a Line \[gray\] drawn in tree (start=(3.9, 2.2), end=(4.6, 1.0)) - low\_detail\_one: a Line \[gray\] drawn in plane (start=(0.0, 2.5), end=(6.1, 2.5)) - low\_detail\_two: a Line \[gray\] drawn in plane (start=(2.5, 2.5), end=(2.5, 4.0)) - low\_main: a Line \[green\] drawn in plane (start=(6.1, 0.0), end=(6.1, 4.0)) - note: a Panel that says "Training impurity chooses splits. Validation evidence chooses how much of the fitted tree to keep." - path: a Table \[text\] that says "$alpha$ Leaves Train error Validation error $0.000$ $9$ $0.000$ $0.250$ $0.015$ $6$ $0.063$ $0.188$ $0.040$ $4$ $0.125$ $0.125$ $0.090$ $2$ $0.250$ $0.188$" (rows=(('$alpha$', 'Leaves', 'Train error', 'Validation error'), ('$0…, header=True) - plane: an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)) - pruned\_left\_leaf: a Polygon \[blue\] drawn in tree (vertices=((1.2, 1.7), (3.8, 1.7), (3.8, 2.5), (1.2, 2.5)), fill\_opacity=0.14) - pruned\_left\_text: a Math \[blue\] that says "$upright("mostly B")$" drawn in tree - pruned\_right\_leaf: a Polygon \[red\] drawn in tree (vertices=((6.2, 1.7), (8.8, 1.7), (8.8, 2.5), (6.2, 2.5)), fill\_opacity=0.14) - pruned\_right\_text: a Math \[red\] that says "$upright("mostly R")$" drawn in tree - red\_points: a Point \[red\] drawn in plane (location=(0.9, 6.3)) - red\_points\_2: a Point \[red\] drawn in plane (location=(1.8, 4.8)) - red\_points\_3: a Point \[red\] drawn in plane (location=(3.3, 6.7)) - red\_points\_4: a Point \[red\] drawn in plane (location=(4.2, 5.2)) - red\_points\_5: a Point \[red\] drawn in plane (location=(5.4, 6.1)) - red\_points\_6: a Point \[red\] drawn in plane (location=(7.2, 4.8)) - red\_points\_7: a Point \[red\] drawn in plane (location=(2.2, 2.8)) - red\_points\_8: a Point \[red\] drawn in plane (location=(6.5, 1.2)) - right\_node: a Polygon \[magenta\] drawn in tree (vertices=((6.3, 3.3), (8.7, 3.3), (8.7, 4.0), (6.3, 4.0)), fill\_opacity=0.1) - right\_text: a Math \[magenta\] that says "$x\_1 \< 6.2?$" drawn in tree - right\_twigs: a Line \[gray\] drawn in tree (start=(7.0, 3.3), end=(6.1, 2.2)) - right\_twigs\_2: a Line \[gray\] drawn in tree (start=(8.0, 3.3), end=(8.9, 2.2)) - right\_twigs\_3: a Line \[gray\] drawn in tree (start=(6.1, 2.2), end=(5.4, 1.0)) - right\_twigs\_4: a Line \[gray\] drawn in tree (start=(6.1, 2.2), end=(6.8, 1.0)) - right\_twigs\_5: a Line \[gray\] drawn in tree (start=(8.9, 2.2), end=(8.2, 1.0)) - right\_twigs\_6: a Line \[gray\] drawn in tree (start=(8.9, 2.2), end=(9.6, 1.0)) - root: a Polygon \[yellow\] drawn in tree (vertices=((3.7, 5.1), (6.3, 5.1), (6.3, 5.8), (3.7, 5.8)), fill\_opacity=0.12) - root\_cut: a Line \[yellow\] drawn in plane (start=(0.0, 4.0), end=(8.0, 4.0)) - root\_text: a Math \[yellow\] that says "$x\_2 \< 4?$" drawn in tree - selection: a Math \[text\] that says "$alpha = 0.040 thin arrow.r thin 4 thin upright("leaves")$" - tree: a Figure (x\_range=(0.0, 10.0), y\_range=(0.0, 6.0), aspect=(5.0, 3.0)) - twig\_nodes: a Point \[gray\] drawn in tree (location=(1.1, 2.2)) - twig\_nodes\_10: a Point \[blue\] drawn in tree (location=(6.8, 1.0)) - twig\_nodes\_11: a Point \[red\] drawn in tree (location=(8.2, 1.0)) - twig\_nodes\_12: a Point \[blue\] drawn in tree (location=(9.6, 1.0)) - twig\_nodes\_2: a Point \[gray\] drawn in tree (location=(3.9, 2.2)) - twig\_nodes\_3: a Point \[gray\] drawn in tree (location=(6.1, 2.2)) - twig\_nodes\_4: a Point \[gray\] drawn in tree (location=(8.9, 2.2)) - twig\_nodes\_5: a Point \[blue\] drawn in tree (location=(0.6, 1.0)) - twig\_nodes\_6: a Point \[red\] drawn in tree (location=(1.8, 1.0)) - twig\_nodes\_7: a Point \[blue\] drawn in tree (location=(3.3, 1.0)) - twig\_nodes\_8: a Point \[red\] drawn in tree (location=(4.6, 1.0)) - twig\_nodes\_9: a Point \[red\] drawn in tree (location=(5.4, 1.0)) #### Beats ##### [05:30.507](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5074166666667) Narration: The fully grown tree has zero training impurity, but it pays for that fit with nine leaves and several thresholds supported by a single unusual observation. Pruning asks whether those extra leaves earn their keep. Board: Empty. Actions: - [05:30.507](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5074166666667): heading is shown on the screen, written out. - [05:30.507](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5074166666667): tree is shown on the screen, written out. - [05:30.507](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5074166666667): plane is shown on the screen, written out. - [05:30.507](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5074166666667): blue\_points is shown on the screen, written out. - [05:30.507](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5074166666667): root is shown on the screen, written out. - [05:30.507](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5074166666667): root\_text is shown on the screen, written out. - [05:30.507](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5074166666667): branch\_left is shown on the screen, written out. - [05:30.507](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5074166666667): branch\_right is shown on the screen, written out. - [05:30.507](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5074166666667): left\_node is shown on the screen, written out. - [05:30.507](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5074166666667): right\_node is shown on the screen, written out. - [05:30.507](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5074166666667): left\_text is shown on the screen, written out. - [05:30.507](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5074166666667): right\_text is shown on the screen, written out. - [05:30.507](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5074166666667): root\_cut is shown on the screen, written out. - [05:30.507](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5074166666667): low\_main is shown on the screen, written out. - [05:30.507](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5074166666667): high\_main is shown on the screen, written out. - [05:30.507](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5074166666667): low\_detail\_one is shown on the screen, written out. - [05:30.507](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5074166666667): low\_detail\_two is shown on the screen, written out. - [05:30.507](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5074166666667): high\_detail\_one is shown on the screen, written out. - [05:30.507](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5074166666667): high\_detail\_two is shown on the screen, written out. - [05:30.507](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5074166666667): high\_detail\_three is shown on the screen, written out. - [05:30.507](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5074166666667): high\_detail\_four is shown on the screen, written out. - [05:30.547](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5474166666667): blue\_points\_2 is shown on the screen, written out. - [05:30.587](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.5874166666667): blue\_points\_3 is shown on the screen, written out. - [05:30.627](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.6274166666667): blue\_points\_4 is shown on the screen, written out. - [05:30.667](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.6674166666667): blue\_points\_5 is shown on the screen, written out. - [05:30.707](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.7074166666667): blue\_points\_6 is shown on the screen, written out. - [05:30.747](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.7474166666667): blue\_points\_7 is shown on the screen, written out. - [05:30.787](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.7874166666667): blue\_points\_8 is shown on the screen, written out. - [05:30.827](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.8274166666667): red\_points is shown on the screen, written out. - [05:30.867](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.8674166666667): red\_points\_2 is shown on the screen, written out. - [05:30.907](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.9074166666667): red\_points\_3 is shown on the screen, written out. - [05:30.947](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.9474166666667): red\_points\_4 is shown on the screen, written out. - [05:30.987](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=330.9874166666667): red\_points\_5 is shown on the screen, written out. - [05:31.027](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=331.0274166666667): red\_points\_6 is shown on the screen, written out. - [05:31.067](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=331.0674166666667): red\_points\_7 is shown on the screen, written out. - [05:31.107](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=331.1074166666667): red\_points\_8 is shown on the screen, written out. - [05:31.307](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=331.3074166666667): left\_twigs is shown on the screen, written out. - [05:31.357](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=331.3574166666667): left\_twigs\_2 is shown on the screen, written out. - [05:31.407](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=331.4074166666667): left\_twigs\_3 is shown on the screen, written out. - [05:31.457](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=331.4574166666667): left\_twigs\_4 is shown on the screen, written out. - [05:31.507](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=331.5074166666667): left\_twigs\_5 is shown on the screen, written out. - [05:31.557](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=331.5574166666667): left\_twigs\_6 is shown on the screen, written out. - [05:31.607](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=331.6074166666667): right\_twigs is shown on the screen, written out. - [05:31.627](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=331.6274166666667): twig\_nodes is shown on the screen, written out. - [05:31.657](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=331.6574166666667): right\_twigs\_2 is shown on the screen, written out. - [05:31.667](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=331.6674166666667): twig\_nodes\_2 is shown on the screen, written out. - [05:31.707](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=331.7074166666667): twig\_nodes\_3 is shown on the screen, written out. - [05:31.707](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=331.7074166666667): right\_twigs\_3 is shown on the screen, written out. - [05:31.747](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=331.7474166666667): twig\_nodes\_4 is shown on the screen, written out. - [05:31.757](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=331.7574166666667): right\_twigs\_4 is shown on the screen, written out. - [05:31.787](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=331.7874166666667): twig\_nodes\_5 is shown on the screen, written out. - [05:31.807](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=331.8074166666667): right\_twigs\_5 is shown on the screen, written out. - [05:31.827](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=331.8274166666667): twig\_nodes\_6 is shown on the screen, written out. - [05:31.857](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=331.8574166666667): right\_twigs\_6 is shown on the screen, written out. - [05:31.867](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=331.8674166666667): twig\_nodes\_7 is shown on the screen, written out. - [05:31.907](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=331.9074166666667): twig\_nodes\_8 is shown on the screen, written out. - [05:31.947](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=331.9474166666667): twig\_nodes\_9 is shown on the screen, written out. - [05:31.987](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=331.9874166666667): twig\_nodes\_10 is shown on the screen, written out. - [05:32.027](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=332.0274166666667): twig\_nodes\_11 is shown on the screen, written out. - [05:32.067](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=332.0674166666667): twig\_nodes\_12 is shown on the screen, written out. - [05:40.097](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=340.0974166666667): criterion is shown on the screen, written out. ##### [05:43.855](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=343.8549166666667) Narration: Cost-complexity pruning gives the trade a precise form. R of T measures the fitted tree's error or impurity. The second term charges alpha for every terminal leaf. Board: criterion — a Math \[text\] that says "$R\_alpha(T) = R(T) + alpha thin \|upright("leaves")(T)\|$"; tree — a Figure (x\_range=(0.0, 10.0), y\_range=(0.0, 6.0), aspect=(5.0, 3.0)); plane — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); heading — a Heading that says "Trade Fit for a Smaller Tree"; blue\_points — a Point \[blue\] drawn in plane (location=(0.8, 1.2)); blue\_points\_2 — a Point \[blue\] drawn in plane (location=(1.5, 2.0)); blue\_points\_3 — a Point \[blue\] drawn in plane (location=(2.8, 1.0)); blue\_points\_4 — a Point \[blue\] drawn in plane (location=(3.6, 2.8)); blue\_points\_5 — a Point \[blue\] drawn in plane (location=(4.5, 1.5)); blue\_points\_6 — a Point \[blue\] drawn in plane (location=(5.6, 2.7)); blue\_points\_7 — a Point \[blue\] drawn in plane (location=(2.4, 5.5)); blue\_points\_8 — a Point \[blue\] drawn in plane (location=(6.8, 6.4)); red\_points — a Point \[red\] drawn in plane (location=(0.9, 6.3)); red\_points\_2 — a Point \[red\] drawn in plane (location=(1.8, 4.8)); red\_points\_3 — a Point \[red\] drawn in plane (location=(3.3, 6.7)); red\_points\_4 — a Point \[red\] drawn in plane (location=(4.2, 5.2)); red\_points\_5 — a Point \[red\] drawn in plane (location=(5.4, 6.1)); red\_points\_6 — a Point \[red\] drawn in plane (location=(7.2, 4.8)); red\_points\_7 — a Point \[red\] drawn in plane (location=(2.2, 2.8)); red\_points\_8 — a Point \[red\] drawn in plane (location=(6.5, 1.2)); root — a Polygon \[yellow\] drawn in tree (vertices=((3.7, 5.1), (6.3, 5.1), (6.3, 5.8), (3.7, 5.8)), fill\_opacity=0.12); root\_text — a Math \[yellow\] that says "$x\_2 \< 4?$" drawn in tree; branch\_left — a Line \[gray\] drawn in tree (start=(4.3, 5.1), end=(2.5, 4.0)); branch\_right — a Line \[gray\] drawn in tree (start=(5.7, 5.1), end=(7.5, 4.0)); left\_node — a Polygon \[green\] drawn in tree (vertices=((1.3, 3.3), (3.7, 3.3), (3.7, 4.0), (1.3, 4.0)), fill\_opacity=0.1); right\_node — a Polygon \[magenta\] drawn in tree (vertices=((6.3, 3.3), (8.7, 3.3), (8.7, 4.0), (6.3, 4.0)), fill\_opacity=0.1); left\_text — a Math \[green\] that says "$x\_1 \< 6.1?$" drawn in tree; right\_text — a Math \[magenta\] that says "$x\_1 \< 6.2?$" drawn in tree; left\_twigs — a Line \[gray\] drawn in tree (start=(2.0, 3.3), end=(1.1, 2.2)); left\_twigs\_2 — a Line \[gray\] drawn in tree (start=(3.0, 3.3), end=(3.9, 2.2)); left\_twigs\_3 — a Line \[gray\] drawn in tree (start=(1.1, 2.2), end=(0.6, 1.0)); left\_twigs\_4 — a Line \[gray\] drawn in tree (start=(1.1, 2.2), end=(1.8, 1.0)); left\_twigs\_5 — a Line \[gray\] drawn in tree (start=(3.9, 2.2), end=(3.3, 1.0)); left\_twigs\_6 — a Line \[gray\] drawn in tree (start=(3.9, 2.2), end=(4.6, 1.0)); right\_twigs — a Line \[gray\] drawn in tree (start=(7.0, 3.3), end=(6.1, 2.2)); right\_twigs\_2 — a Line \[gray\] drawn in tree (start=(8.0, 3.3), end=(8.9, 2.2)); right\_twigs\_3 — a Line \[gray\] drawn in tree (start=(6.1, 2.2), end=(5.4, 1.0)); right\_twigs\_4 — a Line \[gray\] drawn in tree (start=(6.1, 2.2), end=(6.8, 1.0)); right\_twigs\_5 — a Line \[gray\] drawn in tree (start=(8.9, 2.2), end=(8.2, 1.0)); right\_twigs\_6 — a Line \[gray\] drawn in tree (start=(8.9, 2.2), end=(9.6, 1.0)); twig\_nodes — a Point \[gray\] drawn in tree (location=(1.1, 2.2)); twig\_nodes\_2 — a Point \[gray\] drawn in tree (location=(3.9, 2.2)); twig\_nodes\_3 — a Point \[gray\] drawn in tree (location=(6.1, 2.2)); twig\_nodes\_4 — a Point \[gray\] drawn in tree (location=(8.9, 2.2)); twig\_nodes\_5 — a Point \[blue\] drawn in tree (location=(0.6, 1.0)); twig\_nodes\_6 — a Point \[red\] drawn in tree (location=(1.8, 1.0)); twig\_nodes\_7 — a Point \[blue\] drawn in tree (location=(3.3, 1.0)); twig\_nodes\_8 — a Point \[red\] drawn in tree (location=(4.6, 1.0)); twig\_nodes\_9 — a Point \[red\] drawn in tree (location=(5.4, 1.0)); twig\_nodes\_10 — a Point \[blue\] drawn in tree (location=(6.8, 1.0)); twig\_nodes\_11 — a Point \[red\] drawn in tree (location=(8.2, 1.0)); twig\_nodes\_12 — a Point \[blue\] drawn in tree (location=(9.6, 1.0)); root\_cut — a Line \[yellow\] drawn in plane (start=(0.0, 4.0), end=(8.0, 4.0)); low\_main — a Line \[green\] drawn in plane (start=(6.1, 0.0), end=(6.1, 4.0)); high\_main — a Line \[magenta\] drawn in plane (start=(6.2, 4.0), end=(6.2, 8.0)); low\_detail\_one — a Line \[gray\] drawn in plane (start=(0.0, 2.5), end=(6.1, 2.5)); low\_detail\_two — a Line \[gray\] drawn in plane (start=(2.5, 2.5), end=(2.5, 4.0)); high\_detail\_one — a Line \[gray\] drawn in plane (start=(6.2, 5.6), end=(8.0, 5.6)); high\_detail\_two — a Line \[gray\] drawn in plane (start=(2.8, 4.0), end=(2.8, 8.0)); high\_detail\_three — a Line \[gray\] drawn in plane (start=(0.0, 5.1), end=(2.8, 5.1)); high\_detail\_four — a Line \[gray\] drawn in plane (start=(1.5, 5.1), end=(1.5, 8.0)) Actions: - [05:50.379](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=350.3794166666667): criterion (the "R(T)" part) is emphasized. - [05:53.061](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=353.0614166666667): criterion (the "R(T)" part) is no longer emphasized. - [05:53.061](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=353.0614166666667): criterion (the "alpha thin \|upright("leaves")(T)\|" part) is emphasized. - [05:55.557](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=355.5574166666667): criterion (the "alpha thin \|upright("leaves")(T)\|" part) is no longer emphasized. ##### [05:56.157](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=356.1574166666667) Narration: At alpha zero, extra leaves are free, so the pure tree wins. As alpha increases, a weak pair of leaves can cost more than the small reduction in training error that created it. Board: Unchanged from the preceding beat in this scene. Actions: - [05:56.726](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=356.7264166666667): criterion (the "alpha" part) is indicated — a transient flash. - [06:2.821](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=362.8214166666667): low\_detail\_two is indicated — a transient flash. - [06:2.821](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=362.8214166666667): high\_detail\_four is indicated — a transient flash. ##### [06:8.251](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=368.2509166666667) Narration: Prune the weakest lower twigs first. Their narrow thresholds disappear from feature space at the same moment their branches disappear from the tree. The replacement leaf predicts the local majority class. Board: Unchanged from the preceding beat in this scene. Actions: - [06:12.558](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=372.5584166666667): low\_detail\_one is hidden from the screen. - [06:12.558](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=372.5584166666667): low\_detail\_two is hidden from the screen. - [06:15.182](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=375.1824166666667): left\_twigs is hidden from the screen. - [06:15.182](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=375.1824166666667): left\_twigs\_2 is hidden from the screen. - [06:15.182](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=375.1824166666667): left\_twigs\_3 is hidden from the screen. - [06:15.182](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=375.1824166666667): left\_twigs\_4 is hidden from the screen. - [06:15.182](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=375.1824166666667): left\_twigs\_5 is hidden from the screen. - [06:15.182](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=375.1824166666667): left\_twigs\_6 is hidden from the screen. - [06:17.655](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=377.6554166666667): pruned\_left\_leaf is shown on the screen, written out. - [06:19.292](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=379.29241666666667): pruned\_left\_text is shown on the screen, written out. ##### [06:21.227](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=381.2269166666667) Narration: Now prune the corresponding upper twigs. Again, the replacement is not pure on the training set. It deliberately accepts a few mistakes in exchange for a much larger and more stable prediction region. Board: criterion — a Math \[text\] that says "$R\_alpha(T) = R(T) + alpha thin \|upright("leaves")(T)\|$"; tree — a Figure (x\_range=(0.0, 10.0), y\_range=(0.0, 6.0), aspect=(5.0, 3.0)); plane — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); heading — a Heading that says "Trade Fit for a Smaller Tree"; blue\_points — a Point \[blue\] drawn in plane (location=(0.8, 1.2)); blue\_points\_2 — a Point \[blue\] drawn in plane (location=(1.5, 2.0)); blue\_points\_3 — a Point \[blue\] drawn in plane (location=(2.8, 1.0)); blue\_points\_4 — a Point \[blue\] drawn in plane (location=(3.6, 2.8)); blue\_points\_5 — a Point \[blue\] drawn in plane (location=(4.5, 1.5)); blue\_points\_6 — a Point \[blue\] drawn in plane (location=(5.6, 2.7)); blue\_points\_7 — a Point \[blue\] drawn in plane (location=(2.4, 5.5)); blue\_points\_8 — a Point \[blue\] drawn in plane (location=(6.8, 6.4)); red\_points — a Point \[red\] drawn in plane (location=(0.9, 6.3)); red\_points\_2 — a Point \[red\] drawn in plane (location=(1.8, 4.8)); red\_points\_3 — a Point \[red\] drawn in plane (location=(3.3, 6.7)); red\_points\_4 — a Point \[red\] drawn in plane (location=(4.2, 5.2)); red\_points\_5 — a Point \[red\] drawn in plane (location=(5.4, 6.1)); red\_points\_6 — a Point \[red\] drawn in plane (location=(7.2, 4.8)); red\_points\_7 — a Point \[red\] drawn in plane (location=(2.2, 2.8)); red\_points\_8 — a Point \[red\] drawn in plane (location=(6.5, 1.2)); root — a Polygon \[yellow\] drawn in tree (vertices=((3.7, 5.1), (6.3, 5.1), (6.3, 5.8), (3.7, 5.8)), fill\_opacity=0.12); root\_text — a Math \[yellow\] that says "$x\_2 \< 4?$" drawn in tree; branch\_left — a Line \[gray\] drawn in tree (start=(4.3, 5.1), end=(2.5, 4.0)); branch\_right — a Line \[gray\] drawn in tree (start=(5.7, 5.1), end=(7.5, 4.0)); left\_node — a Polygon \[green\] drawn in tree (vertices=((1.3, 3.3), (3.7, 3.3), (3.7, 4.0), (1.3, 4.0)), fill\_opacity=0.1); right\_node — a Polygon \[magenta\] drawn in tree (vertices=((6.3, 3.3), (8.7, 3.3), (8.7, 4.0), (6.3, 4.0)), fill\_opacity=0.1); left\_text — a Math \[green\] that says "$x\_1 \< 6.1?$" drawn in tree; right\_text — a Math \[magenta\] that says "$x\_1 \< 6.2?$" drawn in tree; right\_twigs — a Line \[gray\] drawn in tree (start=(7.0, 3.3), end=(6.1, 2.2)); right\_twigs\_2 — a Line \[gray\] drawn in tree (start=(8.0, 3.3), end=(8.9, 2.2)); right\_twigs\_3 — a Line \[gray\] drawn in tree (start=(6.1, 2.2), end=(5.4, 1.0)); right\_twigs\_4 — a Line \[gray\] drawn in tree (start=(6.1, 2.2), end=(6.8, 1.0)); right\_twigs\_5 — a Line \[gray\] drawn in tree (start=(8.9, 2.2), end=(8.2, 1.0)); right\_twigs\_6 — a Line \[gray\] drawn in tree (start=(8.9, 2.2), end=(9.6, 1.0)); twig\_nodes — a Point \[gray\] drawn in tree (location=(1.1, 2.2)); twig\_nodes\_2 — a Point \[gray\] drawn in tree (location=(3.9, 2.2)); twig\_nodes\_3 — a Point \[gray\] drawn in tree (location=(6.1, 2.2)); twig\_nodes\_4 — a Point \[gray\] drawn in tree (location=(8.9, 2.2)); twig\_nodes\_5 — a Point \[blue\] drawn in tree (location=(0.6, 1.0)); twig\_nodes\_6 — a Point \[red\] drawn in tree (location=(1.8, 1.0)); twig\_nodes\_7 — a Point \[blue\] drawn in tree (location=(3.3, 1.0)); twig\_nodes\_8 — a Point \[red\] drawn in tree (location=(4.6, 1.0)); twig\_nodes\_9 — a Point \[red\] drawn in tree (location=(5.4, 1.0)); twig\_nodes\_10 — a Point \[blue\] drawn in tree (location=(6.8, 1.0)); twig\_nodes\_11 — a Point \[red\] drawn in tree (location=(8.2, 1.0)); twig\_nodes\_12 — a Point \[blue\] drawn in tree (location=(9.6, 1.0)); root\_cut — a Line \[yellow\] drawn in plane (start=(0.0, 4.0), end=(8.0, 4.0)); low\_main — a Line \[green\] drawn in plane (start=(6.1, 0.0), end=(6.1, 4.0)); high\_main — a Line \[magenta\] drawn in plane (start=(6.2, 4.0), end=(6.2, 8.0)); high\_detail\_one — a Line \[gray\] drawn in plane (start=(6.2, 5.6), end=(8.0, 5.6)); high\_detail\_two — a Line \[gray\] drawn in plane (start=(2.8, 4.0), end=(2.8, 8.0)); high\_detail\_three — a Line \[gray\] drawn in plane (start=(0.0, 5.1), end=(2.8, 5.1)); high\_detail\_four — a Line \[gray\] drawn in plane (start=(1.5, 5.1), end=(1.5, 8.0)); pruned\_left\_leaf — a Polygon \[blue\] drawn in tree (vertices=((1.2, 1.7), (3.8, 1.7), (3.8, 2.5), (1.2, 2.5)), fill\_opacity=0.14); pruned\_left\_text — a Math \[blue\] that says "$upright("mostly B")$" drawn in tree Actions: - [06:21.947](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=381.9474166666667): high\_detail\_two is hidden from the screen. - [06:21.947](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=381.9474166666667): high\_detail\_three is hidden from the screen. - [06:21.947](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=381.9474166666667): high\_detail\_four is hidden from the screen. - [06:23.328](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=383.3284166666667): right\_twigs is hidden from the screen. - [06:23.328](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=383.3284166666667): right\_twigs\_2 is hidden from the screen. - [06:23.328](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=383.3284166666667): right\_twigs\_3 is hidden from the screen. - [06:23.328](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=383.3284166666667): right\_twigs\_4 is hidden from the screen. - [06:23.328](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=383.3284166666667): right\_twigs\_5 is hidden from the screen. - [06:23.328](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=383.3284166666667): right\_twigs\_6 is hidden from the screen. - [06:25.325](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=385.3254166666667): pruned\_right\_leaf is shown on the screen, written out. - [06:25.325](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=385.3254166666667): pruned\_right\_text is shown on the screen, written out. ##### [06:34.25](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=394.2499166666667) Narration: The remaining four-leaf tree still captures the broad interaction. The horizontal root separates low from high x two. Within each half, one x one threshold handles the main exception near the far-right edge. Board: criterion — a Math \[text\] that says "$R\_alpha(T) = R(T) + alpha thin \|upright("leaves")(T)\|$"; tree — a Figure (x\_range=(0.0, 10.0), y\_range=(0.0, 6.0), aspect=(5.0, 3.0)); plane — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); heading — a Heading that says "Trade Fit for a Smaller Tree"; blue\_points — a Point \[blue\] drawn in plane (location=(0.8, 1.2)); blue\_points\_2 — a Point \[blue\] drawn in plane (location=(1.5, 2.0)); blue\_points\_3 — a Point \[blue\] drawn in plane (location=(2.8, 1.0)); blue\_points\_4 — a Point \[blue\] drawn in plane (location=(3.6, 2.8)); blue\_points\_5 — a Point \[blue\] drawn in plane (location=(4.5, 1.5)); blue\_points\_6 — a Point \[blue\] drawn in plane (location=(5.6, 2.7)); blue\_points\_7 — a Point \[blue\] drawn in plane (location=(2.4, 5.5)); blue\_points\_8 — a Point \[blue\] drawn in plane (location=(6.8, 6.4)); red\_points — a Point \[red\] drawn in plane (location=(0.9, 6.3)); red\_points\_2 — a Point \[red\] drawn in plane (location=(1.8, 4.8)); red\_points\_3 — a Point \[red\] drawn in plane (location=(3.3, 6.7)); red\_points\_4 — a Point \[red\] drawn in plane (location=(4.2, 5.2)); red\_points\_5 — a Point \[red\] drawn in plane (location=(5.4, 6.1)); red\_points\_6 — a Point \[red\] drawn in plane (location=(7.2, 4.8)); red\_points\_7 — a Point \[red\] drawn in plane (location=(2.2, 2.8)); red\_points\_8 — a Point \[red\] drawn in plane (location=(6.5, 1.2)); root — a Polygon \[yellow\] drawn in tree (vertices=((3.7, 5.1), (6.3, 5.1), (6.3, 5.8), (3.7, 5.8)), fill\_opacity=0.12); root\_text — a Math \[yellow\] that says "$x\_2 \< 4?$" drawn in tree; branch\_left — a Line \[gray\] drawn in tree (start=(4.3, 5.1), end=(2.5, 4.0)); branch\_right — a Line \[gray\] drawn in tree (start=(5.7, 5.1), end=(7.5, 4.0)); left\_node — a Polygon \[green\] drawn in tree (vertices=((1.3, 3.3), (3.7, 3.3), (3.7, 4.0), (1.3, 4.0)), fill\_opacity=0.1); right\_node — a Polygon \[magenta\] drawn in tree (vertices=((6.3, 3.3), (8.7, 3.3), (8.7, 4.0), (6.3, 4.0)), fill\_opacity=0.1); left\_text — a Math \[green\] that says "$x\_1 \< 6.1?$" drawn in tree; right\_text — a Math \[magenta\] that says "$x\_1 \< 6.2?$" drawn in tree; twig\_nodes — a Point \[gray\] drawn in tree (location=(1.1, 2.2)); twig\_nodes\_2 — a Point \[gray\] drawn in tree (location=(3.9, 2.2)); twig\_nodes\_3 — a Point \[gray\] drawn in tree (location=(6.1, 2.2)); twig\_nodes\_4 — a Point \[gray\] drawn in tree (location=(8.9, 2.2)); twig\_nodes\_5 — a Point \[blue\] drawn in tree (location=(0.6, 1.0)); twig\_nodes\_6 — a Point \[red\] drawn in tree (location=(1.8, 1.0)); twig\_nodes\_7 — a Point \[blue\] drawn in tree (location=(3.3, 1.0)); twig\_nodes\_8 — a Point \[red\] drawn in tree (location=(4.6, 1.0)); twig\_nodes\_9 — a Point \[red\] drawn in tree (location=(5.4, 1.0)); twig\_nodes\_10 — a Point \[blue\] drawn in tree (location=(6.8, 1.0)); twig\_nodes\_11 — a Point \[red\] drawn in tree (location=(8.2, 1.0)); twig\_nodes\_12 — a Point \[blue\] drawn in tree (location=(9.6, 1.0)); root\_cut — a Line \[yellow\] drawn in plane (start=(0.0, 4.0), end=(8.0, 4.0)); low\_main — a Line \[green\] drawn in plane (start=(6.1, 0.0), end=(6.1, 4.0)); high\_main — a Line \[magenta\] drawn in plane (start=(6.2, 4.0), end=(6.2, 8.0)); high\_detail\_one — a Line \[gray\] drawn in plane (start=(6.2, 5.6), end=(8.0, 5.6)); pruned\_left\_leaf — a Polygon \[blue\] drawn in tree (vertices=((1.2, 1.7), (3.8, 1.7), (3.8, 2.5), (1.2, 2.5)), fill\_opacity=0.14); pruned\_left\_text — a Math \[blue\] that says "$upright("mostly B")$" drawn in tree; pruned\_right\_leaf — a Polygon \[red\] drawn in tree (vertices=((6.2, 1.7), (8.8, 1.7), (8.8, 2.5), (6.2, 2.5)), fill\_opacity=0.14); pruned\_right\_text — a Math \[red\] that says "$upright("mostly R")$" drawn in tree Actions: - [06:38.766](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=398.7664166666667): root\_cut is indicated — a transient flash. - [06:43.561](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=403.5614166666667): low\_main is indicated — a transient flash. - [06:43.561](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=403.5614166666667): high\_main is indicated — a transient flash. ##### [06:48.445](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=408.4449166666667) Narration: This smaller tree has higher training error by construction. The relevant question is whether it has lower error on observations that did not participate in choosing all those thresholds. Board: Unchanged from the preceding beat in this scene. Actions: - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): criterion is hidden from the screen — left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): heading is hidden from the screen — left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): plane is hidden from the screen — left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): blue\_points is hidden from the screen — plane left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): blue\_points\_2 is hidden from the screen — plane left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): blue\_points\_3 is hidden from the screen — plane left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): blue\_points\_4 is hidden from the screen — plane left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): blue\_points\_5 is hidden from the screen — plane left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): blue\_points\_6 is hidden from the screen — plane left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): blue\_points\_7 is hidden from the screen — plane left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): blue\_points\_8 is hidden from the screen — plane left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): red\_points is hidden from the screen — plane left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): red\_points\_2 is hidden from the screen — plane left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): red\_points\_3 is hidden from the screen — plane left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): red\_points\_4 is hidden from the screen — plane left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): red\_points\_5 is hidden from the screen — plane left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): red\_points\_6 is hidden from the screen — plane left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): red\_points\_7 is hidden from the screen — plane left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): red\_points\_8 is hidden from the screen — plane left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): root\_cut is hidden from the screen — plane left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): low\_main is hidden from the screen — plane left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): high\_main is hidden from the screen — plane left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): high\_detail\_one is hidden from the screen — plane left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): tree is hidden from the screen — left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): root is hidden from the screen — tree left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): root\_text is hidden from the screen — tree left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): branch\_left is hidden from the screen — tree left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): branch\_right is hidden from the screen — tree left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): left\_node is hidden from the screen — tree left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): right\_node is hidden from the screen — tree left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): left\_text is hidden from the screen — tree left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): right\_text is hidden from the screen — tree left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): twig\_nodes is hidden from the screen — tree left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): twig\_nodes\_2 is hidden from the screen — tree left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): twig\_nodes\_3 is hidden from the screen — tree left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): twig\_nodes\_4 is hidden from the screen — tree left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): twig\_nodes\_5 is hidden from the screen — tree left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): twig\_nodes\_6 is hidden from the screen — tree left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): twig\_nodes\_7 is hidden from the screen — tree left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): twig\_nodes\_8 is hidden from the screen — tree left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): twig\_nodes\_9 is hidden from the screen — tree left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): twig\_nodes\_10 is hidden from the screen — tree left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): twig\_nodes\_11 is hidden from the screen — tree left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): twig\_nodes\_12 is hidden from the screen — tree left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): pruned\_left\_leaf is hidden from the screen — tree left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): pruned\_left\_text is hidden from the screen — tree left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): pruned\_right\_leaf is hidden from the screen — tree left the board. - [06:59.334](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=419.3344166666667): pruned\_right\_text is hidden from the screen — tree left the board. ##### [07:0.534](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=420.5344166666667) Narration: A pruning path supplies a nested sequence of subtrees. Here the unpruned tree has nine leaves and no training mistakes, but its validation error is the worst entry in the table. Board: Empty. Actions: - [07:0.534](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=420.5344166666667): heading\_path is shown on the screen, written out. - [07:1.498](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=421.4984166666667): path is shown on the screen, written out. - [07:6.234](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=426.2344166666667): path is shown on the screen, written out. - [07:9.996](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=429.9964166666667): path (the "$0.250$" part) is indicated — a transient flash. ##### [07:12.222](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=432.2219166666667) Narration: A small penalty removes three leaves. Training error rises, validation error falls. A larger penalty leaves four terminal regions, and the validation error reaches its minimum. Board: heading\_path — a Heading that says "Choose Complexity Outside the Training Fit" Actions: - [07:12.779](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=432.7794166666667): path is shown on the screen, written out. - [07:20.36](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=440.3604166666667): path is shown on the screen, written out. - [07:23.622](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=443.6224166666667): path (the "$0.125$" part) is indicated — a transient flash. ##### [07:25.093](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=445.0934166666667) Narration: Push alpha farther and the tree collapses to two leaves. That model is now too simple for the data, so validation error rises again. Pruning is not a ritual of making trees small. It is model selection along a structured complexity path. Board: Unchanged from the preceding beat in this scene. Actions: - [07:26.161](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=446.1614166666667): path is shown on the screen, written out. - [07:32.779](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=452.7794166666667): path (the "$0.188$" part) is indicated — a transient flash. ##### [07:41.808](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=461.8079166666667) Narration: Select alpha using cross-validation, a held-out set, or a nested procedure when tuning itself must be evaluated. Never choose the pruning strength by returning to the same training impurity that rewarded every twig. Board: Unchanged from the preceding beat in this scene. Actions: - [07:42.156](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=462.1564166666667): selection is shown on the screen, written out. - [07:43.317](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=463.3174166666667): path is indicated — a transient flash. - [07:49.923](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=469.9234166666667): note is shown on the screen, written out. ##### [07:55.841](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=475.84091666666666) Narration: Libraries also offer pre-pruning controls such as maximum depth, minimum samples per leaf, minimum impurity decrease, and maximum leaf count. Those prevent growth. Cost-complexity pruning fits first and removes branches afterward. Board: selection — a Math \[text\] that says "$alpha = 0.040 thin arrow.r thin 4 thin upright("leaves")$"; note — a Panel that says "Training impurity chooses splits. Validation evidence chooses how much of the fitted tree to keep."; heading\_path — a Heading that says "Choose Complexity Outside the Training Fit" Actions: - [08:2.655](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=482.6554166666667): note (the "Training impurity" part) is indicated — a transient flash. - [08:11.676](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=491.6764166666667): A box is drawn around selection. ##### [08:13.298](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=493.2984166666667) Narration: Either route trades some bias for less variance. A single pruned tree is often much easier to explain and more stable than the pure tree. The forest will take a different route: keep many unstable trees, then make their instability cancel. Board: Unchanged from the preceding beat in this scene. Actions: - [08:28.902](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=508.9019166666667): heading\_path is hidden from the screen — left the board. - [08:28.902](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=508.9019166666667): note is hidden from the screen — left the board. - [08:28.902](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=508.9019166666667): path is hidden from the screen — left the board. - [08:28.902](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=508.9019166666667): selection is hidden from the screen — left the board. ### Scene 4: [Planting Many Trees](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=509.9435833333333) Span: 08:29.944–11:46.406 (509.9435833333333s–706.4061875s). #### Objects - bootstrap: a Table \[text\] that says "Tree Bootstrap indices Omitted $T\_1$ $1,1,2,4,5,5,7,8$ $3,6$ $T\_2$ $2,3,3,4,6,7,7,8$ $1,5$ $T\_3$ $1,2,4,4,5,6,8,8$ $3,7$" (rows=(('Tree', 'Bootstrap indices', 'Omitted'), ('$T\_1$', '$1,1,2,4,…, header=True) - caption\_one: a Math \[text\] that says "$T\_1$" - caption\_three: a Math \[text\] that says "$T\_3$" - caption\_two: a Math \[text\] that says "$T\_2$" - feature\_law: a Math \[text\] that says "$S\_(b,j) subset \{1, dots, p\}, quad \|S\_(b,j)\| = m$" - features: a Table \[text\] that says "Node Offered features Winning split Root $x\_2, x\_5, x\_8$ $x\_2 \< 4.1$ Left child $x\_1, x\_4, x\_8$ $x\_1 \< 6.0$ Right child $x\_3, x\_5, x\_7$ $x\_5 \< 2.7$" (rows=(('Node', 'Offered features', 'Winning split'), ('Root', '$x\_2,…, header=True) - heading\_features: a Heading that says "A Random Subset at Every Split" - heading\_samples: a Heading that says "Same Dataset, Different Training Samples" - heading\_trees: a Heading that says "Three Plausible Trees" - lines\_one: a Line \[red\] drawn in tree\_one (start=(0.0, 4.0), end=(8.0, 4.0)) - lines\_one\_2: a Line \[red\] drawn in tree\_one (start=(5.8, 0.0), end=(5.8, 4.0)) - lines\_one\_3: a Line \[red\] drawn in tree\_one (start=(2.3, 4.0), end=(2.3, 8.0)) - lines\_one\_4: a Line \[red\] drawn in tree\_one (start=(2.3, 5.4), end=(6.6, 5.4)) - lines\_one\_5: a Line \[red\] drawn in tree\_one (start=(6.6, 4.0), end=(6.6, 8.0)) - lines\_three: a Line \[blue\] drawn in tree\_three (start=(0.0, 4.5), end=(8.0, 4.5)) - lines\_three\_2: a Line \[blue\] drawn in tree\_three (start=(6.2, 0.0), end=(6.2, 4.5)) - lines\_three\_3: a Line \[blue\] drawn in tree\_three (start=(1.8, 4.5), end=(1.8, 8.0)) - lines\_three\_4: a Line \[blue\] drawn in tree\_three (start=(1.8, 6.0), end=(5.0, 6.0)) - lines\_three\_5: a Line \[blue\] drawn in tree\_three (start=(5.0, 4.5), end=(5.0, 8.0)) - lines\_two: a Line \[green\] drawn in tree\_two (start=(3.0, 0.0), end=(3.0, 8.0)) - lines\_two\_2: a Line \[green\] drawn in tree\_two (start=(0.0, 3.4), end=(3.0, 3.4)) - lines\_two\_3: a Line \[green\] drawn in tree\_two (start=(3.0, 4.6), end=(8.0, 4.6)) - lines\_two\_4: a Line \[green\] drawn in tree\_two (start=(5.2, 0.0), end=(5.2, 4.6)) - lines\_two\_5: a Line \[green\] drawn in tree\_two (start=(6.7, 4.6), end=(6.7, 8.0)) - sample\_law: a Math \[text\] that says "$D\_b = upright("sample with replacement")(D, thin n)$" - tree\_one: an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)) - tree\_three: an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)) - tree\_two: an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)) #### Beats ##### [08:29.944](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=509.9435833333333) Narration: A forest does not begin by cloning one fitted tree. It creates many related training problems. For each tree, draw n observations from the original n with replacement. Board: Empty. Actions: - [08:29.944](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=509.9435833333333): heading\_samples is shown on the screen, written out. - [08:37.537](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=517.5365833333333): sample\_law is shown on the screen, written out. ##### [08:42.27](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=522.2695833333333) Narration: With replacement means an observation can appear more than once. In the first sample, indices one and five are repeated, while three and six are absent. The sample still contains n rows. Board: sample\_law — a Math \[text\] that says "$D\_b = upright("sample with replacement")(D, thin n)$"; heading\_samples — a Heading that says "Same Dataset, Different Training Samples" Actions: - [08:47.75](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=527.7495833333334): bootstrap is shown on the screen, written out. - [08:48.969](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=528.9685833333333): bootstrap is shown on the screen, written out. - [08:50.711](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=530.7105833333334): bootstrap (the "$1,1,2,4,5,5,7,8$" part) is emphasized. - [08:56.69](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=536.6895833333333): bootstrap (the "$1,1,2,4,5,5,7,8$" part) is no longer emphasized. ##### [08:57.29](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=537.2895833333333) Narration: A second bootstrap draw repeats different observations and omits different ones. A third does it again. Each tree therefore sees a perturbed empirical distribution, even though all draws came from the same dataset. Board: Unchanged from the preceding beat in this scene. Actions: - [08:57.789](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=537.7885833333333): bootstrap is shown on the screen, written out. - [09:3.258](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=543.2575833333333): bootstrap is shown on the screen, written out. ##### [09:12.867](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=552.8665833333333) Narration: Fitting deep trees to these samples is ordinary bootstrap aggregation, or bagging. It creates diversity because a marginal observation may be duplicated in one sample and unavailable in another. Board: Unchanged from the preceding beat in this scene. Actions: - None. ##### [09:26.458](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=566.4580833333333) Narration: A random forest adds a second source of variation. At every node, it offers the split search only a random subset of the available features. Board: Unchanged from the preceding beat in this scene. Actions: - [09:36.525](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=576.5245833333333): bootstrap is hidden from the screen — left the board. - [09:36.525](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=576.5245833333333): heading\_samples is hidden from the screen — left the board. - [09:36.525](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=576.5245833333333): sample\_law is hidden from the screen — left the board. ##### [09:37.725](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=577.7245833333333) Narration: Here is one tree in an eight-feature problem. The formula records a fresh feature subset at node j. At the root, features two, five, and eight are offered, and feature two supplies the best available impurity reduction. Board: Empty. Actions: - [09:37.725](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=577.7245833333333): heading\_features is shown on the screen, written out. - [09:39.617](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=579.6165833333333): features is shown on the screen, written out. - [09:41.823](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=581.8225833333333): feature\_law is shown on the screen, written out. - [09:45.771](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=585.7705833333333): features is shown on the screen, written out. - [09:49.834](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=589.8335833333333): features (the "$x\_2 \< 4.1$" part) is indicated — a transient flash. ##### [09:54.079](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=594.0790833333333) Narration: The left child gets a fresh draw, features one, four, and eight. The right child gets another draw, features three, five, and seven. Feature availability is local to a node. Board: feature\_law — a Math \[text\] that says "$S\_(b,j) subset \{1, dots, p\}, quad \|S\_(b,j)\| = m$"; heading\_features — a Heading that says "A Random Subset at Every Split" Actions: - [09:54.683](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=594.6825833333334): features is shown on the screen, written out. - [09:55.531](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=595.5305833333333): features (the "column=2" part) is emphasized. - [09:59.177](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=599.1765833333334): features is shown on the screen, written out. - [10:6.224](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=606.2235833333333): features (the "column=2" part) is no longer emphasized. ##### [10:6.824](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=606.8235833333333) Narration: This restriction can force a tree to ignore the dominant predictor at a particular node. That sounds inefficient for one tree. Its purpose is to prevent every tree from making the same early decisions. Board: Unchanged from the preceding beat in this scene. Actions: - [10:7.393](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=607.3925833333333): feature\_law (the "S\_(b,j)" part) is indicated — a transient flash. ##### [10:20.369](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=620.3685833333333) Narration: Bootstrap sampling perturbs the observations. Feature sampling perturbs the available questions. Together they produce trees whose errors are less synchronized. Board: Unchanged from the preceding beat in this scene. Actions: - [10:30.957](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=630.9570833333333): feature\_law is hidden from the screen — left the board. - [10:30.957](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=630.9570833333333): features is hidden from the screen — left the board. - [10:30.957](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=630.9570833333333): heading\_features is hidden from the screen — left the board. ##### [10:32.157](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=632.1570833333333) Narration: These are three trees trained from the same original problem. Each diagram shows the axis-aligned boundaries produced by one bootstrap sample and one sequence of random feature offers. Board: Empty. Actions: - [10:32.157](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=632.1570833333333): heading\_trees is shown on the screen, written out. - [10:32.877](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=632.8765833333333): tree\_one is shown on the screen, written out. - [10:32.877](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=632.8765833333333): tree\_two is shown on the screen, written out. - [10:32.877](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=632.8765833333333): tree\_three is shown on the screen, written out. - [10:39.437](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=639.4365833333334): caption\_one is shown on the screen, written out. - [10:39.692](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=639.6915833333334): caption\_two is shown on the screen, written out. - [10:40.238](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=640.2375833333333): caption\_three is shown on the screen, written out. ##### [10:44.541](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=644.5405833333333) Narration: The first tree chooses a horizontal root and then several local vertical cuts. Its boundary is coherent within each rectangle, but jagged as a whole. Board: tree\_one — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); caption\_one — a Math \[text\] that says "$T\_1$"; tree\_two — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); caption\_two — a Math \[text\] that says "$T\_2$"; tree\_three — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); caption\_three — a Math \[text\] that says "$T\_3$"; heading\_trees — a Heading that says "Three Plausible Trees" Actions: - [10:45.273](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=645.2725833333334): lines\_one is shown on the screen, written out. - [10:45.393](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=645.3925833333333): lines\_one\_2 is shown on the screen, written out. - [10:45.513](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=645.5125833333334): lines\_one\_3 is shown on the screen, written out. - [10:45.633](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=645.6325833333333): lines\_one\_4 is shown on the screen, written out. - [10:45.753](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=645.7525833333333): lines\_one\_5 is shown on the screen, written out. ##### [10:55.485](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=655.4850833333334) Narration: The second tree begins vertically because its sample and offered features differ. It partitions the same feature plane into a visibly different collection of rectangles. Board: tree\_one — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); caption\_one — a Math \[text\] that says "$T\_1$"; tree\_two — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); caption\_two — a Math \[text\] that says "$T\_2$"; tree\_three — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); caption\_three — a Math \[text\] that says "$T\_3$"; heading\_trees — a Heading that says "Three Plausible Trees"; lines\_one — a Line \[red\] drawn in tree\_one (start=(0.0, 4.0), end=(8.0, 4.0)); lines\_one\_2 — a Line \[red\] drawn in tree\_one (start=(5.8, 0.0), end=(5.8, 4.0)); lines\_one\_3 — a Line \[red\] drawn in tree\_one (start=(2.3, 4.0), end=(2.3, 8.0)); lines\_one\_4 — a Line \[red\] drawn in tree\_one (start=(2.3, 5.4), end=(6.6, 5.4)); lines\_one\_5 — a Line \[red\] drawn in tree\_one (start=(6.6, 4.0), end=(6.6, 8.0)) Actions: - [10:56.089](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=656.0885833333334): lines\_two is shown on the screen, written out. - [10:56.209](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=656.2085833333333): lines\_two\_2 is shown on the screen, written out. - [10:56.329](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=656.3285833333334): lines\_two\_3 is shown on the screen, written out. - [10:56.449](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=656.4485833333333): lines\_two\_4 is shown on the screen, written out. - [10:56.569](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=656.5685833333333): lines\_two\_5 is shown on the screen, written out. ##### [11:6.802](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=666.8015833333334) Narration: The third tree returns to a horizontal root, but its thresholds and deeper branches are different again. None of these trees is intended to be the final boundary. Board: tree\_one — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); caption\_one — a Math \[text\] that says "$T\_1$"; tree\_two — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); caption\_two — a Math \[text\] that says "$T\_2$"; tree\_three — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); caption\_three — a Math \[text\] that says "$T\_3$"; heading\_trees — a Heading that says "Three Plausible Trees"; lines\_one — a Line \[red\] drawn in tree\_one (start=(0.0, 4.0), end=(8.0, 4.0)); lines\_one\_2 — a Line \[red\] drawn in tree\_one (start=(5.8, 0.0), end=(5.8, 4.0)); lines\_one\_3 — a Line \[red\] drawn in tree\_one (start=(2.3, 4.0), end=(2.3, 8.0)); lines\_one\_4 — a Line \[red\] drawn in tree\_one (start=(2.3, 5.4), end=(6.6, 5.4)); lines\_one\_5 — a Line \[red\] drawn in tree\_one (start=(6.6, 4.0), end=(6.6, 8.0)); lines\_two — a Line \[green\] drawn in tree\_two (start=(3.0, 0.0), end=(3.0, 8.0)); lines\_two\_2 — a Line \[green\] drawn in tree\_two (start=(0.0, 3.4), end=(3.0, 3.4)); lines\_two\_3 — a Line \[green\] drawn in tree\_two (start=(3.0, 4.6), end=(8.0, 4.6)); lines\_two\_4 — a Line \[green\] drawn in tree\_two (start=(5.2, 0.0), end=(5.2, 4.6)); lines\_two\_5 — a Line \[green\] drawn in tree\_two (start=(6.7, 4.6), end=(6.7, 8.0)) Actions: - [11:7.359](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=667.3585833333334): lines\_three is shown on the screen, written out. - [11:7.479](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=667.4785833333333): lines\_three\_2 is shown on the screen, written out. - [11:7.599](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=667.5985833333334): lines\_three\_3 is shown on the screen, written out. - [11:7.719](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=667.7185833333333): lines\_three\_4 is shown on the screen, written out. - [11:7.839](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=667.8385833333333): lines\_three\_5 is shown on the screen, written out. ##### [11:17.154](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=677.1540833333333) Narration: For classification, each tree supplies class probabilities from its reached leaf, commonly the class proportions among that leaf's training samples. The forest averages those probabilities and then chooses a class. Board: tree\_one — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); caption\_one — a Math \[text\] that says "$T\_1$"; tree\_two — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); caption\_two — a Math \[text\] that says "$T\_2$"; tree\_three — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); caption\_three — a Math \[text\] that says "$T\_3$"; heading\_trees — a Heading that says "Three Plausible Trees"; lines\_one — a Line \[red\] drawn in tree\_one (start=(0.0, 4.0), end=(8.0, 4.0)); lines\_one\_2 — a Line \[red\] drawn in tree\_one (start=(5.8, 0.0), end=(5.8, 4.0)); lines\_one\_3 — a Line \[red\] drawn in tree\_one (start=(2.3, 4.0), end=(2.3, 8.0)); lines\_one\_4 — a Line \[red\] drawn in tree\_one (start=(2.3, 5.4), end=(6.6, 5.4)); lines\_one\_5 — a Line \[red\] drawn in tree\_one (start=(6.6, 4.0), end=(6.6, 8.0)); lines\_two — a Line \[green\] drawn in tree\_two (start=(3.0, 0.0), end=(3.0, 8.0)); lines\_two\_2 — a Line \[green\] drawn in tree\_two (start=(0.0, 3.4), end=(3.0, 3.4)); lines\_two\_3 — a Line \[green\] drawn in tree\_two (start=(3.0, 4.6), end=(8.0, 4.6)); lines\_two\_4 — a Line \[green\] drawn in tree\_two (start=(5.2, 0.0), end=(5.2, 4.6)); lines\_two\_5 — a Line \[green\] drawn in tree\_two (start=(6.7, 4.6), end=(6.7, 8.0)); lines\_three — a Line \[blue\] drawn in tree\_three (start=(0.0, 4.5), end=(8.0, 4.5)); lines\_three\_2 — a Line \[blue\] drawn in tree\_three (start=(6.2, 0.0), end=(6.2, 4.5)); lines\_three\_3 — a Line \[blue\] drawn in tree\_three (start=(1.8, 4.5), end=(1.8, 8.0)); lines\_three\_4 — a Line \[blue\] drawn in tree\_three (start=(1.8, 6.0), end=(5.0, 6.0)); lines\_three\_5 — a Line \[blue\] drawn in tree\_three (start=(5.0, 4.5), end=(5.0, 8.0)) Actions: - None. ##### [11:31.001](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=691.0005833333333) Narration: For regression, the same architecture averages numeric predictions. Bootstrap samples and random feature subsets still serve the same purpose: build individually flexible models whose mistakes are not identical. Board: Unchanged from the preceding beat in this scene. Actions: - [11:45.365](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=705.3645208333332): caption\_one is hidden from the screen — left the board. - [11:45.365](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=705.3645208333332): caption\_three is hidden from the screen — left the board. - [11:45.365](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=705.3645208333332): caption\_two is hidden from the screen — left the board. - [11:45.365](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=705.3645208333332): heading\_trees is hidden from the screen — left the board. - [11:45.365](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=705.3645208333332): tree\_one is hidden from the screen — left the board. - [11:45.365](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=705.3645208333332): lines\_one is hidden from the screen — tree\_one left the board. - [11:45.365](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=705.3645208333332): lines\_one\_2 is hidden from the screen — tree\_one left the board. - [11:45.365](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=705.3645208333332): lines\_one\_3 is hidden from the screen — tree\_one left the board. - [11:45.365](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=705.3645208333332): lines\_one\_4 is hidden from the screen — tree\_one left the board. - [11:45.365](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=705.3645208333332): lines\_one\_5 is hidden from the screen — tree\_one left the board. - [11:45.365](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=705.3645208333332): tree\_three is hidden from the screen — left the board. - [11:45.365](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=705.3645208333332): lines\_three is hidden from the screen — tree\_three left the board. - [11:45.365](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=705.3645208333332): lines\_three\_2 is hidden from the screen — tree\_three left the board. - [11:45.365](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=705.3645208333332): lines\_three\_3 is hidden from the screen — tree\_three left the board. - [11:45.365](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=705.3645208333332): lines\_three\_4 is hidden from the screen — tree\_three left the board. - [11:45.365](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=705.3645208333332): lines\_three\_5 is hidden from the screen — tree\_three left the board. - [11:45.365](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=705.3645208333332): tree\_two is hidden from the screen — left the board. - [11:45.365](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=705.3645208333332): lines\_two is hidden from the screen — tree\_two left the board. - [11:45.365](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=705.3645208333332): lines\_two\_2 is hidden from the screen — tree\_two left the board. - [11:45.365](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=705.3645208333332): lines\_two\_3 is hidden from the screen — tree\_two left the board. - [11:45.365](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=705.3645208333332): lines\_two\_4 is hidden from the screen — tree\_two left the board. - [11:45.365](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=705.3645208333332): lines\_two\_5 is hidden from the screen — tree\_two left the board. ### Scene 5: [Averaging High-Variance Trees](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=706.4061875) Span: 11:46.406–15:44.665 (706.4061875s–944.6647083333332s). #### Objects - blue\_points: a Point \[blue\] drawn in plane (location=(0.8, 1.2)) - blue\_points\_2: a Point \[blue\] drawn in plane (location=(1.5, 2.0)) - blue\_points\_3: a Point \[blue\] drawn in plane (location=(2.8, 1.0)) - blue\_points\_4: a Point \[blue\] drawn in plane (location=(3.6, 2.8)) - blue\_points\_5: a Point \[blue\] drawn in plane (location=(4.5, 1.5)) - blue\_points\_6: a Point \[blue\] drawn in plane (location=(5.6, 2.7)) - blue\_points\_7: a Point \[blue\] drawn in plane (location=(2.4, 5.5)) - blue\_points\_8: a Point \[blue\] drawn in plane (location=(6.8, 6.4)) - correlated: a Math \[text\] that says "$upright("Var")(overline(T)) = sigma^2 (rho + frac(1-rho, B))$" - count: a Math \[text\] that says "$B = 1$" - decision: a Math \[text\] that says "$hat(y)(x) = upright("arg max")\_k thin hat(p)\_(B,k)(x)$" - heading\_boundary: a Heading that says "Average the Votes" - heading\_variance: a Heading that says "What Averaging Can and Cannot Remove" - hundred\_trees: a Line \[green\] drawn in plane (start=(0.0, 4.7), end=(0.8, 4.7)) - hundred\_trees\_10: a Line \[green\] drawn in plane (start=(4.0, 4.5), end=(4.0, 4.4)) - hundred\_trees\_11: a Line \[green\] drawn in plane (start=(4.0, 4.4), end=(4.8, 4.4)) - hundred\_trees\_12: a Line \[green\] drawn in plane (start=(4.8, 4.4), end=(4.8, 4.3)) - hundred\_trees\_13: a Line \[green\] drawn in plane (start=(4.8, 4.3), end=(5.6, 4.3)) - hundred\_trees\_14: a Line \[green\] drawn in plane (start=(5.6, 4.3), end=(5.6, 4.2)) - hundred\_trees\_15: a Line \[green\] drawn in plane (start=(5.6, 4.2), end=(6.4, 4.2)) - hundred\_trees\_16: a Line \[green\] drawn in plane (start=(6.4, 4.2), end=(6.4, 4.3)) - hundred\_trees\_17: a Line \[green\] drawn in plane (start=(6.4, 4.3), end=(7.2, 4.3)) - hundred\_trees\_18: a Line \[green\] drawn in plane (start=(7.2, 4.3), end=(7.2, 4.5)) - hundred\_trees\_19: a Line \[green\] drawn in plane (start=(7.2, 4.5), end=(8.0, 4.5)) - hundred\_trees\_2: a Line \[green\] drawn in plane (start=(0.8, 4.7), end=(0.8, 4.9)) - hundred\_trees\_3: a Line \[green\] drawn in plane (start=(0.8, 4.9), end=(1.6, 4.9)) - hundred\_trees\_4: a Line \[green\] drawn in plane (start=(1.6, 4.9), end=(1.6, 5.0)) - hundred\_trees\_5: a Line \[green\] drawn in plane (start=(1.6, 5.0), end=(2.4, 5.0)) - hundred\_trees\_6: a Line \[green\] drawn in plane (start=(2.4, 5.0), end=(2.4, 4.8)) - hundred\_trees\_7: a Line \[green\] drawn in plane (start=(2.4, 4.8), end=(3.2, 4.8)) - hundred\_trees\_8: a Line \[green\] drawn in plane (start=(3.2, 4.8), end=(3.2, 4.5)) - hundred\_trees\_9: a Line \[green\] drawn in plane (start=(3.2, 4.5), end=(4.0, 4.5)) - independent: a Math \[text\] that says "$upright("Var")(overline(T)) = frac(sigma^2, B)$" - limit: a Math \[text\] that says "$B arrow.r infinity: thin upright("Var")(overline(T)) arrow.r rho sigma^2$" - mixed\_errors: a Table \[text\] that says "Tree $T\_1$ $T\_2$ $T\_3$ $T\_4$ $T\_5$ Error $+1$ $-1$ $+1$ $-1$ $0$" (rows=(('Tree', '$T\_1$', '$T\_2$', '$T\_3$', '$T\_4$', '$T\_5$'), ('Error…) - mixed\_label: a Tex \[text\] that says "Decorrelated errors" - mixed\_mean: a Math \[text\] that says "$upright("mean error") = 0$" - one\_tree: a Line \[gray\] drawn in plane (start=(0.0, 4.8), end=(1.2, 4.8)) - one\_tree\_10: a Line \[gray\] drawn in plane (start=(6.5, 3.6), end=(6.5, 5.2)) - one\_tree\_11: a Line \[gray\] drawn in plane (start=(6.5, 5.2), end=(8.0, 5.2)) - one\_tree\_2: a Line \[gray\] drawn in plane (start=(1.2, 4.8), end=(1.2, 6.2)) - one\_tree\_3: a Line \[gray\] drawn in plane (start=(1.2, 6.2), end=(2.3, 6.2)) - one\_tree\_4: a Line \[gray\] drawn in plane (start=(2.3, 6.2), end=(2.3, 3.3)) - one\_tree\_5: a Line \[gray\] drawn in plane (start=(2.3, 3.3), end=(3.6, 3.3)) - one\_tree\_6: a Line \[gray\] drawn in plane (start=(3.6, 3.3), end=(3.6, 5.7)) - one\_tree\_7: a Line \[gray\] drawn in plane (start=(3.6, 5.7), end=(5.1, 5.7)) - one\_tree\_8: a Line \[gray\] drawn in plane (start=(5.1, 5.7), end=(5.1, 3.6)) - one\_tree\_9: a Line \[gray\] drawn in plane (start=(5.1, 3.6), end=(6.5, 3.6)) - plane: an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)) - point: a Point \[yellow\] drawn in plane (location=(2.5, 4.8)) - point\_2: a Point \[yellow\] drawn in plane (location=(2.7, 4.8)) - red\_points: a Point \[red\] drawn in plane (location=(0.9, 6.3)) - red\_points\_2: a Point \[red\] drawn in plane (location=(1.8, 4.8)) - red\_points\_3: a Point \[red\] drawn in plane (location=(3.3, 6.7)) - red\_points\_4: a Point \[red\] drawn in plane (location=(4.2, 5.2)) - red\_points\_5: a Point \[red\] drawn in plane (location=(5.4, 6.1)) - red\_points\_6: a Point \[red\] drawn in plane (location=(7.2, 4.8)) - red\_points\_7: a Point \[red\] drawn in plane (location=(2.2, 2.8)) - red\_points\_8: a Point \[red\] drawn in plane (location=(6.5, 1.2)) - same\_errors: a Table \[text\] that says "Tree $T\_1$ $T\_2$ $T\_3$ $T\_4$ $T\_5$ Error $+1$ $+1$ $+1$ $+1$ $+1$" (rows=(('Tree', '$T\_1$', '$T\_2$', '$T\_3$', '$T\_4$', '$T\_5$'), ('Error…) - same\_label: a Tex \[text\] that says "Highly correlated errors" - same\_mean: a Math \[text\] that says "$upright("mean error") = +1$" - ten\_trees: a Line \[yellow\] drawn in plane (start=(0.0, 4.9), end=(1.5, 4.9)) - ten\_trees\_10: a Line \[yellow\] drawn in plane (start=(6.8, 4.1), end=(6.8, 4.6)) - ten\_trees\_11: a Line \[yellow\] drawn in plane (start=(6.8, 4.6), end=(8.0, 4.6)) - ten\_trees\_2: a Line \[yellow\] drawn in plane (start=(1.5, 4.9), end=(1.5, 5.3)) - ten\_trees\_3: a Line \[yellow\] drawn in plane (start=(1.5, 5.3), end=(2.7, 5.3)) - ten\_trees\_4: a Line \[yellow\] drawn in plane (start=(2.7, 5.3), end=(2.7, 4.2)) - ten\_trees\_5: a Line \[yellow\] drawn in plane (start=(2.7, 4.2), end=(4.1, 4.2)) - ten\_trees\_6: a Line \[yellow\] drawn in plane (start=(4.1, 4.2), end=(4.1, 4.7)) - ten\_trees\_7: a Line \[yellow\] drawn in plane (start=(4.1, 4.7), end=(5.5, 4.7)) - ten\_trees\_8: a Line \[yellow\] drawn in plane (start=(5.5, 4.7), end=(5.5, 4.1)) - ten\_trees\_9: a Line \[yellow\] drawn in plane (start=(5.5, 4.1), end=(6.8, 4.1)) - vote: a Math \[text\] that says "$hat(p)\_B(x) = frac(1,B) sum\_(b=1)^B hat(p)\_b(x)$" #### Beats ##### [11:46.406](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=706.4061875) Narration: Return to the labelled plane. One deep tree supplies one jagged boundary. It reacts strongly to the particular observations and random choices that shaped its branches. Board: Empty. Actions: - [11:46.406](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=706.4061875): heading\_boundary is shown on the screen, written out. - [11:46.406](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=706.4061875): plane is shown on the screen, written out. - [11:46.406](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=706.4061875): blue\_points is shown on the screen, written out. - [11:46.446](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=706.4461875): blue\_points\_2 is shown on the screen, written out. - [11:46.486](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=706.4861875): blue\_points\_3 is shown on the screen, written out. - [11:46.526](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=706.5261875): blue\_points\_4 is shown on the screen, written out. - [11:46.566](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=706.5661875): blue\_points\_5 is shown on the screen, written out. - [11:46.606](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=706.6061875): blue\_points\_6 is shown on the screen, written out. - [11:46.646](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=706.6461875): blue\_points\_7 is shown on the screen, written out. - [11:46.686](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=706.6861875): blue\_points\_8 is shown on the screen, written out. - [11:46.726](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=706.7261875): red\_points is shown on the screen, written out. - [11:46.766](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=706.7661875): red\_points\_2 is shown on the screen, written out. - [11:46.806](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=706.8061875): red\_points\_3 is shown on the screen, written out. - [11:46.846](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=706.8461875): red\_points\_4 is shown on the screen, written out. - [11:46.886](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=706.8861875): red\_points\_5 is shown on the screen, written out. - [11:46.926](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=706.9261875): red\_points\_6 is shown on the screen, written out. - [11:46.966](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=706.9661874999999): red\_points\_7 is shown on the screen, written out. - [11:47.006](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=707.0061875): red\_points\_8 is shown on the screen, written out. - [11:48.682](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=708.6821874999999): plane moves to a new place on the board. - [11:48.682](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=708.6821874999999): vote is shown on the screen, written out. - [11:48.682](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=708.6821874999999): count is shown on the screen, written out. - [11:50.737](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=710.7371875): one\_tree is shown on the screen, written out. - [11:50.797](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=710.7971875): one\_tree\_2 is shown on the screen, written out. - [11:50.857](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=710.8571875): one\_tree\_3 is shown on the screen, written out. - [11:50.917](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=710.9171875): one\_tree\_4 is shown on the screen, written out. - [11:50.977](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=710.9771875): one\_tree\_5 is shown on the screen, written out. - [11:51.037](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=711.0371875): one\_tree\_6 is shown on the screen, written out. - [11:51.097](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=711.0971875): one\_tree\_7 is shown on the screen, written out. - [11:51.131](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=711.1311875): decision is shown on the screen, written out. - [11:51.157](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=711.1571875): one\_tree\_8 is shown on the screen, written out. - [11:51.217](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=711.2171875): one\_tree\_9 is shown on the screen, written out. - [11:51.277](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=711.2771875): one\_tree\_10 is shown on the screen, written out. - [11:51.337](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=711.3371875): one\_tree\_11 is shown on the screen, written out. ##### [11:58.291](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=718.2911875) Narration: For each query point x, tree b supplies a class probability. The forest averages those probabilities. Its displayed boundary is where the winning averaged class changes. Board: vote — a Math \[text\] that says "$hat(p)\_B(x) = frac(1,B) sum\_(b=1)^B hat(p)\_b(x)$"; decision — a Math \[text\] that says "$hat(y)(x) = upright("arg max")\_k thin hat(p)\_(B,k)(x)$"; count — a Math \[text\] that says "$B = 1$"; plane — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); heading\_boundary — a Heading that says "Average the Votes"; blue\_points — a Point \[blue\] drawn in plane (location=(0.8, 1.2)); blue\_points\_2 — a Point \[blue\] drawn in plane (location=(1.5, 2.0)); blue\_points\_3 — a Point \[blue\] drawn in plane (location=(2.8, 1.0)); blue\_points\_4 — a Point \[blue\] drawn in plane (location=(3.6, 2.8)); blue\_points\_5 — a Point \[blue\] drawn in plane (location=(4.5, 1.5)); blue\_points\_6 — a Point \[blue\] drawn in plane (location=(5.6, 2.7)); blue\_points\_7 — a Point \[blue\] drawn in plane (location=(2.4, 5.5)); blue\_points\_8 — a Point \[blue\] drawn in plane (location=(6.8, 6.4)); red\_points — a Point \[red\] drawn in plane (location=(0.9, 6.3)); red\_points\_2 — a Point \[red\] drawn in plane (location=(1.8, 4.8)); red\_points\_3 — a Point \[red\] drawn in plane (location=(3.3, 6.7)); red\_points\_4 — a Point \[red\] drawn in plane (location=(4.2, 5.2)); red\_points\_5 — a Point \[red\] drawn in plane (location=(5.4, 6.1)); red\_points\_6 — a Point \[red\] drawn in plane (location=(7.2, 4.8)); red\_points\_7 — a Point \[red\] drawn in plane (location=(2.2, 2.8)); red\_points\_8 — a Point \[red\] drawn in plane (location=(6.5, 1.2)); one\_tree — a Line \[gray\] drawn in plane (start=(0.0, 4.8), end=(1.2, 4.8)); one\_tree\_2 — a Line \[gray\] drawn in plane (start=(1.2, 4.8), end=(1.2, 6.2)); one\_tree\_3 — a Line \[gray\] drawn in plane (start=(1.2, 6.2), end=(2.3, 6.2)); one\_tree\_4 — a Line \[gray\] drawn in plane (start=(2.3, 6.2), end=(2.3, 3.3)); one\_tree\_5 — a Line \[gray\] drawn in plane (start=(2.3, 3.3), end=(3.6, 3.3)); one\_tree\_6 — a Line \[gray\] drawn in plane (start=(3.6, 3.3), end=(3.6, 5.7)); one\_tree\_7 — a Line \[gray\] drawn in plane (start=(3.6, 5.7), end=(5.1, 5.7)); one\_tree\_8 — a Line \[gray\] drawn in plane (start=(5.1, 5.7), end=(5.1, 3.6)); one\_tree\_9 — a Line \[gray\] drawn in plane (start=(5.1, 3.6), end=(6.5, 3.6)); one\_tree\_10 — a Line \[gray\] drawn in plane (start=(6.5, 3.6), end=(6.5, 5.2)); one\_tree\_11 — a Line \[gray\] drawn in plane (start=(6.5, 5.2), end=(8.0, 5.2)) Actions: - [12:0.532](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=720.5321875): vote (the "hat(p)\_b(x)" part) is emphasized. - [12:4.235](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=724.2351874999999): vote (the "frac(1,B) sum\_(b=1)^B" part) is emphasized. - [12:4.235](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=724.2351874999999): vote (the "hat(p)\_b(x)" part) is no longer emphasized. - [12:7.451](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=727.4511875): vote (the "frac(1,B) sum\_(b=1)^B" part) is no longer emphasized. ##### [12:11.197](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=731.1971874999999) Narration: Add trees until B equals ten. Each individual boundary is still made of rectangular steps, but their idiosyncratic notches occur at different places. Majority support removes many notches that only one tree wanted. Board: Unchanged from the preceding beat in this scene. Actions: - [12:11.545](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=731.5451875): one\_tree is hidden from the screen. - [12:11.545](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=731.5451875): one\_tree\_2 is hidden from the screen. - [12:11.545](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=731.5451875): one\_tree\_3 is hidden from the screen. - [12:11.545](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=731.5451875): one\_tree\_4 is hidden from the screen. - [12:11.545](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=731.5451875): one\_tree\_5 is hidden from the screen. - [12:11.545](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=731.5451875): one\_tree\_6 is hidden from the screen. - [12:11.545](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=731.5451875): one\_tree\_7 is hidden from the screen. - [12:11.545](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=731.5451875): one\_tree\_8 is hidden from the screen. - [12:11.545](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=731.5451875): one\_tree\_9 is hidden from the screen. - [12:11.545](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=731.5451875): one\_tree\_10 is hidden from the screen. - [12:11.545](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=731.5451875): one\_tree\_11 is hidden from the screen. - [12:13.31](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=733.3101875): count becomes "$B = 10$". - [12:13.31](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=733.3101875): ten\_trees is shown on the screen, written out. - [12:13.37](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=733.3701874999999): ten\_trees\_2 is shown on the screen, written out. - [12:13.43](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=733.4301875): ten\_trees\_3 is shown on the screen, written out. - [12:13.49](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=733.4901874999999): ten\_trees\_4 is shown on the screen, written out. - [12:13.55](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=733.5501875): ten\_trees\_5 is shown on the screen, written out. - [12:13.61](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=733.6101874999999): ten\_trees\_6 is shown on the screen, written out. - [12:13.67](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=733.6701875): ten\_trees\_7 is shown on the screen, written out. - [12:13.73](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=733.7301874999999): ten\_trees\_8 is shown on the screen, written out. - [12:13.79](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=733.7901875): ten\_trees\_9 is shown on the screen, written out. - [12:13.85](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=733.8501875): ten\_trees\_10 is shown on the screen, written out. - [12:13.91](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=733.9101875): ten\_trees\_11 is shown on the screen, written out. ##### [12:26.809](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=746.8091875) Narration: Now average one hundred trees. The aggregate is still a staircase if we inspect it finely enough, because every contributor is a tree. At this scale, however, it follows the broad class structure rather than every isolated observation. Board: vote — a Math \[text\] that says "$hat(p)\_B(x) = frac(1,B) sum\_(b=1)^B hat(p)\_b(x)$"; decision — a Math \[text\] that says "$hat(y)(x) = upright("arg max")\_k thin hat(p)\_(B,k)(x)$"; count — a Math \[text\] that says "$B = 1$"; plane — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); heading\_boundary — a Heading that says "Average the Votes"; blue\_points — a Point \[blue\] drawn in plane (location=(0.8, 1.2)); blue\_points\_2 — a Point \[blue\] drawn in plane (location=(1.5, 2.0)); blue\_points\_3 — a Point \[blue\] drawn in plane (location=(2.8, 1.0)); blue\_points\_4 — a Point \[blue\] drawn in plane (location=(3.6, 2.8)); blue\_points\_5 — a Point \[blue\] drawn in plane (location=(4.5, 1.5)); blue\_points\_6 — a Point \[blue\] drawn in plane (location=(5.6, 2.7)); blue\_points\_7 — a Point \[blue\] drawn in plane (location=(2.4, 5.5)); blue\_points\_8 — a Point \[blue\] drawn in plane (location=(6.8, 6.4)); red\_points — a Point \[red\] drawn in plane (location=(0.9, 6.3)); red\_points\_2 — a Point \[red\] drawn in plane (location=(1.8, 4.8)); red\_points\_3 — a Point \[red\] drawn in plane (location=(3.3, 6.7)); red\_points\_4 — a Point \[red\] drawn in plane (location=(4.2, 5.2)); red\_points\_5 — a Point \[red\] drawn in plane (location=(5.4, 6.1)); red\_points\_6 — a Point \[red\] drawn in plane (location=(7.2, 4.8)); red\_points\_7 — a Point \[red\] drawn in plane (location=(2.2, 2.8)); red\_points\_8 — a Point \[red\] drawn in plane (location=(6.5, 1.2)); ten\_trees — a Line \[yellow\] drawn in plane (start=(0.0, 4.9), end=(1.5, 4.9)); ten\_trees\_2 — a Line \[yellow\] drawn in plane (start=(1.5, 4.9), end=(1.5, 5.3)); ten\_trees\_3 — a Line \[yellow\] drawn in plane (start=(1.5, 5.3), end=(2.7, 5.3)); ten\_trees\_4 — a Line \[yellow\] drawn in plane (start=(2.7, 5.3), end=(2.7, 4.2)); ten\_trees\_5 — a Line \[yellow\] drawn in plane (start=(2.7, 4.2), end=(4.1, 4.2)); ten\_trees\_6 — a Line \[yellow\] drawn in plane (start=(4.1, 4.2), end=(4.1, 4.7)); ten\_trees\_7 — a Line \[yellow\] drawn in plane (start=(4.1, 4.7), end=(5.5, 4.7)); ten\_trees\_8 — a Line \[yellow\] drawn in plane (start=(5.5, 4.7), end=(5.5, 4.1)); ten\_trees\_9 — a Line \[yellow\] drawn in plane (start=(5.5, 4.1), end=(6.8, 4.1)); ten\_trees\_10 — a Line \[yellow\] drawn in plane (start=(6.8, 4.1), end=(6.8, 4.6)); ten\_trees\_11 — a Line \[yellow\] drawn in plane (start=(6.8, 4.6), end=(8.0, 4.6)) Actions: - [12:28.516](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=748.5161875): count becomes "$B = 100$". - [12:28.516](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=748.5161875): ten\_trees is hidden from the screen. - [12:28.516](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=748.5161875): ten\_trees\_2 is hidden from the screen. - [12:28.516](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=748.5161875): ten\_trees\_3 is hidden from the screen. - [12:28.516](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=748.5161875): ten\_trees\_4 is hidden from the screen. - [12:28.516](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=748.5161875): ten\_trees\_5 is hidden from the screen. - [12:28.516](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=748.5161875): ten\_trees\_6 is hidden from the screen. - [12:28.516](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=748.5161875): ten\_trees\_7 is hidden from the screen. - [12:28.516](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=748.5161875): ten\_trees\_8 is hidden from the screen. - [12:28.516](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=748.5161875): ten\_trees\_9 is hidden from the screen. - [12:28.516](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=748.5161875): ten\_trees\_10 is hidden from the screen. - [12:28.516](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=748.5161875): ten\_trees\_11 is hidden from the screen. - [12:28.516](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=748.5161875): hundred\_trees is shown on the screen, written out. - [12:28.556](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=748.5561875): hundred\_trees\_2 is shown on the screen, written out. - [12:28.596](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=748.5961875): hundred\_trees\_3 is shown on the screen, written out. - [12:28.636](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=748.6361875): hundred\_trees\_4 is shown on the screen, written out. - [12:28.676](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=748.6761875): hundred\_trees\_5 is shown on the screen, written out. - [12:28.716](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=748.7161874999999): hundred\_trees\_6 is shown on the screen, written out. - [12:28.756](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=748.7561875): hundred\_trees\_7 is shown on the screen, written out. - [12:28.796](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=748.7961875): hundred\_trees\_8 is shown on the screen, written out. - [12:28.836](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=748.8361874999999): hundred\_trees\_9 is shown on the screen, written out. - [12:28.876](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=748.8761875): hundred\_trees\_10 is shown on the screen, written out. - [12:28.916](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=748.9161875): hundred\_trees\_11 is shown on the screen, written out. - [12:28.956](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=748.9561874999999): hundred\_trees\_12 is shown on the screen, written out. - [12:28.996](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=748.9961875): hundred\_trees\_13 is shown on the screen, written out. - [12:29.036](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=749.0361875): hundred\_trees\_14 is shown on the screen, written out. - [12:29.076](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=749.0761875): hundred\_trees\_15 is shown on the screen, written out. - [12:29.116](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=749.1161875): hundred\_trees\_16 is shown on the screen, written out. - [12:29.156](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=749.1561875): hundred\_trees\_17 is shown on the screen, written out. - [12:29.196](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=749.1961875): hundred\_trees\_18 is shown on the screen, written out. - [12:29.236](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=749.2361875): hundred\_trees\_19 is shown on the screen, written out. ##### [12:42.56](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=762.5596875) Narration: Calling this boundary smoother does not mean that a forest fits a smooth analytic function. It means the averaged prediction varies more stably across nearby points and fewer decisions depend on one tree's narrow rectangular accident. Board: vote — a Math \[text\] that says "$hat(p)\_B(x) = frac(1,B) sum\_(b=1)^B hat(p)\_b(x)$"; decision — a Math \[text\] that says "$hat(y)(x) = upright("arg max")\_k thin hat(p)\_(B,k)(x)$"; count — a Math \[text\] that says "$B = 1$"; plane — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); heading\_boundary — a Heading that says "Average the Votes"; blue\_points — a Point \[blue\] drawn in plane (location=(0.8, 1.2)); blue\_points\_2 — a Point \[blue\] drawn in plane (location=(1.5, 2.0)); blue\_points\_3 — a Point \[blue\] drawn in plane (location=(2.8, 1.0)); blue\_points\_4 — a Point \[blue\] drawn in plane (location=(3.6, 2.8)); blue\_points\_5 — a Point \[blue\] drawn in plane (location=(4.5, 1.5)); blue\_points\_6 — a Point \[blue\] drawn in plane (location=(5.6, 2.7)); blue\_points\_7 — a Point \[blue\] drawn in plane (location=(2.4, 5.5)); blue\_points\_8 — a Point \[blue\] drawn in plane (location=(6.8, 6.4)); red\_points — a Point \[red\] drawn in plane (location=(0.9, 6.3)); red\_points\_2 — a Point \[red\] drawn in plane (location=(1.8, 4.8)); red\_points\_3 — a Point \[red\] drawn in plane (location=(3.3, 6.7)); red\_points\_4 — a Point \[red\] drawn in plane (location=(4.2, 5.2)); red\_points\_5 — a Point \[red\] drawn in plane (location=(5.4, 6.1)); red\_points\_6 — a Point \[red\] drawn in plane (location=(7.2, 4.8)); red\_points\_7 — a Point \[red\] drawn in plane (location=(2.2, 2.8)); red\_points\_8 — a Point \[red\] drawn in plane (location=(6.5, 1.2)); hundred\_trees — a Line \[green\] drawn in plane (start=(0.0, 4.7), end=(0.8, 4.7)); hundred\_trees\_2 — a Line \[green\] drawn in plane (start=(0.8, 4.7), end=(0.8, 4.9)); hundred\_trees\_3 — a Line \[green\] drawn in plane (start=(0.8, 4.9), end=(1.6, 4.9)); hundred\_trees\_4 — a Line \[green\] drawn in plane (start=(1.6, 4.9), end=(1.6, 5.0)); hundred\_trees\_5 — a Line \[green\] drawn in plane (start=(1.6, 5.0), end=(2.4, 5.0)); hundred\_trees\_6 — a Line \[green\] drawn in plane (start=(2.4, 5.0), end=(2.4, 4.8)); hundred\_trees\_7 — a Line \[green\] drawn in plane (start=(2.4, 4.8), end=(3.2, 4.8)); hundred\_trees\_8 — a Line \[green\] drawn in plane (start=(3.2, 4.8), end=(3.2, 4.5)); hundred\_trees\_9 — a Line \[green\] drawn in plane (start=(3.2, 4.5), end=(4.0, 4.5)); hundred\_trees\_10 — a Line \[green\] drawn in plane (start=(4.0, 4.5), end=(4.0, 4.4)); hundred\_trees\_11 — a Line \[green\] drawn in plane (start=(4.0, 4.4), end=(4.8, 4.4)); hundred\_trees\_12 — a Line \[green\] drawn in plane (start=(4.8, 4.4), end=(4.8, 4.3)); hundred\_trees\_13 — a Line \[green\] drawn in plane (start=(4.8, 4.3), end=(5.6, 4.3)); hundred\_trees\_14 — a Line \[green\] drawn in plane (start=(5.6, 4.3), end=(5.6, 4.2)); hundred\_trees\_15 — a Line \[green\] drawn in plane (start=(5.6, 4.2), end=(6.4, 4.2)); hundred\_trees\_16 — a Line \[green\] drawn in plane (start=(6.4, 4.2), end=(6.4, 4.3)); hundred\_trees\_17 — a Line \[green\] drawn in plane (start=(6.4, 4.3), end=(7.2, 4.3)); hundred\_trees\_18 — a Line \[green\] drawn in plane (start=(7.2, 4.3), end=(7.2, 4.5)); hundred\_trees\_19 — a Line \[green\] drawn in plane (start=(7.2, 4.5), end=(8.0, 4.5)) Actions: - [12:50.953](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=770.9531875): point is shown on the screen, grown. - [12:50.953](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=770.9531875): point\_2 is shown on the screen, grown. - [12:52.953](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=772.9531875): point is hidden from the screen. - [12:52.953](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=772.9531875): point\_2 is hidden from the screen. ##### [12:56.882](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=776.8821875) Narration: Why should averaging unstable models work? Imagine each tree prediction as a useful signal plus a zero-mean fitting error. Averaging keeps the shared signal. Errors that point in different directions can cancel. Board: Unchanged from the preceding beat in this scene. Actions: - [12:57.892](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=777.8921875): vote (the "frac(1,B)" part) is indicated — a transient flash. ##### [13:12.32](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=792.3196875) Narration: If the tree errors were independent and each had variance sigma squared, the mean of B trees would have variance sigma squared over B. Double the number of trees and this variance contribution halves. Board: Unchanged from the preceding beat in this scene. Actions: - [13:14.073](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=794.0731875): independent is shown on the screen, written out. - [13:19.657](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=799.6571875): independent (the "frac(sigma^2, B)" part) is indicated — a transient flash. ##### [13:26.364](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=806.3636875) Narration: Real tree errors are not independent. They use the same original dataset, and strong predictors can make them discover similar branches. Let rho represent their average pairwise error correlation. Board: vote — a Math \[text\] that says "$hat(p)\_B(x) = frac(1,B) sum\_(b=1)^B hat(p)\_b(x)$"; decision — a Math \[text\] that says "$hat(y)(x) = upright("arg max")\_k thin hat(p)\_(B,k)(x)$"; count — a Math \[text\] that says "$B = 1$"; independent — a Math \[text\] that says "$upright("Var")(overline(T)) = frac(sigma^2, B)$"; plane — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); heading\_boundary — a Heading that says "Average the Votes"; blue\_points — a Point \[blue\] drawn in plane (location=(0.8, 1.2)); blue\_points\_2 — a Point \[blue\] drawn in plane (location=(1.5, 2.0)); blue\_points\_3 — a Point \[blue\] drawn in plane (location=(2.8, 1.0)); blue\_points\_4 — a Point \[blue\] drawn in plane (location=(3.6, 2.8)); blue\_points\_5 — a Point \[blue\] drawn in plane (location=(4.5, 1.5)); blue\_points\_6 — a Point \[blue\] drawn in plane (location=(5.6, 2.7)); blue\_points\_7 — a Point \[blue\] drawn in plane (location=(2.4, 5.5)); blue\_points\_8 — a Point \[blue\] drawn in plane (location=(6.8, 6.4)); red\_points — a Point \[red\] drawn in plane (location=(0.9, 6.3)); red\_points\_2 — a Point \[red\] drawn in plane (location=(1.8, 4.8)); red\_points\_3 — a Point \[red\] drawn in plane (location=(3.3, 6.7)); red\_points\_4 — a Point \[red\] drawn in plane (location=(4.2, 5.2)); red\_points\_5 — a Point \[red\] drawn in plane (location=(5.4, 6.1)); red\_points\_6 — a Point \[red\] drawn in plane (location=(7.2, 4.8)); red\_points\_7 — a Point \[red\] drawn in plane (location=(2.2, 2.8)); red\_points\_8 — a Point \[red\] drawn in plane (location=(6.5, 1.2)); hundred\_trees — a Line \[green\] drawn in plane (start=(0.0, 4.7), end=(0.8, 4.7)); hundred\_trees\_2 — a Line \[green\] drawn in plane (start=(0.8, 4.7), end=(0.8, 4.9)); hundred\_trees\_3 — a Line \[green\] drawn in plane (start=(0.8, 4.9), end=(1.6, 4.9)); hundred\_trees\_4 — a Line \[green\] drawn in plane (start=(1.6, 4.9), end=(1.6, 5.0)); hundred\_trees\_5 — a Line \[green\] drawn in plane (start=(1.6, 5.0), end=(2.4, 5.0)); hundred\_trees\_6 — a Line \[green\] drawn in plane (start=(2.4, 5.0), end=(2.4, 4.8)); hundred\_trees\_7 — a Line \[green\] drawn in plane (start=(2.4, 4.8), end=(3.2, 4.8)); hundred\_trees\_8 — a Line \[green\] drawn in plane (start=(3.2, 4.8), end=(3.2, 4.5)); hundred\_trees\_9 — a Line \[green\] drawn in plane (start=(3.2, 4.5), end=(4.0, 4.5)); hundred\_trees\_10 — a Line \[green\] drawn in plane (start=(4.0, 4.5), end=(4.0, 4.4)); hundred\_trees\_11 — a Line \[green\] drawn in plane (start=(4.0, 4.4), end=(4.8, 4.4)); hundred\_trees\_12 — a Line \[green\] drawn in plane (start=(4.8, 4.4), end=(4.8, 4.3)); hundred\_trees\_13 — a Line \[green\] drawn in plane (start=(4.8, 4.3), end=(5.6, 4.3)); hundred\_trees\_14 — a Line \[green\] drawn in plane (start=(5.6, 4.3), end=(5.6, 4.2)); hundred\_trees\_15 — a Line \[green\] drawn in plane (start=(5.6, 4.2), end=(6.4, 4.2)); hundred\_trees\_16 — a Line \[green\] drawn in plane (start=(6.4, 4.2), end=(6.4, 4.3)); hundred\_trees\_17 — a Line \[green\] drawn in plane (start=(6.4, 4.3), end=(7.2, 4.3)); hundred\_trees\_18 — a Line \[green\] drawn in plane (start=(7.2, 4.3), end=(7.2, 4.5)); hundred\_trees\_19 — a Line \[green\] drawn in plane (start=(7.2, 4.5), end=(8.0, 4.5)) Actions: - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): count is hidden from the screen — left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): decision is hidden from the screen — left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): heading\_boundary is hidden from the screen — left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): independent is hidden from the screen — left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): plane is hidden from the screen — left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): blue\_points is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): blue\_points\_2 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): blue\_points\_3 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): blue\_points\_4 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): blue\_points\_5 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): blue\_points\_6 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): blue\_points\_7 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): blue\_points\_8 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): red\_points is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): red\_points\_2 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): red\_points\_3 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): red\_points\_4 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): red\_points\_5 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): red\_points\_6 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): red\_points\_7 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): red\_points\_8 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): hundred\_trees is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): hundred\_trees\_2 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): hundred\_trees\_3 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): hundred\_trees\_4 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): hundred\_trees\_5 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): hundred\_trees\_6 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): hundred\_trees\_7 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): hundred\_trees\_8 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): hundred\_trees\_9 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): hundred\_trees\_10 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): hundred\_trees\_11 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): hundred\_trees\_12 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): hundred\_trees\_13 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): hundred\_trees\_14 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): hundred\_trees\_15 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): hundred\_trees\_16 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): hundred\_trees\_17 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): hundred\_trees\_18 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): hundred\_trees\_19 is hidden from the screen — plane left the board. - [13:39.483](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=819.4826875): vote is hidden from the screen — left the board. ##### [13:40.683](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=820.6826874999999) Narration: Here is the extreme problem. Five trees all make an error of plus one on the same case. Their average error is still plus one. Repetition did not remove a shared mistake. Board: Empty. Actions: - [13:40.683](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=820.6826874999999): heading\_variance is shown on the screen, written out. - [13:43.26](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=823.2601875): same\_label is shown on the screen, written out. - [13:43.631](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=823.6311875): same\_errors is shown on the screen, written out. - [13:45.524](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=825.5241874999999): same\_errors is shown on the screen, written out. - [13:48.252](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=828.2521875): same\_mean is shown on the screen, written out. ##### [13:54.379](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=834.3786875) Narration: On the right, errors differ across trees. Positive and negative errors offset, and their average is zero in this small illustration. Diversity is useful when it concerns errors, not merely visual differences between diagrams. Board: same\_label — a Tex \[text\] that says "Highly correlated errors"; same\_mean — a Math \[text\] that says "$upright("mean error") = +1$"; heading\_variance — a Heading that says "What Averaging Can and Cannot Remove" Actions: - [13:55.087](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=835.0871875): mixed\_label is shown on the screen, written out. - [13:57.037](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=837.0371875): mixed\_errors is shown on the screen, written out. - [13:58.314](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=838.3141875): mixed\_errors is shown on the screen, written out. - [14:2.111](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=842.1111874999999): mixed\_mean is shown on the screen, written out. - [14:11.143](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=851.1431875): mixed\_errors is hidden from the screen — left the board. - [14:11.143](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=851.1431875): mixed\_label is hidden from the screen — left the board. - [14:11.143](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=851.1431875): mixed\_mean is hidden from the screen — left the board. - [14:11.143](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=851.1431875): same\_errors is hidden from the screen — left the board. - [14:11.143](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=851.1431875): same\_label is hidden from the screen — left the board. - [14:11.143](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=851.1431875): same\_mean is hidden from the screen — left the board. - [14:11.143](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=851.1431875): independent is shown on the screen, faded in — cast on this board again. - [14:11.143](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=851.1431875): vote is shown on the screen, faded in — cast on this board again. ##### [14:11.743](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=851.7431875) Narration: With equal variance and average correlation rho, the ensemble variance is sigma squared times rho plus one minus rho over B. The second part shrinks as trees are added. The correlated part does not. Board: vote — a Math \[text\] that says "$hat(p)\_B(x) = frac(1,B) sum\_(b=1)^B hat(p)\_b(x)$"; independent — a Math \[text\] that says "$upright("Var")(overline(T)) = frac(sigma^2, B)$"; heading\_variance — a Heading that says "What Averaging Can and Cannot Remove" Actions: - [14:13.821](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=853.8211875): correlated is shown on the screen, written out. - [14:23.55](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=863.5501875): correlated (the "frac(1-rho, B)" part) is emphasized. - [14:26.058](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=866.0581875): correlated (the "frac(1-rho, B)" part) is no longer emphasized. - [14:26.058](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=866.0581875): correlated (the "rho" part) is emphasized. - [14:27.939](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=867.9391875): correlated (the "rho" part) is no longer emphasized. ##### [14:28.539](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=868.5391875) Narration: Take the number of trees toward infinity. The variance approaches rho sigma squared, not zero. If rho is close to one, a huge forest behaves like repeated copies of one unstable tree. Board: vote — a Math \[text\] that says "$hat(p)\_B(x) = frac(1,B) sum\_(b=1)^B hat(p)\_b(x)$"; independent — a Math \[text\] that says "$upright("Var")(overline(T)) = frac(sigma^2, B)$"; heading\_variance — a Heading that says "What Averaging Can and Cannot Remove"; correlated — a Math \[text\] that says "$upright("Var")(overline(T)) = sigma^2 (rho + frac(1-rho, B))$" Actions: - [14:30.385](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=870.3851875): limit is shown on the screen, written out. - [14:33.067](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=873.0671875): limit (the "rho sigma^2" part) is indicated — a transient flash. ##### [14:42.862](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=882.8621875) Narration: This is why decorrelation is essential. Bootstrap samples alter which observations drive the branches. Random feature subsets prevent one dominant predictor from forcing the same root and early splits in every tree. Board: vote — a Math \[text\] that says "$hat(p)\_B(x) = frac(1,B) sum\_(b=1)^B hat(p)\_b(x)$"; independent — a Math \[text\] that says "$upright("Var")(overline(T)) = frac(sigma^2, B)$"; heading\_variance — a Heading that says "What Averaging Can and Cannot Remove"; correlated — a Math \[text\] that says "$upright("Var")(overline(T)) = sigma^2 (rho + frac(1-rho, B))$"; limit — a Math \[text\] that says "$B arrow.r infinity: thin upright("Var")(overline(T)) arrow.r rho sigma^2$" Actions: - [14:43.744](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=883.7441875): correlated (the "rho" part) is indicated — a transient flash. ##### [14:57.94](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=897.9396875) Narration: There is a trade-off. Offering fewer features can weaken each individual tree by denying it useful predictors. But if that loss is modest and the correlation falls substantially, the average can generalize better. Board: Unchanged from the preceding beat in this scene. Actions: - [15:2.015](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=902.0151874999999): vote (the "hat(p)\_b(x)" part) is indicated — a transient flash. ##### [15:12.727](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=912.7266875) Narration: Adding trees mainly reduces Monte Carlo noise in the fitted ensemble. It does not repair severe bias, leakage, bad labels, a shifted deployment population, or a feature set that contains no useful signal. Board: Unchanged from the preceding beat in this scene. Actions: - [15:13.284](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=913.2841874999999): independent (the "B" part) is indicated — a transient flash. ##### [15:27.247](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=927.2471875) Narration: The forest's advantage is therefore specific. Deep trees provide flexible, high-variance base predictions. Resampling and feature randomness make their errors less alike. Averaging then removes the part of the variance that is not shared. Board: Unchanged from the preceding beat in this scene. Actions: - [15:42.351](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=942.3511874999999): A box is drawn around correlated. - [15:43.623](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=943.6230416666666): correlated is hidden from the screen — left the board. - [15:43.623](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=943.6230416666666): heading\_variance is hidden from the screen — left the board. - [15:43.623](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=943.6230416666666): independent is hidden from the screen — left the board. - [15:43.623](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=943.6230416666666): limit is hidden from the screen — left the board. - [15:43.623](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=943.6230416666666): vote is hidden from the screen — left the board. ### Scene 6: [From the Idea to the Library](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=944.6647083333332) Span: 15:44.665–20:0.191 (944.6647083333332s–1200.1914375s). #### Objects - diagnostics: a Table \[text\] that says "Term Practical reading Out-of-bag score Predictions from trees that omitted each row Feature importance A summary to investigate, not a causal effect Permutation importance Loss after disrupting one feature Class weight A change to fitting…" (rows=(('Term', 'Practical reading'), ('Out-of-bag score', 'Predictio…, header=True) - final\_law: a Math \[text\] that says "$upright("strong trees") + upright("low error correlation") + upright("averaging")$" - forest\_controls: a Table \[text\] that says "Forest control Effect \`bootstrap\` Observation resampling \`max\_features\` Correlation and tree strength \`n\_estimators\` Monte Carlo stability \`n\_jobs\` Compute, not statistics" (rows=(('Forest control', 'Effect'), ('\`bootstrap\`', 'Observation res…, header=True) - heading\_close: a Heading that says "What to Carry Away" - heading\_controls: a Heading that says "Translate the Picture into Parameters" - heading\_diagnostics: a Heading that says "Diagnostics with Precise Meanings" - heading\_workflow: a Heading that says "A Practical Fitting Sequence" - plane: an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)) - point: a Point \[yellow\] drawn in plane (location=(5.4, 4.5)) - probability\_note: a Panel that says "A forest probability is an average of leaf-level estimates. It may rank well without being calibrated well." - simple\_boundary: a Line \[green\] drawn in plane (start=(0.0, 4.4), end=(3.0, 4.4)) - simple\_boundary\_2: a Line \[green\] drawn in plane (start=(3.0, 4.4), end=(3.0, 4.7)) - simple\_boundary\_3: a Line \[green\] drawn in plane (start=(3.0, 4.7), end=(5.5, 4.7)) - simple\_boundary\_4: a Line \[green\] drawn in plane (start=(5.5, 4.7), end=(5.5, 4.2)) - simple\_boundary\_5: a Line \[green\] drawn in plane (start=(5.5, 4.2), end=(8.0, 4.2)) - steps: a Block \[text\] that says "Define the prediction target, split strategy, and evaluation metric before fitting. Fit a small tree first; inspect depth, leaf sizes, impurity gains, and obvious leakage. Tune structural regularization with validation data, not training p…" - train\_points: a Point \[blue\] drawn in plane (location=(1.0, 1.2)) - train\_points\_10: a Point \[red\] drawn in plane (location=(7.0, 6.2)) - train\_points\_2: a Point \[blue\] drawn in plane (location=(2.0, 2.4)) - train\_points\_3: a Point \[blue\] drawn in plane (location=(3.4, 1.7)) - train\_points\_4: a Point \[blue\] drawn in plane (location=(4.8, 2.6)) - train\_points\_5: a Point \[blue\] drawn in plane (location=(6.2, 1.8)) - train\_points\_6: a Point \[red\] drawn in plane (location=(1.2, 6.1)) - train\_points\_7: a Point \[red\] drawn in plane (location=(2.5, 5.4)) - train\_points\_8: a Point \[red\] drawn in plane (location=(4.1, 6.5)) - train\_points\_9: a Point \[red\] drawn in plane (location=(5.8, 5.2)) - tree\_controls: a Table \[text\] that says "Tree control Effect \`max\_depth\`, \`max\_leaf\_nodes\` Global complexity \`min\_samples\_leaf\` Small-region stability \`min\_impurity\_decrease\` Required local gain \`ccp\_alpha\` Cost-complexity subtree" (rows=(('Tree control', 'Effect'), ('\`max\_depth\`, \`max\_leaf\_nodes\`', …, header=True) #### Beats ##### [15:44.665](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=944.6647083333332) Narration: Here is a practical sequence that preserves the logic we have developed. Start with the prediction problem, not the estimator. Fix the target, evaluation unit, split strategy, and metric before fitting. Board: Empty. Actions: - [15:44.665](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=944.6647083333332): heading\_workflow is shown on the screen, written out. - [15:44.665](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=944.6647083333332): plane is shown on the screen, written out. - [15:44.665](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=944.6647083333332): train\_points is shown on the screen, written out. - [15:44.715](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=944.7147083333332): train\_points\_2 is shown on the screen, written out. - [15:44.765](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=944.7647083333333): train\_points\_3 is shown on the screen, written out. - [15:44.815](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=944.8147083333332): train\_points\_4 is shown on the screen, written out. - [15:44.865](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=944.8647083333333): train\_points\_5 is shown on the screen, written out. - [15:44.915](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=944.9147083333332): train\_points\_6 is shown on the screen, written out. - [15:44.965](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=944.9647083333332): train\_points\_7 is shown on the screen, written out. - [15:45.015](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=945.0147083333333): train\_points\_8 is shown on the screen, written out. - [15:45.065](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=945.0647083333332): train\_points\_9 is shown on the screen, written out. - [15:45.115](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=945.1147083333333): train\_points\_10 is shown on the screen, written out. - [15:45.722](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=945.7217083333333): steps is shown on the screen, written out. - [15:53.442](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=953.4417083333333): steps (the "Define the prediction target" part) is emphasized. - [15:58.434](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=958.4342083333332): steps (the "Define the prediction target" part) is no longer emphasized. ##### [15:59.034](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=959.0342083333333) Narration: A random row split is wrong whenever rows from the same customer, patient, device, location, or future time can leak information across folds. Group and temporal structure belong in the validation design. Board: steps — a Block \[text\] that says "Define the prediction target, split strategy, and evaluation metric before fitting. Fit a small tree first; inspect depth, leaf sizes, impurity gains, and obvious leakage. Tune structural regularization with validation data, not training p…"; plane — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); heading\_workflow — a Heading that says "A Practical Fitting Sequence"; train\_points — a Point \[blue\] drawn in plane (location=(1.0, 1.2)); train\_points\_2 — a Point \[blue\] drawn in plane (location=(2.0, 2.4)); train\_points\_3 — a Point \[blue\] drawn in plane (location=(3.4, 1.7)); train\_points\_4 — a Point \[blue\] drawn in plane (location=(4.8, 2.6)); train\_points\_5 — a Point \[blue\] drawn in plane (location=(6.2, 1.8)); train\_points\_6 — a Point \[red\] drawn in plane (location=(1.2, 6.1)); train\_points\_7 — a Point \[red\] drawn in plane (location=(2.5, 5.4)); train\_points\_8 — a Point \[red\] drawn in plane (location=(4.1, 6.5)); train\_points\_9 — a Point \[red\] drawn in plane (location=(5.8, 5.2)); train\_points\_10 — a Point \[red\] drawn in plane (location=(7.0, 6.2)) Actions: - [16:0.381](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=960.3807083333332): steps (the "split strategy" part) is emphasized. - [16:13.199](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=973.1987083333332): steps (the "split strategy" part) is no longer emphasized. ##### [16:13.799](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=973.7987083333333) Narration: Fit a small decision tree before the forest. Its splits expose coding mistakes, target proxies, implausible thresholds, and feature interactions that a large ensemble can conceal behind a good aggregate score. Board: Unchanged from the preceding beat in this scene. Actions: - [16:14.426](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=974.4257083333332): steps (the "Fit a small tree first" part) is emphasized. - [16:17.387](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=977.3867083333332): simple\_boundary is shown on the screen, written out. - [16:17.467](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=977.4667083333333): simple\_boundary\_2 is shown on the screen, written out. - [16:17.547](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=977.5467083333332): simple\_boundary\_3 is shown on the screen, written out. - [16:17.627](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=977.6267083333332): simple\_boundary\_4 is shown on the screen, written out. - [16:17.707](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=977.7067083333333): simple\_boundary\_5 is shown on the screen, written out. - [16:27.371](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=987.3712083333332): steps (the "Fit a small tree first" part) is no longer emphasized. ##### [16:27.971](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=987.9712083333333) Narration: Then tune tree structure against validation performance. Maximum depth limits path length. Minimum samples per leaf demands evidence in each terminal region. Cost-complexity alpha removes weak fitted branches. Board: steps — a Block \[text\] that says "Define the prediction target, split strategy, and evaluation metric before fitting. Fit a small tree first; inspect depth, leaf sizes, impurity gains, and obvious leakage. Tune structural regularization with validation data, not training p…"; plane — an Axes (x\_range=(0.0, 8.0), y\_range=(0.0, 8.0), aspect=(1.0, 1.0)); heading\_workflow — a Heading that says "A Practical Fitting Sequence"; train\_points — a Point \[blue\] drawn in plane (location=(1.0, 1.2)); train\_points\_2 — a Point \[blue\] drawn in plane (location=(2.0, 2.4)); train\_points\_3 — a Point \[blue\] drawn in plane (location=(3.4, 1.7)); train\_points\_4 — a Point \[blue\] drawn in plane (location=(4.8, 2.6)); train\_points\_5 — a Point \[blue\] drawn in plane (location=(6.2, 1.8)); train\_points\_6 — a Point \[red\] drawn in plane (location=(1.2, 6.1)); train\_points\_7 — a Point \[red\] drawn in plane (location=(2.5, 5.4)); train\_points\_8 — a Point \[red\] drawn in plane (location=(4.1, 6.5)); train\_points\_9 — a Point \[red\] drawn in plane (location=(5.8, 5.2)); train\_points\_10 — a Point \[red\] drawn in plane (location=(7.0, 6.2)); simple\_boundary — a Line \[green\] drawn in plane (start=(0.0, 4.4), end=(3.0, 4.4)); simple\_boundary\_2 — a Line \[green\] drawn in plane (start=(3.0, 4.4), end=(3.0, 4.7)); simple\_boundary\_3 — a Line \[green\] drawn in plane (start=(3.0, 4.7), end=(5.5, 4.7)); simple\_boundary\_4 — a Line \[green\] drawn in plane (start=(5.5, 4.7), end=(5.5, 4.2)); simple\_boundary\_5 — a Line \[green\] drawn in plane (start=(5.5, 4.2), end=(8.0, 4.2)) Actions: - [16:28.761](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=988.7607083333332): steps (the "Tune structural regularization" part) is emphasized. - [16:38.687](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=998.6867083333333): point is shown on the screen, grown. - [16:40.687](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1000.6867083333333): point is hidden from the screen. - [16:43.981](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1003.9812083333333): steps (the "Tune structural regularization" part) is no longer emphasized. ##### [16:44.581](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1004.5812083333333) Narration: For a forest, raise the number of estimators until the validation metric and predictions stabilize. More trees usually increase compute rather than overfitting in the familiar single-tree sense, but returns become negligible. Board: Unchanged from the preceding beat in this scene. Actions: - [16:45.998](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1005.9977083333332): steps (the "increase tree count" part) is emphasized. - [16:58.49](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1018.4897083333333): steps (the "increase tree count" part) is no longer emphasized. ##### [16:59.09](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1019.0897083333332) Narration: Finally inspect more than one aggregate score. Check important subgroups, threshold-sensitive decisions, probability calibration, drift, and the cost of the errors the model actually makes. Board: Unchanged from the preceding beat in this scene. Actions: - [17:0.135](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1020.1347083333333): steps (the "Inspect subgroup errors" part) is emphasized. - [17:11.756](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1031.7562083333332): heading\_workflow is hidden from the screen — left the board. - [17:11.756](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1031.7562083333332): plane is hidden from the screen — left the board. - [17:11.756](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1031.7562083333332): train\_points is hidden from the screen — plane left the board. - [17:11.756](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1031.7562083333332): train\_points\_2 is hidden from the screen — plane left the board. - [17:11.756](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1031.7562083333332): train\_points\_3 is hidden from the screen — plane left the board. - [17:11.756](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1031.7562083333332): train\_points\_4 is hidden from the screen — plane left the board. - [17:11.756](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1031.7562083333332): train\_points\_5 is hidden from the screen — plane left the board. - [17:11.756](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1031.7562083333332): train\_points\_6 is hidden from the screen — plane left the board. - [17:11.756](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1031.7562083333332): train\_points\_7 is hidden from the screen — plane left the board. - [17:11.756](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1031.7562083333332): train\_points\_8 is hidden from the screen — plane left the board. - [17:11.756](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1031.7562083333332): train\_points\_9 is hidden from the screen — plane left the board. - [17:11.756](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1031.7562083333332): train\_points\_10 is hidden from the screen — plane left the board. - [17:11.756](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1031.7562083333332): simple\_boundary is hidden from the screen — plane left the board. - [17:11.756](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1031.7562083333332): simple\_boundary\_2 is hidden from the screen — plane left the board. - [17:11.756](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1031.7562083333332): simple\_boundary\_3 is hidden from the screen — plane left the board. - [17:11.756](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1031.7562083333332): simple\_boundary\_4 is hidden from the screen — plane left the board. - [17:11.756](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1031.7562083333332): simple\_boundary\_5 is hidden from the screen — plane left the board. - [17:11.756](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1031.7562083333332): steps is hidden from the screen — left the board. - [17:11.756](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1031.7562083333332): steps (the "Inspect subgroup errors" part) is no longer emphasized. ##### [17:12.356](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1032.3562083333331) Narration: Maximum depth and maximum leaf nodes limit global tree size. They are coarse controls on how many successive rectangles the tree may create. Board: Empty. Actions: - [17:12.356](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1032.3562083333331): heading\_controls is shown on the screen, written out. - [17:12.705](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1032.7047083333332): tree\_controls is shown on the screen, written out. - [17:12.705](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1032.7047083333332): tree\_controls is shown on the screen, written out. - [17:12.705](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1032.7047083333332): tree\_controls (the "\`max\_depth\`, \`max\_leaf\_nodes\`" part) is indicated — a transient flash. ##### [17:22.256](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1042.2557083333331) Narration: Minimum samples per leaf directly attacks tiny regions. Minimum impurity decrease requires a split to earn enough local gain before it is allowed. Board: heading\_controls — a Heading that says "Translate the Picture into Parameters" Actions: - [17:22.604](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1042.6037083333333): tree\_controls is shown on the screen, written out. - [17:24.915](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1044.9147083333332): tree\_controls (the "\`min\_samples\_leaf\`" part) is indicated — a transient flash. - [17:26.83](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1046.8297083333332): tree\_controls is shown on the screen, written out. ##### [17:32.747](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1052.7472083333332) Narration: Cost-complexity alpha names the post-pruning penalty we used earlier. Its numerical scale depends on the data, weights, impurity, and implementation, so tune it through the supplied pruning path rather than by folklore. Board: Unchanged from the preceding beat in this scene. Actions: - [17:34.187](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1054.1867083333332): tree\_controls is shown on the screen, written out. - [17:36.01](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1056.0097083333333): tree\_controls (the "\`ccp\_alpha\`" part) is indicated — a transient flash. ##### [17:48.417](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1068.4172083333333) Narration: Bootstrap turns observation resampling on or off. Max features controls the random candidate subset at each split. Lowering max features usually reduces correlation, but can also weaken individual trees. Board: Unchanged from the preceding beat in this scene. Actions: - [17:48.766](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1068.7657083333331): forest\_controls is shown on the screen, written out. - [17:48.766](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1068.7657083333331): forest\_controls is shown on the screen, written out. - [17:52.887](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1072.8867083333332): forest\_controls is shown on the screen, written out. - [17:59.632](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1079.6317083333333): forest\_controls (the "\`max\_features\`" part) is indicated — a transient flash. ##### [18:3.645](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1083.6452083333334) Narration: The number of estimators is the ensemble size B. Parallel-job settings change wall-clock cost, not the fitted statistical objective. Do not confuse faster execution with stronger regularization. Board: Unchanged from the preceding beat in this scene. Actions: - [18:4.156](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1084.1557083333332): forest\_controls is shown on the screen, written out. - [18:7.57](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1087.5697083333332): forest\_controls is shown on the screen, written out. - [18:10.658](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1090.6577083333332): forest\_controls (the "Compute, not statistics" part) is indicated — a transient flash. - [18:17.542](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1097.5422083333333): forest\_controls is hidden from the screen — left the board. - [18:17.542](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1097.5422083333333): heading\_controls is hidden from the screen — left the board. - [18:17.542](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1097.5422083333333): tree\_controls is hidden from the screen — left the board. ##### [18:18.142](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1098.1422083333332) Narration: Classification probabilities deserve separate validation. A forest averages leaf class proportions, which can rank cases very effectively while remaining overconfident or underconfident as probabilities. Board: Empty. Actions: - [18:18.142](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1098.1422083333332): heading\_diagnostics is shown on the screen, written out. - [18:19.315](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1099.3147083333333): probability\_note is shown on the screen, written out. - [18:27.604](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1107.6037083333333): probability\_note (the "calibrated" part) is indicated — a transient flash. ##### [18:31.803](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1111.8032083333333) Narration: Out-of-bag predictions use, for each training row, only trees whose bootstrap samples omitted that row. They provide a convenient internal diagnostic, but they do not override grouped, temporal, or external validation requirements. Board: probability\_note — a Panel that says "A forest probability is an average of leaf-level estimates. It may rank well without being calibrated well."; heading\_diagnostics — a Heading that says "Diagnostics with Precise Meanings" Actions: - [18:32.21](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1112.2097083333333): diagnostics is shown on the screen, written out. - [18:32.21](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1112.2097083333333): diagnostics is shown on the screen, written out. - [18:37.167](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1117.1667083333332): diagnostics (the "omitted each row" part) is indicated — a transient flash. ##### [18:47.392](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1127.3917083333333) Narration: Impurity-based feature importance summarizes how fitted splits used a feature. It can favour variables with many available thresholds and divide credit awkwardly among correlated predictors. It is not a causal effect. Board: Unchanged from the preceding beat in this scene. Actions: - [18:48.715](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1128.7147083333334): diagnostics is shown on the screen, written out. - [19:0.105](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1140.1047083333333): diagnostics (the "not a causal effect" part) is indicated — a transient flash. ##### [19:2.284](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1142.2837083333334) Narration: Permutation importance asks how predictive performance changes after one feature is disrupted. It is often closer to the operational question, but correlated features can substitute for one another and hide each other's importance. Board: Unchanged from the preceding beat in this scene. Actions: - [19:2.69](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1142.6897083333333): diagnostics is shown on the screen, written out. - [19:4.861](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1144.8607083333332): diagnostics (the "Loss" part) is indicated — a transient flash. ##### [19:16.874](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1156.8737083333333) Narration: Class weights alter the fitting objective. They do not decide which deployment metric matters. A random seed makes the stochastic fit reproducible; it does not make sampling uncertainty disappear. Board: Unchanged from the preceding beat in this scene. Actions: - [19:17.176](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1157.1757083333332): diagnostics is shown on the screen, written out. - [19:22.888](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1162.8877083333332): diagnostics is shown on the screen, written out. - [19:29.029](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1169.0292083333334): diagnostics is hidden from the screen — left the board. - [19:29.029](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1169.0292083333334): heading\_diagnostics is hidden from the screen — left the board. - [19:29.029](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1169.0292083333334): probability\_note is hidden from the screen — left the board. ##### [19:30.229](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1170.2292083333332) Narration: The whole lecture can be compressed into three ingredients. Trees must be strong enough to capture useful structure. Their errors must be sufficiently decorrelated. Averaging then reduces the unshared variance. Board: Empty. Actions: - [19:30.229](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1170.2292083333332): heading\_close is shown on the screen, written out. - [19:32.366](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1172.3657083333333): final\_law is shown on the screen, written out. - [19:34.711](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1174.7107083333333): final\_law (the "upright("strong trees")" part) is indicated — a transient flash. - [19:38.589](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1178.5887083333332): final\_law (the "upright("low error correlation")" part) is indicated — a transient flash. - [19:40.04](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1180.0397083333332): final\_law (the "upright("averaging")" part) is indicated — a transient flash. ##### [19:43.728](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1183.7277083333333) Narration: One tree turns impurity reduction into an interpretable hierarchy of rectangles. Pruning makes that hierarchy less fragile. A random forest keeps many flexible trees, makes their mistakes less alike, and averages what remains. Board: final\_law — a Math \[text\] that says "$upright("strong trees") + upright("low error correlation") + upright("averaging")$"; heading\_close — a Heading that says "What to Carry Away" Actions: - [19:57.103](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1197.1027083333333): A box is drawn around final\_law. - [19:59.15](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1199.1497708333334): final\_law is hidden from the screen — left the board. - [19:59.15](https://academa.ai/lectures/decision-trees-and-why-forests-beat-them?t=1199.1497708333334): heading\_close is hidden from the screen — left the board.