WEBVTT 00:00:00.070 --> 00:00:03.077 Here is the entire training problem in miniature. 00:00:03.390 --> 00:00:05.838 Each dot is one labelled observation, 00:00:05.921 --> 00:00:10.496 blue or red, and each position contains two measured features. 00:00:10.693 --> 00:00:13.863 We want a rule that predicts the colour of a new point. 00:00:14.567 --> 00:00:18.419 A decision tree will not draw a diagonal or fit a smooth curve. 00:00:18.502 --> 00:00:24.064 At one node it chooses one feature, one threshold, and one yes-or-no question. 00:00:24.168 --> 00:00:28.128 That question cuts the current region with an axis-aligned line. 00:00:28.739 --> 00:00:31.038 But which question should it ask first? 00:00:31.350 --> 00:00:36.123 The usual classification answer is the split that reduces impurity most. 00:00:36.319 --> 00:00:41.858 Impurity is not model error. It measures how mixed the labels are inside a node. 00:00:42.655 --> 00:00:46.937 For Gini impurity, subtract the squared class proportions from one. 00:00:47.020 --> 00:00:52.512 At the root we have eight blue and eight red, so both proportions are one half. 00:00:52.709 --> 00:00:57.934 The root impurity is zero point five, its largest possible value for two classes. 00:00:58.731 --> 00:01:02.888 The fitting code now considers thresholds between observed values. 00:01:02.992 --> 00:01:05.710 Let us compare two representative candidates. 00:01:05.906 --> 00:01:10.609 A vertical cut at x one equals four leaves both children evenly mixed. 00:01:11.313 --> 00:01:14.982 Its weighted child impurity is still zero point five. 00:01:15.179 --> 00:01:18.869 Subtract that from the root impurity and the gain is zero. 00:01:18.952 --> 00:01:21.354 The cut changed the addresses of the points, 00:01:21.437 --> 00:01:23.713 but learned nothing about their labels. 00:01:24.765 --> 00:01:27.367 Now try x two less than four. 00:01:27.471 --> 00:01:30.290 Below the line are six blue and two red. 00:01:30.373 --> 00:01:33.276 Above it are two blue and six red. 00:01:33.380 --> 00:01:36.118 Each child has Gini impurity three eighths, 00:01:36.201 --> 00:01:38.989 and their weighted average is also three eighths. 00:01:39.901 --> 00:01:43.743 The reduction is zero point five minus zero point three seven five, 00:01:43.826 --> 00:01:46.206 which is zero point one two five. 00:01:46.403 --> 00:01:49.118 That is larger than the vertical candidate's zero, 00:01:49.201 --> 00:01:51.570 so this horizontal question wins. 00:01:52.367 --> 00:01:53.702 The first branch of the tree 00:01:53.702 --> 00:01:58.068 and the first pair of rectangles are the same decision written in two languages. 00:01:58.264 --> 00:01:59.459 The root asks 00:01:59.542 --> 00:02:01.330 x two less than four. 00:02:01.527 --> 00:02:03.812 The left child receives the lower rectangle, 00:02:03.895 --> 00:02:06.462 and the right child receives the upper one. 00:02:07.375 --> 00:02:09.303 Notice what has not happened. 00:02:09.500 --> 00:02:13.399 We have not classified everything correctly, and neither child is pure. 00:02:13.482 --> 00:02:15.909 We have only made the labels less mixed. 00:02:16.013 --> 00:02:20.495 Training a tree means repeating this exact competition inside each child. 00:02:21.407 --> 00:02:24.821 That is the basic fitting loop behind the library call. 00:02:25.018 --> 00:02:29.558 Enumerate legal feature thresholds, score their weighted impurity, choose the best 00:02:29.558 --> 00:02:34.098 gain, partition the observations, and repeat on the resulting nodes. 00:02:35.173 --> 00:02:37.241 Continue from the root split. 00:02:37.437 --> 00:02:41.223 The tree on the left and feature space on the right will grow together. 00:02:41.419 --> 00:02:44.322 A new internal node must always correspond 00:02:44.322 --> 00:02:49.478 to a new axis-aligned cut inside exactly one existing rectangle. 00:02:50.390 --> 00:02:52.446 Start in the lower rectangle. 00:02:52.550 --> 00:02:56.393 It contains six blue observations and two red ones. 00:02:56.590 --> 00:03:00.373 The best available gain first separates the far-right red point 00:03:00.456 --> 00:03:03.487 with x one less than six point one. 00:03:04.284 --> 00:03:08.348 The right child is pure, but the large left child is not. 00:03:08.545 --> 00:03:11.424 Within that child, a horizontal cut at x 00:03:11.424 --> 00:03:17.253 two equals two point five creates a pure blue strip below and a mixed strip above. 00:03:18.050 --> 00:03:21.961 That upper strip is mixed because of one red observation at two point two, 00:03:22.044 --> 00:03:23.414 two point eight. 00:03:23.611 --> 00:03:26.421 Another vertical threshold isolates it. 00:03:26.618 --> 00:03:28.601 The training algorithm is rewarded, 00:03:28.684 --> 00:03:31.808 because two new leaves become perfectly pure. 00:03:32.721 --> 00:03:35.380 Now do the same work above the root. 00:03:35.484 --> 00:03:37.073 Most points there are red, 00:03:37.156 --> 00:03:40.698 but two blue exceptions force the recursion to continue. 00:03:40.894 --> 00:03:45.075 The first upper split separates the far-right pair from the rest. 00:03:45.871 --> 00:03:48.682 The far-right pair still disagrees. 00:03:48.878 --> 00:03:53.987 A horizontal cut isolates the blue point at the top from the red point below. 00:03:54.184 --> 00:03:58.863 Again, training impurity falls to zero in both resulting leaves. 00:03:59.660 --> 00:04:02.935 The left upper region has its own blue exception. 00:04:03.131 --> 00:04:07.961 A split at x one equals two point eight narrows the search, and another 00:04:07.961 --> 00:04:13.140 horizontal split at x two equals five point one separates one red point. 00:04:14.053 --> 00:04:15.225 One final threshold 00:04:15.225 --> 00:04:20.415 at x one equals one point five isolates the remaining upper blue point. 00:04:20.612 --> 00:04:24.200 Every terminal region now contains only one class. 00:04:24.397 --> 00:04:27.776 Consequently every leaf has Gini impurity zero. 00:04:28.573 --> 00:04:31.348 On the training set, this looks perfect. 00:04:31.452 --> 00:04:34.181 Every observation is classified correctly. 00:04:34.378 --> 00:04:37.409 But look at the geometry required to achieve it: 00:04:37.605 --> 00:04:39.879 thin strips, short corridors, 00:04:39.962 --> 00:04:44.235 and thresholds whose only purpose is to rescue one exceptional dot. 00:04:45.450 --> 00:04:46.831 A new point can cross one 00:04:46.831 --> 00:04:51.566 of those arbitrary thresholds after an imperceptible change in a feature. 00:04:51.649 --> 00:04:54.378 Its predicted class then jumps, even 00:04:54.378 --> 00:04:58.732 though the training labels gave us almost no evidence that such a jump should exist. 00:04:59.784 --> 00:05:02.315 This is the characteristic strength and weakness 00:05:02.315 --> 00:05:04.731 of an unconstrained decision tree. 00:05:05.345 --> 00:05:09.932 It has low bias because it can represent complicated interactions. 00:05:10.129 --> 00:05:11.243 It also has high 00:05:11.243 --> 00:05:16.097 variance because a few observations can rearrange entire branches and rectangles. 00:05:16.894 --> 00:05:21.910 Pure leaves are therefore a training condition, not evidence of a useful model. 00:05:22.223 --> 00:05:26.343 The next question is whether every branch earns its complexity on data 00:05:26.426 --> 00:05:28.470 that did not choose the branch. 00:05:29.585 --> 00:05:33.857 The fully grown tree has zero training impurity, but it pays for that fit 00:05:33.857 --> 00:05:39.082 with nine leaves and several thresholds supported by a single unusual observation. 00:05:39.697 --> 00:05:43.192 Pruning asks whether those extra leaves earn their keep. 00:05:44.244 --> 00:05:48.146 Cost-complexity pruning gives the trade a precise form. 00:05:48.342 --> 00:05:51.977 R of T measures the fitted tree's error or impurity. 00:05:52.174 --> 00:05:55.622 The second term charges alpha for every terminal leaf. 00:05:56.326 --> 00:06:00.797 At alpha zero, extra leaves are free, so the pure tree wins. 00:06:00.993 --> 00:06:05.463 As alpha increases, a weak pair of leaves can cost more than the small reduction 00:06:05.463 --> 00:06:07.554 in training error that created it. 00:06:08.350 --> 00:06:11.021 Prune the weakest lower twigs first. 00:06:11.125 --> 00:06:12.808 Their narrow thresholds disappear 00:06:12.808 --> 00:06:17.197 from feature space at the same moment their branches disappear from the tree. 00:06:17.394 --> 00:06:20.773 The replacement leaf predicts the local majority class. 00:06:21.570 --> 00:06:24.531 Now prune the corresponding upper twigs. 00:06:24.728 --> 00:06:28.339 Again, the replacement is not pure on the training set. 00:06:28.443 --> 00:06:32.495 It deliberately accepts a few mistakes in exchange for a much larger 00:06:32.495 --> 00:06:34.713 and more stable prediction region. 00:06:35.510 --> 00:06:39.388 The remaining four-leaf tree still captures the broad interaction. 00:06:39.585 --> 00:06:43.347 The horizontal root separates low from high x two. 00:06:43.544 --> 00:06:44.647 Within each half, 00:06:44.647 --> 00:06:49.420 one x one threshold handles the main exception near the far-right edge. 00:06:50.216 --> 00:06:53.766 This smaller tree has higher training error by construction. 00:06:53.850 --> 00:06:57.554 The relevant question is whether it has lower error on observations 00:06:57.554 --> 00:07:01.246 that did not participate in choosing all those thresholds. 00:07:02.643 --> 00:07:06.370 A pruning path supplies a nested sequence of subtrees. 00:07:06.567 --> 00:07:10.059 Here the unpruned tree has nine leaves and no training mistakes, 00:07:10.143 --> 00:07:13.534 but its validation error is the worst entry in the table. 00:07:14.330 --> 00:07:16.944 A small penalty removes three leaves. 00:07:17.257 --> 00:07:20.856 Training error rises, validation error falls. 00:07:21.053 --> 00:07:23.697 A larger penalty leaves four terminal regions, 00:07:23.781 --> 00:07:26.534 and the validation error reaches its minimum. 00:07:27.446 --> 00:07:31.255 Push alpha farther and the tree collapses to two leaves. 00:07:31.452 --> 00:07:36.573 That model is now too simple for the data, so validation error rises again. 00:07:36.677 --> 00:07:39.986 Pruning is not a ritual of making trees small. 00:07:40.090 --> 00:07:43.910 It is model selection along a structured complexity path. 00:07:44.707 --> 00:07:47.224 Select alpha using cross-validation, 00:07:47.308 --> 00:07:52.625 a held-out set, or a nested procedure when tuning itself must be evaluated. 00:07:52.822 --> 00:07:54.401 Never choose the pruning strength 00:07:54.401 --> 00:07:58.326 by returning to the same training impurity that rewarded every twig. 00:07:59.122 --> 00:08:03.299 Libraries also offer pre-pruning controls such as maximum depth, 00:08:03.383 --> 00:08:09.119 minimum samples per leaf, minimum impurity decrease, and maximum leaf count. 00:08:09.432 --> 00:08:11.244 Those prevent growth. 00:08:11.441 --> 00:08:16.074 Cost-complexity pruning fits first and removes branches afterward. 00:08:16.870 --> 00:08:20.052 Either route trades some bias for less variance. 00:08:20.156 --> 00:08:23.731 A single pruned tree is often much easier to explain 00:08:23.731 --> 00:08:26.193 and more stable than the pure tree. 00:08:26.645 --> 00:08:28.782 The forest will take a different route: 00:08:28.886 --> 00:08:33.009 keep many unstable trees, then make their instability cancel. 00:08:34.080 --> 00:08:37.305 A forest does not begin by cloning one fitted tree. 00:08:37.389 --> 00:08:40.048 It creates many related training problems. 00:08:40.152 --> 00:08:45.087 For each tree, draw n observations from the original n with replacement. 00:08:46.301 --> 00:08:50.621 With replacement means an observation can appear more than once. 00:08:50.818 --> 00:08:51.828 In the first sample, 00:08:51.828 --> 00:08:55.645 indices one and five are repeated, while three and six are absent. 00:08:55.729 --> 00:08:58.203 The sample still contains n rows. 00:08:58.628 --> 00:09:03.060 A second bootstrap draw repeats different observations and omits different ones. 00:09:03.144 --> 00:09:04.759 A third does it again. 00:09:04.955 --> 00:09:08.505 Each tree therefore sees a perturbed empirical distribution, 00:09:08.589 --> 00:09:11.377 even though all draws came from the same dataset. 00:09:12.289 --> 00:09:13.264 Fitting deep trees 00:09:13.264 --> 00:09:17.491 to these samples is ordinary bootstrap aggregation, or bagging. 00:09:17.595 --> 00:09:21.228 It creates diversity because a marginal observation may be duplicated 00:09:21.228 --> 00:09:23.865 in one sample and unavailable in another. 00:09:24.777 --> 00:09:28.238 A random forest adds a second source of variation. 00:09:28.342 --> 00:09:29.430 At every node, 00:09:29.514 --> 00:09:34.008 it offers the split search only a random subset of the available features. 00:09:35.405 --> 00:09:38.308 Here is one tree in an eight-feature problem. 00:09:38.412 --> 00:09:42.012 The formula records a fresh feature subset at node j. 00:09:42.208 --> 00:09:45.979 At the root, features two, five, and eight are offered, 00:09:46.063 --> 00:09:49.884 and feature two supplies the best available impurity reduction. 00:09:50.680 --> 00:09:55.058 The left child gets a fresh draw, features one, four, and eight. 00:09:55.255 --> 00:09:59.679 The right child gets another draw, features three, five, and seven. 00:09:59.875 --> 00:10:02.662 Feature availability is local to a node. 00:10:03.459 --> 00:10:06.753 This restriction can force a tree to ignore the dominant predictor 00:10:06.837 --> 00:10:08.379 at a particular node. 00:10:08.463 --> 00:10:10.635 That sounds inefficient for one tree. 00:10:10.831 --> 00:10:15.058 Its purpose is to prevent every tree from making the same early decisions. 00:10:15.855 --> 00:10:18.921 Bootstrap sampling perturbs the observations. 00:10:19.117 --> 00:10:22.020 Feature sampling perturbs the available questions. 00:10:22.333 --> 00:10:26.072 Together they produce trees whose errors are less synchronized. 00:10:27.469 --> 00:10:30.802 These are three trees trained from the same original problem. 00:10:30.998 --> 00:10:35.363 Each diagram shows the axis-aligned boundaries produced by one bootstrap 00:10:35.363 --> 00:10:38.952 sample and one sequence of random feature offers. 00:10:40.004 --> 00:10:44.857 The first tree chooses a horizontal root and then several local vertical cuts. 00:10:45.054 --> 00:10:49.594 Its boundary is coherent within each rectangle, but jagged as a whole. 00:10:50.391 --> 00:10:54.826 The second tree begins vertically because its sample and offered features differ. 00:10:55.023 --> 00:10:57.005 It partitions the same feature plane 00:10:57.089 --> 00:10:59.946 into a visibly different collection of rectangles. 00:11:00.743 --> 00:11:03.376 The third tree returns to a horizontal root, 00:11:03.460 --> 00:11:06.700 but its thresholds and deeper branches are different again. 00:11:06.897 --> 00:11:09.881 None of these trees is intended to be the final boundary. 00:11:10.678 --> 00:11:15.191 For classification, each tree supplies class probabilities from its reached leaf, 00:11:15.275 --> 00:11:18.817 commonly the class proportions among that leaf's training samples. 00:11:19.014 --> 00:11:22.765 The forest averages those probabilities and then chooses a class. 00:11:23.561 --> 00:11:27.707 For regression, the same architecture averages numeric predictions. 00:11:27.904 --> 00:11:32.676 Bootstrap samples and random feature subsets still serve the same purpose: 00:11:32.873 --> 00:11:36.891 build individually flexible models whose mistakes are not identical. 00:11:37.989 --> 00:11:39.882 Return to the labelled plane. 00:11:40.496 --> 00:11:44.386 One deep tree supplies one jagged boundary. It reacts strongly 00:11:44.386 --> 00:11:48.624 to the particular observations and random choices that shaped its branches. 00:11:49.328 --> 00:11:53.880 For each query point x, tree b supplies a class probability. 00:11:53.984 --> 00:11:56.864 The forest averages those probabilities. 00:11:57.060 --> 00:12:00.974 Its displayed boundary is where the winning averaged class changes. 00:12:01.770 --> 00:12:04.499 Add trees until B equals ten. 00:12:04.696 --> 00:12:08.154 Each individual boundary is still made of rectangular steps, 00:12:08.237 --> 00:12:12.081 but their idiosyncratic notches occur at different places. 00:12:12.277 --> 00:12:16.458 Majority support removes many notches that only one tree wanted. 00:12:17.254 --> 00:12:19.647 Now average one hundred trees. 00:12:19.750 --> 00:12:23.092 The aggregate is still a staircase if we inspect it finely enough, 00:12:23.175 --> 00:12:25.672 because every contributor is a tree. 00:12:25.869 --> 00:12:27.377 At this scale, however, 00:12:27.460 --> 00:12:31.977 it follows the broad class structure rather than every isolated observation. 00:12:33.191 --> 00:12:36.198 Calling this boundary smoother does not mean that a forest fits 00:12:36.198 --> 00:12:38.289 a smooth analytic function. 00:12:38.485 --> 00:12:42.373 It means the averaged prediction varies more stably across nearby points 00:12:42.456 --> 00:12:47.007 and fewer decisions depend on one tree's narrow rectangular accident. 00:12:47.804 --> 00:12:50.835 Why should averaging unstable models work? 00:12:51.032 --> 00:12:56.246 Imagine each tree prediction as a useful signal plus a zero-mean fitting error. 00:12:56.442 --> 00:12:58.823 Averaging keeps the shared signal. 00:12:58.927 --> 00:13:02.248 Errors that point in different directions can cancel. 00:13:03.045 --> 00:13:07.315 If the tree errors were independent and each had variance sigma squared, 00:13:07.398 --> 00:13:11.172 the mean of B trees would have variance sigma squared over B. 00:13:11.276 --> 00:13:14.911 Double the number of trees and this variance contribution halves. 00:13:16.125 --> 00:13:18.596 Real tree errors are not independent. 00:13:18.679 --> 00:13:20.744 They use the same original dataset, 00:13:20.827 --> 00:13:24.369 and strong predictors can make them discover similar branches. 00:13:24.566 --> 00:13:28.653 Let rho represent their average pairwise error correlation. 00:13:30.166 --> 00:13:32.164 Here is the extreme problem. 00:13:32.360 --> 00:13:37.191 Five trees all make an error of plus one on the same case. 00:13:37.387 --> 00:13:39.731 Their average error is still plus one. 00:13:39.814 --> 00:13:42.589 Repetition did not remove a shared mistake. 00:13:43.386 --> 00:13:46.997 On the right, errors differ across trees. 00:13:47.194 --> 00:13:49.723 Positive and negative errors offset, 00:13:49.806 --> 00:13:53.406 and their average is zero in this small illustration. 00:13:53.602 --> 00:13:54.903 Diversity is useful 00:13:54.903 --> 00:13:59.710 when it concerns errors, not merely visual differences between diagrams. 00:14:00.507 --> 00:14:04.058 With equal variance and average correlation rho, 00:14:04.141 --> 00:14:06.626 the ensemble variance is sigma squared 00:14:07.380 --> 00:14:11.641 times rho plus one minus rho over B. 00:14:11.815 --> 00:14:14.727 The second part shrinks as trees are added. 00:14:14.810 --> 00:14:17.017 The correlated part does not. 00:14:17.535 --> 00:14:20.403 Take the number of trees toward infinity. 00:14:20.565 --> 00:14:24.513 The variance approaches rho sigma squared, not zero. 00:14:24.826 --> 00:14:26.635 If rho is close to one, 00:14:26.718 --> 00:14:31.166 a huge forest behaves like repeated copies of one unstable tree. 00:14:31.962 --> 00:14:34.619 This is why decorrelation is essential. 00:14:34.702 --> 00:14:38.743 Bootstrap samples alter which observations drive the branches. 00:14:38.940 --> 00:14:40.577 Random feature subsets prevent 00:14:40.577 --> 00:14:45.906 one dominant predictor from forcing the same root and early splits in every tree. 00:14:46.703 --> 00:14:48.329 There is a trade-off. 00:14:48.433 --> 00:14:51.670 Offering fewer features can weaken each individual tree 00:14:51.753 --> 00:14:54.192 by denying it useful predictors. 00:14:54.807 --> 00:14:58.764 But if that loss is modest and the correlation falls substantially, 00:14:58.847 --> 00:15:01.077 the average can generalize better. 00:15:01.873 --> 00:15:06.321 Adding trees mainly reduces Monte Carlo noise in the fitted ensemble. 00:15:06.517 --> 00:15:11.277 It does not repair severe bias, leakage, bad labels, a shifted 00:15:11.277 --> 00:15:15.922 deployment population, or a feature set that contains no useful signal. 00:15:16.718 --> 00:15:19.819 The forest's advantage is therefore specific. 00:15:20.016 --> 00:15:24.347 Deep trees provide flexible, high-variance base predictions. 00:15:24.660 --> 00:15:29.142 Resampling and feature randomness make their errors less alike. 00:15:29.594 --> 00:15:33.473 Averaging then removes the part of the variance that is not shared. 00:15:34.515 --> 00:15:38.301 Here is a practical sequence that preserves the logic we have developed. 00:15:38.915 --> 00:15:42.480 Start with the prediction problem, not the estimator. 00:15:42.677 --> 00:15:48.390 Fix the target, evaluation unit, split strategy, and metric before fitting. 00:15:49.186 --> 00:15:54.060 A random row split is wrong whenever rows from the same customer, patient, 00:15:54.144 --> 00:15:59.880 device, location, or future time can leak information across folds. 00:16:00.494 --> 00:16:04.326 Group and temporal structure belong in the validation design. 00:16:05.123 --> 00:16:08.421 Fit a small decision tree before the forest. 00:16:08.734 --> 00:16:14.095 Its splits expose coding mistakes, target proxies, implausible thresholds, 00:16:14.179 --> 00:16:15.316 and feature interactions 00:16:15.316 --> 00:16:19.671 that a large ensemble can conceal behind a good aggregate score. 00:16:20.886 --> 00:16:24.508 Then tune tree structure against validation performance. 00:16:24.705 --> 00:16:26.482 Maximum depth limits 00:16:27.097 --> 00:16:28.467 path length. 00:16:28.664 --> 00:16:32.914 Minimum samples per leaf demands evidence in each terminal region. 00:16:33.111 --> 00:16:37.024 Cost-complexity alpha removes weak fitted branches. 00:16:37.821 --> 00:16:39.759 For a forest, raise the number 00:16:39.759 --> 00:16:43.905 of estimators until the validation metric and predictions stabilize. 00:16:44.102 --> 00:16:47.724 More trees usually increase compute rather than overfitting 00:16:47.724 --> 00:16:52.113 in the familiar single-tree sense, but returns become negligible. 00:16:53.328 --> 00:16:56.487 Finally inspect more than one aggregate score. 00:16:56.799 --> 00:17:01.478 Check important subgroups, threshold-sensitive decisions, probability 00:17:01.478 --> 00:17:06.762 calibration, drift, and the cost of the errors the model actually makes. 00:17:07.558 --> 00:17:12.203 Maximum depth and maximum leaf nodes limit global tree size. 00:17:12.400 --> 00:17:17.021 They are coarse controls on how many successive rectangles the tree may create. 00:17:17.818 --> 00:17:21.430 Minimum samples per leaf directly attacks tiny regions. 00:17:21.742 --> 00:17:25.643 Minimum impurity decrease requires a split to earn enough local gain 00:17:25.643 --> 00:17:27.246 before it is allowed. 00:17:28.461 --> 00:17:33.187 Cost-complexity alpha names the post-pruning penalty we used earlier. 00:17:33.383 --> 00:17:38.803 Its numerical scale depends on the data, weights, impurity, and implementation, 00:17:38.887 --> 00:17:43.044 so tune it through the supplied pruning path rather than by folklore. 00:17:43.840 --> 00:17:47.661 Bootstrap turns observation resampling on or off. 00:17:47.857 --> 00:17:51.956 Max features controls the random candidate subset at each split. 00:17:52.153 --> 00:17:55.261 Lowering max features usually reduces correlation, 00:17:55.345 --> 00:17:58.040 but can also weaken individual trees. 00:17:58.836 --> 00:18:01.957 The number of estimators is the ensemble size B. 00:18:02.041 --> 00:18:05.556 Parallel-job settings change wall-clock cost, 00:18:05.640 --> 00:18:08.067 not the fitted statistical objective. 00:18:08.519 --> 00:18:12.514 Do not confuse faster execution with stronger regularization. 00:18:13.310 --> 00:18:16.887 Classification probabilities deserve separate validation. 00:18:17.084 --> 00:18:21.925 A forest averages leaf class proportions, which can rank cases very effectively 00:18:21.925 --> 00:18:26.140 while remaining overconfident or underconfident as probabilities. 00:18:27.053 --> 00:18:28.907 Out-of-bag predictions use, 00:18:28.991 --> 00:18:33.938 for each training row, only trees whose bootstrap samples omitted that row. 00:18:34.251 --> 00:18:37.524 They provide a convenient internal diagnostic, but they 00:18:37.524 --> 00:18:42.703 do not override grouped, temporal, or external validation requirements. 00:18:43.616 --> 00:18:45.749 Impurity-based feature importance 00:18:45.833 --> 00:18:49.004 summarizes how fitted splits used a feature. 00:18:49.107 --> 00:18:50.953 It can favour variables with many 00:18:50.953 --> 00:18:55.865 available thresholds and divide credit awkwardly among correlated predictors. 00:18:55.969 --> 00:18:58.048 It is not a causal effect. 00:18:58.752 --> 00:19:00.632 Permutation importance asks how 00:19:00.632 --> 00:19:04.383 predictive performance changes after one feature is disrupted. 00:19:04.487 --> 00:19:08.632 It is often closer to the operational question, but correlated features 00:19:08.632 --> 00:19:12.302 can substitute for one another and hide each other's importance. 00:19:13.354 --> 00:19:16.222 Class weights alter the fitting objective. 00:19:16.326 --> 00:19:19.577 They do not decide which deployment metric matters. 00:19:19.774 --> 00:19:23.652 A random seed makes the stochastic fit reproducible; 00:19:23.849 --> 00:19:26.926 it does not make sampling uncertainty disappear. 00:19:28.137 --> 00:19:31.644 The whole lecture can be compressed into three ingredients. 00:19:31.748 --> 00:19:34.988 Trees must be strong enough to capture useful structure. 00:19:35.092 --> 00:19:38.204 Their errors must be sufficiently decorrelated. 00:19:38.308 --> 00:19:41.652 Averaging then reduces the unshared variance. 00:19:42.449 --> 00:19:47.535 One tree turns impurity reduction into an interpretable hierarchy of rectangles. 00:19:47.731 --> 00:19:50.681 Pruning makes that hierarchy less fragile. 00:19:50.785 --> 00:19:55.914 A random forest keeps many flexible trees, makes their mistakes less alike, 00:19:55.998 --> 00:19:58.216 and averages what remains.