{"version":1,"lectureId":"01M14TYW7K99A7C00QDVC7B6Y4","attempt":2,"publication":{"slug":"support-vector-machines-and-the-widest-corridor","title":"Support Vector Machines: Maximum-Margin Separation and the Kernel Trick","subject":"machine-learning","summary":"A geometric introduction to support vector machines, beginning with the widest separating corridor and deriving the hard-margin optimization problem from point-to-hyperplane distance. The lecture identifies support vectors as active constraints, extends the model to overlapping classes with slack variables and soft margins, then develops a concrete nonlinear example through an explicit quadratic feature lift. It concludes by deriving the kernel trick and showing how training and prediction depend only on kernel evaluations involving the support vectors.","metaDescription":"A visual introduction to maximum-margin classifiers, support vectors, soft margins, nonlinear feature maps, and the kernel trick.","transcript":"Suppose red and blue are two classes in a training set. We want a line that puts every red point on one side and every blue point on the other. This data is linearly separable, so at least one such line exists. In fact, one separating line is easy to draw. Every red point lies above it, and every blue point lies below it. But this line runs rather close to the blue class. Another line also separates the data, but now it passes close to the red class. A tilted line works too. Correct classification alone does not tell us which of these choices is preferable. Imagine placing an empty corridor around a separator. Its walls remain parallel to the line, and we widen them until one wall would hit red or the other would hit blue. The corridor's width measures how much room the classifier has before a training point changes sides. Now slide the centre line inside that gap. If it moves down, the lower clearance shrinks. If it moves up, the upper clearance shrinks. The best position balances the nearest points on the two sides and maximizes the smaller clearance. The points that stop the two walls are special. Three red points touch the upper wall, and two blue points touch the lower wall. These are the support vectors, because they physically support the widest corridor. Points farther away do not determine the final position. We could move one of those distant points slightly without changing either wall. Move a support vector, however, and the maximum-margin separator generally has to move with it. So an SVM does not merely search for zero training errors. Among all separating hyperplanes, it chooses the one with the largest geometric margin. Next we will turn this corridor into an optimization problem. A line in two dimensions, or a hyperplane in any dimension, can be written as w dot x plus b equals zero. The vector w is perpendicular to the separator, while b shifts it without changing its orientation. The expression w dot x plus b is a signed score. Positive scores lie on one side, negative scores lie on the other, and its sign becomes the predicted class. Let each training label y sub i be plus one for red and minus one for blue. Correct classification then has one compact condition: y sub i times the score must be positive. There is a scaling ambiguity. Multiplying both w and b by any positive number leaves the zero set unchanged, so it leaves the separating hyperplane unchanged. We use that freedom to make the closest signed scores equal to one. After this normalization, every red point must have score at least plus one, and every blue point must have score at most minus one. The support vectors are precisely the points where the combined inequality becomes an equality. The perpendicular distance from a point x to the central hyperplane is the absolute score divided by the length of w. Each wall has score one in magnitude, so each wall is one over the length of w from the centre. The full corridor therefore has width two over the length of w. To make that width large, we must make the length of w small. Squaring it changes neither the minimizing choice nor the geometry. We have reached the hard-margin optimization problem. Minimize one half w dot w, which is one half the squared length of the normal vector. The minimization is subject to one constraint per training example: y sub i times w dot x sub i plus b must be at least one. These inequalities say that every point lies outside the empty corridor on its proper side. Only a few of those constraints are tight at the solution. Those active constraints belong to the support vectors. The remaining points satisfy their inequalities with room to spare and do not pin the optimum. This completes the geometric derivation. A wide corridor became a small normal vector, correct classification became a family of inequalities, and the points touching equality became the support vectors. Real classes often overlap. A red example may sit deep among blue examples, while a blue example may sit among red ones. No line can put every point on its requested side in this training set. If we insist on the hard-margin constraints, the optimization problem has no feasible answer. The red point below and the blue point above cannot both satisfy the same perfect separation. A soft-margin SVM relaxes each constraint with a nonnegative slack variable xi sub i. Instead of demanding a signed score of at least one, it permits the score to fall short by xi sub i. Draw the central separator and its two margin walls. A point outside its correct wall needs zero slack. A point inside the corridor needs some slack, because it has not reached the wall required by its label. This red point is not merely inside the margin. It lies on the blue side of the separator, so its slack exceeds one. The same is true in reverse for this blue point. Slack records the degree of each violation rather than pretending the data is perfectly separable. The new objective contains two costs. One half w dot w still rewards a wide margin. The sum of the slacks penalizes points that enter the margin or cross the separator, and the constant C controls how heavily that penalty is counted. This is not permission to ignore errors for free. Every violation raises the objective. The optimizer accepts a violation only when avoiding it would require a sufficiently narrower or more contorted decision boundary. With a smaller C, slack is relatively inexpensive. The optimizer is more willing to tolerate unusual training points in exchange for a broad, stable corridor. With a larger C, violations are expensive. The separator bends its linear choice as far as it can toward fitting difficult examples, often leaving a narrower margin. The drawing is schematic, but the trade is exact: margin width against total slack. Small C can underreact to genuine structure. Very large C can overreact to noise or mislabeled examples. In practice C is a regularization hyperparameter, chosen by validation rather than by demanding zero training error. Support vectors still matter in the soft-margin model. They now include points touching a margin wall and points lying inside or beyond it. These are the examples that exert force on the final decision boundary. Now consider a different training set. The red examples form a compact inner cluster, while the blue examples make a ring around them. This is a simple nonlinear pattern with no noise and no overlap. Try a slanted line. It can place some blue points opposite the red cluster, but blue points remain on both sides. Rotating the line merely changes which part of the ring is misclassified. Move the line above the centre instead. The red points may all fall below it, but so do the blue points around the lower half of the ring. No translation fixes the problem either. There is a geometric reason, not just a failure of imagination. Join the outer blue points to form their convex hull. The entire red cluster lies inside that hull. A straight line divides the plane into two half-planes. If every blue vertex lies in one half-plane, then every convex combination of those vertices lies there too. The red points inside their hull therefore cannot all occupy the opposite half-plane. The useful feature here is not left versus right or up versus down. It is distance from the centre. Square that distance and call it q of x: x one squared plus x two squared. Every red point has a small value of q. Every blue point has a much larger value. A threshold at two separates them perfectly: red below two, blue above two. Back in the original plane, the threshold q equals two is a circle, not a line. So the data is nonlinear in its original coordinates. The next move is to change coordinates rather than abandon maximum-margin separation. A feature map replaces each original input by a new vector. For this problem, map x one and x two to three quadratic features: x one squared, square root two times x one x two, and x two squared. At first I have placed the points in a flat copy of their familiar arrangement. Now apply the feature map. Each point moves to the three coordinates computed from its original two. The inner red points remain near the origin of feature space. The outer blue points rise onto a distant sheet because x one squared plus x two squared is large for every point on the ring. In this three-dimensional space, one flat plane separates the classes. Its equation is feature one plus feature three equals two. Red points lie below it, and blue points lie above it. This is ordinary linear separation again. We can place parallel margin planes around this separator, maximize their distance, and obtain support vectors exactly as before. The only change is that the vectors now live in feature space. Now compare the two spaces. On the left is the original input plane. On the right is the lifted feature space with its flat separating plane. The plane above is feature one plus feature three equals two. Substitute the definitions of those features and the same condition becomes x one squared plus x two squared equals two below. That equation draws this circle in the input plane. A linear decision surface in feature space has become a nonlinear decision boundary in the original coordinates. The construction works, but it seems expensive. A richer feature map may contain hundreds, millions, or even infinitely many coordinates. Writing and storing every lifted vector would defeat the purpose. The escape comes from noticing what the SVM calculation actually asks of those lifted vectors. It does not need to inspect their coordinates one at a time. It needs inner products between them. Return to the quadratic feature map. To compare two lifted examples phi of x and phi of z, the SVM needs their inner product. Let us calculate that inner product once and see what remains. Multiply matching lifted coordinates. The first contribution is x one squared z one squared. The middle contribution contains the factor two, and the last is x two squared z two squared. Those three terms form a familiar square. They equal x one z one plus x two z two, all squared. But x one z one plus x two z two is simply the ordinary dot product x dot z. Therefore the lifted inner product is the square of one dot product in the original two-dimensional space. Define K of x z to be that squared dot product. Evaluating K gives exactly the number we would have obtained by constructing both three-dimensional feature vectors and taking their inner product. That replacement is the kernel trick. We behave as if the data had been lifted, while every computation stays in the original input space. The feature coordinates disappear, but their geometry remains in the kernel values. Why are inner products enough? At the optimum, the normal vector in feature space can be written as a weighted sum of lifted training examples. Each weight is alpha sub i times the label y sub i. To classify a new input, begin with w dot phi of x plus b. Substitute the weighted expansion of w, and the calculation becomes a sum of inner products between training features and the new feature vector. Now replace every lifted inner product by K of x sub i and x. Prediction can be performed without forming w and without explicitly forming phi of any example. Most coefficients alpha sub i are zero. Only support vectors have nonzero coefficients and contribute to the final sum. The same small set of geometrically decisive points now makes prediction sparse. Training can also be written entirely in terms of pairwise kernels. The dual objective contains K of x sub i and x sub j for pairs of training examples, together with nonnegative coefficients whose label-weighted sum is zero. The optimization therefore needs a matrix of pairwise kernel values, often called the Gram matrix. Once those values are available, the algorithm can operate as though it knew every lifted coordinate. A kernel supplies exactly the required feature-space inner product. Different kernels encode different notions of similarity. The linear kernel is the ordinary dot product, so it gives the original linear SVM. A polynomial kernel raises a shifted dot product to a degree d. Our quadratic example used degree two with no shift. Higher degrees represent richer interactions among the original coordinates. The radial basis kernel decreases exponentially with squared distance between two inputs. It corresponds to a very rich feature space and produces flexible local boundaries, with gamma controlling how quickly similarity falls away. A kernel is not magic permission to use any similarity formula. It must behave like a genuine inner product in some feature space. When it does, the maximum-margin optimization remains mathematically well formed. Let us connect the whole argument. First, maximum margin chooses the separator with the widest empty corridor. The nearest points become support vectors because their constraints pin that corridor in place. When classes overlap, slack variables measure margin violations, and C trades a wider corridor against the cost of those violations. When no straight boundary works, a feature map can lift the examples into a space where a hyperplane does work. The decision surface is linear there even though it is curved in the original coordinates. Finally, the kernel trick computes only the inner products required by training and prediction. It preserves the lifted geometry without paying the cost of explicitly writing the lifted vectors. That is the support vector machine in one connected picture: maximize geometric separation, let a small set of points determine the answer, tolerate overlap when necessary, and use kernels when the useful separation lives in a nonlinear feature space.","watch":{"version":1,"scenes":[{"title":"The Widest Empty Corridor","start":0,"end":113.69085416666667,"objects":{"axes":"an Axes (x_range=(0.0, 6.0), y_range=(0.0, 4.8), aspect=(6.0, 4.8))","best_separator":"a Line [green] labelled \"w dot x + b = 0\" drawn in axes (start=(0.0, 2.0), end=(6.0, 2.0))","candidate_high":"a Line [yellow] labelled \"upright(\"separator\")\" drawn in axes (start=(0.0, 2.55), end=(6.0, 2.35))","candidate_low":"a Line [yellow] labelled \"upright(\"separator\")\" drawn in axes (start=(0.0, 1.55), end=(6.0, 1.75))","candidate_tilted":"a Line [yellow] labelled \"upright(\"separator\")\" drawn in axes (start=(0.0, 1.65), end=(6.0, 2.25))","corridor":"a Region [yellow] drawn in axes (predicates=(<function <lambda> at 0x2ab79a3ae020>,), x_range=(0.0, 6.0), y_range=(1.0, 3.0))","lower_wall":"a Line [gray] labelled \"w dot x + b = -1\" drawn in axes (start=(0.0, 1.0), end=(6.0, 1.0), dashed=True)","margin_width":"a Math [text] that says \"$upright(\"corridor width\") = frac(2, sqrt(w dot w))$\"","negative_points":"a Point [blue] drawn in axes (location=(0.8, 0.5))","negative_points_2":"a Point [blue] drawn in axes (location=(2.0, 1.0))","negative_points_3":"a Point [blue] drawn in axes (location=(3.0, 0.4))","negative_points_4":"a Point [blue] drawn in axes (location=(4.0, 1.0))","negative_points_5":"a Point [blue] drawn in axes (location=(5.2, 0.5))","positive_points":"a Point [red] drawn in axes (location=(1.0, 3.0))","positive_points_2":"a Point [red] drawn in axes (location=(2.0, 4.0))","positive_points_3":"a Point [red] drawn in axes (location=(3.0, 3.0))","positive_points_4":"a Point [red] drawn in axes (location=(4.0, 4.2))","positive_points_5":"a Point [red] drawn in axes (location=(5.0, 3.0))","question":"a Panel that says \"Many lines separate these two classes. Which separator should a classifier trust?\"","support_definition":"a Panel that says \"The training points that touch the two walls of the maximum-margin corridor are the support vectors.\"","upper_wall":"a Line [gray] labelled \"w dot x + b = 1\" drawn in axes (start=(0.0, 3.0), end=(6.0, 3.0), dashed=True)"},"beats":[{"start":0,"say":"Suppose red and blue are two classes in a training set. We want a line that puts every red point on one side and every blue point on the other. This data is linearly separable, so at least one such line exists.","live":[],"does":[[0,"question is shown on the screen, written out."],[0,"axes is shown on the screen, written out."],[0.522,"positive_points is shown on the screen, written out."],[0.522,"positive_points_2 is shown on the screen, written out."],[0.522,"positive_points_3 is shown on the screen, written out."],[0.522,"positive_points_4 is shown on the screen, written out."],[0.522,"positive_points_5 is shown on the screen, written out."],[0.824,"negative_points is shown on the screen, written out."],[0.824,"negative_points_2 is shown on the screen, written out."],[0.824,"negative_points_3 is shown on the screen, written out."],[0.824,"negative_points_4 is shown on the screen, written out."],[0.824,"negative_points_5 is shown on the screen, written out."]]},{"start":14.346,"say":"In fact, one separating line is easy to draw. Every red point lies above it, and every blue point lies below it. But this line runs rather close to the blue class.","live":["axes","question","positive_points","positive_points_2","positive_points_3","positive_points_4","positive_points_5","negative_points","negative_points_2","negative_points_3","negative_points_4","negative_points_5"],"does":[[15.53,"candidate_low is shown on the screen, written out."],[24.006,"candidate_low is indicated — a transient flash."]]},{"start":26.2775,"say":"Another line also separates the data, but now it passes close to the red class. A tilted line works too. Correct classification alone does not tell us which of these choices is preferable.","live":["axes","question","positive_points","positive_points_2","positive_points_3","positive_points_4","positive_points_5","negative_points","negative_points_2","negative_points_3","negative_points_4","negative_points_5","candidate_low"],"does":[[26.2775,"candidate_low is hidden from the screen."],[26.626,"candidate_high is shown on the screen, written out."],[32.373000000000005,"candidate_high is hidden from the screen."],[32.373000000000005,"candidate_tilted is shown on the screen, written out."],[37.94500000000001,"candidate_tilted is hidden from the screen."]]},{"start":39.486,"say":"Imagine placing an empty corridor around a separator. Its walls remain parallel to the line, and we widen them until one wall would hit red or the other would hit blue. The corridor's width measures how much room the classifier has before a training point changes sides.","live":["axes","question","positive_points","positive_points_2","positive_points_3","positive_points_4","positive_points_5","negative_points","negative_points_2","negative_points_3","negative_points_4","negative_points_5"],"does":[[41.099999999999994,"corridor is shown on the screen, written out."],[41.900999999999996,"best_separator is shown on the screen, written out."],[43.433,"lower_wall is shown on the screen, written out."],[43.433,"upper_wall is shown on the screen, written out."]]},{"start":56.6885,"say":"Now slide the centre line inside that gap. If it moves down, the lower clearance shrinks. If it moves up, the upper clearance shrinks. The best position balances the nearest points on the two sides and maximizes the smaller clearance.","live":["axes","question","positive_points","positive_points_2","positive_points_3","positive_points_4","positive_points_5","negative_points","negative_points_2","negative_points_3","negative_points_4","negative_points_5","best_separator","corridor","lower_wall","upper_wall"],"does":[[60.427,"candidate_low is shown on the screen, written out."],[63.272,"candidate_low is hidden from the screen."],[63.272,"candidate_high is shown on the screen, written out."],[66.012,"candidate_high is hidden from the screen."],[66.662,"best_separator is indicated — a transient flash."]]},{"start":71.5455,"say":"The points that stop the two walls are special. Three red points touch the upper wall, and two blue points touch the lower wall. These are the support vectors, because they physically support the widest corridor.","live":null,"does":[[73.763,"axes moves to a new place on the board."],[73.763,"support_definition is shown on the screen, written out."],[75.331,"positive_points is indicated — a transient flash."],[75.331,"positive_points_3 is indicated — a transient flash."],[75.331,"positive_points_5 is indicated — a transient flash."],[77.50200000000001,"negative_points_2 is indicated — a transient flash."],[77.50200000000001,"negative_points_4 is indicated — a transient flash."]]},{"start":84.58,"say":"Points farther away do not determine the final position. We could move one of those distant points slightly without changing either wall. Move a support vector, however, and the maximum-margin separator generally has to move with it.","live":["support_definition","axes","question","positive_points","positive_points_2","positive_points_3","positive_points_4","positive_points_5","negative_points","negative_points_2","negative_points_3","negative_points_4","negative_points_5","best_separator","corridor","lower_wall","upper_wall"],"does":[[85.27600000000001,"positive_points_2 is indicated — a transient flash."],[89.41,"negative_points is indicated — a transient flash."],[93.044,"positive_points is indicated — a transient flash."]]},{"start":98.78699999999999,"say":"So an SVM does not merely search for zero training errors. Among all separating hyperplanes, it chooses the one with the largest geometric margin. Next we will turn this corridor into an optimization problem.","live":null,"does":[[106.41499999999999,"best_separator is indicated — a transient flash."],[106.937,"margin_width is shown on the screen, written out."],[112.6491875,"axes is hidden from the screen — left the board."],[112.6491875,"positive_points is hidden from the screen — axes left the board."],[112.6491875,"positive_points_2 is hidden from the screen — axes left the board."],[112.6491875,"positive_points_3 is hidden from the screen — axes left the board."],[112.6491875,"positive_points_4 is hidden from the screen — axes left the board."],[112.6491875,"positive_points_5 is hidden from the screen — axes left the board."],[112.6491875,"negative_points is hidden from the screen — axes left the board."],[112.6491875,"negative_points_2 is hidden from the screen — axes left the board."],[112.6491875,"negative_points_3 is hidden from the screen — axes left the board."],[112.6491875,"negative_points_4 is hidden from the screen — axes left the board."],[112.6491875,"negative_points_5 is hidden from the screen — axes left the board."],[112.6491875,"best_separator is hidden from the screen — axes left the board."],[112.6491875,"corridor is hidden from the screen — axes left the board."],[112.6491875,"lower_wall is hidden from the screen — axes left the board."],[112.6491875,"upper_wall is hidden from the screen — axes left the board."],[112.6491875,"margin_width is hidden from the screen — left the board."],[112.6491875,"question is hidden from the screen — left the board."],[112.6491875,"support_definition is hidden from the screen — left the board."]]}]},{"title":"From Geometry to Optimisation","start":113.69085416666667,"end":271.34333333333336,"objects":{"axes":"an Axes (x_range=(0.0, 6.0), y_range=(0.0, 4.8), aspect=(6.0, 4.8))","constraint":"a Math [text] that says \"$y_i (w dot x_i + b) >= 1, quad i = 1, ..., n$\"","decision":"a Math [text] that says \"$hat(y) = op(\"sign\")(w dot x + b)$\"","distance":"a Math [text] that says \"$upright(\"distance\") = frac(abs(w dot x + b), sqrt(w dot w))$\"","full_margin":"a Line [magenta] labelled \"frac(2, sqrt(w dot w))\" drawn in axes (start=(5.65, 1.0), end=(5.65, 3.0))","hard_margin":"a Panel that says \"Choose the smallest squared normal vector subject to every labelled point lying on or beyond its correct margin wall.\"","heading_problem":"a Heading that says \"The Hard-Margin Problem\"","heading_score":"a Heading that says \"Score, Distance, and Normalization\"","lower_wall":"a Line [gray] labelled \"-1\" drawn in axes (start=(0.0, 1.0), end=(6.0, 1.0), dashed=True)","margin":"a Math [text] that says \"$M = frac(2, sqrt(w dot w))$\"","negative_points":"a Point [blue] drawn in axes (location=(0.8, 0.5))","negative_points_2":"a Point [blue] drawn in axes (location=(2.0, 1.0))","negative_points_3":"a Point [blue] drawn in axes (location=(3.0, 0.4))","negative_points_4":"a Point [blue] drawn in axes (location=(4.0, 1.0))","negative_points_5":"a Point [blue] drawn in axes (location=(5.2, 0.5))","normal":"a Vector [yellow] labelled \"w\" drawn in axes (start=(3.0, 2.0), end=(3.0, 3.0))","normalized_constraint":"a Math [text] that says \"$y_i (w dot x_i + b) >= 1$\"","objective":"a Math [text] that says \"$min_(w,b) frac(1, 2) w dot w$\"","positive_points":"a Point [red] drawn in axes (location=(1.0, 3.0))","positive_points_2":"a Point [red] drawn in axes (location=(2.0, 4.0))","positive_points_3":"a Point [red] drawn in axes (location=(3.0, 3.0))","positive_points_4":"a Point [red] drawn in axes (location=(4.0, 4.2))","positive_points_5":"a Point [red] drawn in axes (location=(5.0, 3.0))","score":"a Math [text] that says \"$s(x) = w dot x + b$\"","separator":"a Line [green] labelled \"w dot x + b = 0\" drawn in axes (start=(0.0, 2.0), end=(6.0, 2.0))","signed_constraint":"a Math [text] that says \"$y_i (w dot x_i + b) > 0$\"","upper_wall":"a Line [gray] labelled \"1\" drawn in axes (start=(0.0, 3.0), end=(6.0, 3.0), dashed=True)"},"beats":[{"start":113.69085416666667,"say":"A line in two dimensions, or a hyperplane in any dimension, can be written as w dot x plus b equals zero. The vector w is perpendicular to the separator, while b shifts it without changing its orientation.","live":[],"does":[[113.69085416666667,"heading_score is shown on the screen, written out."],[113.88785416666667,"axes is shown on the screen, written out."],[114.50385416666667,"positive_points is shown on the screen, written out."],[114.50385416666667,"positive_points_2 is shown on the screen, written out."],[114.50385416666667,"positive_points_3 is shown on the screen, written out."],[114.50385416666667,"positive_points_4 is shown on the screen, written out."],[114.50385416666667,"positive_points_5 is shown on the screen, written out."],[114.50385416666667,"negative_points is shown on the screen, written out."],[114.50385416666667,"negative_points_2 is shown on the screen, written out."],[114.50385416666667,"negative_points_3 is shown on the screen, written out."],[114.50385416666667,"negative_points_4 is shown on the screen, written out."],[114.50385416666667,"negative_points_5 is shown on the screen, written out."],[120.15785416666667,"separator is shown on the screen, written out."],[122.85085416666666,"normal is shown on the screen, written out."]]},{"start":128.53635416666665,"say":"The expression w dot x plus b is a signed score. Positive scores lie on one side, negative scores lie on the other, and its sign becomes the predicted class.","live":["axes","heading_score","positive_points","positive_points_2","positive_points_3","positive_points_4","positive_points_5","negative_points","negative_points_2","negative_points_3","negative_points_4","negative_points_5","separator","normal"],"does":[[131.75285416666668,"axes moves to a new place on the board."],[131.75285416666668,"score is shown on the screen, written out."],[137.41785416666667,"The segment (0.0, 2.0) to (6.0, 2.0) in axes is lit up."],[138.59085416666667,"decision is shown on the screen, written out."],[140.83135416666667,"axes: retire a lit segment (unemphasize_line)."]]},{"start":141.43135416666667,"say":"Let each training label y sub i be plus one for red and minus one for blue. Correct classification then has one compact condition: y sub i times the score must be positive.","live":["score","decision","axes","heading_score","positive_points","positive_points_2","positive_points_3","positive_points_4","positive_points_5","negative_points","negative_points_2","negative_points_3","negative_points_4","negative_points_5","separator","normal"],"does":[[141.43135416666667,"signed_constraint is shown on the screen, written out."],[151.63685416666667,"signed_constraint (the \"y_i\" part) is emphasized."],[154.43485416666667,"signed_constraint (the \"w dot x_i + b\" part) is emphasized."],[154.43485416666667,"signed_constraint (the \"y_i\" part) is no longer emphasized."],[156.18835416666667,"signed_constraint (the \"w dot x_i + b\" part) is no longer emphasized."]]},{"start":156.78835416666666,"say":"There is a scaling ambiguity. Multiplying both w and b by any positive number leaves the zero set unchanged, so it leaves the separating hyperplane unchanged. We use that freedom to make the closest signed scores equal to one.","live":["score","decision","signed_constraint","axes","heading_score","positive_points","positive_points_2","positive_points_3","positive_points_4","positive_points_5","negative_points","negative_points_2","negative_points_3","negative_points_4","negative_points_5","separator","normal"],"does":[[171.45185416666666,"normalized_constraint is shown on the screen, written out."],[171.45185416666666,"upper_wall is shown on the screen, written out."],[171.45185416666666,"lower_wall is shown on the screen, written out."]]},{"start":172.81835416666667,"say":"After this normalization, every red point must have score at least plus one, and every blue point must have score at most minus one. The support vectors are precisely the points where the combined inequality becomes an equality.","live":["score","decision","signed_constraint","normalized_constraint","axes","heading_score","positive_points","positive_points_2","positive_points_3","positive_points_4","positive_points_5","negative_points","negative_points_2","negative_points_3","negative_points_4","negative_points_5","separator","normal","upper_wall","lower_wall"],"does":[[175.08185416666666,"positive_points is indicated — a transient flash."],[175.08185416666666,"positive_points_3 is indicated — a transient flash."],[175.08185416666666,"positive_points_5 is indicated — a transient flash."],[176.30085416666668,"normalized_constraint (the \">= 1\" part) is emphasized."],[178.12385416666666,"negative_points_2 is indicated — a transient flash."],[178.12385416666666,"negative_points_4 is indicated — a transient flash."],[186.78535416666665,"normalized_constraint (the \">= 1\" part) is no longer emphasized."]]},{"start":187.38535416666667,"say":"The perpendicular distance from a point x to the central hyperplane is the absolute score divided by the length of w. Each wall has score one in magnitude, so each wall is one over the length of w from the centre.","live":null,"does":[[188.62785416666668,"distance is shown on the screen, written out."],[192.41185416666667,"distance (the \"abs(w dot x + b)\" part) is emphasized."],[193.30585416666668,"distance (the \"abs(w dot x + b)\" part) is no longer emphasized."],[193.30585416666668,"distance (the \"sqrt(w dot w)\" part) is emphasized."],[201.34035416666666,"distance (the \"sqrt(w dot w)\" part) is no longer emphasized."]]},{"start":201.94035416666668,"say":"The full corridor therefore has width two over the length of w. To make that width large, we must make the length of w small. Squaring it changes neither the minimizing choice nor the geometry.","live":["score","decision","signed_constraint","normalized_constraint","distance","axes","heading_score","positive_points","positive_points_2","positive_points_3","positive_points_4","positive_points_5","negative_points","negative_points_2","negative_points_3","negative_points_4","negative_points_5","separator","normal","upper_wall","lower_wall"],"does":[[203.91385416666668,"margin is shown on the screen, written out."],[204.16985416666668,"full_margin is shown on the screen, written out."],[204.87785416666668,"margin (the \"sqrt(w dot w)\" part) is emphasized."],[215.27985416666667,"margin moves to a new place on the board."],[215.27985416666667,"decision is hidden from the screen — left the board."],[215.27985416666667,"distance is hidden from the screen — left the board."],[215.27985416666667,"heading_score is hidden from the screen — left the board."],[215.27985416666667,"normalized_constraint is hidden from the screen — left the board."],[215.27985416666667,"score is hidden from the screen — left the board."],[215.27985416666667,"signed_constraint is hidden from the screen — left the board."],[215.27985416666667,"margin (the \"sqrt(w dot w)\" part) is no longer emphasized."]]},{"start":215.8798541666667,"say":"We have reached the hard-margin optimization problem. Minimize one half w dot w, which is one half the squared length of the normal vector.","live":["margin","axes","positive_points","positive_points_2","positive_points_3","positive_points_4","positive_points_5","negative_points","negative_points_2","negative_points_3","negative_points_4","negative_points_5","separator","normal","upper_wall","lower_wall","full_margin"],"does":[[215.8798541666667,"heading_problem is shown on the screen, written out."],[219.9778541666667,"objective is shown on the screen, written out."],[223.56585416666667,"objective (the \"w dot w\" part) is emphasized."],[225.73685416666666,"objective (the \"w dot w\" part) is no longer emphasized."]]},{"start":226.33685416666668,"say":"The minimization is subject to one constraint per training example: y sub i times w dot x sub i plus b must be at least one. These inequalities say that every point lies outside the empty corridor on its proper side.","live":["margin","axes","positive_points","positive_points_2","positive_points_3","positive_points_4","positive_points_5","negative_points","negative_points_2","negative_points_3","negative_points_4","negative_points_5","separator","normal","upper_wall","lower_wall","full_margin","objective","heading_problem"],"does":[[228.64685416666669,"constraint is shown on the screen, written out."],[229.16985416666668,"constraint (the \"i = 1, ..., n\" part) is emphasized."],[237.19185416666667,"constraint (the \"i = 1, ..., n\" part) is no longer emphasized."]]},{"start":242.88885416666668,"say":"Only a few of those constraints are tight at the solution. Those active constraints belong to the support vectors. The remaining points satisfy their inequalities with room to spare and do not pin the optimum.","live":["margin","axes","positive_points","positive_points_2","positive_points_3","positive_points_4","positive_points_5","negative_points","negative_points_2","negative_points_3","negative_points_4","negative_points_5","separator","normal","upper_wall","lower_wall","full_margin","objective","constraint","heading_problem"],"does":[[247.09185416666668,"hard_margin is shown on the screen, written out."],[248.55485416666667,"positive_points is indicated — a transient flash."],[248.55485416666667,"positive_points_3 is indicated — a transient flash."],[248.55485416666667,"positive_points_5 is indicated — a transient flash."],[248.55485416666667,"negative_points_2 is indicated — a transient flash."],[248.55485416666667,"negative_points_4 is indicated — a transient flash."]]},{"start":256.6313541666667,"say":"This completes the geometric derivation. A wide corridor became a small normal vector, correct classification became a family of inequalities, and the points touching equality became the support vectors.","live":["margin","axes","positive_points","positive_points_2","positive_points_3","positive_points_4","positive_points_5","negative_points","negative_points_2","negative_points_3","negative_points_4","negative_points_5","separator","normal","upper_wall","lower_wall","full_margin","objective","constraint","hard_margin","heading_problem"],"does":[[260.1258541666667,"A box is drawn around margin."],[261.51885416666676,"objective is indicated — a transient flash."],[265.72185416666673,"constraint is indicated — a transient flash."],[270.3016666666667,"axes is hidden from the screen — left the board."],[270.3016666666667,"positive_points is hidden from the screen — axes left the board."],[270.3016666666667,"positive_points_2 is hidden from the screen — axes left the board."],[270.3016666666667,"positive_points_3 is hidden from the screen — axes left the board."],[270.3016666666667,"positive_points_4 is hidden from the screen — axes left the board."],[270.3016666666667,"positive_points_5 is hidden from the screen — axes left the board."],[270.3016666666667,"negative_points is hidden from the screen — axes left the board."],[270.3016666666667,"negative_points_2 is hidden from the screen — axes left the board."],[270.3016666666667,"negative_points_3 is hidden from the screen — axes left the board."],[270.3016666666667,"negative_points_4 is hidden from the screen — axes left the board."],[270.3016666666667,"negative_points_5 is hidden from the screen — axes left the board."],[270.3016666666667,"separator is hidden from the screen — axes left the board."],[270.3016666666667,"normal is hidden from the screen — axes left the board."],[270.3016666666667,"upper_wall is hidden from the screen — axes left the board."],[270.3016666666667,"lower_wall is hidden from the screen — axes left the board."],[270.3016666666667,"full_margin is hidden from the screen — axes left the board."],[270.3016666666667,"constraint is hidden from the screen — left the board."],[270.3016666666667,"hard_margin is hidden from the screen — left the board."],[270.3016666666667,"heading_problem is hidden from the screen — left the board."],[270.3016666666667,"margin is hidden from the screen — left the board."],[270.3016666666667,"objective is hidden from the screen — left the board."]]}]},{"title":"When the Classes Overlap","start":271.34333333333336,"end":439.78439583333335,"objects":{"axes":"an Axes (x_range=(0.0, 6.0), y_range=(0.0, 4.8), aspect=(6.0, 4.8))","blue_slack":"a Line [magenta] labelled \"xi_j\" drawn in axes (start=(1.5, 3.3), end=(1.5, 1.0), dashed=True)","hard_constraint":"a Math [text] that says \"$y_i (w dot x_i + b) >= 1$\"","heading_tradeoff":"a Heading that says \"What the Parameter $C$ Trades\"","high_blue":"a Point [blue] drawn in high_c (location=(0.8, 0.5))","high_blue_2":"a Point [blue] drawn in high_c (location=(1.5, 3.3))","high_blue_3":"a Point [blue] drawn in high_c (location=(2.5, 1.0))","high_blue_4":"a Point [blue] drawn in high_c (location=(4.0, 0.8))","high_blue_5":"a Point [blue] drawn in high_c (location=(5.2, 1.2))","high_c":"a Figure (x_range=(0.0, 6.0), y_range=(0.0, 4.8), aspect=(6.0, 4.8))","high_label":"a Tex [text] that says \"Larger $C$: violations cost more\"","high_line":"a Line [green] drawn in high_c (start=(0.0, 1.45), end=(6.0, 2.55))","high_lower":"a Line [gray] drawn in high_c (start=(0.0, 0.45), end=(6.0, 1.55), dashed=True)","high_red":"a Point [red] drawn in high_c (location=(0.8, 3.4))","high_red_2":"a Point [red] drawn in high_c (location=(2.0, 4.0))","high_red_3":"a Point [red] drawn in high_c (location=(3.0, 3.0))","high_red_4":"a Point [red] drawn in high_c (location=(4.2, 3.6))","high_red_5":"a Point [red] drawn in high_c (location=(5.0, 0.7))","high_upper":"a Line [gray] drawn in high_c (start=(0.0, 2.45), end=(6.0, 3.55), dashed=True)","interpretation":"a Panel that says \"Slack measures how far a training example falls inside its required margin wall. Values greater than one correspond to misclassification.\"","low_blue":"a Point [blue] drawn in low_c (location=(0.8, 0.5))","low_blue_2":"a Point [blue] drawn in low_c (location=(1.5, 3.3))","low_blue_3":"a Point [blue] drawn in low_c (location=(2.5, 1.0))","low_blue_4":"a Point [blue] drawn in low_c (location=(4.0, 0.8))","low_blue_5":"a Point [blue] drawn in low_c (location=(5.2, 1.2))","low_c":"a Figure (x_range=(0.0, 6.0), y_range=(0.0, 4.8), aspect=(6.0, 4.8))","low_label":"a Tex [text] that says \"Smaller $C$: wider margin, more tolerance\"","low_line":"a Line [green] drawn in low_c (start=(0.0, 2.0), end=(6.0, 2.0))","low_lower":"a Line [gray] drawn in low_c (start=(0.0, 1.0), end=(6.0, 1.0), dashed=True)","low_red":"a Point [red] drawn in low_c (location=(0.8, 3.4))","low_red_2":"a Point [red] drawn in low_c (location=(2.0, 4.0))","low_red_3":"a Point [red] drawn in low_c (location=(3.0, 3.0))","low_red_4":"a Point [red] drawn in low_c (location=(4.2, 3.6))","low_red_5":"a Point [red] drawn in low_c (location=(5.0, 0.7))","low_upper":"a Line [gray] drawn in low_c (start=(0.0, 3.0), end=(6.0, 3.0), dashed=True)","lower_wall":"a Line [gray] labelled \"-1\" drawn in axes (start=(0.0, 1.0), end=(6.0, 1.0), dashed=True)","negative_points":"a Point [blue] drawn in axes (location=(0.8, 0.5))","negative_points_2":"a Point [blue] drawn in axes (location=(1.5, 3.3))","negative_points_3":"a Point [blue] drawn in axes (location=(2.5, 1.0))","negative_points_4":"a Point [blue] drawn in axes (location=(4.0, 0.8))","negative_points_5":"a Point [blue] drawn in axes (location=(5.2, 1.2))","positive_points":"a Point [red] drawn in axes (location=(0.8, 3.4))","positive_points_2":"a Point [red] drawn in axes (location=(2.0, 4.0))","positive_points_3":"a Point [red] drawn in axes (location=(3.0, 3.0))","positive_points_4":"a Point [red] drawn in axes (location=(4.2, 3.6))","positive_points_5":"a Point [red] drawn in axes (location=(5.0, 0.7))","question":"a Panel that says \"What should the classifier do when no hyperplane separates every training point?\"","red_slack":"a Line [magenta] labelled \"xi_i\" drawn in axes (start=(5.0, 0.7), end=(5.0, 3.0), dashed=True)","separator":"a Line [green] labelled \"w dot x + b = 0\" drawn in axes (start=(0.0, 2.0), end=(6.0, 2.0))","soft_constraint":"a Math [text] that says \"$y_i (w dot x_i + b) >= 1 - xi_i, quad xi_i >= 0$\"","soft_objective":"a Math [text] that says \"$min_(w,b,xi) frac(1, 2) w dot w + C sum_i xi_i$\"","upper_wall":"a Line [gray] labelled \"1\" drawn in axes (start=(0.0, 3.0), end=(6.0, 3.0), dashed=True)"},"beats":[{"start":271.34333333333336,"say":"Real classes often overlap. A red example may sit deep among blue examples, while a blue example may sit among red ones. No line can put every point on its requested side in this training set.","live":[],"does":[[271.34333333333336,"question is shown on the screen, written out."],[271.34333333333336,"axes is shown on the screen, written out."],[274.08333333333337,"positive_points is shown on the screen, written out."],[274.08333333333337,"positive_points_2 is shown on the screen, written out."],[274.08333333333337,"positive_points_3 is shown on the screen, written out."],[274.08333333333337,"positive_points_4 is shown on the screen, written out."],[274.08333333333337,"positive_points_5 is shown on the screen, written out."],[275.9173333333334,"negative_points is shown on the screen, written out."],[275.9173333333334,"negative_points_2 is shown on the screen, written out."],[275.9173333333334,"negative_points_3 is shown on the screen, written out."],[275.9173333333334,"negative_points_4 is shown on the screen, written out."],[275.9173333333334,"negative_points_5 is shown on the screen, written out."]]},{"start":285.41083333333336,"say":"If we insist on the hard-margin constraints, the optimization problem has no feasible answer. The red point below and the blue point above cannot both satisfy the same perfect separation.","live":["axes","question","positive_points","positive_points_2","positive_points_3","positive_points_4","positive_points_5","negative_points","negative_points_2","negative_points_3","negative_points_4","negative_points_5"],"does":[[287.79133333333334,"axes moves to a new place on the board."],[287.79133333333334,"hard_constraint is shown on the screen, written out."],[292.37733333333335,"positive_points_5 is indicated — a transient flash."],[293.5733333333334,"negative_points_2 is indicated — a transient flash."]]},{"start":298.4918333333334,"say":"A soft-margin SVM relaxes each constraint with a nonnegative slack variable xi sub i. Instead of demanding a signed score of at least one, it permits the score to fall short by xi sub i.","live":["hard_constraint","axes","question","positive_points","positive_points_2","positive_points_3","positive_points_4","positive_points_5","negative_points","negative_points_2","negative_points_3","negative_points_4","negative_points_5"],"does":[[300.50033333333334,"soft_constraint is shown on the screen, written out."],[303.11233333333337,"soft_constraint (the \"xi_i\" part) is emphasized."],[312.73783333333336,"soft_constraint (the \"xi_i\" part) is no longer emphasized."]]},{"start":313.3378333333334,"say":"Draw the central separator and its two margin walls. A point outside its correct wall needs zero slack. A point inside the corridor needs some slack, because it has not reached the wall required by its label.","live":["hard_constraint","soft_constraint","axes","question","positive_points","positive_points_2","positive_points_3","positive_points_4","positive_points_5","negative_points","negative_points_2","negative_points_3","negative_points_4","negative_points_5"],"does":[[314.46433333333334,"separator is shown on the screen, written out."],[316.0663333333334,"lower_wall is shown on the screen, written out."],[316.0663333333334,"upper_wall is shown on the screen, written out."],[321.89433333333335,"interpretation is shown on the screen, written out."]]},{"start":328.3113333333334,"say":"This red point is not merely inside the margin. It lies on the blue side of the separator, so its slack exceeds one. The same is true in reverse for this blue point. Slack records the degree of each violation rather than pretending the data is perfectly separable.","live":["hard_constraint","soft_constraint","interpretation","axes","question","positive_points","positive_points_2","positive_points_3","positive_points_4","positive_points_5","negative_points","negative_points_2","negative_points_3","negative_points_4","negative_points_5","separator","lower_wall","upper_wall"],"does":[[328.96133333333336,"red_slack is shown on the screen, written out."],[334.8593333333334,"interpretation (the \"greater than one\" part) is emphasized."],[338.08633333333336,"blue_slack is shown on the screen, written out."],[345.15683333333334,"interpretation (the \"greater than one\" part) is no longer emphasized."]]},{"start":345.75683333333336,"say":"The new objective contains two costs. One half w dot w still rewards a wide margin. The sum of the slacks penalizes points that enter the margin or cross the separator, and the constant C controls how heavily that penalty is counted.","live":["hard_constraint","soft_constraint","interpretation","axes","question","positive_points","positive_points_2","positive_points_3","positive_points_4","positive_points_5","negative_points","negative_points_2","negative_points_3","negative_points_4","negative_points_5","separator","lower_wall","upper_wall","red_slack","blue_slack"],"does":[[346.34933333333333,"soft_objective is shown on the screen, written out."],[351.48033333333336,"soft_objective (the \"frac(1, 2) w dot w\" part) is emphasized."],[354.4413333333334,"soft_objective (the \"C sum_i xi_i\" part) is emphasized."],[354.4413333333334,"soft_objective (the \"frac(1, 2) w dot w\" part) is no longer emphasized."],[362.21933333333334,"soft_objective (the \"C sum_i xi_i\" part) is no longer emphasized."]]},{"start":362.81933333333336,"say":"This is not permission to ignore errors for free. Every violation raises the objective. The optimizer accepts a violation only when avoiding it would require a sufficiently narrower or more contorted decision boundary.","live":["hard_constraint","soft_constraint","soft_objective","interpretation","axes","question","positive_points","positive_points_2","positive_points_3","positive_points_4","positive_points_5","negative_points","negative_points_2","negative_points_3","negative_points_4","negative_points_5","separator","lower_wall","upper_wall","red_slack","blue_slack"],"does":[[377.55283333333335,"axes is hidden from the screen — left the board."],[377.55283333333335,"positive_points is hidden from the screen — axes left the board."],[377.55283333333335,"positive_points_2 is hidden from the screen — axes left the board."],[377.55283333333335,"positive_points_3 is hidden from the screen — axes left the board."],[377.55283333333335,"positive_points_4 is hidden from the screen — axes left the board."],[377.55283333333335,"positive_points_5 is hidden from the screen — axes left the board."],[377.55283333333335,"negative_points is hidden from the screen — axes left the board."],[377.55283333333335,"negative_points_2 is hidden from the screen — axes left the board."],[377.55283333333335,"negative_points_3 is hidden from the screen — axes left the board."],[377.55283333333335,"negative_points_4 is hidden from the screen — axes left the board."],[377.55283333333335,"negative_points_5 is hidden from the screen — axes left the board."],[377.55283333333335,"separator is hidden from the screen — axes left the board."],[377.55283333333335,"lower_wall is hidden from the screen — axes left the board."],[377.55283333333335,"upper_wall is hidden from the screen — axes left the board."],[377.55283333333335,"red_slack is hidden from the screen — axes left the board."],[377.55283333333335,"blue_slack is hidden from the screen — axes left the board."],[377.55283333333335,"hard_constraint is hidden from the screen — left the board."],[377.55283333333335,"interpretation is hidden from the screen — left the board."],[377.55283333333335,"question is hidden from the screen — left the board."],[377.55283333333335,"soft_constraint is hidden from the screen — left the board."],[377.55283333333335,"soft_objective is hidden from the screen — left the board."]]},{"start":378.1528333333334,"say":"With a smaller C, slack is relatively inexpensive. The optimizer is more willing to tolerate unusual training points in exchange for a broad, stable corridor.","live":[],"does":[[378.1528333333334,"heading_tradeoff is shown on the screen, written out."],[378.79133333333334,"low_c is shown on the screen, written out."],[378.79133333333334,"low_label is shown on the screen, written out."],[385.53633333333335,"low_red is shown on the screen, written out."],[385.53633333333335,"low_red_2 is shown on the screen, written out."],[385.53633333333335,"low_red_3 is shown on the screen, written out."],[385.53633333333335,"low_red_4 is shown on the screen, written out."],[385.53633333333335,"low_red_5 is shown on the screen, written out."],[385.53633333333335,"low_blue is shown on the screen, written out."],[385.53633333333335,"low_blue_2 is shown on the screen, written out."],[385.53633333333335,"low_blue_3 is shown on the screen, written out."],[385.53633333333335,"low_blue_4 is shown on the screen, written out."],[385.53633333333335,"low_blue_5 is shown on the screen, written out."],[388.61333333333334,"low_line is shown on the screen, written out."],[388.61333333333334,"low_upper is shown on the screen, written out."],[388.61333333333334,"low_lower is shown on the screen, written out."]]},{"start":390.28133333333335,"say":"With a larger C, violations are expensive. The separator bends its linear choice as far as it can toward fitting difficult examples, often leaving a narrower margin. The drawing is schematic, but the trade is exact: margin width against total slack.","live":["low_label","low_c","heading_tradeoff","low_red","low_red_2","low_red_3","low_red_4","low_red_5","low_blue","low_blue_2","low_blue_3","low_blue_4","low_blue_5","low_line","low_upper","low_lower"],"does":[[391.04733333333337,"high_c is shown on the screen, written out."],[391.04733333333337,"high_label is shown on the screen, written out."],[394.22933333333333,"high_line is shown on the screen, written out."],[394.22933333333333,"high_upper is shown on the screen, written out."],[394.22933333333333,"high_lower is shown on the screen, written out."],[398.14133333333336,"high_red is shown on the screen, written out."],[398.14133333333336,"high_red_2 is shown on the screen, written out."],[398.14133333333336,"high_red_3 is shown on the screen, written out."],[398.14133333333336,"high_red_4 is shown on the screen, written out."],[398.14133333333336,"high_red_5 is shown on the screen, written out."],[398.14133333333336,"high_blue is shown on the screen, written out."],[398.14133333333336,"high_blue_2 is shown on the screen, written out."],[398.14133333333336,"high_blue_3 is shown on the screen, written out."],[398.14133333333336,"high_blue_4 is shown on the screen, written out."],[398.14133333333336,"high_blue_5 is shown on the screen, written out."]]},{"start":408.15683333333334,"say":"Small C can underreact to genuine structure. Very large C can overreact to noise or mislabeled examples. In practice C is a regularization hyperparameter, chosen by validation rather than by demanding zero training error.","live":["low_label","low_c","high_label","high_c","heading_tradeoff","low_red","low_red_2","low_red_3","low_red_4","low_red_5","low_blue","low_blue_2","low_blue_3","low_blue_4","low_blue_5","low_line","low_upper","low_lower","high_red","high_red_2","high_red_3","high_red_4","high_red_5","high_blue","high_blue_2","high_blue_3","high_blue_4","high_blue_5","high_line","high_upper","high_lower"],"does":[[408.45833333333337,"low_line is indicated — a transient flash."],[412.34833333333336,"high_line is indicated — a transient flash."]]},{"start":424.91783333333336,"say":"Support vectors still matter in the soft-margin model. They now include points touching a margin wall and points lying inside or beyond it. These are the examples that exert force on the final decision boundary.","live":null,"does":[[429.8643333333334,"low_red_3 is indicated — a transient flash."],[429.8643333333334,"low_red_5 is indicated — a transient flash."],[429.8643333333334,"low_blue_2 is indicated — a transient flash."],[429.8643333333334,"low_blue_3 is indicated — a transient flash."],[438.74272916666666,"heading_tradeoff is hidden from the screen — left the board."],[438.74272916666666,"high_c is hidden from the screen — left the board."],[438.74272916666666,"high_red is hidden from the screen — high_c left the board."],[438.74272916666666,"high_red_2 is hidden from the screen — high_c left the board."],[438.74272916666666,"high_red_3 is hidden from the screen — high_c left the board."],[438.74272916666666,"high_red_4 is hidden from the screen — high_c left the board."],[438.74272916666666,"high_red_5 is hidden from the screen — high_c left the board."],[438.74272916666666,"high_blue is hidden from the screen — high_c left the board."],[438.74272916666666,"high_blue_2 is hidden from the screen — high_c left the board."],[438.74272916666666,"high_blue_3 is hidden from the screen — high_c left the board."],[438.74272916666666,"high_blue_4 is hidden from the screen — high_c left the board."],[438.74272916666666,"high_blue_5 is hidden from the screen — high_c left the board."],[438.74272916666666,"high_line is hidden from the screen — high_c left the board."],[438.74272916666666,"high_upper is hidden from the screen — high_c left the board."],[438.74272916666666,"high_lower is hidden from the screen — high_c left the board."],[438.74272916666666,"high_label is hidden from the screen — left the board."],[438.74272916666666,"low_c is hidden from the screen — left the board."],[438.74272916666666,"low_red is hidden from the screen — low_c left the board."],[438.74272916666666,"low_red_2 is hidden from the screen — low_c left the board."],[438.74272916666666,"low_red_3 is hidden from the screen — low_c left the board."],[438.74272916666666,"low_red_4 is hidden from the screen — low_c left the board."],[438.74272916666666,"low_red_5 is hidden from the screen — low_c left the board."],[438.74272916666666,"low_blue is hidden from the screen — low_c left the board."],[438.74272916666666,"low_blue_2 is hidden from the screen — low_c left the board."],[438.74272916666666,"low_blue_3 is hidden from the screen — low_c left the board."],[438.74272916666666,"low_blue_4 is hidden from the screen — low_c left the board."],[438.74272916666666,"low_blue_5 is hidden from the screen — low_c left the board."],[438.74272916666666,"low_line is hidden from the screen — low_c left the board."],[438.74272916666666,"low_upper is hidden from the screen — low_c left the board."],[438.74272916666666,"low_lower is hidden from the screen — low_c left the board."],[438.74272916666666,"low_label is hidden from the screen — left the board."]]}]},{"title":"No Straight Line Will Do","start":439.78439583333335,"end":546.5005625,"objects":{"axes":"an Axes (x_range=(-3.0, 3.0), y_range=(-3.0, 3.0), aspect=(1.0, 1.0))","boundary_rule":"a Math [text] that says \"$q(x) = 2$\"","first_try":"a Line [yellow] drawn in axes (start=(-3.0, -0.8), end=(3.0, 0.8))","inner_points":"a Point [red] drawn in axes (location=(0.6, 0.0))","inner_points_2":"a Point [red] drawn in axes (location=(-0.6, 0.0))","inner_points_3":"a Point [red] drawn in axes (location=(0.0, 0.6))","inner_points_4":"a Point [red] drawn in axes (location=(0.0, -0.6))","inner_rule":"a Math [text] that says \"$q(x) < 2$\"","nonlinear_boundary":"a Circle [green] labelled \"x_1^2 + x_2^2 = 2\" drawn in axes (radius=1.4142135623730951)","outer_hull":"a Polygon [gray] drawn in axes (vertices=((2.0, 0.0), (1.4, 1.4), (0.0, 2.0), (-1.4, 1.4), (-2.0, 0.0), …, fill_opacity=0.12)","outer_points":"a Point [blue] drawn in axes (location=(2.0, 0.0))","outer_points_2":"a Point [blue] drawn in axes (location=(1.4, 1.4))","outer_points_3":"a Point [blue] drawn in axes (location=(0.0, 2.0))","outer_points_4":"a Point [blue] drawn in axes (location=(-1.4, 1.4))","outer_points_5":"a Point [blue] drawn in axes (location=(-2.0, 0.0))","outer_points_6":"a Point [blue] drawn in axes (location=(-1.4, -1.4))","outer_points_7":"a Point [blue] drawn in axes (location=(0.0, -2.0))","outer_points_8":"a Point [blue] drawn in axes (location=(1.4, -1.4))","outer_rule":"a Math [text] that says \"$q(x) > 2$\"","question":"a Panel that says \"Can one straight line separate the inner red cluster from the outer blue ring?\"","radius_score":"a Math [text] that says \"$q(x) = x_1^2 + x_2^2$\"","second_try":"a Line [yellow] drawn in axes (start=(-1.1, -3.0), end=(1.1, 3.0))","third_try":"a Line [yellow] drawn in axes (start=(-3.0, 1.25), end=(3.0, 1.25))"},"beats":[{"start":439.78439583333335,"say":"Now consider a different training set. The red examples form a compact inner cluster, while the blue examples make a ring around them. This is a simple nonlinear pattern with no noise and no overlap.","live":[],"does":[[439.78439583333335,"question is shown on the screen, written out."],[439.78439583333335,"axes is shown on the screen, written out."],[442.59439583333335,"inner_points is shown on the screen, written out."],[442.59439583333335,"inner_points_2 is shown on the screen, written out."],[442.59439583333335,"inner_points_3 is shown on the screen, written out."],[442.59439583333335,"inner_points_4 is shown on the screen, written out."],[445.50839583333334,"outer_points is shown on the screen, written out."],[445.50839583333334,"outer_points_2 is shown on the screen, written out."],[445.50839583333334,"outer_points_3 is shown on the screen, written out."],[445.50839583333334,"outer_points_4 is shown on the screen, written out."],[445.50839583333334,"outer_points_5 is shown on the screen, written out."],[445.50839583333334,"outer_points_6 is shown on the screen, written out."],[445.50839583333334,"outer_points_7 is shown on the screen, written out."],[445.50839583333334,"outer_points_8 is shown on the screen, written out."]]},{"start":452.86539583333337,"say":"Try a slanted line. It can place some blue points opposite the red cluster, but blue points remain on both sides. Rotating the line merely changes which part of the ring is misclassified.","live":["axes","question","inner_points","inner_points_2","inner_points_3","inner_points_4","outer_points","outer_points_2","outer_points_3","outer_points_4","outer_points_5","outer_points_6","outer_points_7","outer_points_8"],"does":[[453.64339583333333,"first_try is shown on the screen, written out."],[461.02739583333334,"first_try is hidden from the screen."],[461.02739583333334,"second_try is shown on the screen, written out."]]},{"start":465.74839583333335,"say":"Move the line above the centre instead. The red points may all fall below it, but so do the blue points around the lower half of the ring. No translation fixes the problem either.","live":["axes","question","inner_points","inner_points_2","inner_points_3","inner_points_4","outer_points","outer_points_2","outer_points_3","outer_points_4","outer_points_5","outer_points_6","outer_points_7","outer_points_8","second_try"],"does":[[465.74839583333335,"second_try is hidden from the screen."],[466.6773958333333,"third_try is shown on the screen, written out."],[475.90739583333334,"third_try is hidden from the screen."]]},{"start":477.30839583333335,"say":"There is a geometric reason, not just a failure of imagination. Join the outer blue points to form their convex hull. The entire red cluster lies inside that hull.","live":["axes","question","inner_points","inner_points_2","inner_points_3","inner_points_4","outer_points","outer_points_2","outer_points_3","outer_points_4","outer_points_5","outer_points_6","outer_points_7","outer_points_8"],"does":[[483.64739583333335,"outer_hull is shown on the screen, drawn."],[485.76039583333335,"inner_points is indicated — a transient flash."],[485.76039583333335,"inner_points_2 is indicated — a transient flash."],[485.76039583333335,"inner_points_3 is indicated — a transient flash."],[485.76039583333335,"inner_points_4 is indicated — a transient flash."]]},{"start":488.64689583333336,"say":"A straight line divides the plane into two half-planes. If every blue vertex lies in one half-plane, then every convex combination of those vertices lies there too. The red points inside their hull therefore cannot all occupy the opposite half-plane.","live":["axes","question","inner_points","inner_points_2","inner_points_3","inner_points_4","outer_points","outer_points_2","outer_points_3","outer_points_4","outer_points_5","outer_points_6","outer_points_7","outer_points_8","outer_hull"],"does":[[495.27639583333337,"outer_hull is indicated — a transient flash."]]},{"start":504.01489583333336,"say":"The useful feature here is not left versus right or up versus down. It is distance from the centre. Square that distance and call it q of x: x one squared plus x two squared.","live":null,"does":[[513.2683958333333,"axes moves to a new place on the board."],[513.2683958333333,"radius_score is shown on the screen, written out."],[515.6243958333333,"radius_score (the \"x_1^2 + x_2^2\" part) is emphasized."],[518.2368958333334,"radius_score (the \"x_1^2 + x_2^2\" part) is no longer emphasized."]]},{"start":518.8368958333333,"say":"Every red point has a small value of q. Every blue point has a much larger value. A threshold at two separates them perfectly: red below two, blue above two.","live":["radius_score","axes","question","inner_points","inner_points_2","inner_points_3","inner_points_4","outer_points","outer_points_2","outer_points_3","outer_points_4","outer_points_5","outer_points_6","outer_points_7","outer_points_8","outer_hull"],"does":[[519.5453958333334,"inner_rule is shown on the screen, written out."],[520.3113958333333,"inner_points is indicated — a transient flash."],[520.3113958333333,"inner_points_2 is indicated — a transient flash."],[520.3113958333333,"inner_points_3 is indicated — a transient flash."],[520.3113958333333,"inner_points_4 is indicated — a transient flash."],[522.6213958333334,"outer_rule is shown on the screen, written out."],[523.6663958333334,"outer_points is indicated — a transient flash."],[523.6663958333334,"outer_points_2 is indicated — a transient flash."],[523.6663958333334,"outer_points_3 is indicated — a transient flash."],[523.6663958333334,"outer_points_4 is indicated — a transient flash."],[523.6663958333334,"outer_points_5 is indicated — a transient flash."],[523.6663958333334,"outer_points_6 is indicated — a transient flash."],[523.6663958333334,"outer_points_7 is indicated — a transient flash."],[523.6663958333334,"outer_points_8 is indicated — a transient flash."]]},{"start":530.8493958333333,"say":"Back in the original plane, the threshold q equals two is a circle, not a line. So the data is nonlinear in its original coordinates. The next move is to change coordinates rather than abandon maximum-margin separation.","live":["radius_score","inner_rule","outer_rule","axes","question","inner_points","inner_points_2","inner_points_3","inner_points_4","outer_points","outer_points_2","outer_points_3","outer_points_4","outer_points_5","outer_points_6","outer_points_7","outer_points_8","outer_hull"],"does":[[533.8333958333334,"boundary_rule is shown on the screen, written out."],[534.7743958333333,"nonlinear_boundary is shown on the screen, drawn."],[537.5483958333334,"nonlinear_boundary is indicated — a transient flash."],[545.4588958333334,"axes is hidden from the screen — left the board."],[545.4588958333334,"inner_points is hidden from the screen — axes left the board."],[545.4588958333334,"inner_points_2 is hidden from the screen — axes left the board."],[545.4588958333334,"inner_points_3 is hidden from the screen — axes left the board."],[545.4588958333334,"inner_points_4 is hidden from the screen — axes left the board."],[545.4588958333334,"outer_points is hidden from the screen — axes left the board."],[545.4588958333334,"outer_points_2 is hidden from the screen — axes left the board."],[545.4588958333334,"outer_points_3 is hidden from the screen — axes left the board."],[545.4588958333334,"outer_points_4 is hidden from the screen — axes left the board."],[545.4588958333334,"outer_points_5 is hidden from the screen — axes left the board."],[545.4588958333334,"outer_points_6 is hidden from the screen — axes left the board."],[545.4588958333334,"outer_points_7 is hidden from the screen — axes left the board."],[545.4588958333334,"outer_points_8 is hidden from the screen — axes left the board."],[545.4588958333334,"outer_hull is hidden from the screen — axes left the board."],[545.4588958333334,"nonlinear_boundary is hidden from the screen — axes left the board."],[545.4588958333334,"boundary_rule is hidden from the screen — left the board."],[545.4588958333334,"inner_rule is hidden from the screen — left the board."],[545.4588958333334,"outer_rule is hidden from the screen — left the board."],[545.4588958333334,"question is hidden from the screen — left the board."],[545.4588958333334,"radius_score is hidden from the screen — left the board."]]}]},{"title":"Lifting into Feature Space","start":546.5005625,"end":679.4129583333333,"objects":{"circle":"a Circle [green] labelled \"x_1^2 + x_2^2 = 2\" drawn in flat (radius=1.4142135623730951)","feature_map":"a Math [text] that says \"$phi(x) = (x_1^2, thin sqrt(2) x_1 x_2, thin x_2^2)$\"","flat":"an Axes (x_range=(-3.0, 3.0), y_range=(-3.0, 3.0), aspect=(1.0, 1.0))","flat_inner":"a Point [red] drawn in flat (location=(0.6, 0.0))","flat_inner_2":"a Point [red] drawn in flat (location=(-0.6, 0.0))","flat_inner_3":"a Point [red] drawn in flat (location=(0.0, 0.6))","flat_inner_4":"a Point [red] drawn in flat (location=(0.0, -0.6))","flat_outer":"a Point [blue] drawn in flat (location=(2.0, 0.0))","flat_outer_2":"a Point [blue] drawn in flat (location=(1.4, 1.4))","flat_outer_3":"a Point [blue] drawn in flat (location=(0.0, 2.0))","flat_outer_4":"a Point [blue] drawn in flat (location=(-1.4, 1.4))","flat_outer_5":"a Point [blue] drawn in flat (location=(-2.0, 0.0))","flat_outer_6":"a Point [blue] drawn in flat (location=(-1.4, -1.4))","flat_outer_7":"a Point [blue] drawn in flat (location=(0.0, -2.0))","flat_outer_8":"a Point [blue] drawn in flat (location=(1.4, -1.4))","heading_lift":"a Heading that says \"Lift the Data into Three Features\"","heading_return":"a Heading that says \"A Plane Above, a Curve Below\"","lift":"a VariableNumber","lifted_points":"a Point [red] drawn in space (location=((((1.0 - lift) * 0.6) + ((lift * 0.6) * 0.6)), (((1.0 - lift) …)","lifted_points_10":"a Point [blue] drawn in space (location=((((1.0 - lift) * -1.4) + ((lift * -1.4) * -1.4)), (((1.0 - lif…)","lifted_points_11":"a Point [blue] drawn in space (location=((((1.0 - lift) * 0.0) + ((lift * 0.0) * 0.0)), (((1.0 - lift) …)","lifted_points_12":"a Point [blue] drawn in space (location=((((1.0 - lift) * 1.4) + ((lift * 1.4) * 1.4)), (((1.0 - lift) …)","lifted_points_2":"a Point [red] drawn in space (location=((((1.0 - lift) * -0.6) + ((lift * -0.6) * -0.6)), (((1.0 - lif…)","lifted_points_3":"a Point [red] drawn in space (location=((((1.0 - lift) * 0.0) + ((lift * 0.0) * 0.0)), (((1.0 - lift) …)","lifted_points_4":"a Point [red] drawn in space (location=((((1.0 - lift) * 0.0) + ((lift * 0.0) * 0.0)), (((1.0 - lift) …)","lifted_points_5":"a Point [blue] drawn in space (location=((((1.0 - lift) * 2.0) + ((lift * 2.0) * 2.0)), (((1.0 - lift) …)","lifted_points_6":"a Point [blue] drawn in space (location=((((1.0 - lift) * 1.4) + ((lift * 1.4) * 1.4)), (((1.0 - lift) …)","lifted_points_7":"a Point [blue] drawn in space (location=((((1.0 - lift) * 0.0) + ((lift * 0.0) * 0.0)), (((1.0 - lift) …)","lifted_points_8":"a Point [blue] drawn in space (location=((((1.0 - lift) * -1.4) + ((lift * -1.4) * -1.4)), (((1.0 - lif…)","lifted_points_9":"a Point [blue] drawn in space (location=((((1.0 - lift) * -2.0) + ((lift * -2.0) * -2.0)), (((1.0 - lif…)","plane_rule":"a Math [text] that says \"$phi_1 + phi_3 = 2$\"","same_rule":"a Math [text] that says \"$x_1^2 + x_2^2 = 2$\"","separating_plane":"a Plane [green] labelled \"phi_1 + phi_3 = 2\" drawn in space (point=(1.0, 0.0, 1.0), normal=(1.0, 0.0, 1.0), size=5.0)","space":"an Axes3D (x_range=(-2.6, 4.4), y_range=(-3.2, 3.2), z_range=(-0.6, 4.4))","tex":"a Tex [text] that says \"Original input space\"","tex_2":"a Tex [text] that says \"Quadratic feature space\""},"beats":[{"start":546.5005625,"say":"A feature map replaces each original input by a new vector. For this problem, map x one and x two to three quadratic features: x one squared, square root two times x one x two, and x two squared.","live":[],"does":[[546.5005625,"heading_lift is shown on the screen, written out."],[546.5005625,"space is shown on the screen, written out."],[546.6865625,"feature_map is shown on the screen, written out."],[548.7765625,"lifted_points is shown on the screen, written out."],[548.7765625,"lifted_points_2 is shown on the screen, written out."],[548.7765625,"lifted_points_3 is shown on the screen, written out."],[548.7765625,"lifted_points_4 is shown on the screen, written out."],[548.7765625,"lifted_points_5 is shown on the screen, written out."],[548.7765625,"lifted_points_6 is shown on the screen, written out."],[548.7765625,"lifted_points_7 is shown on the screen, written out."],[548.7765625,"lifted_points_8 is shown on the screen, written out."],[548.7765625,"lifted_points_9 is shown on the screen, written out."],[548.7765625,"lifted_points_10 is shown on the screen, written out."],[548.7765625,"lifted_points_11 is shown on the screen, written out."],[548.7765625,"lifted_points_12 is shown on the screen, written out."]]},{"start":562.9825625,"say":"At first I have placed the points in a flat copy of their familiar arrangement. Now apply the feature map. Each point moves to the three coordinates computed from its original two.","live":["feature_map","space","heading_lift","lifted_points","lifted_points_2","lifted_points_3","lifted_points_4","lifted_points_5","lifted_points_6","lifted_points_7","lifted_points_8","lifted_points_9","lifted_points_10","lifted_points_11","lifted_points_12"],"does":[[568.9165625,"lifted_points is redrawn as the numbers it depends on change."],[568.9165625,"lifted_points_2 is redrawn as the numbers it depends on change."],[568.9165625,"lifted_points_3 is redrawn as the numbers it depends on change."],[568.9165625,"lifted_points_4 is redrawn as the numbers it depends on change."],[568.9165625,"lifted_points_5 is redrawn as the numbers it depends on change."],[568.9165625,"lifted_points_6 is redrawn as the numbers it depends on change."],[568.9165625,"lifted_points_7 is redrawn as the numbers it depends on change."],[568.9165625,"lifted_points_8 is redrawn as the numbers it depends on change."],[568.9165625,"lifted_points_9 is redrawn as the numbers it depends on change."],[568.9165625,"lifted_points_10 is redrawn as the numbers it depends on change."],[568.9165625,"lifted_points_11 is redrawn as the numbers it depends on change."],[568.9165625,"lifted_points_12 is redrawn as the numbers it depends on change."],[568.9165625,"lift ticks to 1.0."]]},{"start":575.1705625,"say":"The inner red points remain near the origin of feature space. The outer blue points rise onto a distant sheet because x one squared plus x two squared is large for every point on the ring.","live":null,"does":[[575.9715625,"lifted_points is indicated — a transient flash."],[575.9715625,"lifted_points_2 is indicated — a transient flash."],[575.9715625,"lifted_points_3 is indicated — a transient flash."],[575.9715625,"lifted_points_4 is indicated — a transient flash."],[577.9915625,"space turns in its own slot."],[580.1045625,"lifted_points_5 is indicated — a transient flash."],[580.1045625,"lifted_points_6 is indicated — a transient flash."],[580.1045625,"lifted_points_7 is indicated — a transient flash."],[580.1045625,"lifted_points_8 is indicated — a transient flash."],[580.1045625,"lifted_points_9 is indicated — a transient flash."],[580.1045625,"lifted_points_10 is indicated — a transient flash."],[580.1045625,"lifted_points_11 is indicated — a transient flash."],[580.1045625,"lifted_points_12 is indicated — a transient flash."]]},{"start":588.6690625,"say":"In this three-dimensional space, one flat plane separates the classes. Its equation is feature one plus feature three equals two. Red points lie below it, and blue points lie above it.","live":null,"does":[[591.5945624999999,"separating_plane is shown on the screen, written out."],[594.1025625,"plane_rule is shown on the screen, written out."],[598.5255625,"lifted_points is indicated — a transient flash."],[598.5255625,"lifted_points_2 is indicated — a transient flash."],[598.5255625,"lifted_points_3 is indicated — a transient flash."],[598.5255625,"lifted_points_4 is indicated — a transient flash."],[600.4415625,"lifted_points_5 is indicated — a transient flash."],[600.4415625,"lifted_points_6 is indicated — a transient flash."],[600.4415625,"lifted_points_7 is indicated — a transient flash."],[600.4415625,"lifted_points_8 is indicated — a transient flash."],[600.4415625,"lifted_points_9 is indicated — a transient flash."],[600.4415625,"lifted_points_10 is indicated — a transient flash."],[600.4415625,"lifted_points_11 is indicated — a transient flash."],[600.4415625,"lifted_points_12 is indicated — a transient flash."]]},{"start":602.8060625,"say":"This is ordinary linear separation again. We can place parallel margin planes around this separator, maximize their distance, and obtain support vectors exactly as before. The only change is that the vectors now live in feature space.","live":["feature_map","plane_rule","space","heading_lift","lifted_points","lifted_points_2","lifted_points_3","lifted_points_4","lifted_points_5","lifted_points_6","lifted_points_7","lifted_points_8","lifted_points_9","lifted_points_10","lifted_points_11","lifted_points_12","separating_plane"],"does":[[604.2345625,"separating_plane is indicated — a transient flash."],[618.2365625,"space moves to a new place on the board."],[618.2365625,"feature_map is hidden from the screen — left the board."],[618.2365625,"heading_lift is hidden from the screen — left the board."],[618.2365625,"plane_rule is hidden from the screen — left the board."]]},{"start":618.8365625,"say":"Now compare the two spaces. On the left is the original input plane. On the right is the lifted feature space with its flat separating plane.","live":["space","lifted_points","lifted_points_2","lifted_points_3","lifted_points_4","lifted_points_5","lifted_points_6","lifted_points_7","lifted_points_8","lifted_points_9","lifted_points_10","lifted_points_11","lifted_points_12","separating_plane"],"does":[[618.8365625,"heading_return is shown on the screen, written out."],[621.8785625,"flat is shown on the screen, written out."],[622.4005625,"flat_inner is shown on the screen, written out."],[622.4005625,"flat_inner_2 is shown on the screen, written out."],[622.4005625,"flat_inner_3 is shown on the screen, written out."],[622.4005625,"flat_inner_4 is shown on the screen, written out."],[622.4005625,"flat_outer is shown on the screen, written out."],[622.4005625,"flat_outer_2 is shown on the screen, written out."],[622.4005625,"flat_outer_3 is shown on the screen, written out."],[622.4005625,"flat_outer_4 is shown on the screen, written out."],[622.4005625,"flat_outer_5 is shown on the screen, written out."],[622.4005625,"flat_outer_6 is shown on the screen, written out."],[622.4005625,"flat_outer_7 is shown on the screen, written out."],[622.4005625,"flat_outer_8 is shown on the screen, written out."]]},{"start":628.7360625,"say":"The plane above is feature one plus feature three equals two. Substitute the definitions of those features and the same condition becomes x one squared plus x two squared equals two below.","live":["space","lifted_points","lifted_points_2","lifted_points_3","lifted_points_4","lifted_points_5","lifted_points_6","lifted_points_7","lifted_points_8","lifted_points_9","lifted_points_10","lifted_points_11","lifted_points_12","separating_plane","flat","heading_return","flat_inner","flat_inner_2","flat_inner_3","flat_inner_4","flat_outer","flat_outer_2","flat_outer_3","flat_outer_4","flat_outer_5","flat_outer_6","flat_outer_7","flat_outer_8"],"does":[[633.2985625,"same_rule is shown on the screen, written out."],[637.3615625,"same_rule (the \"x_1^2 + x_2^2\" part) is emphasized."],[640.9030625,"same_rule (the \"x_1^2 + x_2^2\" part) is no longer emphasized."]]},{"start":641.5030624999999,"say":"That equation draws this circle in the input plane. A linear decision surface in feature space has become a nonlinear decision boundary in the original coordinates.","live":["space","lifted_points","lifted_points_2","lifted_points_3","lifted_points_4","lifted_points_5","lifted_points_6","lifted_points_7","lifted_points_8","lifted_points_9","lifted_points_10","lifted_points_11","lifted_points_12","separating_plane","flat","same_rule","heading_return","flat_inner","flat_inner_2","flat_inner_3","flat_inner_4","flat_outer","flat_outer_2","flat_outer_3","flat_outer_4","flat_outer_5","flat_outer_6","flat_outer_7","flat_outer_8"],"does":[[643.1285625,"circle is shown on the screen, drawn."],[648.5965625,"circle is indicated — a transient flash."]]},{"start":652.5630625,"say":"The construction works, but it seems expensive. A richer feature map may contain hundreds, millions, or even infinitely many coordinates. Writing and storing every lifted vector would defeat the purpose.","live":["space","lifted_points","lifted_points_2","lifted_points_3","lifted_points_4","lifted_points_5","lifted_points_6","lifted_points_7","lifted_points_8","lifted_points_9","lifted_points_10","lifted_points_11","lifted_points_12","separating_plane","flat","same_rule","heading_return","flat_inner","flat_inner_2","flat_inner_3","flat_inner_4","flat_outer","flat_outer_2","flat_outer_3","flat_outer_4","flat_outer_5","flat_outer_6","flat_outer_7","flat_outer_8","circle"],"does":[]},{"start":666.4220625,"say":"The escape comes from noticing what the SVM calculation actually asks of those lifted vectors. It does not need to inspect their coordinates one at a time. It needs inner products between them.","live":null,"does":[[671.0885625,"lifted_points is indicated — a transient flash."],[671.0885625,"lifted_points_2 is indicated — a transient flash."],[671.0885625,"lifted_points_3 is indicated — a transient flash."],[671.0885625,"lifted_points_4 is indicated — a transient flash."],[671.0885625,"lifted_points_5 is indicated — a transient flash."],[671.0885625,"lifted_points_6 is indicated — a transient flash."],[671.0885625,"lifted_points_7 is indicated — a transient flash."],[671.0885625,"lifted_points_8 is indicated — a transient flash."],[671.0885625,"lifted_points_9 is indicated — a transient flash."],[671.0885625,"lifted_points_10 is indicated — a transient flash."],[671.0885625,"lifted_points_11 is indicated — a transient flash."],[671.0885625,"lifted_points_12 is indicated — a transient flash."],[678.3712916666666,"flat is hidden from the screen — left the board."],[678.3712916666666,"flat_inner is hidden from the screen — flat left the board."],[678.3712916666666,"flat_inner_2 is hidden from the screen — flat left the board."],[678.3712916666666,"flat_inner_3 is hidden from the screen — flat left the board."],[678.3712916666666,"flat_inner_4 is hidden from the screen — flat left the board."],[678.3712916666666,"flat_outer is hidden from the screen — flat left the board."],[678.3712916666666,"flat_outer_2 is hidden from the screen — flat left the board."],[678.3712916666666,"flat_outer_3 is hidden from the screen — flat left the board."],[678.3712916666666,"flat_outer_4 is hidden from the screen — flat left the board."],[678.3712916666666,"flat_outer_5 is hidden from the screen — flat left the board."],[678.3712916666666,"flat_outer_6 is hidden from the screen — flat left the board."],[678.3712916666666,"flat_outer_7 is hidden from the screen — flat left the board."],[678.3712916666666,"flat_outer_8 is hidden from the screen — flat left the board."],[678.3712916666666,"circle is hidden from the screen — flat left the board."],[678.3712916666666,"heading_return is hidden from the screen — left the board."],[678.3712916666666,"same_rule is hidden from the screen — left the board."],[678.3712916666666,"space is hidden from the screen — left the board."],[678.3712916666666,"lifted_points is hidden from the screen — space left the board."],[678.3712916666666,"lifted_points_2 is hidden from the screen — space left the board."],[678.3712916666666,"lifted_points_3 is hidden from the screen — space left the board."],[678.3712916666666,"lifted_points_4 is hidden from the screen — space left the board."],[678.3712916666666,"lifted_points_5 is hidden from the screen — space left the board."],[678.3712916666666,"lifted_points_6 is hidden from the screen — space left the board."],[678.3712916666666,"lifted_points_7 is hidden from the screen — space left the board."],[678.3712916666666,"lifted_points_8 is hidden from the screen — space left the board."],[678.3712916666666,"lifted_points_9 is hidden from the screen — space left the board."],[678.3712916666666,"lifted_points_10 is hidden from the screen — space left the board."],[678.3712916666666,"lifted_points_11 is hidden from the screen — space left the board."],[678.3712916666666,"lifted_points_12 is hidden from the screen — space left the board."],[678.3712916666666,"separating_plane is hidden from the screen — space left the board."]]}]},{"title":"The Kernel Trick","start":679.4129583333333,"end":964.8744166666667,"objects":{"boundary":"a Circle [green] drawn in plot (radius=1.4142135624)","classifier":"a Derivation [text] that says \"$f(x) &= w dot phi(x) + b \\ &= sum_i alpha_i y_i (phi(x_i) dot phi(x)) + b \\ &= sum_(i in upright(\"SV\")) alpha_i y_i K(x_i,x) + b$\"","dual_constraints":"a Math [text] that says \"$alpha_i >= 0, quad sum_i alpha_i y_i = 0$\"","dual_objective":"a Math [text] that says \"$max_alpha sum_i alpha_i - frac(1, 2) sum_(i,j) alpha_i alpha_j y_i y_j K(x_i,x_j)$\"","heading":"a Heading that says \"Common Kernel Choices\"","heading_classifier":"a Heading that says \"The Classifier Needs Only Kernels\"","heading_identity":"a Heading that says \"Compute the Lifted Inner Product Directly\"","heading_recap":"a Heading that says \"The Complete SVM Story\"","heading_training":"a Heading that says \"Training Uses a Kernel Matrix\"","kernel":"a Math [text] that says \"$K(x,z) = (x dot z)^2$\"","kernel_definition":"a Panel that says \"A kernel returns an inner product in some feature space: $K(x,z)=phi(x) dot phi(z)$.\"","kernel_derivation":"a Derivation [text] that says \"$phi(x) = (x_1^2, thin sqrt(2) x_1 x_2, thin x_2^2) \\ phi(x) dot phi(z) &= x_1^2 z_1^2 + 2 x_1 x_2 z_1 z_2 + x_2^2 z_2^2 \\ &= (x_1 z_1 + x_2 z_2)^2 \\ &= (x dot z)^2$\"","linear_kernel":"a Math [text] that says \"$K(x,z) = x dot z$\"","ordinary_inner":"a Point [gray] drawn in plot (location=(0.45, 0.0))","ordinary_inner_2":"a Point [gray] drawn in plot (location=(-0.45, 0.0))","ordinary_outer":"a Point [gray] drawn in plot (location=(2.25, 0.0))","ordinary_outer_2":"a Point [gray] drawn in plot (location=(-2.25, 0.0))","ordinary_outer_3":"a Point [gray] drawn in plot (location=(0.0, 2.25))","ordinary_outer_4":"a Point [gray] drawn in plot (location=(0.0, -2.25))","plot":"an Axes (x_range=(-3.0, 3.0), y_range=(-3.0, 3.0), aspect=(1.0, 1.0))","polynomial_kernel":"a Math [text] that says \"$K(x,z) = (x dot z + c)^d$\"","rbf_kernel":"a Math [text] that says \"$K(x,z) = exp(-gamma (x-z) dot (x-z))$\"","recap":"a Block [text] that says \"Choose a separator with a wide margin. The closest training points become support vectors. Slack variables trade margin width against violations. A feature map can make nonlinear data linearly separable. A kernel computes feature-space inn…\"","support_inner":"a Point [red] drawn in plot (location=(0.75, 0.0))","support_inner_2":"a Point [red] drawn in plot (location=(-0.75, 0.0))","support_inner_3":"a Point [red] drawn in plot (location=(0.0, 0.75))","support_inner_4":"a Point [red] drawn in plot (location=(0.0, -0.75))","support_outer":"a Point [blue] drawn in plot (location=(1.4, 1.4))","support_outer_2":"a Point [blue] drawn in plot (location=(-1.4, 1.4))","support_outer_3":"a Point [blue] drawn in plot (location=(-1.4, -1.4))","support_outer_4":"a Point [blue] drawn in plot (location=(1.4, -1.4))","w_expansion":"a Math [text] that says \"$w = sum_i alpha_i y_i phi(x_i)$\""},"beats":[{"start":679.4129583333333,"say":"Return to the quadratic feature map. To compare two lifted examples phi of x and phi of z, the SVM needs their inner product. Let us calculate that inner product once and see what remains.","live":[],"does":[[679.4129583333333,"heading_identity is shown on the screen, written out."],[679.4129583333333,"plot is shown on the screen, written out."],[680.0749583333334,"boundary is shown on the screen, written out."],[680.6669583333334,"kernel_derivation is shown on the screen, written out."],[683.1749583333333,"ordinary_inner is shown on the screen, written out."],[683.1749583333333,"ordinary_inner_2 is shown on the screen, written out."],[683.1749583333333,"support_inner is shown on the screen, written out."],[683.1749583333333,"support_inner_2 is shown on the screen, written out."],[683.1749583333333,"support_inner_3 is shown on the screen, written out."],[683.1749583333333,"support_inner_4 is shown on the screen, written out."],[683.1749583333333,"ordinary_outer is shown on the screen, written out."],[683.1749583333333,"ordinary_outer_2 is shown on the screen, written out."],[683.1749583333333,"ordinary_outer_3 is shown on the screen, written out."],[683.1749583333333,"ordinary_outer_4 is shown on the screen, written out."],[683.1749583333333,"support_outer is shown on the screen, written out."],[683.1749583333333,"support_outer_2 is shown on the screen, written out."],[683.1749583333333,"support_outer_3 is shown on the screen, written out."],[683.1749583333333,"support_outer_4 is shown on the screen, written out."]]},{"start":693.1904583333334,"say":"Multiply matching lifted coordinates. The first contribution is x one squared z one squared. The middle contribution contains the factor two, and the last is x two squared z two squared.","live":["plot","heading_identity","boundary","ordinary_inner","ordinary_inner_2","support_inner","support_inner_2","support_inner_3","support_inner_4","ordinary_outer","ordinary_outer_2","ordinary_outer_3","ordinary_outer_4","support_outer","support_outer_2","support_outer_3","support_outer_4"],"does":[[693.5389583333333,"kernel_derivation is shown on the screen, written out."],[696.4519583333333,"kernel_derivation (the \"x_1^2 z_1^2\" part) is emphasized."],[700.7829583333333,"kernel_derivation (the \"2 x_1 x_2 z_1 z_2\" part) is emphasized."],[700.7829583333333,"kernel_derivation (the \"x_1^2 z_1^2\" part) is no longer emphasized."],[703.6619583333334,"kernel_derivation (the \"2 x_1 x_2 z_1 z_2\" part) is no longer emphasized."],[703.6619583333334,"kernel_derivation (the \"x_2^2 z_2^2\" part) is emphasized."],[706.6344583333333,"kernel_derivation (the \"x_2^2 z_2^2\" part) is no longer emphasized."]]},{"start":707.2344583333334,"say":"Those three terms form a familiar square. They equal x one z one plus x two z two, all squared.","live":null,"does":[[709.3239583333333,"kernel_derivation is shown on the screen, written out."],[712.1919583333333,"kernel_derivation (the \"x_1 z_1 + x_2 z_2\" part) is emphasized."],[715.4889583333334,"kernel_derivation (the \"x_1 z_1 + x_2 z_2\" part) is no longer emphasized."]]},{"start":716.0889583333334,"say":"But x one z one plus x two z two is simply the ordinary dot product x dot z. Therefore the lifted inner product is the square of one dot product in the original two-dimensional space.","live":null,"does":[[720.3959583333333,"kernel_derivation is shown on the screen, written out."],[724.8079583333333,"kernel_derivation (the \"(x dot z)^2\" part) is emphasized."],[728.3724583333334,"kernel_derivation (the \"(x dot z)^2\" part) is no longer emphasized."]]},{"start":728.9724583333333,"say":"Define K of x z to be that squared dot product. Evaluating K gives exactly the number we would have obtained by constructing both three-dimensional feature vectors and taking their inner product.","live":null,"does":[[729.3209583333334,"kernel is shown on the screen, written out."],[735.4159583333334,"A box is drawn around kernel."]]},{"start":742.7734583333333,"say":"That replacement is the kernel trick. We behave as if the data had been lifted, while every computation stays in the original input space. The feature coordinates disappear, but their geometry remains in the kernel values.","live":["kernel","plot","heading_identity","boundary","ordinary_inner","ordinary_inner_2","support_inner","support_inner_2","support_inner_3","support_inner_4","ordinary_outer","ordinary_outer_2","ordinary_outer_3","ordinary_outer_4","support_outer","support_outer_2","support_outer_3","support_outer_4"],"does":[[744.3989583333333,"kernel is indicated — a transient flash."],[757.2504583333333,"kernel moves to a new place on the board."],[757.2504583333333,"heading_identity is hidden from the screen — left the board."],[757.2504583333333,"kernel_derivation is hidden from the screen — left the board."],[757.2504583333333,"The box around kernel is lifted."]]},{"start":757.8504583333333,"say":"Why are inner products enough? At the optimum, the normal vector in feature space can be written as a weighted sum of lifted training examples. Each weight is alpha sub i times the label y sub i.","live":["kernel","plot","boundary","ordinary_inner","ordinary_inner_2","support_inner","support_inner_2","support_inner_3","support_inner_4","ordinary_outer","ordinary_outer_2","ordinary_outer_3","ordinary_outer_4","support_outer","support_outer_2","support_outer_3","support_outer_4"],"does":[[757.8504583333333,"heading_classifier is shown on the screen, written out."],[764.3639583333334,"w_expansion is shown on the screen, written out."],[767.6259583333333,"w_expansion (the \"alpha_i\" part) is indicated — a transient flash."],[769.6699583333334,"w_expansion (the \"y_i\" part) is indicated — a transient flash."]]},{"start":772.1734583333333,"say":"To classify a new input, begin with w dot phi of x plus b. Substitute the weighted expansion of w, and the calculation becomes a sum of inner products between training features and the new feature vector.","live":["kernel","plot","boundary","ordinary_inner","ordinary_inner_2","support_inner","support_inner_2","support_inner_3","support_inner_4","ordinary_outer","ordinary_outer_2","ordinary_outer_3","ordinary_outer_4","support_outer","support_outer_2","support_outer_3","support_outer_4","w_expansion","heading_classifier"],"does":[[774.4839583333334,"classifier is shown on the screen, written out."],[778.1529583333333,"classifier is shown on the screen, written out."],[782.4479583333333,"classifier (the \"phi(x_i) dot phi(x)\" part) is emphasized."],[786.1054583333333,"classifier (the \"phi(x_i) dot phi(x)\" part) is no longer emphasized."]]},{"start":786.7054583333334,"say":"Now replace every lifted inner product by K of x sub i and x. Prediction can be performed without forming w and without explicitly forming phi of any example.","live":null,"does":[[787.2859583333334,"classifier is shown on the screen, written out."],[789.2019583333333,"classifier (the \"K(x_i,x)\" part) is emphasized."],[797.6424583333334,"classifier (the \"K(x_i,x)\" part) is no longer emphasized."]]},{"start":798.2424583333334,"say":"Most coefficients alpha sub i are zero. Only support vectors have nonzero coefficients and contribute to the final sum. The same small set of geometrically decisive points now makes prediction sparse.","live":null,"does":[[801.0639583333333,"ordinary_inner is dimmed out."],[801.0639583333333,"ordinary_inner_2 is dimmed out."],[801.0639583333333,"ordinary_outer is dimmed out."],[801.0639583333333,"ordinary_outer_2 is dimmed out."],[801.0639583333333,"ordinary_outer_3 is dimmed out."],[801.0639583333333,"ordinary_outer_4 is dimmed out."],[802.6889583333333,"support_inner is indicated — a transient flash."],[802.6889583333333,"support_inner_2 is indicated — a transient flash."],[802.6889583333333,"support_inner_3 is indicated — a transient flash."],[802.6889583333333,"support_inner_4 is indicated — a transient flash."],[802.6889583333333,"support_outer is indicated — a transient flash."],[802.6889583333333,"support_outer_2 is indicated — a transient flash."],[802.6889583333333,"support_outer_3 is indicated — a transient flash."],[802.6889583333333,"support_outer_4 is indicated — a transient flash."],[812.7894583333333,"kernel moves to a new place on the board."],[812.7894583333333,"classifier is hidden from the screen — left the board."],[812.7894583333333,"heading_classifier is hidden from the screen — left the board."],[812.7894583333333,"plot is hidden from the screen — left the board."],[812.7894583333333,"boundary is hidden from the screen — plot left the board."],[812.7894583333333,"ordinary_inner is hidden from the screen — plot left the board."],[812.7894583333333,"ordinary_inner_2 is hidden from the screen — plot left the board."],[812.7894583333333,"support_inner is hidden from the screen — plot left the board."],[812.7894583333333,"support_inner_2 is hidden from the screen — plot left the board."],[812.7894583333333,"support_inner_3 is hidden from the screen — plot left the board."],[812.7894583333333,"support_inner_4 is hidden from the screen — plot left the board."],[812.7894583333333,"ordinary_outer is hidden from the screen — plot left the board."],[812.7894583333333,"ordinary_outer_2 is hidden from the screen — plot left the board."],[812.7894583333333,"ordinary_outer_3 is hidden from the screen — plot left the board."],[812.7894583333333,"ordinary_outer_4 is hidden from the screen — plot left the board."],[812.7894583333333,"support_outer is hidden from the screen — plot left the board."],[812.7894583333333,"support_outer_2 is hidden from the screen — plot left the board."],[812.7894583333333,"support_outer_3 is hidden from the screen — plot left the board."],[812.7894583333333,"support_outer_4 is hidden from the screen — plot left the board."],[812.7894583333333,"w_expansion is hidden from the screen — left the board."]]},{"start":813.3894583333333,"say":"Training can also be written entirely in terms of pairwise kernels. The dual objective contains K of x sub i and x sub j for pairs of training examples, together with nonnegative coefficients whose label-weighted sum is zero.","live":["kernel"],"does":[[813.3894583333333,"heading_training is shown on the screen, written out."],[818.2779583333333,"dual_objective is shown on the screen, written out."],[822.6539583333333,"dual_objective (the \"K(x_i,x_j)\" part) is emphasized."],[825.1389583333333,"dual_constraints is shown on the screen, written out."],[828.9239583333333,"dual_objective (the \"K(x_i,x_j)\" part) is no longer emphasized."]]},{"start":829.5239583333333,"say":"The optimization therefore needs a matrix of pairwise kernel values, often called the Gram matrix. Once those values are available, the algorithm can operate as though it knew every lifted coordinate. A kernel supplies exactly the required feature-space inner product.","live":["kernel","dual_objective","dual_constraints","heading_training"],"does":[[829.5239583333333,"kernel_definition is shown on the screen, written out."],[846.4624583333333,"dual_constraints is hidden from the screen — left the board."],[846.4624583333333,"dual_objective is hidden from the screen — left the board."],[846.4624583333333,"heading_training is hidden from the screen — left the board."],[846.4624583333333,"kernel is hidden from the screen — left the board."],[846.4624583333333,"kernel_definition is hidden from the screen — left the board."]]},{"start":847.0624583333333,"say":"Different kernels encode different notions of similarity. The linear kernel is the ordinary dot product, so it gives the original linear SVM.","live":[],"does":[[851.1379583333334,"linear_kernel is shown on the screen, written out."],[852.7979583333333,"linear_kernel (the \"x dot z\" part) is emphasized."],[856.4434583333333,"linear_kernel (the \"x dot z\" part) is no longer emphasized."]]},{"start":857.0434583333333,"say":"A polynomial kernel raises a shifted dot product to a degree d. Our quadratic example used degree two with no shift. Higher degrees represent richer interactions among the original coordinates.","live":["linear_kernel"],"does":[[857.5539583333333,"polynomial_kernel is shown on the screen, written out."],[860.4219583333334,"polynomial_kernel (the \"d\" part) is emphasized."],[869.8844583333333,"polynomial_kernel (the \"d\" part) is no longer emphasized."]]},{"start":870.4844583333334,"say":"The radial basis kernel decreases exponentially with squared distance between two inputs. It corresponds to a very rich feature space and produces flexible local boundaries, with gamma controlling how quickly similarity falls away.","live":["linear_kernel","polynomial_kernel"],"does":[[871.0179583333334,"rbf_kernel is shown on the screen, written out."],[881.5139583333333,"rbf_kernel (the \"gamma\" part) is emphasized."],[884.9854583333333,"rbf_kernel (the \"gamma\" part) is no longer emphasized."]]},{"start":885.5854583333333,"say":"A kernel is not magic permission to use any similarity formula. It must behave like a genuine inner product in some feature space. When it does, the maximum-margin optimization remains mathematically well formed.","live":["linear_kernel","polynomial_kernel","rbf_kernel"],"does":[[891.8889583333333,"linear_kernel is indicated — a transient flash."],[900.2019583333333,"linear_kernel is hidden from the screen — left the board."],[900.2019583333333,"polynomial_kernel is hidden from the screen — left the board."],[900.2019583333333,"rbf_kernel is hidden from the screen — left the board."]]},{"start":900.8019583333333,"say":"Let us connect the whole argument. First, maximum margin chooses the separator with the widest empty corridor. The nearest points become support vectors because their constraints pin that corridor in place.","live":[],"does":[[900.8019583333333,"heading_recap is shown on the screen, written out."],[900.8019583333333,"recap is shown on the screen, written out."],[900.8019583333333,"classifier is shown on the screen, written out."],[901.0019583333333,"classifier is shown on the screen, written out."],[901.2019583333333,"classifier is shown on the screen, written out."]]},{"start":914.4749583333333,"say":"When classes overlap, slack variables measure margin violations, and C trades a wider corridor against the cost of those violations.","live":["recap","heading_recap"],"does":[]},{"start":923.5964583333333,"say":"When no straight boundary works, a feature map can lift the examples into a space where a hyperplane does work. The decision surface is linear there even though it is curved in the original coordinates.","live":null,"does":[]},{"start":936.5029583333333,"say":"Finally, the kernel trick computes only the inner products required by training and prediction. It preserves the lifted geometry without paying the cost of explicitly writing the lifted vectors.","live":null,"does":[]},{"start":947.9699583333334,"say":"That is the support vector machine in one connected picture: maximize geometric separation, let a small set of points determine the answer, tolerate overlap when necessary, and use kernels when the useful separation lives in a nonlinear feature space.","live":null,"does":[[950.4199583333334,"A box is drawn around classifier."],[963.83275,"classifier is hidden from the screen — left the board."],[963.83275,"heading_recap is hidden from the screen — left the board."],[963.83275,"recap is hidden from the screen — left the board."]]}]}]},"durationSeconds":965,"chapters":[{"title":"The Widest Empty Corridor","startSeconds":0,"narration":"Suppose red and blue are two classes in a training set. We want a line that puts every red point on one side and every blue point on the other. This data is linearly separable, so at least one such line exists. In fact, one separating line is easy to draw. Every red point lies above it, and every blue point lies below it. But this line runs rather close to the blue class. Another line also separates the data, but now it passes close to the red class. A tilted line works too. Correct classification alone does not tell us which of these choices is preferable. Imagine placing an empty corridor around a separator. Its walls remain parallel to the line, and we widen them until one wall would hit red or the other would hit blue. The corridor's width measures how much room the classifier has before a training point changes sides. Now slide the centre line inside that gap. If it moves down, the lower clearance shrinks. If it moves up, the upper clearance shrinks. The best position balances the nearest points on the two sides and maximizes the smaller clearance. The points that stop the two walls are special. Three red points touch the upper wall, and two blue points touch the lower wall. These are the support vectors, because they physically support the widest corridor. Points farther away do not determine the final position. We could move one of those distant points slightly without changing either wall. Move a support vector, however, and the maximum-margin separator generally has to move with it. So an SVM does not merely search for zero training errors. Among all separating hyperplanes, it chooses the one with the largest geometric margin. Next we will turn this corridor into an optimization problem."},{"title":"From Geometry to Optimisation","startSeconds":113.69085416666667,"narration":"A line in two dimensions, or a hyperplane in any dimension, can be written as w dot x plus b equals zero. The vector w is perpendicular to the separator, while b shifts it without changing its orientation. The expression w dot x plus b is a signed score. Positive scores lie on one side, negative scores lie on the other, and its sign becomes the predicted class. Let each training label y sub i be plus one for red and minus one for blue. Correct classification then has one compact condition: y sub i times the score must be positive. There is a scaling ambiguity. Multiplying both w and b by any positive number leaves the zero set unchanged, so it leaves the separating hyperplane unchanged. We use that freedom to make the closest signed scores equal to one. After this normalization, every red point must have score at least plus one, and every blue point must have score at most minus one. The support vectors are precisely the points where the combined inequality becomes an equality. The perpendicular distance from a point x to the central hyperplane is the absolute score divided by the length of w. Each wall has score one in magnitude, so each wall is one over the length of w from the centre. The full corridor therefore has width two over the length of w. To make that width large, we must make the length of w small. Squaring it changes neither the minimizing choice nor the geometry. We have reached the hard-margin optimization problem. Minimize one half w dot w, which is one half the squared length of the normal vector. The minimization is subject to one constraint per training example: y sub i times w dot x sub i plus b must be at least one. These inequalities say that every point lies outside the empty corridor on its proper side. Only a few of those constraints are tight at the solution. Those active constraints belong to the support vectors. The remaining points satisfy their inequalities with room to spare and do not pin the optimum. This completes the geometric derivation. A wide corridor became a small normal vector, correct classification became a family of inequalities, and the points touching equality became the support vectors."},{"title":"When the Classes Overlap","startSeconds":271.34333333333336,"narration":"Real classes often overlap. A red example may sit deep among blue examples, while a blue example may sit among red ones. No line can put every point on its requested side in this training set. If we insist on the hard-margin constraints, the optimization problem has no feasible answer. The red point below and the blue point above cannot both satisfy the same perfect separation. A soft-margin SVM relaxes each constraint with a nonnegative slack variable xi sub i. Instead of demanding a signed score of at least one, it permits the score to fall short by xi sub i. Draw the central separator and its two margin walls. A point outside its correct wall needs zero slack. A point inside the corridor needs some slack, because it has not reached the wall required by its label. This red point is not merely inside the margin. It lies on the blue side of the separator, so its slack exceeds one. The same is true in reverse for this blue point. Slack records the degree of each violation rather than pretending the data is perfectly separable. The new objective contains two costs. One half w dot w still rewards a wide margin. The sum of the slacks penalizes points that enter the margin or cross the separator, and the constant C controls how heavily that penalty is counted. This is not permission to ignore errors for free. Every violation raises the objective. The optimizer accepts a violation only when avoiding it would require a sufficiently narrower or more contorted decision boundary. With a smaller C, slack is relatively inexpensive. The optimizer is more willing to tolerate unusual training points in exchange for a broad, stable corridor. With a larger C, violations are expensive. The separator bends its linear choice as far as it can toward fitting difficult examples, often leaving a narrower margin. The drawing is schematic, but the trade is exact: margin width against total slack. Small C can underreact to genuine structure. Very large C can overreact to noise or mislabeled examples. In practice C is a regularization hyperparameter, chosen by validation rather than by demanding zero training error. Support vectors still matter in the soft-margin model. They now include points touching a margin wall and points lying inside or beyond it. These are the examples that exert force on the final decision boundary."},{"title":"No Straight Line Will Do","startSeconds":439.78439583333335,"narration":"Now consider a different training set. The red examples form a compact inner cluster, while the blue examples make a ring around them. This is a simple nonlinear pattern with no noise and no overlap. Try a slanted line. It can place some blue points opposite the red cluster, but blue points remain on both sides. Rotating the line merely changes which part of the ring is misclassified. Move the line above the centre instead. The red points may all fall below it, but so do the blue points around the lower half of the ring. No translation fixes the problem either. There is a geometric reason, not just a failure of imagination. Join the outer blue points to form their convex hull. The entire red cluster lies inside that hull. A straight line divides the plane into two half-planes. If every blue vertex lies in one half-plane, then every convex combination of those vertices lies there too. The red points inside their hull therefore cannot all occupy the opposite half-plane. The useful feature here is not left versus right or up versus down. It is distance from the centre. Square that distance and call it q of x: x one squared plus x two squared. Every red point has a small value of q. Every blue point has a much larger value. A threshold at two separates them perfectly: red below two, blue above two. Back in the original plane, the threshold q equals two is a circle, not a line. So the data is nonlinear in its original coordinates. The next move is to change coordinates rather than abandon maximum-margin separation."},{"title":"Lifting into Feature Space","startSeconds":546.5005625,"narration":"A feature map replaces each original input by a new vector. For this problem, map x one and x two to three quadratic features: x one squared, square root two times x one x two, and x two squared. At first I have placed the points in a flat copy of their familiar arrangement. Now apply the feature map. Each point moves to the three coordinates computed from its original two. The inner red points remain near the origin of feature space. The outer blue points rise onto a distant sheet because x one squared plus x two squared is large for every point on the ring. In this three-dimensional space, one flat plane separates the classes. Its equation is feature one plus feature three equals two. Red points lie below it, and blue points lie above it. This is ordinary linear separation again. We can place parallel margin planes around this separator, maximize their distance, and obtain support vectors exactly as before. The only change is that the vectors now live in feature space. Now compare the two spaces. On the left is the original input plane. On the right is the lifted feature space with its flat separating plane. The plane above is feature one plus feature three equals two. Substitute the definitions of those features and the same condition becomes x one squared plus x two squared equals two below. That equation draws this circle in the input plane. A linear decision surface in feature space has become a nonlinear decision boundary in the original coordinates. The construction works, but it seems expensive. A richer feature map may contain hundreds, millions, or even infinitely many coordinates. Writing and storing every lifted vector would defeat the purpose. The escape comes from noticing what the SVM calculation actually asks of those lifted vectors. It does not need to inspect their coordinates one at a time. It needs inner products between them."},{"title":"The Kernel Trick","startSeconds":679.4129583333333,"narration":"Return to the quadratic feature map. To compare two lifted examples phi of x and phi of z, the SVM needs their inner product. Let us calculate that inner product once and see what remains. Multiply matching lifted coordinates. The first contribution is x one squared z one squared. The middle contribution contains the factor two, and the last is x two squared z two squared. Those three terms form a familiar square. They equal x one z one plus x two z two, all squared. But x one z one plus x two z two is simply the ordinary dot product x dot z. Therefore the lifted inner product is the square of one dot product in the original two-dimensional space. Define K of x z to be that squared dot product. Evaluating K gives exactly the number we would have obtained by constructing both three-dimensional feature vectors and taking their inner product. That replacement is the kernel trick. We behave as if the data had been lifted, while every computation stays in the original input space. The feature coordinates disappear, but their geometry remains in the kernel values. Why are inner products enough? At the optimum, the normal vector in feature space can be written as a weighted sum of lifted training examples. Each weight is alpha sub i times the label y sub i. To classify a new input, begin with w dot phi of x plus b. Substitute the weighted expansion of w, and the calculation becomes a sum of inner products between training features and the new feature vector. Now replace every lifted inner product by K of x sub i and x. Prediction can be performed without forming w and without explicitly forming phi of any example. Most coefficients alpha sub i are zero. Only support vectors have nonzero coefficients and contribute to the final sum. The same small set of geometrically decisive points now makes prediction sparse. Training can also be written entirely in terms of pairwise kernels. The dual objective contains K of x sub i and x sub j for pairs of training examples, together with nonnegative coefficients whose label-weighted sum is zero. The optimization therefore needs a matrix of pairwise kernel values, often called the Gram matrix. Once those values are available, the algorithm can operate as though it knew every lifted coordinate. A kernel supplies exactly the required feature-space inner product. Different kernels encode different notions of similarity. The linear kernel is the ordinary dot product, so it gives the original linear SVM. A polynomial kernel raises a shifted dot product to a degree d. Our quadratic example used degree two with no shift. Higher degrees represent richer interactions among the original coordinates. The radial basis kernel decreases exponentially with squared distance between two inputs. It corresponds to a very rich feature space and produces flexible local boundaries, with gamma controlling how quickly similarity falls away. A kernel is not magic permission to use any similarity formula. It must behave like a genuine inner product in some feature space. When it does, the maximum-margin optimization remains mathematically well formed. Let us connect the whole argument. First, maximum margin chooses the separator with the widest empty corridor. The nearest points become support vectors because their constraints pin that corridor in place. When classes overlap, slack variables measure margin violations, and C trades a wider corridor against the cost of those violations. When no straight boundary works, a feature map can lift the examples into a space where a hyperplane does work. The decision surface is linear there even though it is curved in the original coordinates. Finally, the kernel trick computes only the inner products required by training and prediction. It preserves the lifted geometry without paying the cost of explicitly writing the lifted vectors. That is the support vector machine in one connected picture: maximize geometric separation, let a small set of points determine the answer, tolerate overlap when necessary, and use kernels when the useful separation lives in a nonlinear feature space."}]}}