{"version":1,"lectureId":"01M14TYV1ZG691V3XR3GYR0P92","attempt":1,"publication":{"slug":"principal-component-analysis-seen-geometrically","title":"PCA from Geometric Intuition to Eigenvectors","subject":"machine-learning","summary":"A geometric explanation of principal component analysis for working data scientists. Rotate a direction through an elongated point cloud, watch projected variance reach its maximum, and build the remaining orthogonal components. The lecture then derives the covariance eigenvalue problem as the algebraic form of that same search, before using eigenvalues for scree plots, component selection, and low-rank reconstruction. It closes with a direct account of why measurement units can dominate PCA and when scaling or standardization is needed.","metaDescription":"Understand PCA geometrically, connect maximum projected variance to covariance eigenvectors, and use scree plots, reconstruction, and scaling.","transcript":"Suppose you use PCA every week. You call a library, ask for two components, and get a transformed table. But what did the algorithm actually choose? Here is the question that organizes the whole method: if we flatten a centered cloud onto one line, which line preserves the most variation? Begin with a two-dimensional data set. Each blue point is one observation, and the two axes are two measured variables. The cloud is not round. It has a long direction running roughly from lower left to upper right. PCA starts by centering each variable. Subtract its sample mean, so the red mean moves to the origin. Centering matters because PCA is supposed to describe variation around the typical observation, not distance from an arbitrary zero. Now choose a candidate direction u. We require unit length, not because short arrows are virtuous, but because otherwise we could manufacture unlimited variance merely by multiplying the direction by a huge number. Project every observation onto that line. Each gray segment drops one blue point to its green projected position. The signed coordinate along the line is its score, z sub i, equal to u transpose times the centered observation. Those scores form a one-dimensional data set. Measure their variance in the ordinary way: square each score, average the squares, and call the result V of u. At the moment the line is horizontal. The green points have some spread, and the marker reports variance four. That is neither especially small nor the largest value available. Turn the candidate line toward the visible long axis of the cloud. The projected points separate, so their one-dimensional variance rises. It reaches seven at this direction, where the spread is greatest. Continue turning. The projections draw closer together again. On the vertical axis the variance has fallen back to four, even though no data point has moved. Keep turning toward the short axis of the cloud. The green points bunch tightly around the origin, and the variance falls to one. This direction preserves very little of the cloud's visible spread. Now return to the maximum. The data stayed fixed throughout. Only the question we asked of them changed: how much of their variation can be seen from this direction? This is PCA's central geometric operation. It searches over all unit directions, projects the centered observations, and chooses the direction whose scores have the largest variance. That maximizing direction is the first principal component. A component is a direction in variable space. The projected coordinate is the score of an observation on that component. Those are related objects, but they are not the same thing. Write the first component as v one: the unit direction that maximizes projected variance. Its variance, seven in this example, tells us how much variation the component captures. After choosing v one, PCA looks for another direction that captures as much remaining variance as possible without repeating the first answer. In two dimensions, there is only one perpendicular direction available. That green direction is the second principal component. The cloud is narrow along it, so its variance is only one. The first component describes the long way through the cloud, and the second describes the remaining side-to-side variation. The two component directions are perpendicular and have unit length. Together they form a rotated coordinate system. PCA has not bent the data or invented a nonlinear feature. It has rotated the axes to line up with the directions of greatest and least variation. So the first answer to carry forward is geometric. PCA asks where the centered cloud is widest, calls that direction component one, and then repeats the question in directions perpendicular to what it has already chosen. A real data set may have fifty variables, five thousand, or five million. We cannot rotate a literal line through all those dimensions on a screen, but the geometric question does not change. Use three dimensions as the bridge. This cloud has a long spatial direction, plus smaller deviations in two independent sideways directions. The first search is exactly the one we just performed. Among every unit direction in the space, find the one on which the projected coordinates have maximum variance. The red arrow is the answer. Projecting onto it compresses every observation to one score. Across all observations, those scores vary more than scores along any other unit direction. Now remove the possibility of choosing that answer again. Every direction perpendicular to v one lies in this gray plane through the origin. Search inside that plane for the direction with greatest projected variance. The green arrow is component two. It is the strongest remaining direction after component one has been excluded. In three dimensions, one perpendicular direction remains after the first two have been fixed. That yellow arrow becomes component three. In p dimensions the process continues the same way. Component k maximizes projected variance subject to unit length and orthogonality to every earlier component. Collect the first k component directions as columns of V sub k. Multiplying the centered data matrix X by V sub k produces the score matrix Z sub k. Each row is now the same observation expressed in principal-component coordinates. Because the component directions are mutually perpendicular unit vectors, V transpose V is the identity. This orthonormal structure is why moving into component coordinates and reconstructing from them remains simple. There cannot be more useful components than variables. There also cannot be more than n minus one nonzero directions of variation after n observations have been centered. The crucial point is that higher-dimensional PCA is not a new algorithmic idea. It is the same sequence of questions: widest direction first, then the widest perpendicular direction left, then the next. We now have the geometry. Only at this point do we need the covariance matrix, because it gives an efficient algebraic way to answer exactly the optimization question we have already understood. Return to the centered two-dimensional cloud. We already know the geometric problem: choose a unit direction u that makes the projected scores vary as much as possible. For one centered observation x sub i, its coordinate on u is the dot product u transpose x sub i. Call that score z sub i. The scores are centered too, so their variance is the average squared score. Substitute the dot product into that familiar calculation. A scalar square can be written as u transpose x sub i, times x sub i transpose u. Put the direction vectors outside the observation-specific middle. The average of those middle outer products is the covariance matrix Sigma. With centered observations stored in the rows of X, Sigma is one over n times X transpose X. So the variance seen along u is exactly the quadratic form u transpose Sigma u. This line is not a different objective. It is the projected spread from the rotating picture, written compactly. We want to maximize that quadratic form while keeping u at unit length. Use a Lagrange multiplier to attach the constraint u transpose u equals one. At a maximum, the derivative with respect to u is zero. Differentiating gives two Sigma u minus two lambda u. Cancel the common factor of two and rearrange. Sigma u equals lambda u. The maximizing direction must therefore be an eigenvector of the covariance matrix. This is the promised restatement. The geometry asked for a direction that keeps its identity while covariance assigns it a variance. The algebra calls that direction an eigenvector. Multiply the eigenvalue equation on the left by u transpose. Because u has unit length, u transpose Sigma u equals lambda. The eigenvalue is the projected variance along its eigenvector. Therefore the first component is the eigenvector with the largest eigenvalue. The second is the perpendicular eigenvector with the next eigenvalue, and so on in descending order. Covariance matrices are symmetric and positive semidefinite. Their eigenvectors can be chosen orthonormal, and their eigenvalues are nonnegative. Those facts are exactly what the geometric construction required. Now calculate the example. Use the covariance matrix with fours on the diagonal and threes off the diagonal. Its positive covariance says the two variables tend to rise and fall together. Eigenvalues make Sigma minus lambda I singular. Subtract lambda from both diagonal entries and set the determinant to zero. Multiply the main diagonal and subtract the other product. We get four minus lambda squared, minus nine, equal to zero. Expand the square and collect terms. The characteristic polynomial is lambda squared minus eight lambda plus seven. Factor it. Lambda minus seven times lambda minus one equals zero. The two eigenvalues are seven and one, exactly the maximum and minimum projected variances from the rotating line. The eigenvector for seven points along one, one. The other eigenvector points along one, minus one. Normalize both by dividing by square root two. Seven of the total eight variance units lie along component one. So one component explains eighty-seven point five percent of this cloud's variance. The eigenvalue calculation has returned every fact we first discovered by rotating and projecting. PCA orders components from most variance to least, but a model still needs a decision: how many components should we keep? The eigenvalues give the evidence for that decision. A scree plot puts the component number along the horizontal axis and its eigenvalue on the vertical axis. Here are five eigenvalues in descending order. The first point is high because component one captures a great deal of variation. The second captures less. After the third, the remaining eigenvalues are small. An individual explained-variance ratio divides one eigenvalue by the sum of all eigenvalues. It reports the fraction of total variance assigned to that component. Cumulative explained variance adds the first k eigenvalues before dividing by the total. It asks how much variation remains visible after reducing the data to k component coordinates. With one component, this example retains sixty-seven point six percent. With two, the total reaches eighty-eight point nine percent. With three components it reaches ninety-six point six percent. Components four and five add very little. The visible bend near component three is often called the elbow. One common heuristic keeps components before the curve settles into a shallow tail. But the elbow is not a theorem and ninety-five percent is not a universal law. Keep enough components for the downstream task. For prediction, choose k with cross-validation. For compression, choose an acceptable reconstruction error. For interpretation, check stability and domain meaning. A scree plot is diagnostic evidence, not an automatic command. A weak elbow may honestly mean there is no clean low-dimensional cutoff. Now make component retention concrete. In principal-component coordinates, a centered observation is built from one contribution along each component direction. This observation has a large score on component one, a smaller score on component two, and a still smaller score on component three. The general reconstruction formula starts at the original mean and adds back the first k score times direction contributions. Keep only component one. Starting from the mean, move along v one by the first score. The green point is the rank-one reconstruction. It preserves the strongest coordinate and discards everything in the omitted perpendicular subspace. The red dashed segment is the information lost for this observation. Now retain component two as well. Add the yellow contribution from the first reconstruction. The new point moves closer to the original. The remaining error lies entirely along component three. If we retain that final red contribution, the reconstruction reaches the original point exactly. The displayed error falls from one point three four with one component to zero point six with two. Every additional component can only reduce squared reconstruction error on the training data. Averaged over the data set, the squared error after keeping k components equals the sum of the omitted eigenvalues. The scree plot and reconstruction error are therefore two readings of the same tradeoff. Keeping a component means preserving one orthogonal pattern of variation. Dropping it means replacing every observation's score on that pattern by zero, then mapping the reduced coordinates back to the original variable space. One warning belongs in every practical explanation of PCA. PCA does not understand variables. It understands numbers, and it rewards whichever direction has the greatest numerical variance. On the left, salary deviation is measured in dollars and age deviation in years. These are centered observations, so subtracting the means has already been done. The salary coordinates contain tens of thousands while the age coordinates contain tens. The raw cloud therefore looks overwhelmingly horizontal, and raw PCA chooses an almost horizontal first component. That result does not prove salary is the most important scientific variable. It proves salary has the largest variance in the numbers handed to the algorithm. On the right, each variable has been centered and divided by its sample standard deviation. Both begin with variance one, so their relationship, rather than their original unit sizes, determines the component. The two answers differ even though they describe the same observations. Scaling has changed the geometry, and changing the geometry changes PCA. Here is the raw covariance matrix. Salary variance is four hundred million, while age variance is one hundred. The larger numerical scale dominates the optimization. Now perform a scientifically meaningless change: write the same salaries in cents instead of dollars. Every salary value is multiplied by one hundred, and its variance is multiplied by ten thousand. Raw PCA can rotate even though nothing about the people changed. That is the blunt warning. PCA maximizes numerical variance. It cannot distinguish signal from noise, importance from nuisance, or measurement scale from genuine structure. Standardization replaces each variable by its centered value divided by its sample standard deviation. The covariance matrix of standardized variables is their correlation matrix. Do not turn that into another automatic ritual. If a variable's original scale carries real meaning, raw covariance may be appropriate. If unlike units should receive equal starting weight, standardization is often necessary. Heavy tails, outliers, counts, and strongly skewed variables may call for robust scaling or a domain-specific transformation. Whatever you choose, document it and validate it as part of the model. PCA is now one connected idea. Center the observations, find the direction of maximum projected variance, continue in perpendicular directions, and compute those directions as covariance eigenvectors. Use eigenvalues to judge compression, reconstruct from retained scores, and never forget that the units help define the answer.","watch":{"version":1,"scenes":[{"title":"Variance in a Direction","start":0,"end":252.26064583333334,"objects":{"angle":"a VariableNumber","candidate":"a Line [yellow] labelled \"u\" drawn in plane (start=((-4.0 * cos(angle)), (-4.0 * sin(angle))), end=((4.0 * cos(angle)), (4.0 * sin(angle))))","centering":"a Math [text] that says \"$x_i arrow.r x_i - overline(x)$\"","centre":"a Point [red] labelled \"overline(x)\" drawn in plane","cloud":"a Point [blue] drawn in plane (location=(-3.1, -2.4), marker_radius=0.075)","cloud_10":"a Point [blue] drawn in plane (location=(2.0, 1.5), marker_radius=0.075)","cloud_11":"a Point [blue] drawn in plane (location=(2.6, 3.0), marker_radius=0.075)","cloud_12":"a Point [blue] drawn in plane (location=(3.1, 2.4), marker_radius=0.075)","cloud_2":"a Point [blue] drawn in plane (location=(-2.7, -3.0), marker_radius=0.075)","cloud_3":"a Point [blue] drawn in plane (location=(-2.2, -1.5), marker_radius=0.075)","cloud_4":"a Point [blue] drawn in plane (location=(-1.6, -2.1), marker_radius=0.075)","cloud_5":"a Point [blue] drawn in plane (location=(-1.1, -0.5), marker_radius=0.075)","cloud_6":"a Point [blue] drawn in plane (location=(-0.5, -1.0), marker_radius=0.075)","cloud_7":"a Point [blue] drawn in plane (location=(0.4, 0.9), marker_radius=0.075)","cloud_8":"a Point [blue] drawn in plane (location=(0.9, 0.4), marker_radius=0.075)","cloud_9":"a Point [blue] drawn in plane (location=(1.4, 2.0), marker_radius=0.075)","constraint":"a Math [text] that says \"$u^T u = 1$\"","heading":"a Heading that says \"Rotate a Direction and Watch the Spread\"","pc1":"a Math [text] that says \"$v_1 = arg max_(u^T u = 1) V(u)$\"","pc2":"a Math [text] that says \"$v_2 bot v_1$\"","pc2_line":"a Line [green] labelled \"v_2\" drawn in plane (start=(2.8, -2.8), end=(-2.8, 2.8))","plane":"an Axes (x_range=(-4.2, 4.2), y_range=(-4.2, 4.2), aspect=(1.0, 1.0))","projected":"a Point [green] drawn in plane (location=((((-3.1 * cos(angle)) + (-2.4 * sin(angle))) * cos(angle)), ((…, marker_radius=0.065)","projected_10":"a Point [green] drawn in plane (location=((((2.0 * cos(angle)) + (1.5 * sin(angle))) * cos(angle)), (((2…, marker_radius=0.065)","projected_11":"a Point [green] drawn in plane (location=((((2.6 * cos(angle)) + (3.0 * sin(angle))) * cos(angle)), (((2…, marker_radius=0.065)","projected_12":"a Point [green] drawn in plane (location=((((3.1 * cos(angle)) + (2.4 * sin(angle))) * cos(angle)), (((3…, marker_radius=0.065)","projected_2":"a Point [green] drawn in plane (location=((((-2.7 * cos(angle)) + (-3.0 * sin(angle))) * cos(angle)), ((…, marker_radius=0.065)","projected_3":"a Point [green] drawn in plane (location=((((-2.2 * cos(angle)) + (-1.5 * sin(angle))) * cos(angle)), ((…, marker_radius=0.065)","projected_4":"a Point [green] drawn in plane (location=((((-1.6 * cos(angle)) + (-2.1 * sin(angle))) * cos(angle)), ((…, marker_radius=0.065)","projected_5":"a Point [green] drawn in plane (location=((((-1.1 * cos(angle)) + (-0.5 * sin(angle))) * cos(angle)), ((…, marker_radius=0.065)","projected_6":"a Point [green] drawn in plane (location=((((-0.5 * cos(angle)) + (-1.0 * sin(angle))) * cos(angle)), ((…, marker_radius=0.065)","projected_7":"a Point [green] drawn in plane (location=((((0.4 * cos(angle)) + (0.9 * sin(angle))) * cos(angle)), (((0…, marker_radius=0.065)","projected_8":"a Point [green] drawn in plane (location=((((0.9 * cos(angle)) + (0.4 * sin(angle))) * cos(angle)), (((0…, marker_radius=0.065)","projected_9":"a Point [green] drawn in plane (location=((((1.4 * cos(angle)) + (2.0 * sin(angle))) * cos(angle)), (((1…, marker_radius=0.065)","question":"a Panel that says \"For a centered cloud of points, which line preserves the greatest amount of variation when the points are projected onto it?\"","residuals":"a Line [gray] drawn in plane (start=(-3.1, -2.4), end=((((-3.1 * cos(angle)) + (-2.4 * sin(angle))) * cos(angle)), ((…, dashed=True)","residuals_10":"a Line [gray] drawn in plane (start=(2.0, 1.5), end=((((2.0 * cos(angle)) + (1.5 * sin(angle))) * cos(angle)), (((2…, dashed=True)","residuals_11":"a Line [gray] drawn in plane (start=(2.6, 3.0), end=((((2.6 * cos(angle)) + (3.0 * sin(angle))) * cos(angle)), (((2…, dashed=True)","residuals_12":"a Line [gray] drawn in plane (start=(3.1, 2.4), end=((((3.1 * cos(angle)) + (2.4 * sin(angle))) * cos(angle)), (((3…, dashed=True)","residuals_2":"a Line [gray] drawn in plane (start=(-2.7, -3.0), end=((((-2.7 * cos(angle)) + (-3.0 * sin(angle))) * cos(angle)), ((…, dashed=True)","residuals_3":"a Line [gray] drawn in plane (start=(-2.2, -1.5), end=((((-2.2 * cos(angle)) + (-1.5 * sin(angle))) * cos(angle)), ((…, dashed=True)","residuals_4":"a Line [gray] drawn in plane (start=(-1.6, -2.1), end=((((-1.6 * cos(angle)) + (-2.1 * sin(angle))) * cos(angle)), ((…, dashed=True)","residuals_5":"a Line [gray] drawn in plane (start=(-1.1, -0.5), end=((((-1.1 * cos(angle)) + (-0.5 * sin(angle))) * cos(angle)), ((…, dashed=True)","residuals_6":"a Line [gray] drawn in plane (start=(-0.5, -1.0), end=((((-0.5 * cos(angle)) + (-1.0 * sin(angle))) * cos(angle)), ((…, dashed=True)","residuals_7":"a Line [gray] drawn in plane (start=(0.4, 0.9), end=((((0.4 * cos(angle)) + (0.9 * sin(angle))) * cos(angle)), (((0…, dashed=True)","residuals_8":"a Line [gray] drawn in plane (start=(0.9, 0.4), end=((((0.9 * cos(angle)) + (0.4 * sin(angle))) * cos(angle)), (((0…, dashed=True)","residuals_9":"a Line [gray] drawn in plane (start=(1.4, 2.0), end=((((1.4 * cos(angle)) + (2.0 * sin(angle))) * cos(angle)), (((1…, dashed=True)","score":"a Math [text] that says \"$z_i = u^T (x_i - overline(x))$\"","variance":"a Math [text] that says \"$V(u) = frac(1, n) sum_i z_i^2$\"","variance_line":"a NumberLine labelled \"V(u)\" (x_range=(0.0, 8.0), include_numbers=True)","variance_marker":"a Point [yellow] labelled \"V(u)\" drawn in variance_line (location=((4.0 + (3.0 * sin((2.0 * angle)))), 0.0))","variance_split":"a Math [text] that says \"$lambda_1 = 7, quad lambda_2 = 1$\""},"beats":[{"start":0,"say":"Suppose you use PCA every week. You call a library, ask for two components, and get a transformed table. But what did the algorithm actually choose? Here is the question that organizes the whole method: if we flatten a centered cloud onto one line, which line preserves the most variation?","live":[],"does":[[0,"question is shown on the screen, written out."],[18.379,"question is hidden from the screen — left the board."]]},{"start":19.579,"say":"Begin with a two-dimensional data set. Each blue point is one observation, and the two axes are two measured variables. The cloud is not round. It has a long direction running roughly from lower left to upper right.","live":null,"does":[[19.579,"plane is shown on the screen, written out."],[22.934,"cloud is shown on the screen, written out."],[22.934,"cloud_2 is shown on the screen, written out."],[22.934,"cloud_3 is shown on the screen, written out."],[22.934,"cloud_4 is shown on the screen, written out."],[22.934,"cloud_5 is shown on the screen, written out."],[22.934,"cloud_6 is shown on the screen, written out."],[22.934,"cloud_7 is shown on the screen, written out."],[22.934,"cloud_8 is shown on the screen, written out."],[22.934,"cloud_9 is shown on the screen, written out."],[22.934,"cloud_10 is shown on the screen, written out."],[22.934,"cloud_11 is shown on the screen, written out."],[22.934,"cloud_12 is shown on the screen, written out."],[28.24,"centre is shown on the screen, written out."]]},{"start":34.366,"say":"PCA starts by centering each variable. Subtract its sample mean, so the red mean moves to the origin. Centering matters because PCA is supposed to describe variation around the typical observation, not distance from an arbitrary zero.","live":["plane","cloud","cloud_2","cloud_3","cloud_4","cloud_5","cloud_6","cloud_7","cloud_8","cloud_9","cloud_10","cloud_11","cloud_12","centre"],"does":[[35.933,"plane moves to a new place on the board."],[35.933,"centering is shown on the screen, written out."],[41.158,"centre is indicated — a transient flash."]]},{"start":50.941500000000005,"say":"Now choose a candidate direction u. We require unit length, not because short arrows are virtuous, but because otherwise we could manufacture unlimited variance merely by multiplying the direction by a huge number.","live":["centering","plane","cloud","cloud_2","cloud_3","cloud_4","cloud_5","cloud_6","cloud_7","cloud_8","cloud_9","cloud_10","cloud_11","cloud_12","centre"],"does":[[52.30000000000001,"candidate is shown on the screen, written out."],[54.64500000000001,"constraint is shown on the screen, written out."]]},{"start":64.23150000000001,"say":"Project every observation onto that line. Each gray segment drops one blue point to its green projected position. The signed coordinate along the line is its score, z sub i, equal to u transpose times the centered observation.","live":["centering","constraint","plane","cloud","cloud_2","cloud_3","cloud_4","cloud_5","cloud_6","cloud_7","cloud_8","cloud_9","cloud_10","cloud_11","cloud_12","centre","candidate"],"does":[[67.72600000000001,"residuals is shown on the screen, written out."],[67.72600000000001,"residuals_2 is shown on the screen, written out."],[67.72600000000001,"residuals_3 is shown on the screen, written out."],[67.72600000000001,"residuals_4 is shown on the screen, written out."],[67.72600000000001,"residuals_5 is shown on the screen, written out."],[67.72600000000001,"residuals_6 is shown on the screen, written out."],[67.72600000000001,"residuals_7 is shown on the screen, written out."],[67.72600000000001,"residuals_8 is shown on the screen, written out."],[67.72600000000001,"residuals_9 is shown on the screen, written out."],[67.72600000000001,"residuals_10 is shown on the screen, written out."],[67.72600000000001,"residuals_11 is shown on the screen, written out."],[67.72600000000001,"residuals_12 is shown on the screen, written out."],[70.02500000000002,"projected is shown on the screen, written out."],[70.02500000000002,"projected_2 is shown on the screen, written out."],[70.02500000000002,"projected_3 is shown on the screen, written out."],[70.02500000000002,"projected_4 is shown on the screen, written out."],[70.02500000000002,"projected_5 is shown on the screen, written out."],[70.02500000000002,"projected_6 is shown on the screen, written out."],[70.02500000000002,"projected_7 is shown on the screen, written out."],[70.02500000000002,"projected_8 is shown on the screen, written out."],[70.02500000000002,"projected_9 is shown on the screen, written out."],[70.02500000000002,"projected_10 is shown on the screen, written out."],[70.02500000000002,"projected_11 is shown on the screen, written out."],[70.02500000000002,"projected_12 is shown on the screen, written out."],[74.19300000000001,"score is shown on the screen, written out."]]},{"start":80.08700000000002,"say":"Those scores form a one-dimensional data set. Measure their variance in the ordinary way: square each score, average the squares, and call the result V of u.","live":["centering","score","constraint","plane","cloud","cloud_2","cloud_3","cloud_4","cloud_5","cloud_6","cloud_7","cloud_8","cloud_9","cloud_10","cloud_11","cloud_12","centre","candidate","residuals","residuals_2","residuals_3","residuals_4","residuals_5","residuals_6","residuals_7","residuals_8","residuals_9","residuals_10","residuals_11","residuals_12","projected","projected_2","projected_3","projected_4","projected_5","projected_6","projected_7","projected_8","projected_9","projected_10","projected_11","projected_12"],"does":[[84.15100000000002,"variance is shown on the screen, written out."],[89.50300000000003,"variance_line is shown on the screen, written out."],[89.50300000000003,"variance_marker is shown on the screen, written out."]]},{"start":91.83300000000001,"say":"At the moment the line is horizontal. The green points have some spread, and the marker reports variance four. That is neither especially small nor the largest value available.","live":["centering","score","variance","constraint","variance_line","plane","cloud","cloud_2","cloud_3","cloud_4","cloud_5","cloud_6","cloud_7","cloud_8","cloud_9","cloud_10","cloud_11","cloud_12","centre","candidate","residuals","residuals_2","residuals_3","residuals_4","residuals_5","residuals_6","residuals_7","residuals_8","residuals_9","residuals_10","residuals_11","residuals_12","projected","projected_2","projected_3","projected_4","projected_5","projected_6","projected_7","projected_8","projected_9","projected_10","projected_11","projected_12","variance_marker"],"does":[[98.14900000000002,"variance_marker is indicated — a transient flash."]]},{"start":103.53250000000001,"say":"Turn the candidate line toward the visible long axis of the cloud. The projected points separate, so their one-dimensional variance rises. It reaches seven at this direction, where the spread is greatest.","live":null,"does":[[103.88100000000001,"candidate is redrawn as the numbers it depends on change."],[103.88100000000001,"residuals is redrawn as the numbers it depends on change."],[103.88100000000001,"residuals_2 is redrawn as the numbers it depends on change."],[103.88100000000001,"residuals_3 is redrawn as the numbers it depends on change."],[103.88100000000001,"residuals_4 is redrawn as the numbers it depends on change."],[103.88100000000001,"residuals_5 is redrawn as the numbers it depends on change."],[103.88100000000001,"residuals_6 is redrawn as the numbers it depends on change."],[103.88100000000001,"residuals_7 is redrawn as the numbers it depends on change."],[103.88100000000001,"residuals_8 is redrawn as the numbers it depends on change."],[103.88100000000001,"residuals_9 is redrawn as the numbers it depends on change."],[103.88100000000001,"residuals_10 is redrawn as the numbers it depends on change."],[103.88100000000001,"residuals_11 is redrawn as the numbers it depends on change."],[103.88100000000001,"residuals_12 is redrawn as the numbers it depends on change."],[103.88100000000001,"projected is redrawn as the numbers it depends on change."],[103.88100000000001,"projected_2 is redrawn as the numbers it depends on change."],[103.88100000000001,"projected_3 is redrawn as the numbers it depends on change."],[103.88100000000001,"projected_4 is redrawn as the numbers it depends on change."],[103.88100000000001,"projected_5 is redrawn as the numbers it depends on change."],[103.88100000000001,"projected_6 is redrawn as the numbers it depends on change."],[103.88100000000001,"projected_7 is redrawn as the numbers it depends on change."],[103.88100000000001,"projected_8 is redrawn as the numbers it depends on change."],[103.88100000000001,"projected_9 is redrawn as the numbers it depends on change."],[103.88100000000001,"projected_10 is redrawn as the numbers it depends on change."],[103.88100000000001,"projected_11 is redrawn as the numbers it depends on change."],[103.88100000000001,"projected_12 is redrawn as the numbers it depends on change."],[103.88100000000001,"variance_marker is redrawn as the numbers it depends on change."],[103.88100000000001,"angle ticks to 0.7853981633974483."],[112.97200000000001,"variance_marker is indicated — a transient flash."]]},{"start":116.47400000000002,"say":"Continue turning. The projections draw closer together again. On the vertical axis the variance has fallen back to four, even though no data point has moved.","live":null,"does":[[116.77600000000002,"candidate is redrawn as the numbers it depends on change."],[116.77600000000002,"residuals is redrawn as the numbers it depends on change."],[116.77600000000002,"residuals_2 is redrawn as the numbers it depends on change."],[116.77600000000002,"residuals_3 is redrawn as the numbers it depends on change."],[116.77600000000002,"residuals_4 is redrawn as the numbers it depends on change."],[116.77600000000002,"residuals_5 is redrawn as the numbers it depends on change."],[116.77600000000002,"residuals_6 is redrawn as the numbers it depends on change."],[116.77600000000002,"residuals_7 is redrawn as the numbers it depends on change."],[116.77600000000002,"residuals_8 is redrawn as the numbers it depends on change."],[116.77600000000002,"residuals_9 is redrawn as the numbers it depends on change."],[116.77600000000002,"residuals_10 is redrawn as the numbers it depends on change."],[116.77600000000002,"residuals_11 is redrawn as the numbers it depends on change."],[116.77600000000002,"residuals_12 is redrawn as the numbers it depends on change."],[116.77600000000002,"projected is redrawn as the numbers it depends on change."],[116.77600000000002,"projected_2 is redrawn as the numbers it depends on change."],[116.77600000000002,"projected_3 is redrawn as the numbers it depends on change."],[116.77600000000002,"projected_4 is redrawn as the numbers it depends on change."],[116.77600000000002,"projected_5 is redrawn as the numbers it depends on change."],[116.77600000000002,"projected_6 is redrawn as the numbers it depends on change."],[116.77600000000002,"projected_7 is redrawn as the numbers it depends on change."],[116.77600000000002,"projected_8 is redrawn as the numbers it depends on change."],[116.77600000000002,"projected_9 is redrawn as the numbers it depends on change."],[116.77600000000002,"projected_10 is redrawn as the numbers it depends on change."],[116.77600000000002,"projected_11 is redrawn as the numbers it depends on change."],[116.77600000000002,"projected_12 is redrawn as the numbers it depends on change."],[116.77600000000002,"variance_marker is redrawn as the numbers it depends on change."],[116.77600000000002,"angle ticks to 1.5707963267948966."],[124.10200000000002,"variance_marker is indicated — a transient flash."]]},{"start":127.43050000000002,"say":"Keep turning toward the short axis of the cloud. The green points bunch tightly around the origin, and the variance falls to one. This direction preserves very little of the cloud's visible spread.","live":null,"does":[[127.77900000000002,"candidate is redrawn as the numbers it depends on change."],[127.77900000000002,"residuals is redrawn as the numbers it depends on change."],[127.77900000000002,"residuals_2 is redrawn as the numbers it depends on change."],[127.77900000000002,"residuals_3 is redrawn as the numbers it depends on change."],[127.77900000000002,"residuals_4 is redrawn as the numbers it depends on change."],[127.77900000000002,"residuals_5 is redrawn as the numbers it depends on change."],[127.77900000000002,"residuals_6 is redrawn as the numbers it depends on change."],[127.77900000000002,"residuals_7 is redrawn as the numbers it depends on change."],[127.77900000000002,"residuals_8 is redrawn as the numbers it depends on change."],[127.77900000000002,"residuals_9 is redrawn as the numbers it depends on change."],[127.77900000000002,"residuals_10 is redrawn as the numbers it depends on change."],[127.77900000000002,"residuals_11 is redrawn as the numbers it depends on change."],[127.77900000000002,"residuals_12 is redrawn as the numbers it depends on change."],[127.77900000000002,"projected is redrawn as the numbers it depends on change."],[127.77900000000002,"projected_2 is redrawn as the numbers it depends on change."],[127.77900000000002,"projected_3 is redrawn as the numbers it depends on change."],[127.77900000000002,"projected_4 is redrawn as the numbers it depends on change."],[127.77900000000002,"projected_5 is redrawn as the numbers it depends on change."],[127.77900000000002,"projected_6 is redrawn as the numbers it depends on change."],[127.77900000000002,"projected_7 is redrawn as the numbers it depends on change."],[127.77900000000002,"projected_8 is redrawn as the numbers it depends on change."],[127.77900000000002,"projected_9 is redrawn as the numbers it depends on change."],[127.77900000000002,"projected_10 is redrawn as the numbers it depends on change."],[127.77900000000002,"projected_11 is redrawn as the numbers it depends on change."],[127.77900000000002,"projected_12 is redrawn as the numbers it depends on change."],[127.77900000000002,"variance_marker is redrawn as the numbers it depends on change."],[127.77900000000002,"angle ticks to 2.356194490192345."],[134.55900000000003,"variance_marker is indicated — a transient flash."]]},{"start":139.88500000000002,"say":"Now return to the maximum. The data stayed fixed throughout. Only the question we asked of them changed: how much of their variation can be seen from this direction?","live":null,"does":[[140.44200000000004,"candidate is redrawn as the numbers it depends on change."],[140.44200000000004,"residuals is redrawn as the numbers it depends on change."],[140.44200000000004,"residuals_2 is redrawn as the numbers it depends on change."],[140.44200000000004,"residuals_3 is redrawn as the numbers it depends on change."],[140.44200000000004,"residuals_4 is redrawn as the numbers it depends on change."],[140.44200000000004,"residuals_5 is redrawn as the numbers it depends on change."],[140.44200000000004,"residuals_6 is redrawn as the numbers it depends on change."],[140.44200000000004,"residuals_7 is redrawn as the numbers it depends on change."],[140.44200000000004,"residuals_8 is redrawn as the numbers it depends on change."],[140.44200000000004,"residuals_9 is redrawn as the numbers it depends on change."],[140.44200000000004,"residuals_10 is redrawn as the numbers it depends on change."],[140.44200000000004,"residuals_11 is redrawn as the numbers it depends on change."],[140.44200000000004,"residuals_12 is redrawn as the numbers it depends on change."],[140.44200000000004,"projected is redrawn as the numbers it depends on change."],[140.44200000000004,"projected_2 is redrawn as the numbers it depends on change."],[140.44200000000004,"projected_3 is redrawn as the numbers it depends on change."],[140.44200000000004,"projected_4 is redrawn as the numbers it depends on change."],[140.44200000000004,"projected_5 is redrawn as the numbers it depends on change."],[140.44200000000004,"projected_6 is redrawn as the numbers it depends on change."],[140.44200000000004,"projected_7 is redrawn as the numbers it depends on change."],[140.44200000000004,"projected_8 is redrawn as the numbers it depends on change."],[140.44200000000004,"projected_9 is redrawn as the numbers it depends on change."],[140.44200000000004,"projected_10 is redrawn as the numbers it depends on change."],[140.44200000000004,"projected_11 is redrawn as the numbers it depends on change."],[140.44200000000004,"projected_12 is redrawn as the numbers it depends on change."],[140.44200000000004,"variance_marker is redrawn as the numbers it depends on change."],[140.44200000000004,"angle ticks to 0.7853981633974483."]]},{"start":150.58550000000002,"say":"This is PCA's central geometric operation. It searches over all unit directions, projects the centered observations, and chooses the direction whose scores have the largest variance.","live":null,"does":[[159.49,"candidate is indicated — a transient flash."],[160.81400000000002,"variance_marker is indicated — a transient flash."]]},{"start":162.81850000000003,"say":"That maximizing direction is the first principal component. A component is a direction in variable space. The projected coordinate is the score of an observation on that component. Those are related objects, but they are not the same thing.","live":null,"does":[[164.96600000000004,"residuals is hidden from the screen."],[164.96600000000004,"residuals_2 is hidden from the screen."],[164.96600000000004,"residuals_3 is hidden from the screen."],[164.96600000000004,"residuals_4 is hidden from the screen."],[164.96600000000004,"residuals_5 is hidden from the screen."],[164.96600000000004,"residuals_6 is hidden from the screen."],[164.96600000000004,"residuals_7 is hidden from the screen."],[164.96600000000004,"residuals_8 is hidden from the screen."],[164.96600000000004,"residuals_9 is hidden from the screen."],[164.96600000000004,"residuals_10 is hidden from the screen."],[164.96600000000004,"residuals_11 is hidden from the screen."],[164.96600000000004,"residuals_12 is hidden from the screen."],[177.93400000000003,"centering is hidden from the screen — left the board."],[177.93400000000003,"constraint is hidden from the screen — left the board."],[177.93400000000003,"score is hidden from the screen — left the board."],[177.93400000000003,"variance is hidden from the screen — left the board."],[177.93400000000003,"variance_line is hidden from the screen — left the board."],[177.93400000000003,"variance_marker is hidden from the screen — variance_line left the board."]]},{"start":179.13400000000001,"say":"Write the first component as v one: the unit direction that maximizes projected variance. Its variance, seven in this example, tells us how much variation the component captures.","live":["plane","cloud","cloud_2","cloud_3","cloud_4","cloud_5","cloud_6","cloud_7","cloud_8","cloud_9","cloud_10","cloud_11","cloud_12","centre","candidate","projected","projected_2","projected_3","projected_4","projected_5","projected_6","projected_7","projected_8","projected_9","projected_10","projected_11","projected_12"],"does":[[179.13400000000001,"heading is shown on the screen, written out."],[180.01700000000005,"pc1 is shown on the screen, written out."],[186.77400000000006,"variance_split is shown on the screen, written out."]]},{"start":191.786,"say":"After choosing v one, PCA looks for another direction that captures as much remaining variance as possible without repeating the first answer. In two dimensions, there is only one perpendicular direction available.","live":["plane","cloud","cloud_2","cloud_3","cloud_4","cloud_5","cloud_6","cloud_7","cloud_8","cloud_9","cloud_10","cloud_11","cloud_12","centre","candidate","projected","projected_2","projected_3","projected_4","projected_5","projected_6","projected_7","projected_8","projected_9","projected_10","projected_11","projected_12","heading","pc1","variance_split"],"does":[[194.967,"pc2_line is shown on the screen, written out."],[202.47800000000004,"pc2 is shown on the screen, written out."]]},{"start":205.11,"say":"That green direction is the second principal component. The cloud is narrow along it, so its variance is only one. The first component describes the long way through the cloud, and the second describes the remaining side-to-side variation.","live":["plane","cloud","cloud_2","cloud_3","cloud_4","cloud_5","cloud_6","cloud_7","cloud_8","cloud_9","cloud_10","cloud_11","cloud_12","centre","candidate","projected","projected_2","projected_3","projected_4","projected_5","projected_6","projected_7","projected_8","projected_9","projected_10","projected_11","projected_12","heading","pc1","variance_split","pc2","pc2_line"],"does":[[205.71400000000006,"pc2_line is indicated — a transient flash."],[213.00500000000005,"variance_split (the \"lambda_1 = 7\" part) is emphasized."],[216.19700000000006,"variance_split (the \"lambda_1 = 7\" part) is no longer emphasized."],[216.19700000000006,"variance_split (the \"lambda_2 = 1\" part) is emphasized."],[219.39000000000001,"variance_split (the \"lambda_2 = 1\" part) is no longer emphasized."]]},{"start":219.99,"say":"The two component directions are perpendicular and have unit length. Together they form a rotated coordinate system. PCA has not bent the data or invented a nonlinear feature. It has rotated the axes to line up with the directions of greatest and least variation.","live":null,"does":[[225.57500000000007,"candidate is indicated — a transient flash."],[233.07500000000007,"pc2_line is indicated — a transient flash."]]},{"start":237.63350000000003,"say":"So the first answer to carry forward is geometric. PCA asks where the centered cloud is widest, calls that direction component one, and then repeats the question in directions perpendicular to what it has already chosen.","live":null,"does":[[251.21897916666668,"heading is hidden from the screen — left the board."],[251.21897916666668,"pc1 is hidden from the screen — left the board."],[251.21897916666668,"pc2 is hidden from the screen — left the board."],[251.21897916666668,"plane is hidden from the screen — left the board."],[251.21897916666668,"cloud is hidden from the screen — plane left the board."],[251.21897916666668,"cloud_2 is hidden from the screen — plane left the board."],[251.21897916666668,"cloud_3 is hidden from the screen — plane left the board."],[251.21897916666668,"cloud_4 is hidden from the screen — plane left the board."],[251.21897916666668,"cloud_5 is hidden from the screen — plane left the board."],[251.21897916666668,"cloud_6 is hidden from the screen — plane left the board."],[251.21897916666668,"cloud_7 is hidden from the screen — plane left the board."],[251.21897916666668,"cloud_8 is hidden from the screen — plane left the board."],[251.21897916666668,"cloud_9 is hidden from the screen — plane left the board."],[251.21897916666668,"cloud_10 is hidden from the screen — plane left the board."],[251.21897916666668,"cloud_11 is hidden from the screen — plane left the board."],[251.21897916666668,"cloud_12 is hidden from the screen — plane left the board."],[251.21897916666668,"centre is hidden from the screen — plane left the board."],[251.21897916666668,"candidate is hidden from the screen — plane left the board."],[251.21897916666668,"projected is hidden from the screen — plane left the board."],[251.21897916666668,"projected_2 is hidden from the screen — plane left the board."],[251.21897916666668,"projected_3 is hidden from the screen — plane left the board."],[251.21897916666668,"projected_4 is hidden from the screen — plane left the board."],[251.21897916666668,"projected_5 is hidden from the screen — plane left the board."],[251.21897916666668,"projected_6 is hidden from the screen — plane left the board."],[251.21897916666668,"projected_7 is hidden from the screen — plane left the board."],[251.21897916666668,"projected_8 is hidden from the screen — plane left the board."],[251.21897916666668,"projected_9 is hidden from the screen — plane left the board."],[251.21897916666668,"projected_10 is hidden from the screen — plane left the board."],[251.21897916666668,"projected_11 is hidden from the screen — plane left the board."],[251.21897916666668,"projected_12 is hidden from the screen — plane left the board."],[251.21897916666668,"pc2_line is hidden from the screen — plane left the board."],[251.21897916666668,"variance_split is hidden from the screen — left the board."]]}]},{"title":"More Than Two Dimensions","start":252.26064583333334,"end":417.72210416666667,"objects":{"axes":"an Axes3D (x_range=(-3.2, 3.2), y_range=(-3.2, 3.2), z_range=(-3.2, 3.2))","basis":"a Math [text] that says \"$V^T V = I$\"","cloud":"a Point [blue] drawn in axes (location=(-2.6, -1.8, -1.2), marker_radius=0.07)","cloud_10":"a Point [blue] drawn in axes (location=(1.8, 1.5, 0.5), marker_radius=0.07)","cloud_11":"a Point [blue] drawn in axes (location=(2.3, 1.4, 1.4), marker_radius=0.07)","cloud_12":"a Point [blue] drawn in axes (location=(2.7, 2.1, 0.8), marker_radius=0.07)","cloud_2":"a Point [blue] drawn in axes (location=(-2.2, -1.8, -0.5), marker_radius=0.07)","cloud_3":"a Point [blue] drawn in axes (location=(-1.8, -1.0, -1.1), marker_radius=0.07)","cloud_4":"a Point [blue] drawn in axes (location=(-1.3, -1.2, 0.0), marker_radius=0.07)","cloud_5":"a Point [blue] drawn in axes (location=(-0.8, -0.2, -0.9), marker_radius=0.07)","cloud_6":"a Point [blue] drawn in axes (location=(-0.3, -0.5, 0.5), marker_radius=0.07)","cloud_7":"a Point [blue] drawn in axes (location=(0.4, 0.1, 0.8), marker_radius=0.07)","cloud_8":"a Point [blue] drawn in axes (location=(0.8, 0.9, 0.1), marker_radius=0.07)","cloud_9":"a Point [blue] drawn in axes (location=(1.3, 0.7, 1.1), marker_radius=0.07)","constraints":"a Math [text] that says \"$u^T u = 1, quad u^T v_j = 0 thin upright(\"for\") thin j < k$\"","first_rule":"a Math [text] that says \"$v_1 = arg max_(u^T u = 1) upright(\"Var\")(X u)$\"","heading":"a Heading that says \"The Same Search in Higher Dimensions\"","limit":"a Math [text] that says \"$k <= min(p, n - 1)$\"","next_rule":"a Math [text] that says \"$v_k = arg max upright(\"Var\")(X u)$\"","orthogonal_plane":"a Plane [gray] labelled \"v_1^bot\" drawn in axes (normal=(2.4, 1.8, 0.9), edge_direction=(-0.6, 0.8, 0.0), size=4.5)","pc1":"a Vector [red] labelled \"v_1\" drawn in axes (start=(0.0, 0.0, 0.0), end=(2.4, 1.8, 0.9))","pc2":"a Vector [green] labelled \"v_2\" drawn in axes (start=(0.0, 0.0, 0.0), end=(-1.2, 1.6, 0.0))","pc3":"a Vector [yellow] labelled \"v_3\" drawn in axes (start=(0.0, 0.0, 0.0), end=(-0.6, -0.45, 2.5))","scores":"a Math [text] that says \"$Z_k = X V_k$\""},"beats":[{"start":252.26064583333334,"say":"A real data set may have fifty variables, five thousand, or five million. We cannot rotate a literal line through all those dimensions on a screen, but the geometric question does not change.","live":[],"does":[[252.26064583333334,"heading is shown on the screen, written out."],[260.2016458333333,"axes is shown on the screen, written out."]]},{"start":265.0861458333333,"say":"Use three dimensions as the bridge. This cloud has a long spatial direction, plus smaller deviations in two independent sideways directions.","live":["axes","heading"],"does":[[268.2666458333333,"cloud is shown on the screen, written out."],[268.2666458333333,"cloud_2 is shown on the screen, written out."],[268.2666458333333,"cloud_3 is shown on the screen, written out."],[268.2666458333333,"cloud_4 is shown on the screen, written out."],[268.2666458333333,"cloud_5 is shown on the screen, written out."],[268.2666458333333,"cloud_6 is shown on the screen, written out."],[268.2666458333333,"cloud_7 is shown on the screen, written out."],[268.2666458333333,"cloud_8 is shown on the screen, written out."],[268.2666458333333,"cloud_9 is shown on the screen, written out."],[268.2666458333333,"cloud_10 is shown on the screen, written out."],[268.2666458333333,"cloud_11 is shown on the screen, written out."],[268.2666458333333,"cloud_12 is shown on the screen, written out."],[269.23064583333337,"axes turns in its own slot."]]},{"start":275.43864583333334,"say":"The first search is exactly the one we just performed. Among every unit direction in the space, find the one on which the projected coordinates have maximum variance.","live":["axes","heading","cloud","cloud_2","cloud_3","cloud_4","cloud_5","cloud_6","cloud_7","cloud_8","cloud_9","cloud_10","cloud_11","cloud_12"],"does":[[276.09964583333334,"axes moves to a new place on the board."],[276.09964583333334,"first_rule is shown on the screen, written out."],[280.60464583333334,"pc1 is shown on the screen, written out."]]},{"start":286.58064583333334,"say":"The red arrow is the answer. Projecting onto it compresses every observation to one score. Across all observations, those scores vary more than scores along any other unit direction.","live":["first_rule","axes","heading","cloud","cloud_2","cloud_3","cloud_4","cloud_5","cloud_6","cloud_7","cloud_8","cloud_9","cloud_10","cloud_11","cloud_12","pc1"],"does":[[287.1846458333333,"pc1 is indicated — a transient flash."]]},{"start":299.84714583333334,"say":"Now remove the possibility of choosing that answer again. Every direction perpendicular to v one lies in this gray plane through the origin.","live":null,"does":[[307.2306458333333,"orthogonal_plane is shown on the screen, written out."],[307.5216458333333,"A quad of data points in axes is lit up."],[309.1466458333333,"axes: retire a lit plane (unemphasize_plane)."]]},{"start":309.74664583333333,"say":"Search inside that plane for the direction with greatest projected variance. The green arrow is component two. It is the strongest remaining direction after component one has been excluded.","live":["first_rule","axes","heading","cloud","cloud_2","cloud_3","cloud_4","cloud_5","cloud_6","cloud_7","cloud_8","cloud_9","cloud_10","cloud_11","cloud_12","pc1","orthogonal_plane"],"does":[[310.09464583333335,"next_rule is shown on the screen, written out."],[314.66964583333333,"pc2 is shown on the screen, written out."],[320.42864583333335,"constraints is shown on the screen, written out."]]},{"start":322.0501458333333,"say":"In three dimensions, one perpendicular direction remains after the first two have been fixed. That yellow arrow becomes component three.","live":["first_rule","next_rule","constraints","axes","heading","cloud","cloud_2","cloud_3","cloud_4","cloud_5","cloud_6","cloud_7","cloud_8","cloud_9","cloud_10","cloud_11","cloud_12","pc1","orthogonal_plane","pc2"],"does":[[328.74864583333334,"pc3 is shown on the screen, written out."]]},{"start":331.6126458333333,"say":"In p dimensions the process continues the same way. Component k maximizes projected variance subject to unit length and orthogonality to every earlier component.","live":["first_rule","next_rule","constraints","axes","heading","cloud","cloud_2","cloud_3","cloud_4","cloud_5","cloud_6","cloud_7","cloud_8","cloud_9","cloud_10","cloud_11","cloud_12","pc1","orthogonal_plane","pc2","pc3"],"does":[[335.5136458333333,"next_rule is indicated — a transient flash."],[339.27564583333333,"constraints (the \"u^T u = 1\" part) is emphasized."],[340.51764583333335,"constraints (the \"u^T u = 1\" part) is no longer emphasized."],[340.51764583333335,"constraints (the \"u^T v_j = 0 thin upright(\"for\") thin j < k\" part) is emphasized."],[343.35014583333333,"constraints (the \"u^T v_j = 0 thin upright(\"for\") thin j < k\" part) is no longer emphasized."]]},{"start":343.95014583333335,"say":"Collect the first k component directions as columns of V sub k. Multiplying the centered data matrix X by V sub k produces the score matrix Z sub k. Each row is now the same observation expressed in principal-component coordinates.","live":null,"does":[[344.29864583333335,"scores is shown on the screen, written out."]]},{"start":362.2671458333333,"say":"Because the component directions are mutually perpendicular unit vectors, V transpose V is the identity. This orthonormal structure is why moving into component coordinates and reconstructing from them remains simple.","live":["first_rule","next_rule","constraints","scores","axes","heading","cloud","cloud_2","cloud_3","cloud_4","cloud_5","cloud_6","cloud_7","cloud_8","cloud_9","cloud_10","cloud_11","cloud_12","pc1","orthogonal_plane","pc2","pc3"],"does":[[368.52464583333335,"basis is shown on the screen, written out."],[370.23164583333335,"pc1 is indicated — a transient flash."],[370.43164583333333,"pc2 is indicated — a transient flash."],[370.6316458333333,"pc3 is indicated — a transient flash."]]},{"start":376.79914583333334,"say":"There cannot be more useful components than variables. There also cannot be more than n minus one nonzero directions of variation after n observations have been centered.","live":["first_rule","next_rule","constraints","scores","basis","axes","heading","cloud","cloud_2","cloud_3","cloud_4","cloud_5","cloud_6","cloud_7","cloud_8","cloud_9","cloud_10","cloud_11","cloud_12","pc1","orthogonal_plane","pc2","pc3"],"does":[[377.3916458333333,"limit is shown on the screen, written out."]]},{"start":389.26514583333335,"say":"The crucial point is that higher-dimensional PCA is not a new algorithmic idea. It is the same sequence of questions: widest direction first, then the widest perpendicular direction left, then the next.","live":["first_rule","next_rule","constraints","scores","basis","limit","axes","heading","cloud","cloud_2","cloud_3","cloud_4","cloud_5","cloud_6","cloud_7","cloud_8","cloud_9","cloud_10","cloud_11","cloud_12","pc1","orthogonal_plane","pc2","pc3"],"does":[[399.0756458333333,"pc1 is indicated — a transient flash."],[399.86464583333327,"pc2 is indicated — a transient flash."],[403.0696458333333,"pc3 is indicated — a transient flash."]]},{"start":404.48214583333333,"say":"We now have the geometry. Only at this point do we need the covariance matrix, because it gives an efficient algebraic way to answer exactly the optimization question we have already understood.","live":null,"does":[[416.68043750000004,"axes is hidden from the screen — left the board."],[416.68043750000004,"cloud is hidden from the screen — axes left the board."],[416.68043750000004,"cloud_2 is hidden from the screen — axes left the board."],[416.68043750000004,"cloud_3 is hidden from the screen — axes left the board."],[416.68043750000004,"cloud_4 is hidden from the screen — axes left the board."],[416.68043750000004,"cloud_5 is hidden from the screen — axes left the board."],[416.68043750000004,"cloud_6 is hidden from the screen — axes left the board."],[416.68043750000004,"cloud_7 is hidden from the screen — axes left the board."],[416.68043750000004,"cloud_8 is hidden from the screen — axes left the board."],[416.68043750000004,"cloud_9 is hidden from the screen — axes left the board."],[416.68043750000004,"cloud_10 is hidden from the screen — axes left the board."],[416.68043750000004,"cloud_11 is hidden from the screen — axes left the board."],[416.68043750000004,"cloud_12 is hidden from the screen — axes left the board."],[416.68043750000004,"pc1 is hidden from the screen — axes left the board."],[416.68043750000004,"orthogonal_plane is hidden from the screen — axes left the board."],[416.68043750000004,"pc2 is hidden from the screen — axes left the board."],[416.68043750000004,"pc3 is hidden from the screen — axes left the board."],[416.68043750000004,"basis is hidden from the screen — left the board."],[416.68043750000004,"constraints is hidden from the screen — left the board."],[416.68043750000004,"first_rule is hidden from the screen — left the board."],[416.68043750000004,"heading is hidden from the screen — left the board."],[416.68043750000004,"limit is hidden from the screen — left the board."],[416.68043750000004,"next_rule is hidden from the screen — left the board."],[416.68043750000004,"scores is hidden from the screen — left the board."]]}]},{"title":"The Covariance Eigenproblem","start":417.72210416666667,"end":657.4953958333333,"objects":{"axes":"a Figure (x_range=(-4.2, 4.2), y_range=(-4.2, 4.2), aspect=(1.0, 1.0))","cloud":"a Point [blue] drawn in axes (location=(-3.1, -2.4), marker_radius=0.07)","cloud_10":"a Point [blue] drawn in axes (location=(2.0, 1.5), marker_radius=0.07)","cloud_11":"a Point [blue] drawn in axes (location=(2.6, 3.0), marker_radius=0.07)","cloud_12":"a Point [blue] drawn in axes (location=(3.1, 2.4), marker_radius=0.07)","cloud_2":"a Point [blue] drawn in axes (location=(-2.7, -3.0), marker_radius=0.07)","cloud_3":"a Point [blue] drawn in axes (location=(-2.2, -1.5), marker_radius=0.07)","cloud_4":"a Point [blue] drawn in axes (location=(-1.6, -2.1), marker_radius=0.07)","cloud_5":"a Point [blue] drawn in axes (location=(-1.1, -0.5), marker_radius=0.07)","cloud_6":"a Point [blue] drawn in axes (location=(-0.5, -1.0), marker_radius=0.07)","cloud_7":"a Point [blue] drawn in axes (location=(0.4, 0.9), marker_radius=0.07)","cloud_8":"a Point [blue] drawn in axes (location=(0.9, 0.4), marker_radius=0.07)","cloud_9":"a Point [blue] drawn in axes (location=(1.4, 2.0), marker_radius=0.07)","covariance":"a Math [text] that says \"$Sigma = frac(1, n) X^T X$\"","d3":"a Math [text] that says \"$&= u^T Sigma u$\"","determinant":"a Math [text] that says \"$op(\"det\")(Sigma - lambda I) = op(\"det\") mat(4-lambda, 3; 3, 4-lambda)$\"","eigenproblem":"a Math [text] that says \"$Sigma u = lambda u$\"","gradient":"a Math [text] that says \"$frac(partial L, partial u) = 2 Sigma u - 2 lambda u = 0$\"","heading_example":"a Heading that says \"The Elongated Cloud, Calculated\"","heading_lagrange":"a Heading that says \"The Maximizer Must Be an Eigenvector\"","heading_share":"a Heading that says \"The Eigenvalues Measure Explained Variance\"","heading_variance":"a Heading that says \"Write the Geometric Question Algebraically\"","lagrangian":"a Math [text] that says \"$L(u, lambda) = u^T Sigma u - lambda (u^T u - 1)$\"","meaning":"a Math [text] that says \"$u^T Sigma u = lambda$\"","polynomial":"a Math [text] that says \"$(4-lambda)^2 - 9 = 0$\"","ranking":"a Math [text] that says \"$lambda_1 >= lambda_2 >= dots >= 0$\"","share":"a Math [text] that says \"$frac(lambda_1, lambda_1 + lambda_2) = frac(7, 8) = 87.5%$\"","sigma":"a Math [text] that says \"$Sigma = mat(4, 3; 3, 4)$\"","v1_line":"a Line [red] labelled \"v_1\" drawn in axes (start=(-3.7, -3.7), end=(3.7, 3.7))","v2_line":"a Line [green] labelled \"v_2\" drawn in axes (start=(3.7, -3.7), end=(-3.7, 3.7))","values":"a Math [text] that says \"$lambda_1 = 7, quad lambda_2 = 1$\"","variance_derivation":"a Derivation [text] that says \"$z_i &= u^T x_i \\ upright(\"Var\")(z) &= frac(1, n) sum_i (u^T x_i)^2 \\ &= frac(1, n) sum_i u^T x_i x_i^T u \\ &= u^T Sigma u$\"","vectors":"a Math [text] that says \"$v_1 = frac(1, sqrt(2)) vec(1, 1), quad v_2 = frac(1, sqrt(2)) vec(1, -1)$\""},"beats":[{"start":417.72210416666667,"say":"Return to the centered two-dimensional cloud. We already know the geometric problem: choose a unit direction u that makes the projected scores vary as much as possible.","live":[],"does":[[417.72210416666667,"heading_variance is shown on the screen, written out."],[419.5221041666667,"axes is shown on the screen, written out."],[419.5221041666667,"cloud is shown on the screen, written out."],[419.5221041666667,"cloud_2 is shown on the screen, written out."],[419.5221041666667,"cloud_3 is shown on the screen, written out."],[419.5221041666667,"cloud_4 is shown on the screen, written out."],[419.5221041666667,"cloud_5 is shown on the screen, written out."],[419.5221041666667,"cloud_6 is shown on the screen, written out."],[419.5221041666667,"cloud_7 is shown on the screen, written out."],[419.5221041666667,"cloud_8 is shown on the screen, written out."],[419.5221041666667,"cloud_9 is shown on the screen, written out."],[419.5221041666667,"cloud_10 is shown on the screen, written out."],[419.5221041666667,"cloud_11 is shown on the screen, written out."],[419.5221041666667,"cloud_12 is shown on the screen, written out."]]},{"start":428.6201041666667,"say":"For one centered observation x sub i, its coordinate on u is the dot product u transpose x sub i. Call that score z sub i.","live":["axes","heading_variance","cloud","cloud_2","cloud_3","cloud_4","cloud_5","cloud_6","cloud_7","cloud_8","cloud_9","cloud_10","cloud_11","cloud_12"],"does":[[437.2931041666667,"axes moves to a new place on the board."],[437.2931041666667,"variance_derivation is shown on the screen, written out."]]},{"start":439.5536041666667,"say":"The scores are centered too, so their variance is the average squared score. Substitute the dot product into that familiar calculation.","live":null,"does":[[442.1661041666667,"variance_derivation is shown on the screen, written out."]]},{"start":449.17460416666665,"say":"A scalar square can be written as u transpose x sub i, times x sub i transpose u. Put the direction vectors outside the observation-specific middle.","live":null,"does":[[450.81110416666667,"variance_derivation is shown on the screen, written out."],[456.90610416666664,"variance_derivation (the \"u^T\" part) is emphasized."],[459.8781041666667,"variance_derivation (the \"u#2\" part) is emphasized."],[459.8781041666667,"variance_derivation (the \"u^T\" part) is no longer emphasized."],[460.3486041666667,"variance_derivation (the \"u#2\" part) is no longer emphasized."]]},{"start":460.94860416666666,"say":"The average of those middle outer products is the covariance matrix Sigma. With centered observations stored in the rows of X, Sigma is one over n times X transpose X.","live":null,"does":[[463.2651041666667,"covariance is shown on the screen, written out."],[464.5531041666667,"variance_derivation is shown on the screen, written out."]]},{"start":472.50860416666666,"say":"So the variance seen along u is exactly the quadratic form u transpose Sigma u. This line is not a different objective. It is the projected spread from the rotating picture, written compactly.","live":["covariance","axes","heading_variance","cloud","cloud_2","cloud_3","cloud_4","cloud_5","cloud_6","cloud_7","cloud_8","cloud_9","cloud_10","cloud_11","cloud_12"],"does":[[474.8131041666667,"variance_derivation (the \"u^T Sigma u\" part) is emphasized."],[485.55210416666665,"covariance is hidden from the screen — left the board."],[485.55210416666665,"heading_variance is hidden from the screen — left the board."],[485.55210416666665,"variance_derivation is hidden from the screen — left the board."],[485.55210416666665,"variance_derivation (the \"u^T Sigma u\" part) is no longer emphasized."]]},{"start":486.1521041666667,"say":"We want to maximize that quadratic form while keeping u at unit length. Use a Lagrange multiplier to attach the constraint u transpose u equals one.","live":["axes","cloud","cloud_2","cloud_3","cloud_4","cloud_5","cloud_6","cloud_7","cloud_8","cloud_9","cloud_10","cloud_11","cloud_12"],"does":[[486.1521041666667,"heading_lagrange is shown on the screen, written out."],[492.0271041666667,"lagrangian is shown on the screen, written out."]]},{"start":497.2011041666667,"say":"At a maximum, the derivative with respect to u is zero. Differentiating gives two Sigma u minus two lambda u.","live":["axes","cloud","cloud_2","cloud_3","cloud_4","cloud_5","cloud_6","cloud_7","cloud_8","cloud_9","cloud_10","cloud_11","cloud_12","lagrangian","heading_lagrange"],"does":[[501.6011041666667,"gradient is shown on the screen, written out."]]},{"start":506.2761041666667,"say":"Cancel the common factor of two and rearrange. Sigma u equals lambda u. The maximizing direction must therefore be an eigenvector of the covariance matrix.","live":["axes","cloud","cloud_2","cloud_3","cloud_4","cloud_5","cloud_6","cloud_7","cloud_8","cloud_9","cloud_10","cloud_11","cloud_12","lagrangian","gradient","heading_lagrange"],"does":[[510.27010416666667,"eigenproblem is shown on the screen, written out."],[515.7731041666667,"A box is drawn around eigenproblem."]]},{"start":519.2291041666667,"say":"This is the promised restatement. The geometry asked for a direction that keeps its identity while covariance assigns it a variance. The algebra calls that direction an eigenvector.","live":["axes","cloud","cloud_2","cloud_3","cloud_4","cloud_5","cloud_6","cloud_7","cloud_8","cloud_9","cloud_10","cloud_11","cloud_12","lagrangian","gradient","eigenproblem","heading_lagrange"],"does":[[523.3161041666667,"v1_line is shown on the screen, written out."]]},{"start":532.2291041666667,"say":"Multiply the eigenvalue equation on the left by u transpose. Because u has unit length, u transpose Sigma u equals lambda. The eigenvalue is the projected variance along its eigenvector.","live":["axes","cloud","cloud_2","cloud_3","cloud_4","cloud_5","cloud_6","cloud_7","cloud_8","cloud_9","cloud_10","cloud_11","cloud_12","lagrangian","gradient","eigenproblem","heading_lagrange","v1_line"],"does":[[533.3201041666666,"meaning is shown on the screen, written out."],[544.6991041666666,"meaning (the \"lambda\" part) is emphasized."],[546.8116041666667,"meaning (the \"lambda\" part) is no longer emphasized."]]},{"start":547.4116041666666,"say":"Therefore the first component is the eigenvector with the largest eigenvalue. The second is the perpendicular eigenvector with the next eigenvalue, and so on in descending order.","live":["axes","cloud","cloud_2","cloud_3","cloud_4","cloud_5","cloud_6","cloud_7","cloud_8","cloud_9","cloud_10","cloud_11","cloud_12","meaning","lagrangian","gradient","eigenproblem","heading_lagrange","v1_line"],"does":[[552.1721041666667,"v2_line is shown on the screen, written out."],[556.5251041666667,"ranking is shown on the screen, written out."]]},{"start":558.6116041666667,"say":"Covariance matrices are symmetric and positive semidefinite. Their eigenvectors can be chosen orthonormal, and their eigenvalues are nonnegative. Those facts are exactly what the geometric construction required.","live":["axes","cloud","cloud_2","cloud_3","cloud_4","cloud_5","cloud_6","cloud_7","cloud_8","cloud_9","cloud_10","cloud_11","cloud_12","meaning","lagrangian","gradient","eigenproblem","ranking","heading_lagrange","v1_line","v2_line"],"does":[[565.2641041666666,"v1_line is indicated — a transient flash."],[565.4641041666666,"v2_line is indicated — a transient flash."],[572.6246041666667,"axes is hidden from the screen — left the board."],[572.6246041666667,"cloud is hidden from the screen — axes left the board."],[572.6246041666667,"cloud_2 is hidden from the screen — axes left the board."],[572.6246041666667,"cloud_3 is hidden from the screen — axes left the board."],[572.6246041666667,"cloud_4 is hidden from the screen — axes left the board."],[572.6246041666667,"cloud_5 is hidden from the screen — axes left the board."],[572.6246041666667,"cloud_6 is hidden from the screen — axes left the board."],[572.6246041666667,"cloud_7 is hidden from the screen — axes left the board."],[572.6246041666667,"cloud_8 is hidden from the screen — axes left the board."],[572.6246041666667,"cloud_9 is hidden from the screen — axes left the board."],[572.6246041666667,"cloud_10 is hidden from the screen — axes left the board."],[572.6246041666667,"cloud_11 is hidden from the screen — axes left the board."],[572.6246041666667,"cloud_12 is hidden from the screen — axes left the board."],[572.6246041666667,"v1_line is hidden from the screen — axes left the board."],[572.6246041666667,"v2_line is hidden from the screen — axes left the board."],[572.6246041666667,"eigenproblem is hidden from the screen — left the board."],[572.6246041666667,"gradient is hidden from the screen — left the board."],[572.6246041666667,"heading_lagrange is hidden from the screen — left the board."],[572.6246041666667,"lagrangian is hidden from the screen — left the board."],[572.6246041666667,"meaning is hidden from the screen — left the board."],[572.6246041666667,"ranking is hidden from the screen — left the board."]]},{"start":573.2246041666667,"say":"Now calculate the example. Use the covariance matrix with fours on the diagonal and threes off the diagonal. Its positive covariance says the two variables tend to rise and fall together.","live":[],"does":[[573.2246041666667,"heading_example is shown on the screen, written out."],[576.5341041666667,"sigma is shown on the screen, written out."]]},{"start":586.5371041666667,"say":"Eigenvalues make Sigma minus lambda I singular. Subtract lambda from both diagonal entries and set the determinant to zero.","live":["sigma","heading_example"],"does":[[591.0541041666667,"determinant is shown on the screen, written out."],[594.4901041666667,"determinant (the \"diagonal=main\" part) is struck through — it is ruled out."],[594.6901041666666,"determinant (the \"diagonal=anti\" part) is struck through — it is ruled out."]]},{"start":596.5876041666667,"say":"Multiply the main diagonal and subtract the other product. We get four minus lambda squared, minus nine, equal to zero.","live":["sigma","determinant","heading_example"],"does":[[599.0371041666667,"The strike through determinant (the \"diagonal=main\" part) is lifted."],[599.2371041666667,"The strike through determinant (the \"diagonal=anti\" part) is lifted."],[602.8221041666667,"polynomial is shown on the screen, written out."]]},{"start":606.7081041666667,"say":"Expand the square and collect terms. The characteristic polynomial is lambda squared minus eight lambda plus seven.","live":["sigma","determinant","polynomial","heading_example"],"does":[[607.0561041666666,"polynomial becomes \"$lambda^2 - 8 lambda + 7 = 0$\"."]]},{"start":615.5396041666667,"say":"Factor it. Lambda minus seven times lambda minus one equals zero. The two eigenvalues are seven and one, exactly the maximum and minimum projected variances from the rotating line.","live":null,"does":[[615.5396041666667,"values is shown on the screen, written out."],[615.8881041666666,"polynomial becomes \"$(lambda - 7)(lambda - 1) = 0$\"."],[618.4771041666667,"values (the \"7\" part) is emphasized."],[620.1491041666666,"values (the \"1\" part) is emphasized."],[620.1491041666666,"values (the \"7\" part) is no longer emphasized."],[629.2396041666666,"values (the \"1\" part) is no longer emphasized."]]},{"start":629.8396041666667,"say":"The eigenvector for seven points along one, one. The other eigenvector points along one, minus one. Normalize both by dividing by square root two.","live":["sigma","determinant","polynomial","values","heading_example"],"does":[[630.4671041666667,"vectors is shown on the screen, written out."],[631.2681041666667,"vectors (the \"frac(1, sqrt(2)) vec(1, 1)\" part) is emphasized."],[634.1821041666667,"vectors (the \"frac(1, sqrt(2)) vec(1, -1)\" part) is emphasized."],[634.1821041666667,"vectors (the \"frac(1, sqrt(2)) vec(1, 1)\" part) is no longer emphasized."],[640.3236041666667,"determinant is hidden from the screen — left the board."],[640.3236041666667,"heading_example is hidden from the screen — left the board."],[640.3236041666667,"polynomial is hidden from the screen — left the board."],[640.3236041666667,"sigma is hidden from the screen — left the board."],[640.3236041666667,"values is hidden from the screen — left the board."],[640.3236041666667,"vectors is hidden from the screen — left the board."],[640.3236041666667,"vectors (the \"frac(1, sqrt(2)) vec(1, -1)\" part) is no longer emphasized."]]},{"start":641.5236041666667,"say":"Seven of the total eight variance units lie along component one. So one component explains eighty-seven point five percent of this cloud's variance. The eigenvalue calculation has returned every fact we first discovered by rotating and projecting.","live":[],"does":[[641.5236041666667,"heading_share is shown on the screen, written out."],[641.8251041666666,"share is shown on the screen, written out."],[642.9751041666666,"A box is drawn around share."],[656.4537291666667,"heading_share is hidden from the screen — left the board."],[656.4537291666667,"share is hidden from the screen — left the board."]]}]},{"title":"Scree Plots and Reconstruction","start":657.4953958333333,"end":879.6621666666666,"objects":{"average_error":"a Math [text] that says \"$frac(1, n) sum_i norm(x_i-hat(x)_(i,k))^2 = sum_(j>k) lambda_j$\"","cumulative":"a Math [text] that says \"$C_k = frac(sum_(j=1)^k lambda_j, sum_l lambda_l)$\"","elbow":"a Point [yellow] labelled \"upright(\"elbow\")\" drawn in scree (location=(3.0, 0.8))","error_one":"a Line [red] drawn in recon_axes (start=(2.2, 0.0, 0.0), end=(2.2, 1.2, 0.6), dashed=True)","error_two":"a Line [red] drawn in recon_axes (start=(2.2, 1.2, 0.0), end=(2.2, 1.2, 0.6), dashed=True)","errors":"a Math [text] that says \"$norm(x-hat(x)_1) = 1.34, quad norm(x-hat(x)_2) = 0.60$\"","first_contribution":"a Vector [green] labelled \"2.2 v_1\" drawn in recon_axes (start=(0.0, 0.0, 0.0), end=(2.2, 0.0, 0.0))","first_reconstruction":"a Point [green] labelled \"hat(x)_1\" drawn in recon_axes (location=(2.2, 0.0, 0.0))","full":"a Math [text] that says \"$x = overline(x) + 2.2 v_1 + 1.2 v_2 + 0.6 v_3$\"","general":"a Math [text] that says \"$hat(x)_k = overline(x) + sum_(j=1)^k z_j v_j$\"","heading_reconstruction":"a Heading that says \"What Keeping Components Actually Does\"","heading_scree":"a Heading that says \"How Many Components Should We Keep?\"","mean":"a Point [gray] labelled \"overline(x)\" drawn in recon_axes (location=(0.0, 0.0, 0.0))","one":"a Math [text] that says \"$hat(x)_1 = overline(x) + 2.2 v_1$\"","original":"a Point [red] labelled \"x\" drawn in recon_axes (location=(2.2, 1.2, 0.6))","ratio":"a Math [text] that says \"$upright(\"explained ratio\")_j = frac(lambda_j, sum_l lambda_l)$\"","recon_axes":"an Axes3D (x_range=(0.0, 2.8), y_range=(0.0, 1.8), z_range=(0.0, 1.2))","scree":"an Axes (x_range=(0.5, 5.5), y_range=(0.0, 8.0), x_ticks_every=1.0)","scree_lines":"a Line [gray] drawn in scree (start=(1.0, 7.0), end=(2.0, 2.2))","scree_lines_2":"a Line [gray] drawn in scree (start=(2.0, 2.2), end=(3.0, 0.8))","scree_lines_3":"a Line [gray] drawn in scree (start=(3.0, 0.8), end=(4.0, 0.25))","scree_lines_4":"a Line [gray] drawn in scree (start=(4.0, 0.25), end=(5.0, 0.1))","scree_points":"a Point [red] labelled \"7.0\" drawn in scree (location=(1.0, 7.0))","scree_points_2":"a Point [green] labelled \"2.2\" drawn in scree (location=(2.0, 2.2))","scree_points_3":"a Point [yellow] labelled \"0.8\" drawn in scree (location=(3.0, 0.8))","scree_points_4":"a Point [blue] labelled \"0.25\" drawn in scree (location=(4.0, 0.25))","scree_points_5":"a Point [magenta] labelled \"0.10\" drawn in scree (location=(5.0, 0.1))","second_contribution":"a Vector [yellow] labelled \"1.2 v_2\" drawn in recon_axes (start=(2.2, 0.0, 0.0), end=(2.2, 1.2, 0.0))","second_reconstruction":"a Point [yellow] labelled \"hat(x)_2\" drawn in recon_axes (location=(2.2, 1.2, 0.0))","table":"a Table [text] that says \"Kept Individual Cumulative 1 67.6% 67.6% 2 21.3% 88.9% 3 7.7% 96.6% 4 2.4% 99.0% 5 1.0% 100%\" (rows=(('Kept', 'Individual', 'Cumulative'), ('1', '67.6%', '67.6%'),…, header=True)","third_contribution":"a Vector [red] labelled \"0.6 v_3\" drawn in recon_axes (start=(2.2, 1.2, 0.0), end=(2.2, 1.2, 0.6))","two":"a Math [text] that says \"$hat(x)_2 = overline(x) + 2.2 v_1 + 1.2 v_2$\""},"beats":[{"start":657.4953958333333,"say":"PCA orders components from most variance to least, but a model still needs a decision: how many components should we keep? The eigenvalues give the evidence for that decision.","live":[],"does":[[657.4953958333333,"heading_scree is shown on the screen, written out."],[665.6223958333333,"scree is shown on the screen, written out."]]},{"start":669.0668958333333,"say":"A scree plot puts the component number along the horizontal axis and its eigenvalue on the vertical axis. Here are five eigenvalues in descending order.","live":["scree","heading_scree"],"does":[[669.8793958333333,"scree_lines is shown on the screen, written out."],[669.8793958333333,"scree_lines_2 is shown on the screen, written out."],[669.8793958333333,"scree_lines_3 is shown on the screen, written out."],[669.8793958333333,"scree_lines_4 is shown on the screen, written out."],[676.9043958333333,"scree_points is shown on the screen, written out."],[676.9043958333333,"scree_points_2 is shown on the screen, written out."],[676.9043958333333,"scree_points_3 is shown on the screen, written out."],[676.9043958333333,"scree_points_4 is shown on the screen, written out."],[676.9043958333333,"scree_points_5 is shown on the screen, written out."]]},{"start":679.9303958333333,"say":"The first point is high because component one captures a great deal of variation. The second captures less. After the third, the remaining eigenvalues are small.","live":["scree","heading_scree","scree_lines","scree_lines_2","scree_lines_3","scree_lines_4","scree_points","scree_points_2","scree_points_3","scree_points_4","scree_points_5"],"does":[[680.4063958333334,"scree_points is indicated — a transient flash."],[685.5033958333333,"scree_points_2 is indicated — a transient flash."],[688.1153958333333,"scree_points_3 is indicated — a transient flash."]]},{"start":691.6878958333333,"say":"An individual explained-variance ratio divides one eigenvalue by the sum of all eigenvalues. It reports the fraction of total variance assigned to that component.","live":null,"does":[[693.8243958333333,"scree moves to a new place on the board."],[693.8243958333333,"ratio is shown on the screen, written out."]]},{"start":703.2013958333333,"say":"Cumulative explained variance adds the first k eigenvalues before dividing by the total. It asks how much variation remains visible after reducing the data to k component coordinates.","live":["ratio","scree","heading_scree","scree_lines","scree_lines_2","scree_lines_3","scree_lines_4","scree_points","scree_points_2","scree_points_3","scree_points_4","scree_points_5"],"does":[[703.6773958333333,"cumulative is shown on the screen, written out."],[711.2003958333333,"table is shown on the screen, written out."]]},{"start":715.8753958333333,"say":"With one component, this example retains sixty-seven point six percent. With two, the total reaches eighty-eight point nine percent.","live":["ratio","cumulative","scree","heading_scree","scree_lines","scree_lines_2","scree_lines_3","scree_lines_4","scree_points","scree_points_2","scree_points_3","scree_points_4","scree_points_5"],"does":[[716.5953958333333,"table is shown on the screen, written out."],[721.5763958333333,"table is shown on the screen, written out."],[723.3643958333333,"table (the \"row=3\" part) is emphasized."],[725.1638958333333,"table (the \"row=3\" part) is no longer emphasized."]]},{"start":725.7638958333333,"say":"With three components it reaches ninety-six point six percent. Components four and five add very little.","live":null,"does":[[726.3563958333333,"table is shown on the screen, written out."],[730.5583958333333,"table is shown on the screen, written out."],[731.0583958333333,"table is shown on the screen, written out."]]},{"start":733.5968958333333,"say":"The visible bend near component three is often called the elbow. One common heuristic keeps components before the curve settles into a shallow tail.","live":null,"does":[[736.8943958333333,"elbow is shown on the screen, written out."]]},{"start":743.3458958333333,"say":"But the elbow is not a theorem and ninety-five percent is not a universal law. Keep enough components for the downstream task. For prediction, choose k with cross-validation. For compression, choose an acceptable reconstruction error. For interpretation, check stability and domain meaning.","live":["ratio","cumulative","scree","heading_scree","scree_lines","scree_lines_2","scree_lines_3","scree_lines_4","scree_points","scree_points_2","scree_points_3","scree_points_4","scree_points_5","elbow"],"does":[[745.3773958333333,"table (the \"96.6%\" part) is indicated — a transient flash."]]},{"start":762.2438958333333,"say":"A scree plot is diagnostic evidence, not an automatic command. A weak elbow may honestly mean there is no clean low-dimensional cutoff.","live":null,"does":[[771.3343958333332,"cumulative is hidden from the screen — left the board."],[771.3343958333332,"heading_scree is hidden from the screen — left the board."],[771.3343958333332,"ratio is hidden from the screen — left the board."],[771.3343958333332,"scree is hidden from the screen — left the board."],[771.3343958333332,"scree_lines is hidden from the screen — scree left the board."],[771.3343958333332,"scree_lines_2 is hidden from the screen — scree left the board."],[771.3343958333332,"scree_lines_3 is hidden from the screen — scree left the board."],[771.3343958333332,"scree_lines_4 is hidden from the screen — scree left the board."],[771.3343958333332,"scree_points is hidden from the screen — scree left the board."],[771.3343958333332,"scree_points_2 is hidden from the screen — scree left the board."],[771.3343958333332,"scree_points_3 is hidden from the screen — scree left the board."],[771.3343958333332,"scree_points_4 is hidden from the screen — scree left the board."],[771.3343958333332,"scree_points_5 is hidden from the screen — scree left the board."],[771.3343958333332,"elbow is hidden from the screen — scree left the board."],[771.3343958333332,"table is hidden from the screen — left the board."]]},{"start":772.5343958333333,"say":"Now make component retention concrete. In principal-component coordinates, a centered observation is built from one contribution along each component direction.","live":[],"does":[[772.5343958333333,"heading_reconstruction is shown on the screen, written out."],[777.3753958333333,"recon_axes is shown on the screen, written out."],[778.4323958333333,"mean is shown on the screen, written out."]]},{"start":783.6768958333333,"say":"This observation has a large score on component one, a smaller score on component two, and a still smaller score on component three.","live":["recon_axes","heading_reconstruction","mean"],"does":[[784.2573958333332,"original is shown on the screen, written out."],[785.5923958333333,"full is shown on the screen, written out."],[786.0913958333333,"recon_axes turns in its own slot."]]},{"start":792.7173958333333,"say":"The general reconstruction formula starts at the original mean and adds back the first k score times direction contributions.","live":["full","recon_axes","heading_reconstruction","mean","original"],"does":[[794.3773958333333,"general is shown on the screen, written out."]]},{"start":800.7013958333333,"say":"Keep only component one. Starting from the mean, move along v one by the first score. The green point is the rank-one reconstruction.","live":["general","full","recon_axes","heading_reconstruction","mean","original"],"does":[[801.7343958333333,"one is shown on the screen, written out."],[804.3353958333332,"first_contribution is shown on the screen, written out."],[807.3653958333333,"first_reconstruction is shown on the screen, written out."],[808.7123958333333,"error_one is shown on the screen, written out."]]},{"start":810.3808958333333,"say":"It preserves the strongest coordinate and discards everything in the omitted perpendicular subspace. The red dashed segment is the information lost for this observation.","live":["general","full","one","recon_axes","heading_reconstruction","mean","original","first_contribution","first_reconstruction","error_one"],"does":[[818.2753958333333,"error_one is indicated — a transient flash."]]},{"start":820.6983958333333,"say":"Now retain component two as well. Add the yellow contribution from the first reconstruction. The new point moves closer to the original.","live":null,"does":[[821.6153958333333,"two is shown on the screen, written out."],[823.9493958333333,"second_contribution is shown on the screen, written out."],[827.6883958333333,"second_reconstruction is shown on the screen, written out."],[827.6883958333333,"error_one is hidden from the screen."],[827.6883958333333,"error_two is shown on the screen, written out."]]},{"start":829.8323958333333,"say":"The remaining error lies entirely along component three. If we retain that final red contribution, the reconstruction reaches the original point exactly.","live":["general","full","one","two","recon_axes","heading_reconstruction","mean","original","first_contribution","first_reconstruction","second_contribution","second_reconstruction","error_two"],"does":[[835.1383958333333,"third_contribution is shown on the screen, written out."],[838.4353958333334,"error_two is hidden from the screen."]]},{"start":840.1498958333333,"say":"The displayed error falls from one point three four with one component to zero point six with two. Every additional component can only reduce squared reconstruction error on the training data.","live":["general","full","one","two","recon_axes","heading_reconstruction","mean","original","first_contribution","first_reconstruction","second_contribution","second_reconstruction","third_contribution"],"does":[[841.1833958333333,"errors is shown on the screen, written out."],[841.9723958333333,"errors (the \"1.34\" part) is emphasized."],[844.3063958333332,"errors (the \"0.60\" part) is emphasized."],[844.3063958333332,"errors (the \"1.34\" part) is no longer emphasized."],[851.4228958333333,"errors (the \"0.60\" part) is no longer emphasized."]]},{"start":852.0228958333332,"say":"Averaged over the data set, the squared error after keeping k components equals the sum of the omitted eigenvalues. The scree plot and reconstruction error are therefore two readings of the same tradeoff.","live":["general","full","one","two","errors","recon_axes","heading_reconstruction","mean","original","first_contribution","first_reconstruction","second_contribution","second_reconstruction","third_contribution"],"does":[[852.3713958333333,"average_error is shown on the screen, written out."],[857.6533958333333,"average_error (the \"sum_(j>k) lambda_j\" part) is emphasized."],[864.7353958333333,"average_error (the \"sum_(j>k) lambda_j\" part) is no longer emphasized."]]},{"start":865.3353958333332,"say":"Keeping a component means preserving one orthogonal pattern of variation. Dropping it means replacing every observation's score on that pattern by zero, then mapping the reduced coordinates back to the original variable space.","live":["general","full","one","two","errors","average_error","recon_axes","heading_reconstruction","mean","original","first_contribution","first_reconstruction","second_contribution","second_reconstruction","third_contribution"],"does":[[878.6205,"average_error is hidden from the screen — left the board."],[878.6205,"errors is hidden from the screen — left the board."],[878.6205,"full is hidden from the screen — left the board."],[878.6205,"general is hidden from the screen — left the board."],[878.6205,"heading_reconstruction is hidden from the screen — left the board."],[878.6205,"one is hidden from the screen — left the board."],[878.6205,"recon_axes is hidden from the screen — left the board."],[878.6205,"mean is hidden from the screen — recon_axes left the board."],[878.6205,"original is hidden from the screen — recon_axes left the board."],[878.6205,"first_contribution is hidden from the screen — recon_axes left the board."],[878.6205,"first_reconstruction is hidden from the screen — recon_axes left the board."],[878.6205,"second_contribution is hidden from the screen — recon_axes left the board."],[878.6205,"second_reconstruction is hidden from the screen — recon_axes left the board."],[878.6205,"third_contribution is hidden from the screen — recon_axes left the board."],[878.6205,"two is hidden from the screen — left the board."]]}]},{"title":"Units Can Choose the Components","start":879.6621666666666,"end":1070.9832291666667,"objects":{"choices":"a Block [text] that says \"Use raw covariance when original scale is meaningful. Standardize when unlike units should begin with equal variance. Validate the scaling choice against the actual downstream task.\"","correlation":"a Math [text] that says \"$Sigma_z = mat(1, 0.20; 0.20, 1)$\"","heading_choice":"a Heading that says \"Units Can Change the Answer\"","heading_compare":"a Heading that says \"PCA Measures Numerical Variance\"","heading_scaling":"a Heading that says \"Scaling Is a Modeling Decision\"","raw_axes":"an Axes (x_range=(-50000.0, 50000.0), y_range=(-30.0, 30.0), x_ticks_every=25000.0)","raw_cloud":"a Point [blue] drawn in raw_axes (location=(-40000.0, -15.0), marker_radius=0.07)","raw_cloud_2":"a Point [blue] drawn in raw_axes (location=(-30000.0, 5.0), marker_radius=0.07)","raw_cloud_3":"a Point [blue] drawn in raw_axes (location=(-15000.0, -8.0), marker_radius=0.07)","raw_cloud_4":"a Point [blue] drawn in raw_axes (location=(-5000.0, 12.0), marker_radius=0.07)","raw_cloud_5":"a Point [blue] drawn in raw_axes (location=(8000.0, -12.0), marker_radius=0.07)","raw_cloud_6":"a Point [blue] drawn in raw_axes (location=(16000.0, 15.0), marker_radius=0.07)","raw_cloud_7":"a Point [blue] drawn in raw_axes (location=(30000.0, -3.0), marker_radius=0.07)","raw_cloud_8":"a Point [blue] drawn in raw_axes (location=(40000.0, 10.0), marker_radius=0.07)","raw_covariance":"a Math [text] that says \"$Sigma_(upright(\"raw\")) = mat(400000000, 40000; 40000, 100)$\"","raw_label":"a Tex [text] that says \"Centered, original units\"","raw_pc":"a Line [red] labelled \"upright(\"PC\")_1\" drawn in raw_axes (start=(-47000.0, -1.0), end=(47000.0, 1.0))","standard_axes":"an Axes (x_range=(-2.6, 2.6), y_range=(-2.6, 2.6), aspect=(1.0, 1.0))","standard_cloud":"a Point [blue] drawn in standard_axes (location=(-2.0, -1.5), marker_radius=0.07)","standard_cloud_2":"a Point [blue] drawn in standard_axes (location=(-1.5, 0.5), marker_radius=0.07)","standard_cloud_3":"a Point [blue] drawn in standard_axes (location=(-0.75, -0.8), marker_radius=0.07)","standard_cloud_4":"a Point [blue] drawn in standard_axes (location=(-0.25, 1.2), marker_radius=0.07)","standard_cloud_5":"a Point [blue] drawn in standard_axes (location=(0.4, -1.2), marker_radius=0.07)","standard_cloud_6":"a Point [blue] drawn in standard_axes (location=(0.8, 1.5), marker_radius=0.07)","standard_cloud_7":"a Point [blue] drawn in standard_axes (location=(1.5, -0.3), marker_radius=0.07)","standard_cloud_8":"a Point [blue] drawn in standard_axes (location=(2.0, 1.0), marker_radius=0.07)","standard_label":"a Tex [text] that says \"Centered and standardized\"","standard_pc":"a Line [green] labelled \"upright(\"PC\")_1\" drawn in standard_axes (start=(-2.3, -1.15), end=(2.3, 1.15))","standardize":"a Math [text] that says \"$z_j = frac(x_j - overline(x)_j, s_j)$\"","unit_change":"a Math [text] that says \"$upright(\"dollars to cents\") arrow.r upright(\"variance\") times 10000$\"","warning":"a Panel that says \"PCA maximizes numerical variance. It does not know whether a large number is important, noisy, or merely written in a larger unit.\""},"beats":[{"start":879.6621666666666,"say":"One warning belongs in every practical explanation of PCA. PCA does not understand variables. It understands numbers, and it rewards whichever direction has the greatest numerical variance.","live":[],"does":[[879.6621666666666,"heading_compare is shown on the screen, written out."],[885.6531666666666,"raw_label is shown on the screen, written out."],[885.6531666666666,"standard_label is shown on the screen, written out."],[887.8011666666666,"raw_axes is shown on the screen, written out."],[887.8011666666666,"standard_axes is shown on the screen, written out."]]},{"start":893.1146666666666,"say":"On the left, salary deviation is measured in dollars and age deviation in years. These are centered observations, so subtracting the means has already been done.","live":["raw_label","raw_axes","standard_label","standard_axes","heading_compare"],"does":[[899.7441666666666,"raw_cloud is shown on the screen, written out."],[899.7441666666666,"raw_cloud_2 is shown on the screen, written out."],[899.7441666666666,"raw_cloud_3 is shown on the screen, written out."],[899.7441666666666,"raw_cloud_4 is shown on the screen, written out."],[899.7441666666666,"raw_cloud_5 is shown on the screen, written out."],[899.7441666666666,"raw_cloud_6 is shown on the screen, written out."],[899.7441666666666,"raw_cloud_7 is shown on the screen, written out."],[899.7441666666666,"raw_cloud_8 is shown on the screen, written out."]]},{"start":905.0576666666666,"say":"The salary coordinates contain tens of thousands while the age coordinates contain tens. The raw cloud therefore looks overwhelmingly horizontal, and raw PCA chooses an almost horizontal first component.","live":["raw_label","raw_axes","standard_label","standard_axes","heading_compare","raw_cloud","raw_cloud_2","raw_cloud_3","raw_cloud_4","raw_cloud_5","raw_cloud_6","raw_cloud_7","raw_cloud_8"],"does":[[913.6961666666666,"raw_pc is shown on the screen, written out."],[917.5391666666666,"raw_pc is indicated — a transient flash."]]},{"start":919.4046666666666,"say":"That result does not prove salary is the most important scientific variable. It proves salary has the largest variance in the numbers handed to the algorithm.","live":["raw_label","raw_axes","standard_label","standard_axes","heading_compare","raw_cloud","raw_cloud_2","raw_cloud_3","raw_cloud_4","raw_cloud_5","raw_cloud_6","raw_cloud_7","raw_cloud_8","raw_pc"],"does":[[926.1841666666667,"raw_pc is indicated — a transient flash."]]},{"start":929.9661666666666,"say":"On the right, each variable has been centered and divided by its sample standard deviation. Both begin with variance one, so their relationship, rather than their original unit sizes, determines the component.","live":null,"does":[[932.8921666666666,"standard_cloud is shown on the screen, written out."],[932.8921666666666,"standard_cloud_2 is shown on the screen, written out."],[932.8921666666666,"standard_cloud_3 is shown on the screen, written out."],[932.8921666666666,"standard_cloud_4 is shown on the screen, written out."],[932.8921666666666,"standard_cloud_5 is shown on the screen, written out."],[932.8921666666666,"standard_cloud_6 is shown on the screen, written out."],[932.8921666666666,"standard_cloud_7 is shown on the screen, written out."],[932.8921666666666,"standard_cloud_8 is shown on the screen, written out."],[943.3061666666666,"standard_pc is shown on the screen, written out."]]},{"start":945.1021666666666,"say":"The two answers differ even though they describe the same observations. Scaling has changed the geometry, and changing the geometry changes PCA.","live":["raw_label","raw_axes","standard_label","standard_axes","heading_compare","raw_cloud","raw_cloud_2","raw_cloud_3","raw_cloud_4","raw_cloud_5","raw_cloud_6","raw_cloud_7","raw_cloud_8","raw_pc","standard_cloud","standard_cloud_2","standard_cloud_3","standard_cloud_4","standard_cloud_5","standard_cloud_6","standard_cloud_7","standard_cloud_8","standard_pc"],"does":[[946.5071666666666,"raw_pc is indicated — a transient flash."],[946.7071666666666,"standard_pc is indicated — a transient flash."],[955.7601666666666,"heading_compare is hidden from the screen — left the board."],[955.7601666666666,"raw_axes is hidden from the screen — left the board."],[955.7601666666666,"raw_cloud is hidden from the screen — raw_axes left the board."],[955.7601666666666,"raw_cloud_2 is hidden from the screen — raw_axes left the board."],[955.7601666666666,"raw_cloud_3 is hidden from the screen — raw_axes left the board."],[955.7601666666666,"raw_cloud_4 is hidden from the screen — raw_axes left the board."],[955.7601666666666,"raw_cloud_5 is hidden from the screen — raw_axes left the board."],[955.7601666666666,"raw_cloud_6 is hidden from the screen — raw_axes left the board."],[955.7601666666666,"raw_cloud_7 is hidden from the screen — raw_axes left the board."],[955.7601666666666,"raw_cloud_8 is hidden from the screen — raw_axes left the board."],[955.7601666666666,"raw_pc is hidden from the screen — raw_axes left the board."],[955.7601666666666,"raw_label is hidden from the screen — left the board."],[955.7601666666666,"standard_axes is hidden from the screen — left the board."],[955.7601666666666,"standard_cloud is hidden from the screen — standard_axes left the board."],[955.7601666666666,"standard_cloud_2 is hidden from the screen — standard_axes left the board."],[955.7601666666666,"standard_cloud_3 is hidden from the screen — standard_axes left the board."],[955.7601666666666,"standard_cloud_4 is hidden from the screen — standard_axes left the board."],[955.7601666666666,"standard_cloud_5 is hidden from the screen — standard_axes left the board."],[955.7601666666666,"standard_cloud_6 is hidden from the screen — standard_axes left the board."],[955.7601666666666,"standard_cloud_7 is hidden from the screen — standard_axes left the board."],[955.7601666666666,"standard_cloud_8 is hidden from the screen — standard_axes left the board."],[955.7601666666666,"standard_pc is hidden from the screen — standard_axes left the board."],[955.7601666666666,"standard_label is hidden from the screen — left the board."]]},{"start":956.9601666666666,"say":"Here is the raw covariance matrix. Salary variance is four hundred million, while age variance is one hundred. The larger numerical scale dominates the optimization.","live":[],"does":[[956.9601666666666,"heading_choice is shown on the screen, written out."],[956.9601666666666,"raw_covariance is shown on the screen, written out."],[962.6491666666666,"raw_covariance (the \"400000000\" part) is emphasized."],[965.0181666666666,"raw_covariance (the \"400000000\" part) is no longer emphasized."]]},{"start":970.4941666666666,"say":"Now perform a scientifically meaningless change: write the same salaries in cents instead of dollars. Every salary value is multiplied by one hundred, and its variance is multiplied by ten thousand. Raw PCA can rotate even though nothing about the people changed.","live":["raw_covariance","heading_choice"],"does":[[976.2521666666667,"unit_change is shown on the screen, written out."],[983.4741666666666,"unit_change (the \"10000\" part) is emphasized."],[989.4421666666666,"unit_change (the \"10000\" part) is no longer emphasized."]]},{"start":990.0421666666666,"say":"That is the blunt warning. PCA maximizes numerical variance. It cannot distinguish signal from noise, importance from nuisance, or measurement scale from genuine structure.","live":["raw_covariance","unit_change","heading_choice"],"does":[[990.0421666666666,"warning is shown on the screen, written out."],[994.0931666666667,"warning (the \"numerical variance\" part) is emphasized."],[1002.7546666666666,"heading_choice is hidden from the screen — left the board."],[1002.7546666666666,"raw_covariance is hidden from the screen — left the board."],[1002.7546666666666,"unit_change is hidden from the screen — left the board."],[1002.7546666666666,"warning is hidden from the screen — left the board."],[1002.7546666666666,"warning (the \"numerical variance\" part) is no longer emphasized."]]},{"start":1003.9546666666666,"say":"Standardization replaces each variable by its centered value divided by its sample standard deviation. The covariance matrix of standardized variables is their correlation matrix.","live":[],"does":[[1003.9546666666666,"heading_scaling is shown on the screen, written out."],[1004.3031666666666,"standardize is shown on the screen, written out."],[1014.4961666666666,"correlation is shown on the screen, written out."]]},{"start":1016.8141666666666,"say":"Do not turn that into another automatic ritual. If a variable's original scale carries real meaning, raw covariance may be appropriate. If unlike units should receive equal starting weight, standardization is often necessary.","live":["standardize","correlation","heading_scaling"],"does":[[1021.4701666666666,"choices is shown on the screen, written out."],[1023.9081666666666,"choices (the \"Use raw covariance when original scale is meaningful.\" part) is emphasized."],[1029.8521666666666,"choices (the \"Standardize when unlike units should begin with equal variance.\" part) is emphasized."],[1029.8521666666666,"choices (the \"Use raw covariance when original scale is meaningful.\" part) is no longer emphasized."],[1032.3596666666667,"choices (the \"Standardize when unlike units should begin with equal variance.\" part) is no longer emphasized."]]},{"start":1032.9596666666666,"say":"Heavy tails, outliers, counts, and strongly skewed variables may call for robust scaling or a domain-specific transformation. Whatever you choose, document it and validate it as part of the model.","live":["standardize","correlation","choices","heading_scaling"],"does":[[1044.8371666666667,"choices (the \"Validate the scaling choice against the actual downstream task.\" part) is emphasized."],[1046.9846666666667,"choices (the \"Validate the scaling choice against the actual downstream task.\" part) is no longer emphasized."]]},{"start":1047.5846666666666,"say":"PCA is now one connected idea. Center the observations, find the direction of maximum projected variance, continue in perpendicular directions, and compute those directions as covariance eigenvectors. Use eigenvalues to judge compression, reconstruct from retained scores, and never forget that the units help define the answer.","live":null,"does":[[1067.8671666666667,"choices (the \"Validate the scaling choice against the actual downstream task.\" part) is indicated — a transient flash."],[1069.9415625,"choices is hidden from the screen — left the board."],[1069.9415625,"correlation is hidden from the screen — left the board."],[1069.9415625,"heading_scaling is hidden from the screen — left the board."],[1069.9415625,"standardize is hidden from the screen — left the board."]]}]}]},"durationSeconds":1071,"chapters":[{"title":"Variance in a Direction","startSeconds":0,"narration":"Suppose you use PCA every week. You call a library, ask for two components, and get a transformed table. But what did the algorithm actually choose? Here is the question that organizes the whole method: if we flatten a centered cloud onto one line, which line preserves the most variation? Begin with a two-dimensional data set. Each blue point is one observation, and the two axes are two measured variables. The cloud is not round. It has a long direction running roughly from lower left to upper right. PCA starts by centering each variable. Subtract its sample mean, so the red mean moves to the origin. Centering matters because PCA is supposed to describe variation around the typical observation, not distance from an arbitrary zero. Now choose a candidate direction u. We require unit length, not because short arrows are virtuous, but because otherwise we could manufacture unlimited variance merely by multiplying the direction by a huge number. Project every observation onto that line. Each gray segment drops one blue point to its green projected position. The signed coordinate along the line is its score, z sub i, equal to u transpose times the centered observation. Those scores form a one-dimensional data set. Measure their variance in the ordinary way: square each score, average the squares, and call the result V of u. At the moment the line is horizontal. The green points have some spread, and the marker reports variance four. That is neither especially small nor the largest value available. Turn the candidate line toward the visible long axis of the cloud. The projected points separate, so their one-dimensional variance rises. It reaches seven at this direction, where the spread is greatest. Continue turning. The projections draw closer together again. On the vertical axis the variance has fallen back to four, even though no data point has moved. Keep turning toward the short axis of the cloud. The green points bunch tightly around the origin, and the variance falls to one. This direction preserves very little of the cloud's visible spread. Now return to the maximum. The data stayed fixed throughout. Only the question we asked of them changed: how much of their variation can be seen from this direction? This is PCA's central geometric operation. It searches over all unit directions, projects the centered observations, and chooses the direction whose scores have the largest variance. That maximizing direction is the first principal component. A component is a direction in variable space. The projected coordinate is the score of an observation on that component. Those are related objects, but they are not the same thing. Write the first component as v one: the unit direction that maximizes projected variance. Its variance, seven in this example, tells us how much variation the component captures. After choosing v one, PCA looks for another direction that captures as much remaining variance as possible without repeating the first answer. In two dimensions, there is only one perpendicular direction available. That green direction is the second principal component. The cloud is narrow along it, so its variance is only one. The first component describes the long way through the cloud, and the second describes the remaining side-to-side variation. The two component directions are perpendicular and have unit length. Together they form a rotated coordinate system. PCA has not bent the data or invented a nonlinear feature. It has rotated the axes to line up with the directions of greatest and least variation. So the first answer to carry forward is geometric. PCA asks where the centered cloud is widest, calls that direction component one, and then repeats the question in directions perpendicular to what it has already chosen."},{"title":"More Than Two Dimensions","startSeconds":252.26064583333334,"narration":"A real data set may have fifty variables, five thousand, or five million. We cannot rotate a literal line through all those dimensions on a screen, but the geometric question does not change. Use three dimensions as the bridge. This cloud has a long spatial direction, plus smaller deviations in two independent sideways directions. The first search is exactly the one we just performed. Among every unit direction in the space, find the one on which the projected coordinates have maximum variance. The red arrow is the answer. Projecting onto it compresses every observation to one score. Across all observations, those scores vary more than scores along any other unit direction. Now remove the possibility of choosing that answer again. Every direction perpendicular to v one lies in this gray plane through the origin. Search inside that plane for the direction with greatest projected variance. The green arrow is component two. It is the strongest remaining direction after component one has been excluded. In three dimensions, one perpendicular direction remains after the first two have been fixed. That yellow arrow becomes component three. In p dimensions the process continues the same way. Component k maximizes projected variance subject to unit length and orthogonality to every earlier component. Collect the first k component directions as columns of V sub k. Multiplying the centered data matrix X by V sub k produces the score matrix Z sub k. Each row is now the same observation expressed in principal-component coordinates. Because the component directions are mutually perpendicular unit vectors, V transpose V is the identity. This orthonormal structure is why moving into component coordinates and reconstructing from them remains simple. There cannot be more useful components than variables. There also cannot be more than n minus one nonzero directions of variation after n observations have been centered. The crucial point is that higher-dimensional PCA is not a new algorithmic idea. It is the same sequence of questions: widest direction first, then the widest perpendicular direction left, then the next. We now have the geometry. Only at this point do we need the covariance matrix, because it gives an efficient algebraic way to answer exactly the optimization question we have already understood."},{"title":"The Covariance Eigenproblem","startSeconds":417.72210416666667,"narration":"Return to the centered two-dimensional cloud. We already know the geometric problem: choose a unit direction u that makes the projected scores vary as much as possible. For one centered observation x sub i, its coordinate on u is the dot product u transpose x sub i. Call that score z sub i. The scores are centered too, so their variance is the average squared score. Substitute the dot product into that familiar calculation. A scalar square can be written as u transpose x sub i, times x sub i transpose u. Put the direction vectors outside the observation-specific middle. The average of those middle outer products is the covariance matrix Sigma. With centered observations stored in the rows of X, Sigma is one over n times X transpose X. So the variance seen along u is exactly the quadratic form u transpose Sigma u. This line is not a different objective. It is the projected spread from the rotating picture, written compactly. We want to maximize that quadratic form while keeping u at unit length. Use a Lagrange multiplier to attach the constraint u transpose u equals one. At a maximum, the derivative with respect to u is zero. Differentiating gives two Sigma u minus two lambda u. Cancel the common factor of two and rearrange. Sigma u equals lambda u. The maximizing direction must therefore be an eigenvector of the covariance matrix. This is the promised restatement. The geometry asked for a direction that keeps its identity while covariance assigns it a variance. The algebra calls that direction an eigenvector. Multiply the eigenvalue equation on the left by u transpose. Because u has unit length, u transpose Sigma u equals lambda. The eigenvalue is the projected variance along its eigenvector. Therefore the first component is the eigenvector with the largest eigenvalue. The second is the perpendicular eigenvector with the next eigenvalue, and so on in descending order. Covariance matrices are symmetric and positive semidefinite. Their eigenvectors can be chosen orthonormal, and their eigenvalues are nonnegative. Those facts are exactly what the geometric construction required. Now calculate the example. Use the covariance matrix with fours on the diagonal and threes off the diagonal. Its positive covariance says the two variables tend to rise and fall together. Eigenvalues make Sigma minus lambda I singular. Subtract lambda from both diagonal entries and set the determinant to zero. Multiply the main diagonal and subtract the other product. We get four minus lambda squared, minus nine, equal to zero. Expand the square and collect terms. The characteristic polynomial is lambda squared minus eight lambda plus seven. Factor it. Lambda minus seven times lambda minus one equals zero. The two eigenvalues are seven and one, exactly the maximum and minimum projected variances from the rotating line. The eigenvector for seven points along one, one. The other eigenvector points along one, minus one. Normalize both by dividing by square root two. Seven of the total eight variance units lie along component one. So one component explains eighty-seven point five percent of this cloud's variance. The eigenvalue calculation has returned every fact we first discovered by rotating and projecting."},{"title":"Scree Plots and Reconstruction","startSeconds":657.4953958333333,"narration":"PCA orders components from most variance to least, but a model still needs a decision: how many components should we keep? The eigenvalues give the evidence for that decision. A scree plot puts the component number along the horizontal axis and its eigenvalue on the vertical axis. Here are five eigenvalues in descending order. The first point is high because component one captures a great deal of variation. The second captures less. After the third, the remaining eigenvalues are small. An individual explained-variance ratio divides one eigenvalue by the sum of all eigenvalues. It reports the fraction of total variance assigned to that component. Cumulative explained variance adds the first k eigenvalues before dividing by the total. It asks how much variation remains visible after reducing the data to k component coordinates. With one component, this example retains sixty-seven point six percent. With two, the total reaches eighty-eight point nine percent. With three components it reaches ninety-six point six percent. Components four and five add very little. The visible bend near component three is often called the elbow. One common heuristic keeps components before the curve settles into a shallow tail. But the elbow is not a theorem and ninety-five percent is not a universal law. Keep enough components for the downstream task. For prediction, choose k with cross-validation. For compression, choose an acceptable reconstruction error. For interpretation, check stability and domain meaning. A scree plot is diagnostic evidence, not an automatic command. A weak elbow may honestly mean there is no clean low-dimensional cutoff. Now make component retention concrete. In principal-component coordinates, a centered observation is built from one contribution along each component direction. This observation has a large score on component one, a smaller score on component two, and a still smaller score on component three. The general reconstruction formula starts at the original mean and adds back the first k score times direction contributions. Keep only component one. Starting from the mean, move along v one by the first score. The green point is the rank-one reconstruction. It preserves the strongest coordinate and discards everything in the omitted perpendicular subspace. The red dashed segment is the information lost for this observation. Now retain component two as well. Add the yellow contribution from the first reconstruction. The new point moves closer to the original. The remaining error lies entirely along component three. If we retain that final red contribution, the reconstruction reaches the original point exactly. The displayed error falls from one point three four with one component to zero point six with two. Every additional component can only reduce squared reconstruction error on the training data. Averaged over the data set, the squared error after keeping k components equals the sum of the omitted eigenvalues. The scree plot and reconstruction error are therefore two readings of the same tradeoff. Keeping a component means preserving one orthogonal pattern of variation. Dropping it means replacing every observation's score on that pattern by zero, then mapping the reduced coordinates back to the original variable space."},{"title":"Units Can Choose the Components","startSeconds":879.6621666666666,"narration":"One warning belongs in every practical explanation of PCA. PCA does not understand variables. It understands numbers, and it rewards whichever direction has the greatest numerical variance. On the left, salary deviation is measured in dollars and age deviation in years. These are centered observations, so subtracting the means has already been done. The salary coordinates contain tens of thousands while the age coordinates contain tens. The raw cloud therefore looks overwhelmingly horizontal, and raw PCA chooses an almost horizontal first component. That result does not prove salary is the most important scientific variable. It proves salary has the largest variance in the numbers handed to the algorithm. On the right, each variable has been centered and divided by its sample standard deviation. Both begin with variance one, so their relationship, rather than their original unit sizes, determines the component. The two answers differ even though they describe the same observations. Scaling has changed the geometry, and changing the geometry changes PCA. Here is the raw covariance matrix. Salary variance is four hundred million, while age variance is one hundred. The larger numerical scale dominates the optimization. Now perform a scientifically meaningless change: write the same salaries in cents instead of dollars. Every salary value is multiplied by one hundred, and its variance is multiplied by ten thousand. Raw PCA can rotate even though nothing about the people changed. That is the blunt warning. PCA maximizes numerical variance. It cannot distinguish signal from noise, importance from nuisance, or measurement scale from genuine structure. Standardization replaces each variable by its centered value divided by its sample standard deviation. The covariance matrix of standardized variables is their correlation matrix. Do not turn that into another automatic ritual. If a variable's original scale carries real meaning, raw covariance may be appropriate. If unlike units should receive equal starting weight, standardization is often necessary. Heavy tails, outliers, counts, and strongly skewed variables may call for robust scaling or a domain-specific transformation. Whatever you choose, document it and validate it as part of the model. PCA is now one connected idea. Center the observations, find the direction of maximum projected variance, continue in perpendicular directions, and compute those directions as covariance eigenvectors. Use eigenvalues to judge compression, reconstruct from retained scores, and never forget that the units help define the answer."}]}}