WEBVTT 00:00:00.070 --> 00:00:02.390 Suppose you use PCA every week. 00:00:02.473 --> 00:00:07.350 You call a library, ask for two components, and get a transformed table. 00:00:07.663 --> 00:00:10.485 But what did the algorithm actually choose? 00:00:10.681 --> 00:00:13.724 Here is the question that organizes the whole method: 00:00:13.827 --> 00:00:16.566 if we flatten a centered cloud onto one line, 00:00:16.649 --> 00:00:19.447 which line preserves the most variation? 00:00:20.960 --> 00:00:23.817 Begin with a two-dimensional data set. 00:00:23.921 --> 00:00:29.192 Each blue point is one observation, and the two axes are two measured variables. 00:00:29.389 --> 00:00:31.140 The cloud is not round. 00:00:31.223 --> 00:00:35.427 It has a long direction running roughly from lower left to upper right. 00:00:36.340 --> 00:00:39.440 PCA starts by centering each variable. 00:00:39.637 --> 00:00:43.701 Subtract its sample mean, so the red mean moves to the origin. 00:00:43.898 --> 00:00:47.357 Centering matters because PCA is supposed to describe variation 00:00:47.357 --> 00:00:51.955 around the typical observation, not distance from an arbitrary zero. 00:00:52.752 --> 00:00:55.225 Now choose a candidate direction u. 00:00:55.422 --> 00:00:58.952 We require unit length, not because short arrows are virtuous, 00:00:58.952 --> 00:01:03.433 but because otherwise we could manufacture unlimited variance merely by multiplying 00:01:03.433 --> 00:01:05.640 the direction by a huge number. 00:01:06.692 --> 00:01:09.758 Project every observation onto that line. 00:01:09.954 --> 00:01:14.437 Each gray segment drops one blue point to its green projected position. 00:01:14.889 --> 00:01:18.997 The signed coordinate along the line is its score, z sub i, 00:01:19.080 --> 00:01:22.784 equal to u transpose times the centered observation. 00:01:23.697 --> 00:01:26.821 Those scores form a one-dimensional data set. 00:01:26.925 --> 00:01:29.384 Measure their variance in the ordinary way: 00:01:29.467 --> 00:01:34.530 square each score, average the squares, and call the result V of u. 00:01:35.327 --> 00:01:37.673 At the moment the line is horizontal. 00:01:37.870 --> 00:01:42.468 The green points have some spread, and the marker reports variance four. 00:01:42.572 --> 00:01:46.566 That is neither especially small nor the largest value available. 00:01:47.363 --> 00:01:51.299 Turn the candidate line toward the visible long axis of the cloud. 00:01:51.403 --> 00:01:55.769 The projected points separate, so their one-dimensional variance rises. 00:01:55.965 --> 00:01:59.705 It reaches seven at this direction, where the spread is greatest. 00:02:00.501 --> 00:02:01.930 Continue turning. 00:02:02.034 --> 00:02:04.763 The projections draw closer together again. 00:02:05.075 --> 00:02:08.499 On the vertical axis the variance has fallen back to four, 00:02:08.582 --> 00:02:10.998 even though no data point has moved. 00:02:11.794 --> 00:02:14.674 Keep turning toward the short axis of the cloud. 00:02:14.778 --> 00:02:19.655 The green points bunch tightly around the origin, and the variance falls to one. 00:02:19.851 --> 00:02:23.579 This direction preserves very little of the cloud's visible spread. 00:02:24.492 --> 00:02:26.382 Now return to the maximum. 00:02:26.465 --> 00:02:28.730 The data stayed fixed throughout. 00:02:28.927 --> 00:02:31.633 Only the question we asked of them changed: 00:02:31.829 --> 00:02:35.290 how much of their variation can be seen from this direction? 00:02:36.202 --> 00:02:39.582 This is PCA's central geometric operation. 00:02:39.894 --> 00:02:44.328 It searches over all unit directions, projects the centered observations, 00:02:44.411 --> 00:02:48.185 and chooses the direction whose scores have the largest variance. 00:02:48.981 --> 00:02:53.046 That maximizing direction is the first principal component. 00:02:53.149 --> 00:02:56.436 A component is a direction in variable space. 00:02:56.632 --> 00:03:00.662 The projected coordinate is the score of an observation on that component. 00:03:00.858 --> 00:03:04.063 Those are related objects, but they are not the same thing. 00:03:05.878 --> 00:03:08.491 Write the first component as v one: 00:03:08.688 --> 00:03:12.369 the unit direction that maximizes projected variance. 00:03:12.565 --> 00:03:15.106 Its variance, seven in this example, 00:03:15.189 --> 00:03:18.360 tells us how much variation the component captures. 00:03:19.156 --> 00:03:23.289 After choosing v one, PCA looks for another direction that captures 00:03:23.289 --> 00:03:27.795 as much remaining variance as possible without repeating the first answer. 00:03:28.247 --> 00:03:32.311 In two dimensions, there is only one perpendicular direction available. 00:03:33.108 --> 00:03:36.232 That green direction is the second principal component. 00:03:36.428 --> 00:03:40.260 The cloud is narrow along it, so its variance is only one. 00:03:40.364 --> 00:03:43.543 The first component describes the long way through the cloud, 00:03:43.626 --> 00:03:47.226 and the second describes the remaining side-to-side variation. 00:03:48.022 --> 00:03:52.180 The two component directions are perpendicular and have unit length. 00:03:52.376 --> 00:03:55.338 Together they form a rotated coordinate system. 00:03:55.650 --> 00:04:00.109 PCA has not bent the data or invented a nonlinear feature. 00:04:00.306 --> 00:04:01.699 It has rotated the axes 00:04:01.699 --> 00:04:05.601 to line up with the directions of greatest and least variation. 00:04:06.397 --> 00:04:09.696 So the first answer to carry forward is geometric. 00:04:09.892 --> 00:04:12.770 PCA asks where the centered cloud is widest, 00:04:12.853 --> 00:04:15.138 calls that direction component one, 00:04:15.221 --> 00:04:16.510 and then repeats the question 00:04:16.510 --> 00:04:19.959 in directions perpendicular to what it has already chosen. 00:04:21.021 --> 00:04:25.418 A real data set may have fifty variables, five thousand, or five million. 00:04:25.502 --> 00:04:29.493 We cannot rotate a literal line through all those dimensions on a screen, 00:04:29.577 --> 00:04:32.411 but the geometric question does not change. 00:04:32.952 --> 00:04:35.461 Use three dimensions as the bridge. 00:04:35.565 --> 00:04:38.012 This cloud has a long spatial direction, 00:04:38.096 --> 00:04:42.241 plus smaller deviations in two independent sideways directions. 00:04:43.038 --> 00:04:46.371 The first search is exactly the one we just performed. 00:04:46.567 --> 00:04:49.130 Among every unit direction in the space, 00:04:49.214 --> 00:04:53.208 find the one on which the projected coordinates have maximum variance. 00:04:53.912 --> 00:04:55.709 The red arrow is the answer. 00:04:55.793 --> 00:04:59.753 Projecting onto it compresses every observation to one score. 00:04:59.949 --> 00:05:01.572 Across all observations, 00:05:01.656 --> 00:05:05.767 those scores vary more than scores along any other unit direction. 00:05:06.564 --> 00:05:10.059 Now remove the possibility of choosing that answer again. 00:05:10.256 --> 00:05:14.122 Every direction perpendicular to v one lies in this gray plane 00:05:14.122 --> 00:05:15.597 through the origin. 00:05:16.510 --> 00:05:20.934 Search inside that plane for the direction with greatest projected variance. 00:05:21.131 --> 00:05:23.183 The green arrow is component two. 00:05:23.267 --> 00:05:27.738 It is the strongest remaining direction after component one has been excluded. 00:05:28.534 --> 00:05:29.832 In three dimensions, 00:05:29.916 --> 00:05:34.503 one perpendicular direction remains after the first two have been fixed. 00:05:34.955 --> 00:05:37.835 That yellow arrow becomes component three. 00:05:38.631 --> 00:05:42.197 In p dimensions the process continues the same way. 00:05:42.393 --> 00:05:48.047 Component k maximizes projected variance subject to unit length and orthogonality 00:05:48.047 --> 00:05:50.173 to every earlier component. 00:05:51.085 --> 00:05:55.881 Collect the first k component directions as columns of V sub k. 00:05:55.985 --> 00:06:02.985 Multiplying the centered data matrix X by V sub k produces the score matrix Z sub k. 00:06:03.183 --> 00:06:05.679 Each row is now the same observation expressed 00:06:05.679 --> 00:06:08.037 in principal-component coordinates. 00:06:08.740 --> 00:06:13.129 Because the component directions are mutually perpendicular unit vectors, 00:06:13.326 --> 00:06:16.078 V transpose V is the identity. 00:06:16.182 --> 00:06:18.330 This orthonormal structure is why moving 00:06:18.330 --> 00:06:22.939 into component coordinates and reconstructing from them remains simple. 00:06:23.852 --> 00:06:26.926 There cannot be more useful components than variables. 00:06:27.010 --> 00:06:32.188 There also cannot be more than n minus one nonzero directions of variation after 00:06:32.188 --> 00:06:34.604 n observations have been centered. 00:06:35.517 --> 00:06:36.376 The crucial point 00:06:36.376 --> 00:06:41.311 is that higher-dimensional PCA is not a new algorithmic idea. 00:06:41.507 --> 00:06:44.016 It is the same sequence of questions: 00:06:44.212 --> 00:06:45.638 widest direction first, 00:06:45.722 --> 00:06:50.146 then the widest perpendicular direction left, then the next. 00:06:51.059 --> 00:06:53.300 We now have the geometry. 00:06:53.497 --> 00:06:56.373 Only at this point do we need the covariance matrix, 00:06:56.457 --> 00:07:00.602 because it gives an efficient algebraic way to answer exactly the optimization 00:07:00.602 --> 00:07:03.029 question we have already understood. 00:07:04.229 --> 00:07:07.005 Return to the centered two-dimensional cloud. 00:07:07.201 --> 00:07:09.429 We already know the geometric problem: 00:07:09.512 --> 00:07:14.842 choose a unit direction u that makes the projected scores vary as much as possible. 00:07:15.638 --> 00:07:18.551 For one centered observation x sub i, 00:07:18.634 --> 00:07:23.150 its coordinate on u is the dot product u transpose x sub i. 00:07:23.347 --> 00:07:25.879 Call that score z sub i. 00:07:26.304 --> 00:07:30.589 The scores are centered too, so their variance is the average squared score. 00:07:30.785 --> 00:07:34.617 Substitute the dot product into that familiar calculation. 00:07:35.414 --> 00:07:39.359 A scalar square can be written as u transpose x sub i, 00:07:39.442 --> 00:07:42.334 times x sub i transpose u. 00:07:42.530 --> 00:07:46.653 Put the direction vectors outside the observation-specific middle. 00:07:47.449 --> 00:07:52.291 The average of those middle outer products is the covariance matrix Sigma. 00:07:52.604 --> 00:07:55.587 With centered observations stored in the rows of X, 00:07:55.670 --> 00:08:00.070 Sigma is one over n times X transpose X. 00:08:00.867 --> 00:08:07.369 So the variance seen along u is exactly the quadratic form u transpose Sigma u. 00:08:07.566 --> 00:08:09.991 This line is not a different objective. 00:08:10.074 --> 00:08:14.626 It is the projected spread from the rotating picture, written compactly. 00:08:15.678 --> 00:08:20.276 We want to maximize that quadratic form while keeping u at unit length. 00:08:20.473 --> 00:08:25.907 Use a Lagrange multiplier to attach the constraint u transpose u equals one. 00:08:26.959 --> 00:08:31.128 At a maximum, the derivative with respect to u is zero. 00:08:31.325 --> 00:08:35.842 Differentiating gives two Sigma u minus two lambda u. 00:08:36.731 --> 00:08:40.041 Cancel the common factor of two and rearrange. 00:08:40.226 --> 00:08:42.850 Sigma u equals lambda u. 00:08:43.035 --> 00:08:48.330 The maximizing direction must therefore be an eigenvector of the covariance matrix. 00:08:49.115 --> 00:08:51.450 This is the promised restatement. 00:08:51.635 --> 00:08:52.831 The geometry asked 00:08:52.831 --> 00:08:57.731 for a direction that keeps its identity while covariance assigns it a variance. 00:08:57.927 --> 00:09:00.993 The algebra calls that direction an eigenvector. 00:09:01.604 --> 00:09:06.191 Multiply the eigenvalue equation on the left by u transpose. 00:09:06.387 --> 00:09:11.496 Because u has unit length, u transpose Sigma u equals lambda. 00:09:11.693 --> 00:09:15.769 The eigenvalue is the projected variance along its eigenvector. 00:09:16.565 --> 00:09:21.257 Therefore the first component is the eigenvector with the largest eigenvalue. 00:09:21.453 --> 00:09:25.456 The second is the perpendicular eigenvector with the next eigenvalue, 00:09:25.539 --> 00:09:27.630 and so on in descending order. 00:09:28.427 --> 00:09:32.677 Covariance matrices are symmetric and positive semidefinite. 00:09:32.873 --> 00:09:35.472 Their eigenvectors can be chosen orthonormal, 00:09:35.555 --> 00:09:38.203 and their eigenvalues are nonnegative. 00:09:38.399 --> 00:09:42.301 Those facts are exactly what the geometric construction required. 00:09:43.097 --> 00:09:44.953 Now calculate the example. 00:09:45.036 --> 00:09:50.471 Use the covariance matrix with fours on the diagonal and threes off the diagonal. 00:09:50.783 --> 00:09:55.335 Its positive covariance says the two variables tend to rise and fall together. 00:09:56.387 --> 00:10:00.359 Eigenvalues make Sigma minus lambda I singular. 00:10:00.555 --> 00:10:05.409 Subtract lambda from both diagonal entries and set the determinant to zero. 00:10:06.206 --> 00:10:09.829 Multiply the main diagonal and subtract the other product. 00:10:10.025 --> 00:10:15.065 We get four minus lambda squared, minus nine, equal to zero. 00:10:15.861 --> 00:10:18.509 Expand the square and collect terms. 00:10:18.706 --> 00:10:24.070 The characteristic polynomial is lambda squared minus eight lambda plus seven. 00:10:24.867 --> 00:10:26.040 Factor it. 00:10:26.237 --> 00:10:30.928 Lambda minus seven times lambda minus one equals zero. 00:10:31.125 --> 00:10:33.724 The two eigenvalues are seven and one, 00:10:33.807 --> 00:10:38.266 exactly the maximum and minimum projected variances from the rotating line. 00:10:39.062 --> 00:10:42.462 The eigenvector for seven points along one, one. 00:10:42.545 --> 00:10:46.052 The other eigenvector points along one, minus one. 00:10:46.249 --> 00:10:49.535 Normalize both by dividing by square root two. 00:10:50.932 --> 00:10:54.833 Seven of the total eight variance units lie along component one. 00:10:55.030 --> 00:10:58.118 So one component explains eighty-seven point five percent 00:10:58.118 --> 00:10:59.895 of this cloud's variance. 00:11:00.092 --> 00:11:01.590 The eigenvalue calculation 00:11:01.590 --> 00:11:06.281 has returned every fact we first discovered by rotating and projecting. 00:11:07.402 --> 00:11:10.604 PCA orders components from most variance to least, 00:11:10.688 --> 00:11:14.903 but a model still needs a decision: how many components should we keep? 00:11:15.099 --> 00:11:18.049 The eigenvalues give the evidence for that decision. 00:11:18.753 --> 00:11:22.407 A scree plot puts the component number along the horizontal axis 00:11:22.491 --> 00:11:25.043 and its eigenvalue on the vertical axis. 00:11:25.127 --> 00:11:28.193 Here are five eigenvalues in descending order. 00:11:28.734 --> 00:11:33.445 The first point is high because component one captures a great deal of variation. 00:11:33.529 --> 00:11:35.596 The second captures less. 00:11:35.793 --> 00:11:39.044 After the third, the remaining eigenvalues are small. 00:11:39.585 --> 00:11:43.544 An individual explained-variance ratio divides one eigenvalue by the sum 00:11:43.544 --> 00:11:45.287 of all eigenvalues. 00:11:45.390 --> 00:11:49.188 It reports the fraction of total variance assigned to that component. 00:11:49.984 --> 00:11:53.130 Cumulative explained variance adds the first k eigenvalues 00:11:53.130 --> 00:11:55.198 before dividing by the total. 00:11:55.394 --> 00:11:59.002 It asks how much variation remains visible after reducing the data 00:11:59.086 --> 00:12:01.177 to k component coordinates. 00:12:02.090 --> 00:12:06.700 With one component, this example retains sixty-seven point six percent. 00:12:06.896 --> 00:12:10.577 With two, the total reaches eighty-eight point nine percent. 00:12:11.060 --> 00:12:14.308 With three components it reaches ninety-six point six percent. 00:12:14.392 --> 00:12:17.156 Components four and five add very little. 00:12:17.860 --> 00:12:21.588 The visible bend near component three is often called the elbow. 00:12:21.691 --> 00:12:24.873 One common heuristic keeps components before the curve settles 00:12:24.873 --> 00:12:26.475 into a shallow tail. 00:12:27.017 --> 00:12:28.535 But the elbow is not a theorem 00:12:28.619 --> 00:12:31.661 and ninety-five percent is not a universal law. 00:12:31.858 --> 00:12:34.541 Keep enough components for the downstream task. 00:12:34.737 --> 00:12:37.649 For prediction, choose k with cross-validation. 00:12:37.733 --> 00:12:41.298 For compression, choose an acceptable reconstruction error. 00:12:41.402 --> 00:12:45.129 For interpretation, check stability and domain meaning. 00:12:45.833 --> 00:12:50.060 A scree plot is diagnostic evidence, not an automatic command. 00:12:50.256 --> 00:12:54.622 A weak elbow may honestly mean there is no clean low-dimensional cutoff. 00:12:55.926 --> 00:12:58.280 Now make component retention concrete. 00:12:58.364 --> 00:13:01.963 In principal-component coordinates, a centered observation is built 00:13:01.963 --> 00:13:05.273 from one contribution along each component direction. 00:13:06.069 --> 00:13:08.992 This observation has a large score on component one, 00:13:09.076 --> 00:13:13.663 a smaller score on component two, and a still smaller score on component three. 00:13:14.088 --> 00:13:17.011 The general reconstruction formula starts at the original mean 00:13:17.095 --> 00:13:20.788 and adds back the first k score times direction contributions. 00:13:21.700 --> 00:13:27.378 Keep only component one. Starting from the mean, move along v one by the first score. 00:13:27.575 --> 00:13:30.502 The green point is the rank-one reconstruction. 00:13:31.205 --> 00:13:34.259 It preserves the strongest coordinate and discards everything 00:13:34.259 --> 00:13:36.976 in the omitted perpendicular subspace. 00:13:37.173 --> 00:13:41.527 The red dashed segment is the information lost for this observation. 00:13:42.231 --> 00:13:44.647 Now retain component two as well. 00:13:44.750 --> 00:13:47.979 Add the yellow contribution from the first reconstruction. 00:13:48.082 --> 00:13:50.777 The new point moves closer to the original. 00:13:51.318 --> 00:13:54.976 The remaining error lies entirely along component three. 00:13:55.173 --> 00:13:57.515 If we retain that final red contribution, 00:13:57.599 --> 00:14:00.793 the reconstruction reaches the original point exactly. 00:14:01.589 --> 00:14:05.397 The displayed error falls from one point three four with one component 00:14:05.397 --> 00:14:07.406 to zero point six with two. 00:14:07.603 --> 00:14:11.106 Every additional component can only reduce squared reconstruction error 00:14:11.190 --> 00:14:12.700 on the training data. 00:14:13.497 --> 00:14:18.130 Averaged over the data set, the squared error after keeping k components equals 00:14:18.130 --> 00:14:20.487 the sum of the omitted eigenvalues. 00:14:20.684 --> 00:14:24.086 The scree plot and reconstruction error are therefore two readings 00:14:24.086 --> 00:14:25.642 of the same tradeoff. 00:14:26.439 --> 00:14:30.712 Keeping a component means preserving one orthogonal pattern of variation. 00:14:30.908 --> 00:14:33.776 Dropping it means replacing every observation's score 00:14:33.776 --> 00:14:35.329 on that pattern by zero, 00:14:35.413 --> 00:14:39.187 then mapping the reduced coordinates back to the original variable space. 00:14:40.298 --> 00:14:44.339 One warning belongs in every practical explanation of PCA. 00:14:44.535 --> 00:14:47.192 PCA does not understand variables. 00:14:47.275 --> 00:14:48.551 It understands numbers, 00:14:48.634 --> 00:14:52.663 and it rewards whichever direction has the greatest numerical variance. 00:14:53.460 --> 00:14:58.801 On the left, salary deviation is measured in dollars and age deviation in years. 00:14:58.997 --> 00:15:00.818 These are centered observations, 00:15:00.901 --> 00:15:04.083 so subtracting the means has already been done. 00:15:04.787 --> 00:15:06.529 The salary coordinates contain tens 00:15:06.529 --> 00:15:09.641 of thousands while the age coordinates contain tens. 00:15:09.837 --> 00:15:13.098 The raw cloud therefore looks overwhelmingly horizontal, 00:15:13.181 --> 00:15:17.304 and raw PCA chooses an almost horizontal first component. 00:15:18.216 --> 00:15:22.768 That result does not prove salary is the most important scientific variable. 00:15:22.965 --> 00:15:27.459 It proves salary has the largest variance in the numbers handed to the algorithm. 00:15:28.255 --> 00:15:29.275 On the right, 00:15:29.358 --> 00:15:33.724 each variable has been centered and divided by its sample standard deviation. 00:15:33.921 --> 00:15:37.367 Both begin with variance one, so their relationship, 00:15:37.450 --> 00:15:41.479 rather than their original unit sizes, determines the component. 00:15:42.531 --> 00:15:46.631 The two answers differ even though they describe the same observations. 00:15:46.827 --> 00:15:51.809 Scaling has changed the geometry, and changing the geometry changes PCA. 00:15:53.205 --> 00:15:55.946 Here is the raw covariance matrix. 00:15:56.050 --> 00:16:00.857 Salary variance is four hundred million, while age variance is one hundred. 00:16:01.169 --> 00:16:04.792 The larger numerical scale dominates the optimization. 00:16:05.589 --> 00:16:08.583 Now perform a scientifically meaningless change: 00:16:08.666 --> 00:16:12.115 write the same salaries in cents instead of dollars. 00:16:12.427 --> 00:16:16.770 Every salary value is multiplied by one hundred, and its variance is multiplied 00:16:16.770 --> 00:16:18.245 by ten thousand. 00:16:18.441 --> 00:16:22.506 Raw PCA can rotate even though nothing about the people changed. 00:16:23.302 --> 00:16:25.207 That is the blunt warning. 00:16:25.520 --> 00:16:28.713 PCA maximizes numerical variance. 00:16:28.910 --> 00:16:31.219 It cannot distinguish signal from noise, 00:16:31.302 --> 00:16:35.552 importance from nuisance, or measurement scale from genuine structure. 00:16:36.948 --> 00:16:39.143 Standardization replaces each variable 00:16:39.143 --> 00:16:43.276 by its centered value divided by its sample standard deviation. 00:16:43.728 --> 00:16:48.803 The covariance matrix of standardized variables is their correlation matrix. 00:16:49.599 --> 00:16:52.700 Do not turn that into another automatic ritual. 00:16:52.897 --> 00:16:56.030 If a variable's original scale carries real meaning, 00:16:56.113 --> 00:16:58.644 raw covariance may be appropriate. 00:16:58.957 --> 00:17:02.090 If unlike units should receive equal starting weight, 00:17:02.173 --> 00:17:04.786 standardization is often necessary. 00:17:05.490 --> 00:17:09.844 Heavy tails, outliers, counts, and strongly skewed 00:17:09.844 --> 00:17:14.860 variables may call for robust scaling or a domain-specific transformation. 00:17:15.056 --> 00:17:19.759 Whatever you choose, document it and validate it as part of the model. 00:17:20.555 --> 00:17:23.377 PCA is now one connected idea. 00:17:23.574 --> 00:17:28.158 Center the observations, find the direction of maximum projected variance, 00:17:28.241 --> 00:17:30.294 continue in perpendicular directions, 00:17:30.377 --> 00:17:34.221 and compute those directions as covariance eigenvectors. 00:17:34.418 --> 00:17:37.680 Use eigenvalues to judge compression, reconstruct 00:17:37.680 --> 00:17:42.325 from retained scores, and never forget that the units help define the answer.