PCA from Geometric Intuition to Eigenvectors

Suppose you use PCA every week. You call a library, ask for two components, and get a transformed table. But what did the algorithm actually choose? Here is the question that organizes the whole method: if we flatten a centered cloud onto one line, which line preserves the most variation? Begin with a two-dimensional data set. Each blue point is one observation, and the two axes are two measured variables. The cloud is not round. It has a long direction running roughly from lower left to upper right. PCA starts by centering each variable. Subtract its sample mean, so the red mean moves to the origin. Centering matters because PCA is supposed to describe variation around the typical observation, not distance from an arbitrary zero. Now choose a candidate direction u. We require unit length, not because short arrows are virtuous, but because otherwise we could manufacture unlimited variance merely by multiplying the direction by a huge number. Project every observation onto that line. Each gray segment drops one blue point to its green projected position. The signed coordinate along the line is its score, z sub i, equal to u transpose times the centered observation. Those scores form a one-dimensional data set. Measure their variance in the ordinary way: square each score, average the squares, and call the result V of u. At the moment the line is horizontal. The green points have some spread, and the marker reports variance four. That is neither especially small nor the largest value available. Turn the candidate line toward the visible long axis of the cloud. The projected points separate, so their one-dimensional variance rises. It reaches seven at this direction, where the spread is greatest. Continue turning. The projections draw closer together again. On the vertical axis the variance has fallen back to four, even though no data point has moved. Keep turning toward the short axis of the cloud. The green points bunch tightly around the origin, and the variance falls to one. This direction preserves very little of the cloud's visible spread. Now return to the maximum. The data stayed fixed throughout. Only the question we asked of them changed: how much of their variation can be seen from this direction? This is PCA's central geometric operation. It searches over all unit directions, projects the centered observations, and chooses the direction whose scores have the largest variance. That maximizing direction is the first principal component. A component is a direction in variable space. The projected coordinate is the score of an observation on that component. Those are related objects, but they are not the same thing. Write the first component as v one: the unit direction that maximizes projected variance. Its variance, seven in this example, tells us how much variation the component captures. After choosing v one, PCA looks for another direction that captures as much remaining variance as possible without repeating the first answer. In two dimensions, there is only one perpendicular direction available. That green direction is the second principal component. The cloud is narrow along it, so its variance is only one. The first component describes the long way through the cloud, and the second describes the remaining side-to-side variation. The two component directions are perpendicular and have unit length. Together they form a rotated coordinate system. PCA has not bent the data or invented a nonlinear feature. It has rotated the axes to line up with the directions of greatest and least variation. So the first answer to carry forward is geometric. PCA asks where the centered cloud is widest, calls that direction component one, and then repeats the question in directions perpendicular to what it has already chosen.

A real data set may have fifty variables, five thousand, or five million. We cannot rotate a literal line through all those dimensions on a screen, but the geometric question does not change. Use three dimensions as the bridge. This cloud has a long spatial direction, plus smaller deviations in two independent sideways directions. The first search is exactly the one we just performed. Among every unit direction in the space, find the one on which the projected coordinates have maximum variance. The red arrow is the answer. Projecting onto it compresses every observation to one score. Across all observations, those scores vary more than scores along any other unit direction. Now remove the possibility of choosing that answer again. Every direction perpendicular to v one lies in this gray plane through the origin. Search inside that plane for the direction with greatest projected variance. The green arrow is component two. It is the strongest remaining direction after component one has been excluded. In three dimensions, one perpendicular direction remains after the first two have been fixed. That yellow arrow becomes component three. In p dimensions the process continues the same way. Component k maximizes projected variance subject to unit length and orthogonality to every earlier component. Collect the first k component directions as columns of V sub k. Multiplying the centered data matrix X by V sub k produces the score matrix Z sub k. Each row is now the same observation expressed in principal-component coordinates. Because the component directions are mutually perpendicular unit vectors, V transpose V is the identity. This orthonormal structure is why moving into component coordinates and reconstructing from them remains simple. There cannot be more useful components than variables. There also cannot be more than n minus one nonzero directions of variation after n observations have been centered. The crucial point is that higher-dimensional PCA is not a new algorithmic idea. It is the same sequence of questions: widest direction first, then the widest perpendicular direction left, then the next. We now have the geometry. Only at this point do we need the covariance matrix, because it gives an efficient algebraic way to answer exactly the optimization question we have already understood.

Return to the centered two-dimensional cloud. We already know the geometric problem: choose a unit direction u that makes the projected scores vary as much as possible. For one centered observation x sub i, its coordinate on u is the dot product u transpose x sub i. Call that score z sub i. The scores are centered too, so their variance is the average squared score. Substitute the dot product into that familiar calculation. A scalar square can be written as u transpose x sub i, times x sub i transpose u. Put the direction vectors outside the observation-specific middle. The average of those middle outer products is the covariance matrix Sigma. With centered observations stored in the rows of X, Sigma is one over n times X transpose X. So the variance seen along u is exactly the quadratic form u transpose Sigma u. This line is not a different objective. It is the projected spread from the rotating picture, written compactly. We want to maximize that quadratic form while keeping u at unit length. Use a Lagrange multiplier to attach the constraint u transpose u equals one. At a maximum, the derivative with respect to u is zero. Differentiating gives two Sigma u minus two lambda u. Cancel the common factor of two and rearrange. Sigma u equals lambda u. The maximizing direction must therefore be an eigenvector of the covariance matrix. This is the promised restatement. The geometry asked for a direction that keeps its identity while covariance assigns it a variance. The algebra calls that direction an eigenvector. Multiply the eigenvalue equation on the left by u transpose. Because u has unit length, u transpose Sigma u equals lambda. The eigenvalue is the projected variance along its eigenvector. Therefore the first component is the eigenvector with the largest eigenvalue. The second is the perpendicular eigenvector with the next eigenvalue, and so on in descending order. Covariance matrices are symmetric and positive semidefinite. Their eigenvectors can be chosen orthonormal, and their eigenvalues are nonnegative. Those facts are exactly what the geometric construction required. Now calculate the example. Use the covariance matrix with fours on the diagonal and threes off the diagonal. Its positive covariance says the two variables tend to rise and fall together. Eigenvalues make Sigma minus lambda I singular. Subtract lambda from both diagonal entries and set the determinant to zero. Multiply the main diagonal and subtract the other product. We get four minus lambda squared, minus nine, equal to zero. Expand the square and collect terms. The characteristic polynomial is lambda squared minus eight lambda plus seven. Factor it. Lambda minus seven times lambda minus one equals zero. The two eigenvalues are seven and one, exactly the maximum and minimum projected variances from the rotating line. The eigenvector for seven points along one, one. The other eigenvector points along one, minus one. Normalize both by dividing by square root two. Seven of the total eight variance units lie along component one. So one component explains eighty-seven point five percent of this cloud's variance. The eigenvalue calculation has returned every fact we first discovered by rotating and projecting.

PCA orders components from most variance to least, but a model still needs a decision: how many components should we keep? The eigenvalues give the evidence for that decision. A scree plot puts the component number along the horizontal axis and its eigenvalue on the vertical axis. Here are five eigenvalues in descending order. The first point is high because component one captures a great deal of variation. The second captures less. After the third, the remaining eigenvalues are small. An individual explained-variance ratio divides one eigenvalue by the sum of all eigenvalues. It reports the fraction of total variance assigned to that component. Cumulative explained variance adds the first k eigenvalues before dividing by the total. It asks how much variation remains visible after reducing the data to k component coordinates. With one component, this example retains sixty-seven point six percent. With two, the total reaches eighty-eight point nine percent. With three components it reaches ninety-six point six percent. Components four and five add very little. The visible bend near component three is often called the elbow. One common heuristic keeps components before the curve settles into a shallow tail. But the elbow is not a theorem and ninety-five percent is not a universal law. Keep enough components for the downstream task. For prediction, choose k with cross-validation. For compression, choose an acceptable reconstruction error. For interpretation, check stability and domain meaning. A scree plot is diagnostic evidence, not an automatic command. A weak elbow may honestly mean there is no clean low-dimensional cutoff. Now make component retention concrete. In principal-component coordinates, a centered observation is built from one contribution along each component direction. This observation has a large score on component one, a smaller score on component two, and a still smaller score on component three. The general reconstruction formula starts at the original mean and adds back the first k score times direction contributions. Keep only component one. Starting from the mean, move along v one by the first score. The green point is the rank-one reconstruction. It preserves the strongest coordinate and discards everything in the omitted perpendicular subspace. The red dashed segment is the information lost for this observation. Now retain component two as well. Add the yellow contribution from the first reconstruction. The new point moves closer to the original. The remaining error lies entirely along component three. If we retain that final red contribution, the reconstruction reaches the original point exactly. The displayed error falls from one point three four with one component to zero point six with two. Every additional component can only reduce squared reconstruction error on the training data. Averaged over the data set, the squared error after keeping k components equals the sum of the omitted eigenvalues. The scree plot and reconstruction error are therefore two readings of the same tradeoff. Keeping a component means preserving one orthogonal pattern of variation. Dropping it means replacing every observation's score on that pattern by zero, then mapping the reduced coordinates back to the original variable space.

One warning belongs in every practical explanation of PCA. PCA does not understand variables. It understands numbers, and it rewards whichever direction has the greatest numerical variance. On the left, salary deviation is measured in dollars and age deviation in years. These are centered observations, so subtracting the means has already been done. The salary coordinates contain tens of thousands while the age coordinates contain tens. The raw cloud therefore looks overwhelmingly horizontal, and raw PCA chooses an almost horizontal first component. That result does not prove salary is the most important scientific variable. It proves salary has the largest variance in the numbers handed to the algorithm. On the right, each variable has been centered and divided by its sample standard deviation. Both begin with variance one, so their relationship, rather than their original unit sizes, determines the component. The two answers differ even though they describe the same observations. Scaling has changed the geometry, and changing the geometry changes PCA. Here is the raw covariance matrix. Salary variance is four hundred million, while age variance is one hundred. The larger numerical scale dominates the optimization. Now perform a scientifically meaningless change: write the same salaries in cents instead of dollars. Every salary value is multiplied by one hundred, and its variance is multiplied by ten thousand. Raw PCA can rotate even though nothing about the people changed. That is the blunt warning. PCA maximizes numerical variance. It cannot distinguish signal from noise, importance from nuisance, or measurement scale from genuine structure. Standardization replaces each variable by its centered value divided by its sample standard deviation. The covariance matrix of standardized variables is their correlation matrix. Do not turn that into another automatic ritual. If a variable's original scale carries real meaning, raw covariance may be appropriate. If unlike units should receive equal starting weight, standardization is often necessary. Heavy tails, outliers, counts, and strongly skewed variables may call for robust scaling or a domain-specific transformation. Whatever you choose, document it and validate it as part of the model. PCA is now one connected idea. Center the observations, find the direction of maximum projected variance, continue in perpendicular directions, and compute those directions as covariance eigenvectors. Use eigenvalues to judge compression, reconstruct from retained scores, and never forget that the units help define the answer.