by Suraj Rampure (firstname.lastname@example.org)
In discussion for Data 100, I ignored the difference between eigenvalues and singular values. They are actually very different, and we'll look at the formal definitions of both and the relationships between them here.
Suppose A∈𝕄n×n is an n×n square matrix, and v∈ℝn is an n-element column vector. Then, if we can say
this implies v is an eigenvector with corresponding eigenvalue λ.
Note, eigenvectors and eigenvalues exist only when A is a square matrix. Let's take a look at why.
Suppose A∈𝕄m×n, meaning A is a matrix with m rows and n columns. If we want to multiply A on the right by a column vector v, that vector must have n elements in order for the dimensions of A and v to allow multiplication. However, when we multiply an m×n matrix by a n×1 vector, the result will be an m×1 vector.
For example, suppose A=[210324−35160] and v=⎡⎣⎢⎢⎢⎢⎢120−13⎤⎦⎥⎥⎥⎥⎥∈ℝ5. The result of this vector multiplication will be ∈ℝ2.
The only way that Av can have the same dimensions as v is if m=n, i.e. A is square.
To find the singular values of A, we first begin by finding the eigenvalues of ATA. If A∈𝕄m×n, then ATA will be an n×n symmetric matrix. Since ATA is square, it has eigenvalues, and furthermore, all of ATA's eigenvalues will be non-negative*.
Suppose λ1,λ2,...,λn are the n eigenvalues of ATA. The singular values of A, then, are σ1=λ1‾‾‾√,σ2=λ2‾‾‾√,...,σn=λn‾‾‾√. In other words, the singular values of A are the square roots of the eigenvalues of ATA. Notice, since λi is non-negative, we can always take the square root.
When are they the same?
There is a very special case in which the singular values of a matrix are the same as the eigenvalues of a matrix.
Claim: If A is a symmetric matrix, i.e. A=AT, then the singular values of A are equal to the absolute values of the eigenvalues of A. In other words, if A is symmetric, then if λ1,λ2,...,λn are the eigenvalues of A, then σ1=|λ1|,σ2=|λ2|,...,σn=|λn|.
Proof: First, we'll show that if λ is an eigenvalue of A, then λ2 is an eigenvalue of A2 for any symmetric matrix A.
Now, we've shown that the eigenvalues of ATA are of the form λ2. The singular values of A are simply the square roots of the eigenvalues of ATA, i.e. λ2‾‾‾√. λ could have originally been negative, so we must say λ2‾‾‾√=|λ|.
This proves that if λ is an eigenvalue of a symmetric matrix A, then |λ| is a singular value of A.
Furthermore, if A is positive semi-definite, meaning it is symmetric AND all of its eigenvalues are non-negative, we can remove the absolute value symbol, and simply state that the eigenvalues and singular values of A overlap.
- Eigenvalues are only defined for square matrices, whereas singular values are defined for all matrices
- Even when a matrix is square, there is no direct relationship between its singular values and eigenvalues
- If a matrix is symmetric (also meaning it is square), then its eigenvalues and singular values are closely related by σ=|λ|
For the purposes of our course, this is relevant when looking at PCA. We find the directions in which our data vary the most by determining and ranking the singular values of our data matrix A (the "directions" that we choose are actually the eigenvectors of ATA). All of this, of course, is done after the columns of A are de-meaned.
Look at this note if you're interested in reading more about singular values and the singular value decomposition (SVD).