Determine Svds of The Following Matrices by Hand Calculation

Singular Value Decomposition (SVD) is a fundamental matrix factorization technique in linear algebra. This guide explains how to compute SVD by hand for any given matrix, including step-by-step instructions, formulas, and practical examples.

What is Singular Value Decomposition (SVD)?

Singular Value Decomposition is a matrix factorization technique that decomposes a matrix into three simpler matrices:

A = U Σ Vᵀ

Where:

A is the original m×n matrix
U is an m×m orthogonal matrix (left singular vectors)
Σ is an m×n diagonal matrix (singular values)
Vᵀ is an n×n orthogonal matrix (right singular vectors)

SVD has applications in data compression, dimensionality reduction, and solving linear systems.

How to Calculate SVD by Hand

Calculating SVD manually involves several steps:

Compute the covariance matrix AᵀA or AAᵀ
Find the eigenvalues of the covariance matrix
Compute the singular values from the eigenvalues
Find the eigenvectors to form U and V
Construct the Σ matrix from the singular values

Note: For non-square matrices, you'll need to compute both AᵀA and AAᵀ to get all singular values and vectors.

Step-by-Step Process

For a given matrix A:

Calculate AᵀA or AAᵀ
Find the eigenvalues λ of the covariance matrix
Compute singular values σ = √λ
Find eigenvectors v for AᵀA or u for AAᵀ
Construct U, Σ, and V matrices

Example Calculation

Let's compute the SVD of the matrix:

A = [1 1; 0 1]

Step 1: Compute AᵀA

AᵀA = [1 0; 1 1] * [1 1; 0 1] = [1 1; 1 2]

Step 2: Find eigenvalues of AᵀA

Solve |AᵀA - λI| = 0:

|1-λ 1; 1 2-λ| = (1-λ)(2-λ) - 1 = λ² - 3λ + 1 = 0

Solutions: λ₁ = 2 + √3, λ₂ = 2 - √3

Step 3: Compute singular values

σ₁ = √(2 + √3) ≈ 1.9319 σ₂ = √(2 - √3) ≈ 0.7320

Step 4: Find eigenvectors for V

For λ₁ = 2 + √3:

(1-λ₁)v₁ + v₂ = 0 → v₁ = -v₂

Choose v₁ = [1; -1] (normalized)

For λ₂ = 2 - √3:

(1-λ₂)v₁ + v₂ = 0 → v₁ = v₂

Choose v₁ = [1; 1] (normalized)

Step 5: Construct Σ and V

Σ = [σ₁ 0; 0 σ₂] ≈ [1.9319 0; 0 0.7320] V = [1 1; -1 1] (normalized columns)

Step 6: Find U

Compute U = A V Σ⁻¹

U = [1 1; 0 1] * [1 1; -1 1] * [1/σ₁ 0; 0 1/σ₂] ≈ [0.5145 0.4855; -0.8575 0.5145]

Interpreting the Results

The SVD provides several useful insights:

The singular values (σ₁, σ₂, ...) represent the importance of each dimension in the data
The columns of U and V represent the principal components of the matrix
The rank of the matrix is equal to the number of non-zero singular values

In our example, the first singular value is much larger than the second, indicating that most of the information in the matrix is captured by the first principal component.

FAQ

What is the difference between SVD and eigenvalue decomposition?: Eigenvalue decomposition only works for square matrices, while SVD works for any m×n matrix. SVD provides more complete information about the matrix structure.
How many singular values does a matrix have?: A matrix has min(m,n) singular values, where m and n are the matrix dimensions.
Can SVD be used for data compression?: Yes, by keeping only the largest singular values and corresponding vectors, you can approximate the original matrix with fewer data points.
What are the computational advantages of SVD?: SVD provides a numerically stable way to solve linear systems, compute matrix inverses, and perform dimensionality reduction.
How is SVD used in machine learning?: SVD is used in principal component analysis (PCA), collaborative filtering, and latent semantic indexing to reduce dimensionality and extract meaningful patterns.