Mastering SVD and Matrix Norms: A Computational Guide Inspired by ENGG7302

Introduction to Advanced Computational Techniques

In engineering and data science, understanding matrix decompositions and norms is essential for solving real-world problems. This tutorial covers core topics from ENGG7302 Advanced Computational Techniques, including singular value decomposition (SVD), matrix norms, least squares, and orthogonal projections. We'll connect these concepts to modern applications like AI model compression, recommendation systems, and financial risk analysis.

Singular Value Decomposition (SVD) Basics

SVD is a factorization of a real or complex matrix: A = U Σ VH, where U and V are unitary matrices and Σ is a diagonal matrix of singular values. The full SVD includes all zero singular values, while the reduced SVD omits zeros. A common exam question asks about uniqueness: U and V are not necessarily unique, but the singular values (Σ) are unique up to ordering. This is important when using SVD for data compression, like in Netflix's recommendation algorithm.

Example: Matrix with Distinct Singular Values

Consider A = [[1,0,0],[0,2,0],[0,0,3]]. Its SVD is straightforward: U = I, Σ = diag(3,2,1), V = I. The Frobenius norm is sqrt(1+4+9)=√14≈3.74, while the 2-norm is 3, 1-norm is 3, ∞-norm is 3. Here, the Frobenius norm is largest, a common exam trap.

Matrix Norms and Condition Numbers

Matrix norms measure the size of a matrix. The 1-norm is the maximum absolute column sum, the ∞-norm is the maximum absolute row sum, the 2-norm equals the largest singular value, and the Frobenius norm is the square root of sum of squares of all elements. Condition number κ(A) = σ_max / σ_min indicates sensitivity to perturbations. In machine learning, a high condition number means your model is unstable—similar to how a small change in a viral app's algorithm can drastically affect user engagement.

Least Squares and Normal Equations

Solving Ax = b when no exact solution exists leads to least squares. The normal equation (AHA)x = AHb gives the solution if A has full column rank. In ENGG7302, you must know that AHA is invertible when the null space of A is empty. This is used in linear regression, e.g., predicting stock prices based on multiple features.

Orthogonal Projections and Decompositions

Orthogonal projection matrices satisfy P = P2 = PH. Unitary matrices satisfy UHU = I. An exam question might ask: which of the given matrices are unitary or orthogonal projections? For example, [[0,0,1]] is not an orthogonal projection because it's not square; [[0,-i],[i,0]] is unitary. These concepts are vital for signal processing and quantum computing.

Topical Example: SVD in AI Model Compression

In 2026, large language models like GPT-4 use SVD to reduce model size. By truncating small singular values, you can compress a model by 50% with minimal accuracy loss. This is analogous to the ENGG7302 question about full vs reduced SVD: the reduced SVD keeps only non-zero singular values, saving memory.

Step-by-Step: Computing SVD for a Simple Matrix

Let A = [[2,0,0],[0,0,0]]. Its full SVD: U = [[1,0],[0,1]], Σ = [[2,0,0],[0,0,0]], V = I. Reduced SVD: U = [[1],[0]], Σ = [2], V = [[1,0,0]]. The pseudo-inverse is A+ = V Σ+ UH = [[0.5,0],[0,0],[0,0]]. Condition number: since smallest singular value is 0, it's infinite (singular matrix). This illustrates why condition numbers matter in numerical stability.

Exam-Style Practice Question

Q: Given vectors u = [0,-1,2]H, v = [1,2,0]H, find inner product and rank of outer product. A: Inner product = -2, outer product rank = 1 (since rank-1 matrix). This tests basic linear algebra skills.

Conclusion

Mastering SVD, norms, and least squares is crucial for advanced engineering computations. Whether you're working on AI, finance, or gaming, these tools provide the foundation. Practice with the ENGG7302 exam questions to solidify your understanding.