Matrix Basics

Fundamental Knowledge

Let $A$ and $B$ be square $n \times n$ matrices. Then all of the following hold:

\cos(\theta) = \frac{x^\intercal y}{\|x\| \|y\|} \quad (AB)^\intercal = B^\intercal A^\intercal \quad (AB)^{-1} = B^{-1} A^{-1} \quad A^{-1}A = AA^{-1} = I \quad \text{rank}(A) + \text{null}(A) = n

Av = \lambda v \implies (A - \lambda I)v = 0 \implies \det(A - \lambda I) = 0 \quad \det(A) = \frac{1}{\det(A^{-1})} \quad \det(A) = \det(A^\intercal)

\det(AB) = \det(A)\det(B) \quad \det(cA) = c^n\det(A) \quad \det(A) = \prod_{i=1}^n \lambda_i \quad \text{trace}(A) = \sum_{i=1}^n A_{ii} = \sum_{i=1}^n \lambda_i

Nonsingular Matrices

A nonsingular matrix is invertible. $A$ ( $n \times n$ ) is nonsingular if and only if any (and therefore all) of the following hold:

Columns of $A$ span $\mathbb{R}^n$ , or equivalently, $\text{rank}(A) = \text{dim}(\text{range}(A)) = n$
$A^\intercal$ is nonsingular
$\det(A) \neq 0$
$Ax = 0$ has only the trivial solution $x = 0$ ; $\text{dim}(\text{nul}(A)) = 0$

Note that if $A = \begin{bmatrix} a & b \\ c & d \end{bmatrix}$ , then $A^{-1} = \frac{1}{\det(A)} \begin{bmatrix} d & -b \\ -c & a \end{bmatrix}$ . Larger inverses may be found via Gauss-Jordan Elimination: $[A \mid I] \xrightarrow{\text{elementary row operations}} [I \mid A^{-1}]$

2D Rotation Matrices

2D Rotation matrices by $\theta$ radians counter-clockwise about the origin are matrices in the form $R_\theta = \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}$ .

Orthogonal Matrices

Orthogonal matrices (unitary matrices in the reals) are square with orthonormal row and column vectors. They are nonsingular and satisfy $Q^\intercal = Q^{-1}$ . Orthogonal matrices can be interpreted as rotation matrices.

Idempotent Matrices

Idempotent matrices are square matrices satisfying $A^2 = A$ . In other words, the effect of applying the linear transformation $A$ twice is the same as applying it once. Projection matrices are Idempotent.

Positive Semi-definite Matrices

Covariance and correlation matrices are always positive semi-definite and positive definite if there is no perfect linear dependence among random variables. Each of the following conditions is a necessary and sufficient condition for $A$ to be positive semi-definite/definite:

Positive Semi-Definite	Positive Definite
$z^\intercal Az \ge 0$ for all column vectors $z$	$z^\intercal Az > 0$ for all nonzero column vectors $z$
All eigenvalues are nonnegative	All eigenvalues are positive
All upper left/lower right submatrices have nonnegative determinants	All upper left/lower right submatrices have positive determinants

Note that if $A$ has negative diagonal elements, then $A$ cannot be positive semi-definite.