Probability Distributions

Probability distributions provide a mathematical framework for modeling the uncertainty inherent in financial markets. They are essential for tasks ranging from asset pricing and risk management to portfolio optimization and algorithmic trading.

I. Foundational Concepts

A Random Variable (RV) is a variable whose value is a numerical outcome of a random phenomenon. RVs are classified as Discrete (countable outcomes, e.g., number of defaults) or Continuous (uncountable outcomes over a range, e.g., asset price).

Moment Generating Functions (MGF)

The Moment Generating Function (MGF), $M_X(\theta) = \mathbb{E}[e^{\theta X}]$, is a powerful tool.

• Utility: The $k$-th moment of the distribution ($\mathbb{E}[X^k]$) can be found by taking the $k$-th derivative of the MGF and evaluating it at $\theta=0$.
• Sum of RVs: The MGF of the sum of independent random variables is the product of their individual MGFs: $M_{X+Y}(\theta) = M_X(\theta) M_Y(\theta)$.

II. Key Distributions in Statistics

The following table summarizes the most critical distributions, their parameters, and their relevance in financial modeling.

Essential Formulas and Theorems

A deep understanding of core statistical principles is crucial for modeling financial markets, pricing derivatives, and managing risk.

I. Core Probability Laws

These laws govern how probabilities are calculated and updated, forming the basis for statistical inference and decision-making under uncertainty.

Conditional Probability, Bayes' Theorem, and Law of Total Probability

Consider events $A_1, \dots, A_n$ which form a partition of the sample space (i.e., they are mutually exclusive and collectively exhaustive) and an event $B$.

II. Moments and Relationships

Moments describe the shape and location of a probability distribution. Understanding their properties is key to manipulating random variables in models.

Law of the Unconscious Statistician (LOTUS)

The expected value of a function of a random variable $g(X)$ can be calculated without first finding the distribution of $Y=g(X)$.

Law of Total Expectation and Variance

These laws are essential for models where one random variable depends on another (e.g., a two-stage process or a mixture model).

Intuitively, the Law of Total Expectation says that if we "average over all averages" of $X$ obtained by some information about $Y$, we obtain the true average. Similarly, the Law of Total Variance says that the true variance comes from two sources: between samples (the first term) and within samples (the second term).

Covariance and Correlation

These measure the linear relationship between two random variables $X$ and $Y$.

Key Properties of Variance and Covariance:

1. $\text{Var}(aX + b) = a^2\text{Var}(X)$
2. $\text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y) + 2\text{Cov}(X, Y)$
3. If $X$ and $Y$ are independent, $\text{Cov}(X, Y) = 0$, and $\text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y)$. Note: The converse is not always true (uncorrelated does not imply independent).

Common Relationships Between Distributions

III. Fundamental Theorems and Inequalities

These theorems provide the theoretical justification for many statistical and financial models, particularly those involving large samples or long time horizons.

Central Limit Theorem (CLT)

Let $X_1, X_2, \dots, X_n$ be a sequence of i.i.d. random variables with mean $\mu$ and finite variance $\sigma^2$. As $n \to \infty$, the distribution of the standardized sample mean approaches the standard normal distribution:

Relevance: Justifies the use of the Normal distribution to model asset returns, as returns are the sum of many small, independent price changes. It also underpins statistical inference (e.g., confidence intervals, hypothesis testing).

Law of Large Numbers (LLN)

The LLN states that as the number of trials increases, the average of the results obtained from a large number of independent and identically distributed random variables converges to the expected value.

Relevance: Guarantees that Monte Carlo simulations will converge to the true expected value as the number of simulations increases.

Markov's and Chebyshev's Inequalities

These inequalities provide bounds on the probability that a random variable deviates from its mean, even when the full distribution is unknown.

IV. Quant Finance Specific Tools

These formulas are indispensable for derivative pricing and continuous-time modeling.

Ito's Lemma

Ito's Lemma is the fundamental rule of differentiation for stochastic processes, particularly those involving Brownian motion (Wiener process). It is the stochastic equivalent of the chain rule in standard calculus.

For a function $G(t, X_t)$ where $X_t$ follows the Ito process $dX_t = \mu(X_t, t) dt + \sigma(X_t, t) dW_t$, the differential $dG$ is:

Relevance: Used to derive the Black-Scholes Partial Differential Equation (PDE) and to find the process followed by a function of an asset price (e.g., the log-price).

Geometric Brownian Motion (GBM)

GBM is the most common model for asset prices $S_t$ in continuous time, assuming log-returns are normally distributed.

• $\mu$: Drift (expected return)
• $\sigma$: Volatility
• $dW_t$: Wiener process (Brownian motion)

The solution for $S_t$ is Lognormal: $S_t = S_0 \exp\left( \left(\mu - \frac{1}{2}\sigma^2\right) t + \sigma W_t \right)$.

Black-Scholes-Merton (BSM) Formula (European Call Option)

The BSM formula provides a closed-form solution for the price of a European call option $C$:

where:

• $S$: Current stock price
• $K$: Strike price
• $r$: Risk-free interest rate
• $T-t$: Time to maturity
• $\sigma$: Volatility of the stock return
• $N(\cdot)$: Cumulative distribution function of the standard normal distribution

Risk-Neutral Valuation

The First Fundamental Theorem of Asset Pricing states that in a market with no arbitrage, there exists at least one risk-neutral measure $\mathbb{Q}$ under which the price of any derivative $V$ is the discounted expected value of its payoff, $V_T$, under this measure.

Relevance: This is the core principle of modern derivative pricing. The BSM formula is derived by applying this principle to the GBM process under the risk-neutral measure. The key change is that the drift $\mu$ of the asset price process is replaced by the risk-free rate $r$.

Markov Chains

Markov Chains are a fundamental tool for modeling systems that transition between a finite number of states, where the future state depends only on the current state, not on the sequence of events that preceded it.

I. Core Definitions and Properties

The Markov Property

A sequence of random variables $X_1, X_2, X_3, \dots$ is a Markov Chain if it satisfies the Markov Property (or memoryless property): the conditional probability distribution of the next state, given the present state and all the past states, depends only on the present state.

Transition Matrix

For a discrete state space $\mathcal{X} = \{x_1, x_2, \dots, x_n\}$, the dynamics of the chain are governed by the $n \times n$ Transition Matrix $P$.

• Entry $P_{ij}$: The probability of transitioning from state $x_i$ to state $x_j$.
• Properties: Each entry $P_{ij} \in [0, 1]$, and the sum of entries for each row must total 1 (i.e., $\sum_{j=1}^n P_{ij} = 1$). This makes $P$ a stochastic matrix.
• $k$-step Transition: The probability of moving from state $i$ to state $j$ in $k$ steps is given by the $(i, j)$-th entry of the matrix $P^k$.

II. Classification of States and Chains

The long-term behavior of a Markov Chain is determined by the properties of its states.

III. Stationary Distribution and Long-Term Behavior

The Stationary Distribution $\pi = (\pi_1, \dots, \pi_n)$ is a probability vector that, once reached, remains unchanged by further transitions.

• Defining Equation:
  $$\pi = \pi P, \quad \sum_{i=1}^n \pi_i = 1$$
• Interpretation: $\pi_i$ is the long-run proportion of time the chain spends in state $x_i$. In finance, this can represent the long-run probability of a market being in a certain regime (e.g., high volatility).
• Existence and Uniqueness: A stationary distribution exists for any finite-state Markov Chain. It is unique if and only if the chain is irreducible.

IV. Absorbing Chains and Expected Hitting Time

An Absorbing State $x_i$ is a state from which the chain cannot leave (i.e., $P_{ii} = 1$). A chain is Absorbing if it has at least one absorbing state and every non-absorbing state can reach an absorbing state.

Expected Time to State (Expected Hitting Time)

To find the expected number of steps $\mu_i$ to reach a target state (often an absorbing state) starting from state $x_i$, we solve a system of linear equations.

For a target state $x_n$ (where $\mu_n = 0$):

• Example: To find the expected time to reach $x_3$ from $x_1$ in a $3 \times 3$ chain:
  $$\mu_1 = 1 + P_{11}\mu_1 + P_{12}\mu_2 + P_{13}\mu_3 \quad (\text{where } \mu_3 = 0)$$
  $$\mu_2 = 1 + P_{21}\mu_1 + P_{22}\mu_2 + P_{23}\mu_3 \quad (\text{where } \mu_3 = 0)$$

Gambler's Ruin Problem (A Classic Absorbing Chain)

This is a classic example of an absorbing Markov Chain where the states are the player's current capital, and the absorbing states are 0 (ruin) and $a+b$ (opponent's ruin).

• Fair Coin ($p=0.5$): The probability of ruin (reaching 0) starting with capital $a$ against an opponent with capital $b$ is:
  $$\mathbb{P}(\text{Ruin}) = \frac{b}{a+b}$$
  (Correction: The probability of ruin is $\frac{b}{a+b}$, not $\frac{a}{a+b}$ as stated in the original content. The probability of winning is $\frac{a}{a+b}$.)
• Unfair Coin ($p \ne 0.5$): Let $\rho = \frac{1-p}{p}$ (the odds ratio of losing to winning). The probability of ruin is:
  $$\mathbb{P}(\text{Ruin}) = \frac{\rho^a - \rho^{a+b}}{1 - \rho^{a+b}}$$
  (Correction: The original formula was for the probability of reaching state $a+b$ starting from $a$ in a slightly different formulation. The standard ruin probability is given above.)

Linear Regression

Linear Regression forms the basis for models like the Capital Asset Pricing Model (CAPM), factor models, and many trading strategies.

I. Simple and Multiple Linear Regression

Model Formulation

The core assumption is a linear relationship between a dependent variable $Y$ and one or more independent variables $X_i$.

• $Y$: Dependent variable (e.g., stock return)
• $X_i$: Independent variables/Predictors (e.g., market return, factors)
• $\beta_0$: Intercept
• $\beta_i$: Regression coefficients (slopes)
• $\epsilon$: Error term (residual), representing unmodeled variation

Ordinary Least Squares (OLS) Estimation

OLS finds the coefficients $\hat{\beta}$ that minimize the Residual Sum of Squares (RSS): $RSS = \sum_{i=1}^m (y_i - \hat{y}_i)^2$.

Matrix Form (Multiple Regression): Given the data matrix $\mathbf{X}$ (including a column of ones for the intercept) and the response vector $\mathbf{y}$, the OLS estimator is:

The variance-covariance matrix of the estimated coefficients is:

where $\sigma^2$ is the variance of the error term, estimated by $\hat{\sigma}^2 = \frac{1}{m-p-1} \sum_{i=1}^m (y_i - \hat{y}_i)^2$.

II. The Gauss-Markov Theorem and OLS Assumptions

The OLS estimator $\hat{\beta}$ is the Best Linear Unbiased Estimator (BLUE) if the following assumptions (the Gauss-Markov assumptions) hold.

Note: The OLS estimator is BLUE under assumptions 1-5. If we add the assumption that $\epsilon \sim N(0, \sigma^2)$, the OLS estimator is also the Maximum Likelihood Estimator (MLE).

III. Model Assessment and Inference

IV. Dealing with Violations and Model Selection

Robust Standard Errors

When Heteroscedasticity or Autocorrelation (or both) are present, the OLS standard errors are biased. Heteroscedasticity-Consistent (HC) Standard Errors (e.g., White's or Newey-West for autocorrelation) are used to correct the standard errors, allowing for valid statistical inference even when the error variance is not constant.

Regularization Methods (Shrinkage)

These methods address the issue of Multicollinearity and Overfitting by adding a penalty term to the OLS objective function, shrinking the coefficients towards zero. This reduces the variance of the coefficient estimates at the cost of introducing a small bias (Bias-Variance Tradeoff).

Bias-Variance Tradeoff

The expected prediction error (EPE) of a model $\hat{f}(x)$ can be decomposed:

• Bias: Error from approximating a real-world function $f$ with a simpler model $\hat{f}$.
• Variance: Error from the model being too sensitive to the training data.
• Tradeoff: More complex models (e.g., high-degree polynomials) have low bias but high variance (overfitting). Simpler models (e.g., OLS) have high bias but low variance (underfitting). Regularization methods aim to find the optimal balance.

Classification

Classification methods are used to predict a discrete outcome, such as whether a stock price will go up or down, a company will default, or a trading signal will be positive or negative.

I. Core Classification Models

1. Logistic Regression (Discriminative Model)

Logistic Regression is a linear model used for binary classification. It models the probability of a class membership using the logistic (sigmoid) function to map a linear combination of predictors to a probability between 0 and 1.

• Log-Odds: The model is linear in the log-odds (or logit):
  $$\ln\left(\frac{\mathbb{P}(Y=1 | \mathbf{x})}{\mathbb{P}(Y=0 | \mathbf{x})}\right) = \beta_0 + \mathbf{x}^\intercal \boldsymbol{\beta}$$
• Estimation: Coefficients $\boldsymbol{\beta}$ are estimated using Maximum Likelihood Estimation (MLE), as there is no closed-form solution.
• Decision Boundary: The decision boundary is linear, defined by $\beta_0 + \mathbf{x}^\intercal \boldsymbol{\beta} = 0$.

2. Discriminant Analysis (Generative Model)

Discriminant Analysis models the distribution of the predictors $\mathbf{X}$ separately for each class $k$, $f_k(\mathbf{x}) = \mathbb{P}(\mathbf{X} = \mathbf{x} | Y = k)$, and then uses Bayes' Theorem to find the posterior probability $\mathbb{P}(Y = k | \mathbf{X} = \mathbf{x})$.

• Linear Discriminant Analysis (LDA): Assumes that $f_k(\mathbf{x})$ is a multivariate Gaussian distribution with a common covariance matrix $\boldsymbol{\Sigma}$ across all classes. This results in a linear decision boundary.
• Quadratic Discriminant Analysis (QDA): Assumes that $f_k(\mathbf{x})$ is a multivariate Gaussian distribution with a unique covariance matrix $\boldsymbol{\Sigma}_k$ for each class. This results in a quadratic decision boundary.

3. $k$-Nearest Neighbors ($k$-NN) (Non-Parametric Model)

$k$-NN is a non-parametric, instance-based learning algorithm. It classifies a new observation by finding the $k$ closest training observations (based on a distance metric like Euclidean distance) and assigning the new observation to the most frequent class among its neighbors.

• Key Parameter: $k$ (number of neighbors). A small $k$ leads to high variance (overfitting), while a large $k$ leads to high bias (underfitting).
• Curse of Dimensionality: $k$-NN performance degrades rapidly as the number of features (dimensions) increases, a common issue in high-dimensional financial data.

4. Naive Bayes

Naive Bayes is a generative model that simplifies the estimation of $f_k(\mathbf{x})$ by making the strong assumption that the predictors are conditionally independent given the class $Y=k$.

• Advantage: Computationally efficient and performs surprisingly well in many real-world applications, especially text classification (e.g., sentiment analysis of news articles).

II. Model Performance Metrics

In classification, simply measuring accuracy is often insufficient, especially with imbalanced datasets (e.g., credit default prediction).

Confusion Matrix

A $2 \times 2$ table summarizing the model's performance on a test set.

Key Metrics

ROC Curve and AUC

• ROC (Receiver Operating Characteristic) Curve: Plots the True Positive Rate (Recall) against the False Positive Rate ($\frac{FP}{FP + TN}$) at various threshold settings.
• AUC (Area Under the Curve): The area under the ROC curve. It represents the probability that the model will rank a randomly chosen positive instance higher than a randomly chosen negative instance.
  • Interpretation: An AUC of 1.0 is a perfect classifier; 0.5 is no better than random guessing. AUC is a robust metric for imbalanced datasets.

Tree Methods

Tree-based methods are powerful, non-linear machine learning techniques widely used for their ability to capture complex interactions and non-linear relationships in data, which are often missed by traditional linear models.

I. Decision Trees (Single Trees)

A Decision Tree partitions the feature space into a set of non-overlapping regions. For any given observation, the prediction is the mean of the response values (for regression) or the most frequent class (for classification) of the training observations that fall into that region.

Splitting Criteria

The process of building a tree involves recursively splitting the data based on the feature and split point that maximizes the "purity" of the resulting nodes.

Advantages and Disadvantages

• Pros: Easy to interpret (white-box model), can handle non-linear relationships, and naturally handles categorical predictors.
• Cons: High variance (small changes in data can lead to a very different tree), prone to overfitting, and generally lower predictive accuracy than ensemble methods.

II. Ensemble Methods (Reducing Variance and Bias)

Ensemble methods combine multiple individual decision trees to improve overall predictive performance and robustness.

1. Bagging (Bootstrap Aggregating)

Bagging is a general-purpose procedure for reducing the variance of a statistical learning method.

• Mechanism:
  1. Generate $B$ bootstrap samples (sampling with replacement) from the original training data.
  2. Train a full, unpruned decision tree on each bootstrap sample.
  3. Aggregate the predictions: average the predictions (regression) or take a majority vote (classification).
• Out-of-Bag (OOB) Error: Since each tree is trained on only about $2/3$ of the data, the remaining $1/3$ (OOB observations) can be used as a validation set to estimate the test error without the need for cross-validation.

2. Random Forests

Random Forests are an improvement over bagging that aims to decorrelate the trees, further reducing variance.

• Mechanism:
  1. Use the bagging procedure (bootstrap samples).
  2. At each split in the tree-building process, only a random subset of $m$ predictors is considered as split candidates, where $m \ll p$ (total number of predictors).
• Hyperparameter $m$: Typically set to $\sqrt{p}$ for classification and $p/3$ for regression. By forcing the algorithm to ignore the strongest predictor in some trees, the resulting trees are less correlated, leading to a greater reduction in variance when averaged.
• Feature Importance: Random Forests provide a robust measure of Variable Importance by calculating the total decrease in node impurity (e.g., Gini index) averaged over all trees.

3. Boosting

Boosting is an ensemble technique that focuses on sequentially building trees to reduce bias.

• Mechanism:
  1. Start with a simple model (e.g., a single tree).
  2. Sequentially fit new trees to the residuals (or pseudo-residuals in generalized boosting) of the previous step. Each new tree attempts to correct the errors of the previous ensemble.
  3. Each new tree's contribution is scaled by a small learning rate $\lambda$ to slow down the learning process, which improves generalization.
• Key Algorithms: AdaBoost (Adaptive Boosting) and Gradient Boosting Machines (GBM), including modern implementations like XGBoost and LightGBM.
• Tradeoff: Boosting generally achieves higher predictive accuracy than bagging/Random Forests but is more prone to overfitting if the learning rate is too high or the number of trees is too large.

Neural Networks

Neural Networks (NNs) and Deep Learning (DL) represent a powerful class of non-linear models capable of learning complex patterns and representations directly from data. While historically less prevalent in finance due to their "black-box" nature and data requirements, they are increasingly used for tasks where non-linearity and high-dimensional data are key.

I. Core Architecture and Mechanics

The Neuron and the Network

A neural network is a composition of simple, interconnected units called neurons or nodes, organized in layers.

• Feedforward Pass: The output of a network is calculated by sequentially applying a linear transformation followed by a non-linear activation function $f(\cdot)$ at each layer.
  $$\mathbf{h}^{(l)} = f^{(l)}(\mathbf{W}^{(l)} \mathbf{h}^{(l-1)} + \mathbf{b}^{(l)})$$
  where $\mathbf{h}^{(l)}$ is the output of layer $l$, $\mathbf{W}^{(l)}$ are the weights, and $\mathbf{b}^{(l)}$ are the biases.
• Universal Approximation Theorem: A feedforward network with a single hidden layer and a non-linear activation function can approximate any continuous function to an arbitrary degree of accuracy. This is the theoretical basis for their power.

Activation Functions

Activation functions introduce the essential non-linearity that allows NNs to model complex relationships.

II. Training the Network

Loss Function and Optimization

Training involves minimizing a Loss Function (or Cost Function) $L(\mathbf{y}, \hat{\mathbf{y}})$ that measures the discrepancy between the network's prediction $\hat{\mathbf{y}}$ and the true value $\mathbf{y}$.

• Regression: Mean Squared Error (MSE).
• Classification: Cross-Entropy Loss (or Log Loss).

Backpropagation and Gradient Descent

The network's parameters ($\mathbf{W}$ and $\mathbf{b}$) are updated iteratively using an optimization algorithm, typically a variant of Stochastic Gradient Descent (SGD).

• Gradient Descent: Updates parameters in the direction opposite to the gradient of the loss function.
• Backpropagation: An efficient algorithm for computing the gradient of the loss function with respect to every weight in the network. It uses the chain rule of calculus to propagate the error signal backward from the output layer to the input layer.

Regularization and Overfitting

Due to the massive number of parameters, NNs are highly susceptible to overfitting.

• Dropout: A regularization technique where randomly selected neurons are temporarily ignored during training. This prevents co-adaptation of neurons and forces the network to learn more robust features.
• Early Stopping: Halting the training process when the performance on a separate validation set begins to degrade, even if the loss on the training set is still decreasing.

III. Specialized Architectures for Finance

The choice of architecture depends heavily on the structure of the financial data.

Large Language Models (LLMs)

Large Language Models (LLMs) are transforming quantitative finance by providing powerful tools for processing unstructured data, generating predictive signals, and enabling autonomous decision-making. Their ability to understand context and reason over vast textual corpora makes them essential for extracting alpha from non-traditional data sources.

I. Core Architecture and Mechanics

The Transformer Architecture

LLMs are built upon the Transformer architecture, which introduced the self-attention mechanism to efficiently process sequential data (text).

• Self-Attention: Allows the model to weigh the importance of different words in the input sequence when processing a specific word. This mechanism is key to capturing long-range dependencies and context, which is crucial for understanding complex financial narratives.
• Encoder-Decoder vs. Decoder-Only:
  • Encoder-Decoder (e.g., BERT): Used for tasks like classification and sequence-to-sequence translation (e.g., summarizing a report).
  • Decoder-Only (e.g., GPT-series): Used for generative tasks, predicting the next token in a sequence, which forms the basis of conversational AI and content generation.

Training Paradigms

LLMs are typically trained in a multi-stage process:

II. LLMs as Predictors: Processing Unstructured Data

The primary role of LLMs in alpha generation is to transform qualitative, unstructured data into quantitative, predictive signals.

1. Sentiment Extraction

LLMs excel at extracting nuanced sentiment from text, moving beyond simple keyword counting.

• Embedding-Based Classifiers: Using pre-trained LLMs (like FinBERT) to generate dense vector representations (embeddings) of financial text, which are then fed into traditional classifiers.
• Prompt-Based Classification: Directly prompting a generative LLM (like GPT-4) to classify the sentiment of a news headline or earnings report, leveraging its advanced reasoning capabilities. This has shown predictive power even after accounting for traditional factors.

2. Factor Generation

LLMs can act as a "factor agent" to generate novel alpha factors.

• Conceptual Factor Discovery: LLMs can be prompted to conceptualize new trading factors based on financial theory and market intuition, and even generate the Python code required to compute them from raw data. This automates the initial, creative phase of factor research.
• Relational Representation: LLMs can extract complex relationships between companies, sectors, or events from text, which can be used to build dynamic Knowledge Graphs for more sophisticated network-based predictions.

III. LLMs as Agents: Autonomous Decision-Making

The most advanced application involves integrating LLMs into multi-agent systems that can autonomously execute complex financial workflows.

• Architecture: LLM-based quant agents typically combine a central LLM (for reasoning and planning) with external Tools (APIs for data retrieval, numerical computation, and order execution).
• Multi-Agent Systems: These frameworks simulate a trading desk, with specialized LLM agents (e.g., a Fundamental Analyst, a Technical Analyst, a Portfolio Manager) collaborating to make decisions. This approach enhances robustness and provides a degree of Explainability through the agents' natural language reasoning chains.
• Financial Decision-Making: Agents can handle the entire alpha pipeline:
  1. Data Processing: Analyze news, reports, and social media.
  2. Prediction: Generate trading signals.
  3. Portfolio Optimization: Use external solvers to determine optimal asset allocation.
  4. Execution: Interact with trading APIs to place orders.

IV. Challenges in Quant Finance

Despite their power, LLMs face unique challenges in the financial domain:

Matrix Basics

Fundamental Knowledge

Let $A$ and $B$ be square $n \times n$ matrices. Then all of the following hold:

Nonsingular Matrices

A nonsingular matrix is invertible. $A$ ($n \times n$) is nonsingular if and only if any (and therefore all) of the following hold:

1. Columns of $A$ span $\mathbb{R}^n$, or equivalently, $\text{rank}(A) = \text{dim}(\text{range}(A)) = n$
2. $A^\intercal$ is nonsingular
3. $\det(A) \neq 0$
4. $Ax = 0$ has only the trivial solution $x = 0$; $\text{dim}(\text{nul}(A)) = 0$

Note that if $A = \begin{bmatrix} a & b \\ c & d \end{bmatrix}$, then $A^{-1} = \frac{1}{\det(A)} \begin{bmatrix} d & -b \\ -c & a \end{bmatrix}$. Larger inverses may be found via Gauss-Jordan Elimination: $[A \mid I] \xrightarrow{\text{elementary row operations}} [I \mid A^{-1}]$

2D Rotation Matrices

2D Rotation matrices by $\theta$ radians counter-clockwise about the origin are matrices in the form $R_\theta = \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}$.

Orthogonal Matrices

Orthogonal matrices (unitary matrices in the reals) are square with orthonormal row and column vectors. They are nonsingular and satisfy $Q^\intercal = Q^{-1}$. Orthogonal matrices can be interpreted as rotation matrices.

Idempotent Matrices

Idempotent matrices are square matrices satisfying $A^2 = A$. In other words, the effect of applying the linear transformation $A$ twice is the same as applying it once. Projection matrices are Idempotent.

Positive Semi-definite Matrices

Covariance and correlation matrices are always positive semi-definite and positive definite if there is no perfect linear dependence among random variables. Each of the following conditions is a necessary and sufficient condition for $A$ to be positive semi-definite/definite:

Note that if $A$ has negative diagonal elements, then $A$ cannot be positive semi-definite.

Matrix Decompositions

Diagonalizable Matrices

$A$ is diagonalizable if and only if it has linearly independent eigenvectors, or equivalently, if the geometric multiplicity and the algebraic multiplicity of all the eigenvalues agree. A special case of this is if $A$ has $n$ distinct eigenvalues. Suppose we have eigenvalues $\lambda_1, \dots, \lambda_n$ and corresponding eigenvectors $v_1, \dots, v_n$. Then

Intuitively, this says that we can find a basis consisting of the eigenvectors of $A$. Useful for computing large powers of $A$, as $A^n = XD^n X^{-1}$. An important example is $A$ being real and symmetric implies $A$ is diagonalizable.

Singular Value Decomposition

SVD is powerful in low-rank approximations of matrices. Unlike eigenvalue decomposition, SVD uses two unique bases (left/right singular vectors). For orthogonal matrices $U (m \times m), V (n \times n)$ and diagonal matrix $\Sigma (m \times n)$ with nonnegative diagonal entries in nonincreasing order, we can write any $m \times n$ matrix $A$ as:

Intuitively, this says that we can express $A$ as a diagonal matrix with suitable choices of (orthogonal) bases.

QR Decomposition

For nonsingular $A$, we can write $A = QR$, where $Q$ is orthogonal and $R$ is an upper triangular matrix with positive diagonal elements. QR decomposition assists in increasing the efficiency of solving $Ax = b$ for nonsingular $A$:

QR decomposition is very useful in efficiently solving large numerical systems and inversion of matrices. Furthermore, it is also used in least-squares when our data is not full rank.

LU and Cholesky Decompositions

For nonsingular $A$, we can write $A = LU$, where $L$ is a lower and $U$ is an upper triangular matrix. This decomposition assists in solving $Ax = b$ as well as computing the determinant:

If $A$ is symmetric positive definite, then $A$ can be expressed as $A = R^\intercal R$ via Cholesky decomposition, where $R$ is an upper triangular matrix with positive diagonal entries. Cholesky decomposition is essentially LU decomposition with $L = U^\intercal$. These decompositions are both useful for solving large linear systems.

Projections

Fix a vector $v \in \mathbb{R}^n$. The projection of $x \in \mathbb{R}^n$ onto $v$ is given by

More generally, if $S = \text{Span}\{v_1, \dots, v_k\} \subseteq \mathbb{R}^n$ has orthogonal basis $\{v_1, \dots, v_k\}$, then the projection of $x \in \mathbb{R}^n$ onto $S$ is given by

The main property is that $\text{proj}_S(x) \in S$ and $x - \text{proj}_S(x)$ is orthogonal to any $s \in S$. Linear Regression can be viewed as a projection of our observed data onto the subspace formed by the span of the collected data.

Calculus Basics

Differentiation

At all points $x$ where the functions and the derivatives are defined,

Integration

Disregarding the $+C$ on all the integrals,