Classification

Classification methods are used to predict a discrete outcome, such as whether a stock price will go up or down, a company will default, or a trading signal will be positive or negative.

I. Core Classification Models

1. Logistic Regression (Discriminative Model)

Logistic Regression is a linear model used for binary classification. It models the probability of a class membership using the logistic (sigmoid) function to map a linear combination of predictors to a probability between 0 and 1.

\mathbb{P}(Y = 1 | \mathbf{X} = \mathbf{x}) = \frac{1}{1 + e^{-(\beta_0 + \mathbf{x}^\intercal \boldsymbol{\beta})}}

Log-Odds: The model is linear in the log-odds (or logit): $\ln\left(\frac{\mathbb{P}(Y=1 | \mathbf{x})}{\mathbb{P}(Y=0 | \mathbf{x})}\right) = \beta_0 + \mathbf{x}^\intercal \boldsymbol{\beta}$
Estimation: Coefficients $\boldsymbol{\beta}$ are estimated using Maximum Likelihood Estimation (MLE), as there is no closed-form solution.
Decision Boundary: The decision boundary is linear, defined by $\beta_0 + \mathbf{x}^\intercal \boldsymbol{\beta} = 0$ .

2. Discriminant Analysis (Generative Model)

Discriminant Analysis models the distribution of the predictors $\mathbf{X}$ separately for each class $k$ , $f_k(\mathbf{x}) = \mathbb{P}(\mathbf{X} = \mathbf{x} | Y = k)$ , and then uses Bayes' Theorem to find the posterior probability $\mathbb{P}(Y = k | \mathbf{X} = \mathbf{x})$ .

\mathbb{P}(Y = k | \mathbf{X} = \mathbf{x}) = \frac{f_k(\mathbf{x})\pi_k}{\sum_{i=1}^K \pi_i f_i(\mathbf{x})}

Linear Discriminant Analysis (LDA): Assumes that $f_k(\mathbf{x})$ is a multivariate Gaussian distribution with a common covariance matrix $\boldsymbol{\Sigma}$ across all classes. This results in a linear decision boundary.
Quadratic Discriminant Analysis (QDA): Assumes that $f_k(\mathbf{x})$ is a multivariate Gaussian distribution with a unique covariance matrix $\boldsymbol{\Sigma}_k$ for each class. This results in a quadratic decision boundary.

3. $k$ -Nearest Neighbors ( $k$ -NN) (Non-Parametric Model)

$k$ -NN is a non-parametric, instance-based learning algorithm. It classifies a new observation by finding the $k$ closest training observations (based on a distance metric like Euclidean distance) and assigning the new observation to the most frequent class among its neighbors.

Key Parameter: $k$ (number of neighbors). A small $k$ leads to high variance (overfitting), while a large $k$ leads to high bias (underfitting).
Curse of Dimensionality: $k$ -NN performance degrades rapidly as the number of features (dimensions) increases, a common issue in high-dimensional financial data.

4. Naive Bayes

Naive Bayes is a generative model that simplifies the estimation of $f_k(\mathbf{x})$ by making the strong assumption that the predictors are conditionally independent given the class $Y=k$ .

f_k(\mathbf{x}) = \mathbb{P}(\mathbf{X} = \mathbf{x} | Y = k) = \prod_{j=1}^p \mathbb{P}(X_j = x_j | Y = k)

Advantage: Computationally efficient and performs surprisingly well in many real-world applications, especially text classification (e.g., sentiment analysis of news articles).

II. Model Performance Metrics

In classification, simply measuring accuracy is often insufficient, especially with imbalanced datasets (e.g., credit default prediction).

Confusion Matrix

A $2 \times 2$ table summarizing the model's performance on a test set.

	Predicted Positive	Predicted Negative
Actual Positive	True Positive (TP)	False Negative (FN)
Actual Negative	False Positive (FP)	True Negative (TN)

Key Metrics

Metric	Formula	Interpretation	Relevance in Finance
Accuracy	$\frac{TP + TN}{TP + TN + FP + FN}$	Overall correctness.	Can be misleading for imbalanced data (e.g., 99% accuracy on 1% default rate).
Precision	$\frac{TP}{TP + FP}$	Of all predicted positives, how many were correct?	Important when the cost of a False Positive is high (e.g., a false trading signal).
Recall (Sensitivity)	$\frac{TP}{TP + FN}$	Of all actual positives, how many were correctly identified?	Important when the cost of a False Negative is high (e.g., failing to predict a default).
F1 Score	$2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$	Harmonic mean of Precision and Recall; a balanced measure.	Used to compare models when both FP and FN costs are significant.

ROC Curve and AUC

ROC (Receiver Operating Characteristic) Curve: Plots the True Positive Rate (Recall) against the False Positive Rate ( $\frac{FP}{FP + TN}$ ) at various threshold settings.
AUC (Area Under the Curve): The area under the ROC curve. It represents the probability that the model will rank a randomly chosen positive instance higher than a randomly chosen negative instance.
- Interpretation: An AUC of 1.0 is a perfect classifier; 0.5 is no better than random guessing. AUC is a robust metric for imbalanced datasets.