Google

Linear regression uniquely presents an optimization problem with a direct analytic solution. By subsuming the bias term into the weight vector $$\mathbf{w}$$—which is done by appending a column of all $$1$$s to the design matrix $$\mathbf{X}$$—the training objective becomes minimizing $$\|\mathbf{y} - \mathbf{X}\mathbf{w}\|^2$$. Setting the derivative of this loss with respect to $$\mathbf{w}$$ equal to $$0$$ yields the intermediate derivative formula $$\partial_{\mathbf{w}} \|\mathbf{y} - \mathbf{X}\mathbf{w}\|^2 = 2 \mathbf{X}^	op (\mathbf{X} \mathbf{w} - \mathbf{y}) = 0$$, which directly gives the normal equation $$\mathbf{X}^	op \mathbf{y} = \mathbf{X}^	op \mathbf{X} \mathbf{w}$$. Solving for $$\mathbf{w}$$ provides the optimal parameters: $$\mathbf{w}^* = (\mathbf{X}^	op \mathbf{X})^{-1}\mathbf{X}^	op \mathbf{y}$$. This closed-form solution exists only if the design matrix has full rank, ensuring that the matrix $$\mathbf{X}^	op \mathbf{X}$$ is invertible.

Linear Regression Analytic Solution

You are tasked with fitting a simple linear model. Given the design matrix `X` (which includes a column of ones for the intercept) and the target vector `y`, calculate the coefficient vector `β̂` using the equation `β̂ = (XᵀX)⁻¹Xᵀy`.

`X = [[1, 1], [1, 3], [1, 2]]`
`y = [3, 7, 5]`

Provide the final coefficient vector `β̂`.

Calculating OLS Coefficients

Based on the case study below, explain why the calculated regression line is a poor fit for the majority of the data. Your explanation should refer to the mathematical objective that the equation $\hat{\beta} = (X^{T} X)^{-1} X^{T} y$ is designed to optimize.

Impact of an Outlier on OLS Coefficients

A data scientist is attempting to find the optimal coefficients ($\hat{\beta}$) for a linear model using the equation $\hat{\beta} = (X^{T} X)^{-1} X^{T} y$, where $X$ is the matrix of input features and $y$ is the vector of target values. The calculation fails, returning a 'singular matrix' error. What is the most likely cause of this error in the dataset represented by matrix $X$?

Learn Before

Related