Linear Regression and Gradient Descent

With a single feature, linear regression approximates the data with a straight line: y = w * x + b, where w is the slope coefficient and b is the intercept. With several features the model takes the form y = w1 * x1 + w2 * x2 + b. The goal is to choose the parameters so that the predictions are as close as possible to the real values, for example to predict a house price from its area and number of rooms.

Closeness is measured by the mean squared error (MSE): MSE = (1/n) * sum((y_i - yhat_i)^2), where yhat_i is the prediction of the model. Because errors are squared, positive and negative deviations do not cancel each other, and large errors are penalized more. In the least squares method the parameters can be computed in a single step with the normal equation: w = (X^T * X)^(-1) * X^T * y; here a column of ones is added to the matrix X, so b also becomes an element of the vector w.

When there are many data points and features, the normal equation becomes expensive; then gradient descent is used. The algorithm computes the gradient of the loss function, that is, the vector of partial derivatives, and updates the parameters with a small step in the direction opposite to the gradient: w = w - lr * dMSE/dw. If the step lr (the learning rate) is too large the process diverges, and if it is too small convergence is slow. For linear regression MSE is a convex function, so there is no local minimum problem, and bringing the features to the same scale speeds up convergence.

Model quality is evaluated on the test set with RMSE (the square root of MSE, in the same units as y) and the coefficient of determination R2, which shows what share of the variance the model explains (the closer to 1, the better). Linear regression assumes that the relationship is roughly linear; outliers strongly affect the result, and correlated features (multicollinearity) make the coefficients unstable. A curved relationship can be modeled with polynomial features, and overfitting can be restrained with regularization (Ridge, Lasso).

Key takeaways

• Linear regression finds the parameters of the model y = w * x + b from data.
• Mean squared error (MSE) measures the difference between a prediction and the real value.
• The normal equation is used for small problems and gradient descent for large data.
• Choosing the learning rate lr well and scaling the features to the same range speed up convergence.
• Quality is assessed with RMSE and R2; outliers and multicollinearity can distort the result.