Least Square Regression

Keywords

least square regression, linear regression, maximum likelihood principle, standard error of the estimate, standard deviation, total sum of the squares, unexplained sum of the squares, correlation coefficient, coefficient of determination, polynomial regression

We can use the least-square regression to generate a curve that can sufficiently describe the relationship between $x$ and $y$. This method minimizes the discrepancy between the data points and curve produced; thus, it can be useful for objectively illustrating the general trend in data.

Linear Regression

Best Criteria

For a set of paired observations: $(x_1,y_1),(x_2,y_2),\dots ,(x_n,y_n)$. We can express the residual error of a particular straight line approximation using the formula

$$e=y-a_0-a_{1}x$$

where $a_0$ and $a_1$ are coefficients characterizing the slope and intercept.¹ Consequently, we can find the line that best fit the given data points using a specific criteria: the line with the smallest sum of squares of residual errors is the best fit (least-squares fit). We can determine this sum using the expression

$$\begin{array}{rl}{S}_r& =\sum _{i=1}^n{e}_i^2\\ & \\ & =\sum _{i=1}^n(y-a_0-a_{1}x{)}^2\end{array}$$

where $n$ is the total number of points.

Least-Squares Fit

The expression for a straight line is

$$y=a_0+a_{1}x+e$$

however, we set $e=0$ to obtain the line with the least-squares. On the other hand, we can solve for $a_0$ and $a_1$ using the formulae:

$$\begin{array}{rl}{a}_1& =\frac{n\Sigma x_i{y}_i-\Sigma x_i\Sigma y_i}{n\Sigma x_i^2-{\left(\Sigma x_i\right)}^2}\\ & \\ a_0& =\bar{y}-a_1\bar{x}\end{array}$$

where $\bar{x}$ and $\bar{y}$ are the means of $x$ and $y$.

Quantification of Error

The least-squares regression provides the best estimate of $a_0$ and $a_1$ when it meets the following criteria:

1. Along the entire range of data, the distance between the regression line and the data points is of similar magnitude.
2. The distribution of the points with respect to the line is normal.²

Furthermore, the regression line’s standard deviation can be obtained—assuming that the criteria are met—using

$$\begin{array}{rlr}{s}_{y/x}=\sqrt{\frac{S_r}{n-2}}& & \text{standard error of the estimate}\end{array}$$

The standard error of the estimate tells us the spread of data around the regression line; in contrast, the standard deviation only tells us the spread of data around the mean. This assists us in assessing the accuracy of our fit, especially when we compare other types of regression methods.

To determine the goodness of our fit:

1. Determine the $S_t$ or total sum of the squares around the mean for the dependent variable.
   • This is the magnitude of the residual error associated with the pre-regression dependent variable.
2. Solve for the regression line.
3. Determine the $S_r$ or the sum of the squares of the residuals around the regression line.
   • This is the magnitude of the residual error that remains after the regression; it is also referred to as the unexplained sum of the squares
4. Solve for the coefficient of determination $r^2$
   • A perfect fit has no $S_r$ and has $r=r^2=1$
   • The result indicates the percent of original uncertainty explained by the linear model

The coefficient of determination can mathematically be expressed as

$$\begin{array}{rlrlr}{r}^2=\frac{S_t-S_r}{S_t}& & ;& & S_t=\sum _{i=1}^n(y_i-\bar{y}{)}^2\end{array}$$

where its square root $r$ is referred to as the correlation coefficient. The correlation coefficient can also be solved using the formula

$$r=\frac{n\sum x_i{y}_i-\left(\sum x_i\right)\left(\sum y_i\right)}{\sqrt{n\sum x_i^2-{\left(\sum x_i\right)}^2}\sqrt{n\sum y_i^2-{\left(\sum y_i\right)}^2}}$$

WARNING

Just because $r$ is close to $1$ does not mean that the fit is necessarily good (like in cases where the relationship between the variables are not even linear).

TIP

If $S_{y/x}<S_y$ , then the linear regression model has merit; else, it does not.

Polynomial Regression

Besides linear regression, we could also apply the least-square procedure to polynomial regression—which can be handy for trends in data that a straight line cannot acceptably represent. For example, we can extend the least-square formula to handle quadratic polynomial using the following equation:

$$S_r=\sum _{i=1}^n(y_i-a_0-a_1{x}_i-a_2{x}_i^2{)}^2$$

To compute for the three unknowns (i.e., $a_0,a_1,$ and $a_2$), we need to solve the following system of three linear equations

$$\begin{array}{rl}(n)a_0+\left(\sum x_i\right)a_1+\left(\sum x_i^2\right)a_2& =\sum y_i\\ \left(\sum x_i\right)a_0+\left(\sum x_i^2\right)a_1+\left(\sum x_i^3\right)a_2& =\sum x_i{y}_i\\ \left(\sum x_i^2\right)a_0+\left(\sum x_i^3\right)a_1+\left(\sum x_i^4\right)a_2& =\sum x_i^2{y}_i\end{array}$$

For an $m^{th}$-order polynomial

$$y=a_0+a_{1}x+a_2{x}^2+\cdots +a_m{x}^m+e$$

we can express its standard error of the estimate as

$$s_{y/x}=\sqrt{\frac{S_r}{n-(m+1)}}$$

Similar to a linear regression, the coefficient of determination is

$$\begin{array}{rlrlrl}{r}^2=\frac{S_t-S_r}{S_t}& & ;& & S_t& =\sum _{i=1}^n(y_i-\bar{y}{)}^2\\ & & & & S_r& =\sum _{i=1}^n{e}_i^2\end{array}$$

Multiple Linear Regression

In a case where there are two independent variables and only a single dependent variable, the regression line becomes a regression plane. For example, the following function:

$$y=a_0+a_1{x}_1+a_2{x}_2+e$$

As with previous cases, we can obtain the best values by taking advantage of the sum of the squares of the residuals,

$$S_r=\sum _{i=1}^n(y_i-a_0-a_1{x}_{1i}-a_2{x}_{2i}{)}^2$$

To obtain the coefficients for the minimum sum of the squares of the residuals, we can use the following equation

$$\left[\begin{array}{ccc}n& \sum x_{1i}& \sum x_{2i}\\ \sum x_{1i}& \sum x_{1i}^2& \sum x_{1i}{x}_{2i}\\ \sum x_{2i}& \sum x_{1i}{x}_{2i}& \sum x_{2i}^2\end{array}\right]=\left\{\begin{array}{c}{a}_0\\ a_1\\ a_2\end{array}\right\}=\left\{\begin{array}{c}\sum y_i\\ \sum x_{1i}{y}_i\\ \sum x_{2i}{y}_i\end{array}\right\}$$

It should be noted that it’s not limited to two variables, multiple linear regression can be extended to $m$ dimensions, as in

$$y=a_0+a_1{x}_1+a_2{x}_2+\cdots +a_m{x}_m+e$$

where the standard error is

$$s_{y/x}=\sqrt{\frac{S_r}{n-(m+1)}}$$

and the coefficient of determination is

$$\begin{array}{rlrlrl}{r}^2=\frac{S_t-S_r}{S_t}& & ;& & S_t& =\sum _{i=1}^n(y_i-\bar{y}{)}^2\end{array}$$

Nonlinear Regression

TODO

Gauss-Newton Method

TODO

Sources

1. Numerical Methods for Engineers by Steven Chapra and Raymond Canale (Chapter 17)

Footnotes

1. Residual error is the difference between the true value and approximated value. ↩

2. normal distribution ↩