Ordinary least squares regression

Examples of Ordinary least squares regression in the following topics:

• Least-Squares Regression

  • The criteria for determining the least squares regression line is that the sum of the squared errors is made as small as possible.
  • The process of fitting the best- fit line is called linear regression.
  • Therefore, this best fit line is called the least squares regression line.
  • Here is a scatter plot that shows a correlation between ordinary test scores and final exam test scores for a statistics class:
  • Ordinary Least Squares (OLS) regression (or simply "regression") is a useful tool for examining the relationship between two or more interval/ratio variables assuming there is a linear relationship between said variables.

• Regression Analysis for Forecast Improvement

  • Regression Analysis is a causal / econometric forecasting method.
  • (Note: If not, weighted least squares or other methods might instead be used).
  • Familiar methods, such as linear regression and ordinary least squares regression, are parametric, in that the regression function is defined in terms of a finite number of unknown parameters that are estimated from the data.
  • Nonparametric regression refers to techniques that allow the regression function to lie in a specified set of functions, which may be infinite-dimensional.
  • The performance of regression analysis methods in practice depends on the form of the data generating process and how it relates to the regression approach being used.

• Coefficient of Determination

  • the regression sum of squares (also called the explained sum of squares); and
  • the sum of squares of residuals, also called the residual sum of squares.
  • where $SS_\text{err}$ is the residual sum of squares and $SS_\text{tot}$ is the total sum of squares.
  • In regression, the $r^2$ coefficient of determination is a statistical measure of how well the regression line approximates the real data points.
  • In many (but not all) instances where $r^2$ is used, the predictors are calculated by ordinary least-squares regression: that is, by minimizing $SS_\text{err}$.

• Estimating the Target Parameter: Point Estimation

  • Another popular estimation approach is the linear least squares method.
  • Linear least squares is an approach fitting a statistical model to data in cases where the desired value provided by the model for any data point is expressed linearly in terms of the unknown parameters of the model (as in regression).
  • In statistics, linear least squares problems correspond to a statistical model called linear regression which arises as a particular form of regression analysis.
  • One basic form of such a model is an ordinary least squares model.
  • Contrast why MLE and linear least squares are popular methods for estimating parameters

• Model Assumptions

  • The following are the major assumptions made by standard linear regression models with standard estimation techniques (e.g. ordinary least squares):
  • This is to say there will be a systematic change in the absolute or squared residuals when plotted against the predicting outcome.
  • In effect, residuals appear clustered and spread apart on their predicted plots for larger and smaller values for points along the linear regression line, and the mean squared error for the model will be wrong.
  • (Actual statistical independence is a stronger condition than mere lack of correlation and is often not needed, although it can be exploited if it is known to hold. ) Some methods (e.g. generalized least squares) are capable of handling correlated errors, although they typically require significantly more data unless some sort of regularization is used to bias the model towards assuming uncorrelated errors.
  • For standard least squares estimation methods, the design matrix $X$ must have full column rank $p$; otherwise, we have a condition known as multicollinearity in the predictor variables.

• Fitting a Curve

  • With linear regression, a line in slope-intercept form, $y=mx+b$ is found that "best fits" the data.
  • The simplest and perhaps most common linear regression model is the ordinary least squares approximation.
  • Example 1:  Write the least squares fit line and then graph the line that best fits the data:  for $n=8$ points: $(-1,0),(0,0),(1,1),(2,2),(3,1),(4,2.5),(5,3) $ and $(6,4)$.
  • Here is a line used by the least squares approximation, $y = 0.554x+0.3025$
  • Model a set of data points as a line using the least squares approximation

• Conditions for the least squares line

  • If there is a nonlinear trend (e.g. left panel of Figure 7.13), an advanced regression method from another book or later course should be applied.
  • The variability of points around the least squares line remains roughly constant.
  • Be cautious about applying regression to data collected sequentially in what is called a time series.
  • Should we have concerns about applying least squares regression to the Elmhurst data in Figure 7.12?
  • Least squares regression can be applied to these data.

• Polynomial Regression

  • For this reason, polynomial regression is considered to be a special case of multiple linear regression.
  • Polynomial regression models are usually fit using the method of least-squares.
  • The least-squares method minimizes the variance of the unbiased estimators of the coefficients, under the conditions of the Gauss–Markov theorem.
  • The least-squares method was published in 1805 by Legendre and in 1809 by Gauss.
  • Although polynomial regression is technically a special case of multiple linear regression, the interpretation of a fitted polynomial regression model requires a somewhat different perspective.

• A Graph of Averages

  • A graph of averages and the least-square regression line are both good ways to summarize the data in a scatterplot.
  • The most common method of doing this is called the "least-squares" method.
  • The least-squares regression line is of the form $\hat{y} = a+bx$, with slope $b = \frac{rs_y}{s_x}$ ($r$ is the correlation coefficient, $s_y$ and $s_x$ are the standard deviations of $y$ and $x$).
  • The points on a graph of averages do not usually line up in a straight line, making it different from the least-squares regression line.
  • The graph of averages plots a typical $y$ value in each interval: some of the points fall above the least-squares regression line, and some of the points fall below that line.

• Introduction to fitting a line by least squares regression

  • In this section, we use least squares regression as a more rigorous approach.