The Least Squares Regression Line

Example:

Suppose you have three points in the plane and want to find the line 

        y = mx + b 

that is closest to the points. Then we want to minimize the sum of the squares of the vertical distances, that is find m and b such that

        d₁² + d₂² + d₃² 

is minimum.  We call the equation 

        y = mx + b

the least squares regression line.  We calculate

        d₁² + d₂² + d₃²  = f = [y₁ - (mx₁+ b)]² + [y₂ - (mx₂+ b)]² + [y₃ - (mx₃+ b)]²  

To minimize we take derivatives to get

        f_m = 2[y₁ - (mx₁+ b)](-x₁) + 2[y₂ - (mx₂+ b)](-x₂) + 2[y₃ - (mx₃+ b)](-x₃) = 0

and

        f_b = -2[y₁ - (mx₁+ b)] - 2[y₂ - (mx₂+ b)] - 2[y₃ - (mx₃+ b)] = 0   

We have

1. x₁(y₁ - (mx₁+ b)) + x₂(y₂ - (mx₂+ b)) + x₃(y₃ - (mx₃+ b)) = 0    
   

2. (y₁ - (mx₁+ b)) + (y₂ - (mx₂+ b)) + (y₃ - (mx₃+ b)) = 0 
   

In S notation, we have

1. Sx_iy_i - mSx_i² - bSx_i = 0
   

2. Sy_i - mSx_i - nb = 0 

Notice here n = 3 since there are three points.  These equations work for an arbitrary number of points.

We can solve to get

     

and

Exercise:  Find the equation of the regression line for

(3,2), (5,1) and (4,3)

Fortunately, many calculators and computer programs can find the equation of the regression line.  In particular, the TI 89 can be used by going to "Apps", Data/Matrix Editor, enter the data and then go to calc.

Application

A biologist has run seven experiments with different amounts of nitrogen and bacteria growth.  The data is shown below

 Find the equation of the least squares regression line using a calculator.  What would you estimate for the amount of bacteria given 5 grams of N?

Solution

Using a computer, we obtained

        y  =  -2.35 + 1.19x

Plugging 5 into the equation gives

        y  =  -2.35 + 1.19(5)  =  3.6

We estimate that there will be about 3.6 grams of bacteria in an environment of 5 grams of nitrogen.  The picture below shows the scatter plot and the points.

Least Squares Regression Quadratic

A line is not always the best model for a set of data.  Often theory or a quick look at the plotted points predict that the data can be best modeled by a nonlinear function.  We will not get into the details here, but the technique of finding the extrema can be used for any model.  Again, machines are especially useful for finding the proper coefficients.

Example

You own an umbrella rental shop by the beach and have collected data on the number of customers at each hour of operation.  You expect that typically, your business begins slow, peaks in the middle of the day and then slows down as the day finishes. Hence you expect that a parabola may be the best model for your business.  The table below shows the number of customers on a Saturday in August.

Use a machine to come up with a least squares regression quadratic and then estimate the number of customers at the 11:00 hour.

Solution

A machine gives

        y  =  .59x² - 6.48x + 18.45

Plugging in 11, gives 18.75 or about 19 customers.

Back to the Math 117 Home Page

Back to the Math Department Home

e-mail Questions and Suggestions