Multiple regression: OLS method
(Mostly from Maddala)
The Ordinary Least Squares method of estimation can easily be extended to models involving two or more explanatory variables, though the algebra becomes progressively more complex. In fact, when dealing with the general regression problem with a large number of variables, we use matrix algebra, but that is beyond the scope of this course.
We illustrate the case of two explanatory variables, X1 and X2, with Y the dependant variable. We therefore have a model
Yi = α + 1X1i + 2X2i + ui
We look for estimators so as to minimise the sum of squared errors,
Differentiating, and setting the partial differentials to zero we get
These three equations are called the “normal equations”. They can be simplified as follows: Equation (1) can be written as
Where the bar over Y, X1 and X2 indicates sample mean. Equation (3) can be written as
Substituting in the value of from (4), we get
A similar equation results from (3) and (4). We can simplify this equation using the following notation. Let us define:
Equation (5) can then be written
Similarly, equation (3) becomes
We can solve these two equations to get:
Where =S11S22 – S122. We may therefore obtain from equation (4).
We can calculate the RSS, ESS and TSS from these estimators in the same way as for simple regression, that is
And, the coefficient of multiple determination is
R2 = ESS/TSS
That is, R2 is the proportion of the variation in Y explained by the regression.
The variances of our estimators are given by
Where r122 is the squared correlation coefficient between X1 and X2. Thus, the greater the correlation between the two explanatory variables, the greater the variance in the estimators, i.e. the...
Please join StudyMode to read the full document