Linear regression

Definition:

• Prediction by Regression Model
• Actual value of $y={\beta }_0+{\beta }_{1}x+ϵ$
  • Regression equation$⟹E(y)={\beta }_0+{\beta }_{1}x$
• Predicted valye of $\hat{y}=b_0+b_{1}x$
  • $\hat{y}$ is the point estimator of $E(y)$
• In practice, ${\beta }_0$ and ${\beta }_1$ are not known, but can be estimated with $b_0$ and $b_1$
  • where $b_0$ and $b_1$ are computed by least squares method
• Some notations:
  • $x^{\ast }$: given value of independent value of $x$
  • $y^{\ast }$: possible values (a range) of dependent value of $y$ when $x=x\ast$
  • ${\hat{y}}^{\ast }=b_0+b_1{x}^{\ast }$: the point estimator $E(y^{\ast })$ and the predictor of an individual value of $y^{\ast }$ when $x=x^{\ast }$

Finding line of best fit:

• There are 2 main methods:
  1. Scattergraph method:
     • Draw a line through data points with about an equal number of points above and below the line.
  2. Linear regression:
     • Using correlation:
       • We need to choose $a$ and $b$ to minimize the Mean square error
       • $E[(Y-(a+bX){)}^2]=E[Y^2]-2aE[Y]-2bE[XY]+a^2+2abE[X]+b^{2}E[X^2]$
       • Taking partial derivative and set them equal to 0 we have $a$ and $b$ when it is at minimum point
         • Cant have maximum due to the nature of the problem
       • $b=\frac{\text{Cov}(X,Y)}{{\sigma }_X^2}$
       • $a=E[Y]-bE[X]$
       • The best linear predictor (lowest mean square error) is: ${\mu }_y+\frac{\rho {\sigma }_y}{{\sigma }_x}(X-{\mu }_x)$
         • Happens when ${\mu }_y=E[Y],{\mu }_y=E[X]$ and $\rho$ is the Correlation of $X$ and $Y$
         • The Mean square error of this predictor is given by $E[(Y-{\mu }_y-\frac{\rho {\sigma }_y}{{\sigma }_x}(X-{\mu }_x{)}^2]={\sigma }_y^2(1-{\rho }^2)$
     • Using $y=a+bX$:
       • $b=\frac{n\sum x_i{y}_i-\sum x_i\sum y_i}{\sqrt{n\sum x_i^2-(\sum x_i{)}^2}}$
       • $a=\frac{\sum y}{n}-\frac{b\sum x}{n}$

Assumptions about $ϵ$ in linear regression model:

1. $E(ϵ)=0$. This implies ${\beta }_0$ and ${\beta }_1$ are constants, and hence $E(y)={\beta }_0+{\beta }_{1}x$
2. The variance of $\epsilon$, denoted by ${\sigma }^2$, is the same for all $x$
3. The values of $\epsilon$ are independent.
4. $\epsilon$ is a normally distributed random variable for all values of $x$

Testing for significance of linear:

• Testing for significance
• If ${\beta }_1=0$, then $E(y)={\beta }_0$. In this case, we would conclude that $x$ and $y$ are not linearly related
• If $\beta 1\ne 0$, we would conclude that the two variables are related.
• The t test is commonly used.
  • It requires an estimate of $\sigma 2$, the variance of $\epsilon$ in the regression model.
• With $\hat{y}=b_0+b_{1}x$, we can use the Mean square error $s^2$ as an estimate ${\sigma }^2$
  • $s^2=\frac{\sum (y_i-{\hat{y}}_i{)}^2}{n-2}$
  • The standard error of the estimate $s=\sqrt{s^2}$ is used to estimate $\sigma$
• $H_0:{\beta }_1=0,H_a:{\beta }_1\ne 0$
• For sample distributin of $b_1$:
  • Expected value: $E(b_1)={\beta }_1$
  • sd: ${\sigma }_{b_1}=\frac{\sigma }{\sqrt{\sum (x_i-\bar{x}{)}^2}}$
    • then we use it to estimate: $s_{b_1}=\frac{s}{\sqrt{\sum (x_i-\bar{x}{)}^2}}$
  • $b_1\sim N({\beta }_1,{\sigma }_{b_1}^2)$
  • $\to ts=\frac{b_1-{\beta }_1}{s_{b_1}}=\frac{b_1}{s_{b_1}}$
  • Reject $H_0$ if $p$-value $\le \alpha$, where $p$ follows a t distribution with $n-2$ degrees of freedom
• Confidence Interval for ${\beta }_1$ is $b_1\pm t_{\alpha /2}{s}_{b_1}$
  • with two-tailed
  • If the CI include 0, we can hypothesize value of ${\beta }_1$ might not be in the CI
  • We can reject $H_0$ as there is a chance the model is not significant enought to be used

Variance of predicting value:

• $Var({\hat{y}}^{\ast })=s_{{\hat{y}}^{\ast }}^2=s^2.[\frac{1}{n}+\frac{(x^{\ast }-\bar{x}{)}^2}{\sum (x_i-\bar{x}{)}^2}]$
  • where $s^2$ is the variance of the all collected $y$
  • Think of it as var of the group to other group

Confidence interval for mean of predicting value $E(y^{\ast })$:

• Confidence interval of the mean of all predicting value when $x=x^{\ast }$
• $CI((1-\alpha )\%)={\hat{y}}^{\ast }\pm t_{\alpha /2}{s}_{{\hat{y}}^{\ast }}$
  • with $n-2$ degree of freedom

Prediction interval for ${\hat{y}}^{\ast }$:

• Prediction
• Confidence interval of the 1 value when $x=x^{\ast }$, thus more uncertainty
• Has variance of Mean square error and Variance of predicting value:
  • $s_{pred}^2=s^2+s_{{\hat{y}}^{\ast }}^2=s^2.[1+\frac{1}{n}+\frac{(x^{\ast }-\bar{x}{)}^2}{\sum (x_i-\bar{x}{)}^2}]$