Least Square Data Fitting

Definition:

• A scalar $y$ and an $n$-vector $x$ are related by model, $y\approx f(x)$ which is $f:{\mathbb{R}}^n\to \mathbb{R}$
  • $x$ is independent variable, the feature vector
  • $y$ is the outcome variable, sth we want to predict
• $x^{(i)}$, $y^{(i)}$ is the $i$th data pair
  • $x_j^{(i)}$ isthe jth component of ith data point $x^{(i)}$
• Choose model $\hat{f}:{\mathbb{R}}^n\to \mathbb{R}$ are basis functions that we choose
  • ${\theta }_i$ are model parameters we can choose
  • $\hat{f}(x)={\theta }_1{f}_1(x)+...+{\theta }_p{f}_p(x)$
• ${\hat{y}}^{(i)}=\hat{f}(x^{(i)})$ is the model prediction of $y^{(i)}$
• Define:
  • $y^d=(y^{(1)},...,y^{(N)})$ as vectors of outcomes
  • ${\hat{y}}^d=({\hat{y}}^{(1)},...,{\hat{y}}^{(N)})$ as vectors of predictions
  • $r^d=(r^{(1)},...,r^{(N)})$ as vectors of residuals
    • for $r_i=y^{(i)}-{\hat{y}}^{(i)}$
    • $\text{rms}=(\frac{(r^{(1)}{)}^2+...+(r^{(N)}{)}^2}{N}{)}^{1/2}$
• Define a matrix mapping $A\in {\mathbb{R}}^{N\times p}$ with $A_{ij}=f_j(x^{(i)})$ so ${\hat{y}}^d=A\theta$
• Then we choose $\theta$ to minize $||r^d|{|}^2=||y^d-{\hat{y}}^d|{|}^2=||y^d-A\theta |{|}^2=||A\theta -y^d|{|}^2$
• Similar to ordinary least square, $\hat{\theta }=(A^{⊺}A{)}^{-1}{A}^{⊺}y$ (if columns of $A$ are linearly independent)
• $||A\hat{\theta }-y|{|}^2/N$ is the minimum mean-square error
• For the weights of $\theta$, we define the first weight $f_1(x)=1$ as other weight are relative to it
• The function $\hat{y}(x)=ax+b$ but we can add another last element in $a$ to incorporate the value of $b$, so we have $\hat{y}(x)=Ax$

For $p=1$:

• Then $f_{(}x)=1$ so the model $\hat{f}(x)={\theta }_1$ is a constant

For $p=2$:

• $f_1(x)=1,f_2(x)=x$
• Model has form $\hat{f}(x)={\theta }_1+{\theta }_{2}x$
• Matrix $A$ has form: $A=\left[\begin{array}{cc}1& x^{(1})\\ 1& x^{(2})\\ .& .\\ 1& x^{(N})\end{array}\right]=\left[\begin{array}{cc}1& x^d\end{array}\right]$
• Then ${\hat{\theta }}_1,{\hat{\theta }}_2$ can be found with $\hat{f}(x)=avg(y^d)+\rho \frac{std(y^d)}{std(x^d)}(x-avg(x^d))$

For $p=$ polynomial:

• $f_i(x)=x^{i-1},i=1,...,p$
• Model is $p-1$ degree $\hat{f}(x)={\theta }_1+{\theta }_{2}x+...+{\theta }_p{x}^{p-1}$, here $x$ is powered, not the element
• $A$ is then Vandermonde Matrix

Least Square Classification

Multi-objective Least Square