Multivariate Normal Distribution

Definition: $X\sim N_p(\mu ,\sum )$

• Normal distribution
• A $p$–dimensional normal density for the random vector $X^{⊺}=(X_1,...,X_p)$ has the form $f(X)=\frac{1}{(2\pi {)}^{p/2}|\sum {|}^{1/2}}{e}^{-(X-\mu {)}^{⊺}{\sum }^{-1}(X-\mu )/2}$
  • as $\mu$ moves, the maximum point moves accordingly
• $\left\{\begin{array}{c}{X}_1\\ X_2\\ ...\\ X_n\end{array}\right\}\sim N(\left\{\begin{array}{c}{\mu }_1\\ {\mu }_2\\ ...\\ {\mu }_n\end{array}\right\},\left\{\begin{array}{cccc}{\sigma }_{X_1}^2& {\sigma }_{X_1,X2}& ...& {\sigma }_{X_1,X_n}\\ {\sigma }_{X_2,X_1}& {\sigma }_{X_2}^2& ...& ...\\ ...& ...& ...& ...\\ {\sigma }_{X_n,X_1}& ...& ...& {\sigma }_{X_n}^2\end{array}\right\})$
  • therefore, $\sum$ is a symmetric matrix with ${\sigma }_{X_1,X_2}={\sigma }_{X_2,X_1}$
  • ${\rho }_{12}={\rho }_{X_1,X_2}={\sigma }_{12}/(\sqrt{{\sigma }_{11}}\sqrt{{\sigma }_{22}})=\text{Corr}(X_1,X_2)$ by definition of Correlation
• Contours of constant density for the $p$-dimensional normal distribution are ellipsoids defined by $X$ such the that $(X-\mu {)}^{⊺}{\Sigma }^{-1}(X-\mu )=c^2$
  • These ellipsoids are centered at $\mu$ and have axes $\pm c\sqrt{{\lambda }_i}{e}_i$
  • where $\sum .e_i={\lambda }_i{e}_i$ are Eigenvector. Eigenvalue for $i=1,...,p$
  • In bivariate, these constant density is an ellipsoids formed when $f(x)=$ a constant and $e_i$ are the basis of that plane, ig?
• The solid ellipsoid of $x$ values satisfying $(x-\mu {)}^{⊺}{\sum }^{-1}(x-\mu )\le {\chi }_p^2(\alpha )$ has probability $1-\alpha$
  • The $p$-variate normal density has a maximum value when the squared distance is zero, that is, when $X=\mu$. Thus, is the point of maximum density, or mode, as well as the expected value of $X$ , or mean.
  • The fact that is the mean of the multivariate normal distribution follows from the symmetry exhibited by the constant-density contours: These contours are centered, or balanced, at $\mu$
• The following are true for a random vector $X$ having a multivariate normal distribution:
  1. Linear combinations of the components of $X$ are normally distributed.
  2. All subsets of the components of X have a (multivariate) normal distribution.
  3. Zero covariance implies that the corresponding components are independently distributed.
  4. The conditional distributions of the components are (multivariate) normal

Proposition:

• If $\sum$ is a positive definite, so that ${\sum }^{-1}$ exists, then $\sum e=\lambda e$ implies ${\sum }^{-1}e=\frac{1}{\lambda }e$ so $(\lambda ,e)$ are Eigenvalue-Eigenvector pair for $\sum$ corresponding to the pair $(1/\lambda ,e)$ for ${\lambda }^{-1}$
  • also, ${\sum }^{-1}$ is positive definite
• Linear combination variable of component of X: If $X$ is distributed as $N_p(\mu ,\Sigma )$, then any linear combination of variables $a^{⊺}X$ is distributed as $N(a^{⊺}\mu ,a^{⊺}\sum .a)$.
  • Also, if $a^{⊺}X$ is distributed as $N(a^{⊺}\mu ,a^{⊺}\sum .a)$ for every $a$, then $X$ must be $N_p(\mu ,\sum )$
• Matrix tranformation variable: If $X$ is distributed as $N_p(\mu ,\sum )$, the $q$ linear combinations $A_{q\times p}{X}_{p\times 1}$ (each equation reprensents 1 row of $A$) are distributed as $N_q(A\mu ,A\sum .A^{⊺})$.
  • $A_{q\times p}{X}_{p\times 1}=\left[\begin{array}{c}{a}_{11}{x}_1+...+a_{1p}{x}_p\\ a_{21}{x}_1+...+a_{2p}{x}_p\\ ...\\ a_{q1}{x}_1+...+a_{qp}{x}_p\end{array}\right]$
    • This produce $q$ new variables then form a vector
  • Also, $X_{p\times 1}+{\mathbf{d}}_{p\times 1}$ where $\mathbf{d}$ is a vector of constants is distrbuted as $N_p(\mu +d,\sum )$
• All subsets of $X$ are normally distributed. If we respectively partition $X$ by $q$ elements, its mean vector $\mu$ and its covariance matrix $\sum$ as
  • $X_{p+1}=\left[\begin{array}{c}{X}_{1(q\times 1)}\\ X_{2((p-q)\times 1)}\end{array}\right],{\mu }_{p\times 1}=\left[\begin{array}{c}{\mu }_{1(q\times 1)}\\ {\mu }_{2((q-p)\times 1)}\end{array}\right]$
  • and ${\sum }_{p\times p}=\left[\begin{array}{cc}{\sum }_{q\times q}^{11}& {\sum }_{q\times (p-q)}^{12}\\ {\sum }_{(p-q)\times q}^{21}& {\sum }_{(p-q)\times (p-q)}^{22}\end{array}\right]$
  • then $X_1$ is distributed as $N_q({\mu }^1,{\sum }^{11})$
• Dependency between combination elements as variable:
  • If $X_{1(q_1\times 1)}$ and $X_{2(q_2\times 1)}$ are independent, then $Cov(X_1,X_2)=0$, a $q_1\times q_2$ matrix of zeros
  • If partition $\left[\begin{array}{c}{X}_1\\ X_2\end{array}\right]$ is $N_{q_1+q_2}(\left[\begin{array}{c}{\mu }_1\\ {\mu }_2\end{array}\right],\left[\begin{array}{cc}{\sum }_{11}& {\sum }_{12}\\ {\sum }_{21}& {\sum }_{22}\end{array}\right])$ then $X_1$ and $X_2$ are independent if and only if ${\sum }_{12}=0$
  • If $X_1$ and $X_2$ are independent and are distributed as $N_{q_1}({\mu }_1,{\sum }_{11})$ and $N_{q_2}({\mu }_2,{\sum }_{22})$ respectively, then $\left[\begin{array}{c}{X}_1\\ X_2\end{array}\right]$ has multivariate normal distribution $N_{q_1+q_2}(\left[\begin{array}{c}{\mu }_1\\ {\mu }_2\end{array}\right],\left[\begin{array}{cc}{\sum }_{11}& 0\\ 0& {\sum }_{22}\end{array}\right])$
• Conditional expectation: Let $X=\left[\begin{array}{c}{X}_1\\ X_2\end{array}\right]$ be distributed as $N_p(\mu ,\sum )$ with $\mu =\left[\begin{array}{c}{\mu }_1\\ {\mu }_2\end{array}\right],\sum =\left[\begin{array}{cc}{\sum }_{11}& {\sum }_{12}\\ {\sum }_{21}& {\sum }_{22}\end{array}\right]$ and $|{\sum }_{22}|>0$. Then the conditional distribution of $X_1$ given that $X_2=x_2$, is normal and has:
  • mean = ${\mu }_1+{\sum }_{12}{\sum }_{22}^{-1}(x_2-{\mu }_2)$
  • variance = ${\sum }_{11}-{\sum }_{12}{\sum }_{22}^{-1}{\sum }_{21}$
  • note that the covariance does not depend on the value $x_2$ of the conditioning variable
• Let $X$ be distributed as $N_p(\mu ,\sum )$ with $|\sum |>0$. Then:
  • $(X-\mu {)}^{⊺}{\sum }^{-1}(X-\mu )$ is distributed as ${\chi }_p^2$, Chi-squared distribution with $p$ df
  • The $N_p(\mu ,\sum )$ distribution assigns probability $1-\alpha$ to the solid ellipsoid $\{x:(x-\mu {)}^{⊺}{\sum }^{-1}(x-\mu )\le {\chi }_p^2(\alpha )\}$ where ${\chi }_p^2(\alpha )$ denotes upper $(100\alpha )$-th percentile of ${\chi }_p^2$ distribution
    • Diagonalizable matrix
• Let $X_1,...,X_n$ be mutually independent with $X_j$ distributed as $N_p({\mu }_i,\sum )$, then $V_1=c_1{X}_1+...+c_n{X}_n$ is distributed as $N_p({\sum }_{j=1}^n{c}_j{\mu }_j,({\sum }_{j=1}^n{c}_j^2\left)\Sigma \right)$
  • Moreover, $V_1$ ad $V_2=b_1{X}_1+...+b_n{X}_n$ are jointly multivariate normal with covariance matrix $\left[\begin{array}{cc}\left({\sum }_{j=1}^n{c}_j^2\right)\Sigma & ({\mathbf{b}}^{⊺}\mathbf{c})\Sigma \\ ({\mathbf{b}}^{⊺}\mathbf{c})\Sigma & {\sum }_{j=1}^n{b}_j^2)\Sigma \end{array}\right]$
  • consequently, $V_1$ and $V_2$ are independent if ${\mathbf{b}}^{⊺}\mathbf{c}=0$

Multivariate Normal Likelihood:

• Assume that the $p\times 1$ vectors $X_1,...,X_n$ represent a random sample from a multivariate normal population with mean vector $\mu$ and covariance matrix $\sum$
• The joint density function of all observations is product of the marginal normal density:
  • $\prod _{j=1}^n\frac{1}{(2\pi {)}^{p/2}|\sum {|}^{1/2}}{e}^{-(x_j-\mu {)}^{⊺}{\sum }^{-1}(x_j-\mu )/2}=\frac{1}{(2\pi {)}^{p/2}|\sum {|}^{1/2}}{e}^{-{\sum }_{j=1}^n(x_j-\mu {)}^{⊺}{\sum }^{-1}(x_j-\mu )/2}$
• A function of $\mu$ and $\Sigma$ for the fixed set of observations $x_1,...,x_j$ is called the likelihood
• Lemma: Let $A$ be a $k\times k$ symmetric matrix and $x$ be a $k\times 1$ vector:
  • $x^{⊺}Ax=tr(x^{⊺}Ax)=tr(Ax{x}^{⊺})$, Trace
  • $tr(A)={\sum }_{i=1}^k{\lambda }_i$
• With that, we can have joint density function $L(\mu ,\sum )=(2\pi {)}^{-np/2}|\sum {|}^{-n/2}\times \exp \{-tr[{\sum }^{-1}({\sum }_{j=1}^n(x_j-\bar{x})(x_j-\bar{x}{)}^{⊺}+n(\bar{x}-\mu )(\bar{x}-\mu {)}^{⊺})]/2\}$

Maximum likelihood estimation of $\mu$ and $\sum$:

• Lemma: Given a $p\times p$ symmetric positive definite matrix $B$ and scalar $b>0$, it follows that $\frac{1}{|\sum {|}^b}\exp \{-tr({\sum }^{-}1B)/2\}\le \frac{1}{|B{|}^b}(2b{)}^{pb}{e}^{-pb}$ for all positive definite $p\times p$ matrix $\sum$, with equality holding only for $\sum =\frac{1}{2b}B$
• Proposition: Let $X_1,...,X_n$ be a random sample from a normal population with mean $\mu$ and covariance $\sum$.
  • Then $\hat{\mu }=\bar{X}$ and $\hat{\sum }=\frac{1}{n}{\sum }_{j=1}^n(X_j-\bar{X})(X_j-\bar{X}{)}^{⊺}=\frac{n}{n-1}S$ are maximum likehood estimators of $\mu ,\sum$
  • Their observed values, $\bar{x}$ and $\frac{1}{n}{\sum }_{j=1}^n(X_j-\bar{X})(X_j-\bar{X}{)}^{⊺}$ are maximum likelihood estimates of $\mu$ and $\sum$
• Maximum likelihood estimators possess an invariance property
  • Let $\hat{\theta }$ be the maximum likelihood estimator of $\hat{\theta }$ and consider estimating the parameter $h(\theta )$ Then the maximum likelihood estimate of $h(\theta )$ is $h(\hat{\theta })$
• Let $X_1,...,X_n$ be a random sample from a multivariate normal population with mean $\mu$ and covariance $\mu$, then $\bar{X}$ and $S=\frac{1}{n-1}{\sum }_{j=1}^n(x_j-\bar{x})(x_j-\bar{x}{)}^{⊺}$ are sufficient statistics

The sampling distribution of $\bar{X}$ and $S$:

• Similar to Central limit theorem
• For the multivariate case, $\bar{X}$ has a normal distribution with mean $\mu$ and covariance matrix $(1/n)\sum$
• The sampling distribution of the sample covariance matrix is called the Wishart distribution, defined as the sum of independent products of multivariate normal random vectors
• $W_m=$ Wishart distribution with $m$ df = distribution of ${\sum }_{j=1}^m{Z}_j{Z}_j^{⊺}$ where $Z_j$ are each independently distributed as $N_p(0,\sum )$
  • denote with $W_p(n,\sum )$
• theorem Let $X_1,...,X_n$ be a random sample of size $n$ from a $p$-variate normal distribution with mean $\mu$ and covariance matrix $\sum$, then:
  • $\bar{X}\sim N_p(\mu ,(1/n)\sum )$
  • $(n-1)S\sim W_{n-1}$
  • $\bar{X}$ and $S$ are independent
• Properties of Wishart distribution:
  • if $A_1$ is distributed as $W_{m_1}(A_1|\sum )$ as independently of $A_2\sim W_{m_2}(A_2|\sum )$ then $A_1+A_2$ is distributed as $W_{m_1+m_2}(A_1+A_2|\sum$)
  • If $A\sim W_m(A|\sum )$ then $CA{C}^{⊺}\sim W_m(CA{C}^{⊺}|C\sum C^{⊺})$

Large-sample be haviour of $\bar{X}$ and $S$

• Let $X_1,...,X_n$ be independent observations from any population with mean $\mu$ and finite covariance $\sum$.
• Then $\sqrt{n}(\bar{X}-\mu )$ has an approximate $N_p(0,\sum )$ distribution for large sample sizes.
  • Hence $n$ should also be large relative to $p$.
• In addition $n(\bar{X}-\mu {)}^{⊺}{S}^{-1}(\bar{X}-\mu )$ is approximately ${\chi }_p^2$ for $n-p$ is large
  • $p$ is dimension of the space

Evaluating multivariate normality:

• Similar to Testing univariate normal distribution but using mean as $\bar{x}$ and covariance matrix $S$
• Use squared generalized distances $d_j^2=(x-\bar{x}{)}^{⊺}{S}^{-1}(x-\bar{x}),j=1,2,...,n$
  • applies for all variables with dimension $p\ge 2$
• When the parent population is multivariate normal and both $n$ and $n-p$ are greater than 30, each of the squared distances $d_j^2$ should behave like a chi-square random variable.
• Although these distances are not independent or exactly chi-square distributed, it is helpful to plot them as if they were. The resulting plot is called a chi-square plot.
• Steps:
  • find $\mu$ by finding all ${\mu }_i$
  • find $S$ by finding all:
    • $S_{ii}=Var(X_i)$, rmb sample variance
    • $S_{ij}=Cov(X_i,X_j)$, rmb sample covariance
  • Construct chi-square plot:
    • order $d_j^2$ from smallest to largest
    • Graph the pairs $(q_{c,p}(j-0.5)/n,d_{(j)}^2)$
      • where $q_{c,p}(j-0.5)/n$ is the $100(j-0.5)/n$ quantile of the chi-square distribution with $p$ degrees of freedom
      • $q_{c,p}(j-0.5)/n={\chi }_p^2((n-j+0.5)/n,p)$ ← use chi square (=CHISQ.INV.RT((n-j+0.5)/n,2))
        • note that j=1,2,…,n
  • Find the Correlation Coefficient of the line, it is our test statistics
  • Compare it against Chi-squared distribution with n dof, reject if $ts<{\chi }_n^2(\alpha )$