Multivariate Gaussian

Multivariate Gaussian:

The univariate Gaussian ( $X \sim N(\mu, \sigma^2$ ) is:

$f(x) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{1}{2\sigma^2}(x-\mu)^2\right), \forall \in \mathbb R.$

The degenerate Gaussian has variance equal to $0$ and hence, $X(\omega)=\mu, \forall \omega \in \Omega.$

The multivariate Gaussian is defined for $X \in \mathbb R^n$ as any linear combination of univariate Gaussian distributions $X_i$ :

$a^T X = \sum_{i=1}^n a_i X_i$ for $\forall a \in \mathbb R^n.$

We will express it as $X\sim N(\mu,\Sigma)$ , where $\mu$ is a vector in $\mathbb R^n$ . The $\mathbb E(X_i)=\mu_i$ ; and the covariance matrix is an $n \times n$ positive semidefinite matrix, such that $\Sigma(X_i, X_j) = \text{Cov}_{ij}.$

A multivariate Gaussian is degenerated if the $\text{det}(\Sigma)=0.$

If the Gaussian distributions (components) are independent, the covariance matrix is:

$\Sigma = \begin{bmatrix}\sigma_1^2&0\\0&\sigma_2^2\end{bmatrix}$

If the variance is $1$ for both Gaussians:

If the variances are different, say $1$ and $4$ :

This is the Matlab code:

pkg load 'statistics' % This only works like this in Octave
mu = [0,0]; %// data
sigma = [1 0; 0 4]; %// data
x = -5:.2:5; %// x axis
y = -5:.2:5; %// y axis

[X Y] = meshgrid(x,y); %// all combinations of x, y
Z = mvnpdf([X(:) Y(:)],mu,sigma); %// compute Gaussian pdf
Z = reshape(Z,size(X)); %// put into same size as X, Y
colormap(jet)
surf(X,Y,Z) %// ... or 3D plot

The density of an $M$ -dimensional multivariate Gaussian is:

$f({\bf X \vert \mu, \Sigma}) = \frac{1}{(2\pi)^{M/2}\,\text{det}(\Sigma)^{1/2} }\text{exp}\left[-\frac{1}{2}({\bf X-\mu})^T \,{\bf \Sigma^{-1}\,(X-\mu)}\right]$

The important part of the formula is the exponent, $({\bf X-\mu})^T \,{\bf \Sigma^{-1}\,(X-\mu)}$ , which is a positive definite quadratic function. The part in front is just a normalizing factor.

A quadratic form in linear algebra is of the form $x^TAx$ , providing the formula for ellipsoids in higher dimensions. And $\Sigma$ can be visualized as an “error elipsoid” around the mean.

Ellipsoids are of the form $x^2/a^2+y^2/b^2+z^2/c^2 = 1$ :

This is Dr. Strang’s example of a $3 \times 3$ positive definite matrix. What he calls “the good matrix”:

$A= \begin{bmatrix}\,\,\,2&-1&0\\-1&\,\,2&-1\\\,\,\,0&-1&2\end{bmatrix}$

Proving that it is positive definite through the $x^TAx$ rule…

$x^TAx=2x_1^2 + 2x_2^2 + 2x_3^2 - 2x_1x_2 - 2x_2x_3 > 0$

we could complete the square to proof that the inequality is true.

We are in four dimensions, being that we have a function. But if we cut through this thing at height one, we get an ellipsoid (a lopsided football) with its axes determined by the eigenvalues in the factorization $Q\Lambda Q^T$ , where $Q$ is the matrix of eigenvectors, and $\Lambda$ the diagonal of the squared eigenvalues, $\lambda_i \geq 0$ . Hence,

$Q\Lambda Q^T= Q\Lambda^{1/2}\Lambda^{1/2} Q^T = (Q\Lambda^{1/2})(Q\Lambda^{1/2})^T=AA^T\tag 1$

Affine Property of the Gaussian:

Any affine transformation $f(x) = AX + b$ of a Gaussian is a Gaussian. If $X \sim N(\mu, \Sigma)$ , $\color{red}{AX + b \sim N(A\mu + b, A\Sigma A^T)}$ .

So if $X_1, X_2, \dots, X_n \sim N(0,1)$ iid, placing them in a vector, we get $X \sim (0, I)$ , and $\color{blue}{Ax + \mu \sim N(\mu ,\Sigma)}$ , where $\Sigma = AA^T$ . This is a form of generating multivariate Gaussians.

Sphering:

“Sphering” turns a Gaussian into a “sphere” (multivariate standard) through an affine transformation. So it converts Gaussians back to standard multivariate normal.

$Y \sim N(\mu, \Sigma)\implies A^{-1}(Y-\mu) \sim N(0,\bf I)$ , where $\Sigma = AA^T$ .

Using equation (1) we can use express the $A$ in $\color{blue}{Y = AX + \mu \sim N(\mu,\Sigma)}$ (blue equation) as $A = Q\Lambda^{1/2}$ , giving $Y = AX + \mu= Q\Lambda^{1/2}X + \mu$ . Now we can apply the red equation to $\Lambda^{1/2}X$ , rendering $\Lambda^{1/2}X + \mu \sim N(\Lambda^{1/2}\times 0 + 0, \Lambda^{1/2} I \Lambda^{1/2}) = N(0, \Lambda)$ . $\Lambda$ is geometrically the degree of stretching of the distribution (the variance). When we multiply by $Q$ (an orthogonal matrix) $Q\Lambda^{1/2}$ we end up with $Q\Lambda^{1/2} \sim N(0, \Sigma)$ . An orthogonal matrix give a reflection or rotation.

So by applying an affine transformation to $X \sim N(0,I)$ we end up stretching and rotating. The $\mu$ centers the multivariate (shif).

MARGINAL DISTRIBUTION:

If we have a multivariate Gaussian $X = [X_1,X_2] \in \mathbb R^2$ with two dimensions, the coordinates are also Gaussian ( $X_1$ and $X_2$ are Gaussian).

Proof:

In general, we can decompose an $n$ -dimension multivariate Gaussian matrix $X \sim N(\mu, \Sigma)$ by getting the first $k$ components, indexed as $a = 2, \dots, k$ . We’ll try to show that these first $k$ components are Gaussian. The rest of the components are indexed by $b = k+1, \dots, n$ . Now $X$ can be expressed as a block vector:

$X= \begin{bmatrix}X_a\\X_b\end{bmatrix}$ with $X_a= \begin{bmatrix}X_1\\\vdots\\X_k\end{bmatrix}$ and $X_b= \begin{bmatrix}X_{k+1}\\\vdots\\X_n\end{bmatrix}$ .

We can decompose $\mu = \begin{bmatrix}\mu_a\\\mu_b\end{bmatrix}$

and $\Sigma$ into the block matrix $\Sigma = \begin{bmatrix}\Sigma_{aa}&\Sigma_{ab}\\\Sigma_{ba}&\Sigma_{bb}\end{bmatrix}$ . This is a block matrix because $\Sigma_{aa}=\small\begin{bmatrix}\sigma^2(X_1)&\dots&\text{cov}(X_1,X_k)\\\vdots&\ddots&\vdots\\\text{cov}(X_k,X_1)&\dots&\sigma^2(X_k)\end{bmatrix}$

The marginalization property states that $X_a \sim N({\bf \mu_a}, \Sigma_{aa})$ is multivariate normal. We can prove it using the affine property and using the projection matrix $A$ (without blocks it is of the form $[1 0]$ or $[0 1]$ ):

$A = \begin{bmatrix} {\color{red}{1}}&0&0&0&0&\dots&0&&0&0&0&\dots&0\\ 0&{\color{red}{1}}&0&0&0&\dots&0&&0&0&0&\dots&0\\ 0&0&{\color{red}{1}}&0&0&\dots&0&&0&0&0&\dots&0\\ 0&0&0&{\color{red}{1}}&0&\dots&0&&0&0&0&\dots&0\\ 0&0&0&0&{\color{red}{1}}&\dots&0&&0&0&0&\dots&0\\\vdots&\vdots&\vdots&\vdots&\vdots&\ddots&\vdots&&\vdots&\vdots&\vdots&\ddots&\vdots\\0&0&0&0&0&\dots&{\color{red}{1}}&&0&0&0&\dots&0\\ \end{bmatrix}$

which is $(k \times n)$ .

And by construction,

$AX=X_a$ which by the affine property (red equation):

Given that $A\mu = \mu_a$ (the projection of the means); and $A\Sigma A^T = \Sigma{aa}$ ,

$AX=A_a \sim N(\mu_a, \Sigma_{aa})$ .

The same can be done with $X_b$ .

####CONDITIONAL DISTRIBUTION:

If $X= (X_1, X_2)^T \in \mathbb R^2 \implies (X_1 \vert X_2 = x_2)$ is Gaussian.

Using the same block matrices as above:

$(X_a \vert X_b =x_b) \sim N(m, D)$ where $m= \mu_a + \Sigma_{ab}\Sigma_{bb}^{-1}(x_b - \mu_b)$ and $D = \Sigma_{aa} - \Sigma_{ab}\Sigma_{bb}^{-1}\Sigma_{ba}$ .

For a full derivation see here and here.

Home Page

NOTE: These are tentative notes on different topics for personal use - expect mistakes and misunderstandings.

Multivariate Gaussian

NOTES ON STATISTICS, PROBABILITY and MATHEMATICS