NOTES ON STATISTICS, PROBABILITY and MATHEMATICS

Moment Generating Function (MGF):

In probability theory and statistics, the moment-generating function of a real-valued random variable is an alternative specification of its probability distribution. Thus, it provides the basis of an alternative route to analytical results compared with working directly with probability density functions or cumulative distribution functions. There are particularly simple results for the moment-generating functions of distributions defined by the weighted sums of random variables. Note, however, that not all random variables have moment-generating functions.

The moment-generating function of a random variable \(X\) is

\[\color{blue}{\displaystyle M_{X}(t):=\mathbb {E} \!\left[e^{tX}\right],\quad t\in \mathbb {R}}\]

MOTIVATION:


Instead of integrating the defining equation to obtain the raw moments, we can

  1. First get this “moment generating function (MGF)”; and
  2. Derive the MGF to get the raw moments.
DEFINITION of MGF:


\[\color{blue}{\Large M_X(t) = \mathbb{E}[e^{\,tX}]}\]

This is calculated differently for discrete distributions (Probability Mass Function, PMF):

\(\color{blue}{\Large M_X(t) = \mathbb{E}[e^{\,tX}]=\displaystyle \sum_{i=1}^{\infty}\, e^{tx_i}\,p_i}\tag{1}\)

as compared to continuous distributions (PDF):

\(\color{blue}{\Large M_X(t) = \mathbb{E}[e^{\,tX}]=\displaystyle \int_{-\infty}^{\infty}\, e^{tx}\,f(x)\, dx}\tag{2}\)

Notice that the MGF is the Laplace transform with \(-s\) replaced by \(t\). Here is the formula for the Laplace transform:

\(\large\mathscr L\{f\}(s)= \mathbb E [e^{-sX}].\)

Remember that the Taylor series of a function \(f(x)\) is the power series:

\[f(a) + \frac{f'(a)}{1!}(x-a) + \frac{f''(a)}{2!}(x-a)^2 + \frac{f'''(a)}{3!}(x-a)^3+\cdots\]

The Taylor series of \(e^x\) at \(a=0\) is:

\[1 + \frac{x^1}{1!} + \frac{x^2}{2!} + \frac{x^3}{3!} + \frac{x^4}{4!} + \cdots = 1 + x + + \frac{x^2}{2} + \frac{x^3}{6} + \frac{x^4}{24} + \cdots= \sum_{n=0}^\infty \frac{x^n}{n!}\]


We now expand the Taylor series of the function \(e^{tX}\) within the expectation operator. If \(a=0\) the series is called a Maclaurin series. Of note, we differentiate with respect to \(t\). Attention: \(X\) is a constant. Hence, \(\large\frac{d}{dt}e^{tX}=Xe^{tX}\); and \(\frac{d^2}{dt^2}e^{tX}=X^2e^{tX}\).


\[ \large \begin{align} \large M_X(t) &= \mathbb{E}[e^{\,tX}]\\ &=\mathbb{E} \left[ \displaystyle \sum_{n=0}^\infty \frac{X^{(n)}\,e^{Xa}}{n!}\,(t-a)^n \right]\vert a\\ &=\mathbb E\left[\frac{e^{Xa}}{0!} + \frac{Xe^{Xa}}{1!}(t - a) + \frac{X^2e^{Xa}}{2!}(t - a)^2 + \frac{X^3e^{Xa}}{3!}(t - a)^3+\cdots \right]\,\vert a \\ \,\,\,\\ &\underset{\text{a=0}}=\,\,\,\mathbb{E}\left[1 + \frac{Xt}{1!} + \frac{X^2t^2}{2!} + \frac{X^3t^3}{3!} + \cdots\right]\\ &=\mathbb{E}\left[1\right] +\mathbb{E}\left[ \frac{Xt}{1!}\right] + \mathbb{E}\left[\frac{X^2t^2}{2!}\right] + \mathbb{E}\left[\frac{X^3t^3}{3!}\right] + \cdots\\ &= 1 + \frac{\mathbb{E} \left[X\right]}{1!} \, t \, + \frac{\mathbb{E} \left[X^2\right]}{2!} \, t^2 \, + \frac{\mathbb{E} \left[X^3\right]}{3!} \, t^3 \, + \cdots \end{align} \]


Now, since we are evaluating at \(t=0\), every term with a \(t\) will be \(0\), but the point is to differentiate the power series over and over, so that every time there is just one term without \(t\).

As shown, the moments are just the coefficient that will “survive” after taking the derivative corresponding to the number of the moment.

The first moment:

\[\large \mathbb{E}[\color{blue}{X}] = \int_{-\infty}^{\infty}\color{blue}{X^1}\,\,\color{green}{\text{pdf}}\,\,\,dx\]

can be found “perched” in the MGF as thefirst derivative evaluated at zero:

\[\large M_X(t) = 1 + \frac{ \color{red}{\mathbb{E}\left[X\right]}}{1!} \, t \, + \frac{\mathbb{E} \left[X^2\right]}{2!} \, t^2 \, + \frac{\mathbb{E} \left[X^3\right]}{3!} \, t^3 \, + \cdots\]

Therefore, differentiating once:

\[M^{'}_X(t=0)= \frac{\mathbb{E} \left[X\right]}{1!}= \color{red}{\mathbb E [{X}]}\]

So what is the point?

We started off with \(\color{red}{e^{\,tX}}\), but we don’t have it any longer:

Instead of going through the integration process for each moment which by LOTUS the expectation is the integral of the function of \(X\) times its pdf:

\[\large \mathbb{E}[\color{blue}{X^k}] = \int_{-\infty}^{\infty}\color{blue}{X^k}\,\,\color{green}{\text{pdf}}\,\,\,dx\]

a process that can be hairy, especially compared to integrating:

\[\large M_X(t) = \mathbb{E}[\color{blue}{e^{\,tX}}]=\displaystyle \int_{-\infty}^{\infty}\, \color{blue}{e^{tX}}\,\color{green}{\text{pdf}}\, dx\]

and later differentiating a polynomial to recover the moments.


Let’s put it to work to derive the second raw moment (for example) for the exponential (remember that pdf of an exponential is \(\lambda\,e^{-\lambda x}\)).

First find this gold mine that is the MGF. Again we could directly calculate the moment by integrating:

\[\ \begin{align} \large \mathbb E\left[ X^2 \right] &= \displaystyle \int_{-\infty}^{\infty} X^2\,\text{pdf}\,dx\\ &= \displaystyle \int_{-\infty}^{\infty} \color{red}{X^2}\color{green}{\lambda\,e^{-\lambda X}}\,dx \end{align} \]

or

\[ \begin{align} \large M_X(t) &=\mathbb{E}[e^{\,tX}]=\displaystyle\int_{-\infty}^{\infty}e^{\,tX}\,\,\text{pdf}\,\,\,dx\\ &= \displaystyle\int_{-\infty}^{\infty}\color{red}{e^{\,tX}}\,\,\,\color{green}{\lambda\,e^{-\lambda x}}\,\,dx\\ &= \lambda \displaystyle\int_{-\infty}^{\infty}e^{\,X(t-\lambda)}\,\,\,dx\\ &= \lambda \left(\left.\frac{e^{\,X(t-\lambda)}}{X(t-\lambda)} \right|_{-\infty}^{\infty}\right)=\frac{\lambda}{t-\lambda}\\ &= \lambda\,(t-\lambda)^{-1}. \end{align} \]

Now we simply have to differentiate and evaluate at zero:


\[ \begin{align}\large M_X^{2}(t = 0) &=\frac{d^2}{dt^2}M_X(t=0)\\ &= \frac{d^2}{dt^2}\lambda\,(t-\lambda)^{-1}=\frac{2}{\lambda^2}. \end{align} \]

Home Page

NOTE: These are tentative notes on different topics for personal use - expect mistakes and misunderstandings.