The general formula for an AR(1) process is Xy=ρXt−1+ϵt with ϵt∼iid(0,σ2).
The variance of Xt will be given by:
Var[Xt]=ρ2Var[Xt−1]+Var[ϵt] because Var[aX]=a2Var[X].
The condition of stationarity implies that Var[Xt]=Var[Xt−1]. Therefore,
Var[Xt]=ρ2Var[Xt]+σ2, since Var[ϵt]=σ2. It follows then that…
(1−ρ2)Var[Xt]=σ2. Therefore:
Var[Xt]=σ21−ρ2
To see the covariance of an AR(1) time series with itself lagged, we can backsubstitute in the Xy=ρXt−1+ϵt equation:
Xy=ρXt−1+ϵt=ρ[ρXt−2+ϵt−1]+ϵt=ρ2Xt−2+ρϵt−1+ϵt Or, expressed differently, Xt+h=ρhXt+h−1∑i=0ρiϵt+h−i.
Therefore the covariance of Xt with itself lagged into the future, Xt+h will be:
Cov(Xt,Xt+h)=Cov(Xt,ρhXt+h−1∑i=0ϵt+h−1). But there is no covariance between Xt and this very last term. Hence,
Cov(Xt,Xt+h)=Cov(Xt,ρhXt)
Cov(Xt,Xt+h)=ρhCov(Xt,Xt)
Cov(Xt,Xt+h)=ρhVar(Xt)
and resorting to Eq. 1:
Cov(Xt,Xt+h)=ρhσ21−ρ2
|ρ|<1
Now we can calculate the correlation:
Corr.(Xt,Xt+h)=Cov(Xt,Xt+h)Var(Xt)
Corr.(Xt,Xt+h)=ρh
This implies that the correlogram (ACF) will give progressively (exponentially decaying) spikes each one corresponding to ρstep.
The partial autocorrelation function (PACF) will be used to distinguish AR(1) from AR(2). It allows us to see the residual correlation after a certain number of lags after removing more proximate lags. So in an AR(1), after the first lag the PACF will not be significant. On the other hand, in an AR(2) process, the second lag will still be significant, because of the relationship Xt=ρ1Xt−1+ρ2Xt−2.
They are of the form Xt=ϵt+θϵt−1 with ϵ∼iid(0,σ2). The variance of Xt will therefore be given by:
Var(Xt)=Var(ϵt+θϵt−1)
Var(Xt)=Var(ϵt)+θ2Var(ϵt−1)
Var(Xt)=σ2+θ2σ2=σ2(1+θ2)
As for the covariance:
Cov(Xt,Xt−1)=Cov(ϵt+θϵt−1,ϵt−1+θϵt−2). Since the errors are independent, the Cov of ϵt, ϵt−1 and ϵt−2 is zero.
Cov(Xt,Xt−1)=θCov(ϵt−1,ϵt−1)
Cov(Xt,Xt−1)=θσ2
Notice that this common element in the covariance expression (ϵt−1) only exist if the lag is 1. For instance,
Cov(Xt,Xt+h)=Cov(ϵt+ϵt+h,ϵt+h+θϵt+h−1)
Only in the case of h=1, we’ll end up dealing with…
Cov(Xt,Xt+h)=Cov(ϵt,θϵt)
Cov(Xt,Xt+h)=θCov(ϵt,ϵt)=θσ2. Otherwise the covariance is zero.
Having the variance and covariance, the correlation will be:
For h=1, Corr.(Xt,Xt+h)=θ1+θ2
If h>1, the correlation is zero. This explains why in a MA(1) process, the only statistically significant spike in the ACF is the first one (aside from the spike of 1 of the correlation with itself included in the ACF).
NOTE: These are tentative notes on different topics for personal use - expect mistakes and misunderstandings.