# Estimation of covariance matrices

Given a sample *X*_{1},..., *X*_{n}
from a random vector *X* ∈ **R**^{p×1} (a *p*×1 column), an unbiased estimator of the covariance matrix

is the sample covariance matrix

where

is the sample mean.
This is true regardless if the random variable *X* has normal distribution or not. The reason for the factor *n-1* rather than *n* is essentially that the mean is not known and is replaced by the sample mean.

The maximum likelihood estimator of the covariance matrix, however, is slightly different. When the random variable *X* is normally distributed, the maximum likelihood estimate is

Clearly, the difference between the unbiased and the maximum likelihood estimator diminishes for large *n*.

The probability distribution of the maximum likelihood estimator of the covariance matrix of a multivariate normal distribution is the Wishart distribution.

## Contents

## The multivariate normal distribution

A random vector *X* ∈ **R**^{p×1} (a *p*×1 "column vector") has a multivariate normal distribution with a nonsingular covariance matrix *Σ* precisely if *Σ* ∈ **R**^{p × p} is a positive-definite matrix and the probability density function of *X* is

where μ ∈ **R**^{p×1} is the expected value. The matrix *Σ* is the higher-dimensional analog of what in one dimension would be the variance.

## Maximum-likelihood estimation

Suppose now that *X*_{1}, ..., *X*_{n} are independent and identically distributed with the distribution above. Based on the observed values *x*_{1}, ..., *x*_{n} of this sample, we wish to estimate *Σ*. (Note that we adhere to the convention of writing random variables as capital letters and data as lower-case letters).

### First steps

It is fairly readily shown that the maximum-likelihood estimate of the mean vector μ is the "sample mean" vector:

See the section on estimation in the article on the normal distribution for details; the process here is similar.

Since the estimate of μ does not depend on *Σ*, we can just substitute it for μ in the likelihood function

and then seek the value of *Σ* that maximizes this (in practice it is easier to work with ).

We have

### The trace of a 1 × 1 matrix

Now we come to the first surprising step.

Regard the scalar as the trace of a 1×1 matrix!

This makes it possible to use the identity tr(*AB*) = tr(*BA*) whenever *A* and *B* are matrices so shaped that both products exist.
We get

(so now we are taking the trace of a *p*×*p* matrix!)

where

is sometimes called the scatter matrix.

### Using the spectral theorem

It follows from the spectral theorem of linear algebra that a positive-definite symmetric matrix *S* has a unique positive-definite symmetric square root *S*^{1/2}. We can again use the "cyclic property" of the trace to write

Let *B* = *S*^{1/2} *Σ*^{−1} *S*^{1/2}. Then the expression above becomes

The positive-definite matrix *B* can be diagonalized, and then the problem of finding the value of *B* that maximizes

reduces to the problem of finding the values of the diagonal entries λ_{1}, ..., λ_{p} that maximize

This is just a calculus problem and we get λ_{i} = *n*, so that *B* = *n I _{p}*, i.e.,

*n*times the

*p*×

*p*identity matrix.

### Concluding steps

Finally we get

i.e., the *p*×*p* "sample covariance matrix"

is the maximum-likelihood estimator of the "population covariance matrix" *Σ*. At this point we are using a capital *X* rather than a lower-case *x* because we are thinking of it "as an estimator rather than as an estimate", i.e., as something random whose probability distribution we could profit by knowing. The random matrix *S* can be shown to have a Wishart distribution with *n* − 1 degrees of freedom.^{[1]} That is:

- .

### Alternative derivation

An alternative derivation of the maximum likelihood estimator can be performed via matrix calculus formulae (see also differential of a determinant and differential of the inverse matrix). It also verifies the aforementioned fact about the maximum likelihood estimate of the mean. Re-write the likelihood in the log form using the trace trick:

The differential of this log-likelihood is

It naturally breaks down into the part related to the estimation of the mean, and to the part related to the estimation of the variance. The first order condition for maximum, , is satisfied when the terms multiplying and are identically zero. Assuming (the maximum likelihood estimate of) is non-singular, the first order condition for the estimate of the mean vector is

which leads to the maximum likelihood estimator

This lets us simplify as defined above. Then the terms involving in can be combined as

The first order condition will hold when the term in the square bracket is (matrix-valued) zero. Pre-multiplying the latter by and dividing by gives

which of course coincides with the canonical derivation given earlier.

## Shrinkage estimation

If the sample size *n* is small and the number of considered variables *p* is large, the
above empirical estimators of covariance and correlation are very unstable.
Specifically, it is possible to furnish estimators that improve considerably
upon the maximum likelihood estimate in terms of mean squared error.
Moreover, for *n* < *p*, the empirical estimate of the covariance matrix becomes
singular, i.e. it cannot be inverted to compute the precision matrix.

As an alternative, many methods have been suggested to improve the estimation of the covariance matrix. All of these approaches rely on the concept of shrinkage. This is implicit in Bayesian methods, in penalized maximum likelihood methods, and explicit in the Stein-type shrinkage approach.

A simple version of a shrinkage estimator of the covariance matrix is constructed as follows. One considers a convex combination of the empirical estimator with some suitable chosen target, e.g., the diagonal matrix. Subsequently, the mixing parameter is selected to maximize the expected accuracy of the shrunken estimator. This can be done by cross-validation, or by using an analytic estimate of the shrinkage intensity. The resulting regularized estimator can be shown to outperform the maximum likelihood estimator for small samples. For large samples, the shrinkage intensity will reduce to zero, hence in this case the shrinkage estimator will be identical to the empirical estimator. Apart from increased efficiency the shrinkage estimate has the additional advantage that it is always positive definite and well conditioned.

A review on this topic is given, e.g., in Schäfer and Strimmer 2005.^{[2]}

A covariance shrinkage estimator is implemented in the R package "corpcor".

## See also

## References

- ↑
K.V. Mardia, J.T. Kent, and J.M. Bibby (1979)
*Multivariate Analysis*, Academic Press. - ↑
J. Schäfer and K. Strimmer (2005)
*A Shrinkage Approach to Large-Scale Covariance Matrix Estimation and Implications for Functional Genomics*, Statistical Applications in Genetics and Molecular Biology: Vol. 4: No. 1, Article 32.