*Bounty: 50*

*Bounty: 50*

Most mixed effects models assume that the random effects $gamma$ follow a $MVN(0, Sigma_{gamma})$ distribution. In some cases, specific structure is put on $Sigma_{gamma}$. For now, let’s just assume it’s an unstructured covariance matrix.

Now, if we don’t apply any priors or penalties on $Sigma_{gamma}$, it seems to me that the likelihood is unbounded. In particular, if we set $gamma = 0$, then as $det(Sigma_{gamma}) rightarrow 0$, the contribution of the log-density of $gamma$ approaches infinity. As long as the other contributions to the likelihood are finite (which is typically the case), this implies the log-likelihood would be unbounded.

Clearly, for the MLE this creates an issue (although I know REML is a more popular alternative). For most MCMC algorithms, having unbounded log-densities can be problematic as well, even if the posterior is still proper.

How is this issue typically handled? Is there a canonical penalty/prior on $Sigma_{gamma}$?