g-prior

In statistics, the g-prior is an objective prior for the regression coefficients of a multiple regression. It was introduced by Arnold Zellner. It is a key tool in Bayes and empirical Bayes variable selection.

Definition

Consider a data set $(x_{1},y_{1}),\ldots ,(x_{n},y_{n})$, where the $x_{i}$ are Euclidean vectors and the $y_{i}$ are scalars. The multiple regression model is formulated as

$y_{i}=x_{i}^{\top }\beta +\varepsilon _{i}.$

where the $\varepsilon _{i}$ are random errors. Zellner's g-prior for $\beta$ is a multivariate normal distribution with covariance matrix proportional to the inverse Fisher information matrix for $\beta$, similar to a Jeffreys prior.

Assume the $\varepsilon _{i}$ are i.i.d. normal with zero mean and variance $\psi ^{-1}$. Let $X$ be the matrix with $i$th row equal to $x_{i}^{\top }$. Then the g-prior for $\beta$ is the multivariate normal distribution with prior mean a hyperparameter $\beta _{0}$ and covariance matrix proportional to $\psi ^{-1}(X^{\top }X)^{-1}$, i.e.,

$\beta |\psi \sim {\text{N}}[\beta _{0},g\psi ^{-1}(X^{\top }X)^{-1}].$

where g is a positive scalar parameter.

Posterior distribution of beta

The posterior distribution of $\beta$ is given as

$\beta |\psi ,x,y\sim {\text{N}}{\Big [}q{\hat {\beta }}+(1-q)\beta _{0},{\frac {q}{\psi }}(X^{\top }X)^{-1}{\Big ]}.$

where $q=g/(1+g)$ and

${\hat {\beta }}=(X^{\top }X)^{-1}X^{\top }y.$

is the maximum likelihood (least squares) estimator of $\beta$. The vector of regression coefficients $\beta$ can be estimated by its posterior mean under the g-prior, i.e., as the weighted average of the maximum likelihood estimator and $\beta _{0}$,

${\tilde {\beta }}=q{\hat {\beta }}+(1-q)\beta _{0}.$

Clearly, as g →∞, the posterior mean converges to the maximum likelihood estimator.

Selection of g

Estimation of g is slightly less straightforward than estimation of $\beta$. A variety of methods have been proposed, including Bayes and empirical Bayes estimators.