Efficient Variational Inference for Gaussian Process Regression Networks (2013)Permalink

AbstractPermalink

( Multi-output regression ) correlation between $Y$s may vary with input space

GPRNs (Gaussian Process Regression Networks)

• flexible
• intractable

Thus, propose 2 efficient VI methods for GPRNs

(1) GPRN-MF

• adopts mean-field with full Gaussian over GPRN’s parameters

  
  

(2) GPRN-NPV

• non-parametric VI
• derive analytical forms of ELBO
• closed-form updates of parameters
• $O(N)$ for parameter’s covariances

1. IntroductionPermalink

Challenge in multi-output :

• 1) develop flexible models able to capture the dependencies between $Y$s
• 2) efficient inference

Various non-probabilistic approaches have been developed.

It is crucial to have full posterior probabilities

GP have proved very effective tools for single & multiple output

GP-based methods :

• before) assume that the dependencies between the $Y$s are fixed

  ( = independent of the input space )

• after ) correlation between $Y$s can be spatially adaptive

  $\rightarrow$ GAUSSIAN PROCESS REGRESSION NETWORKS (GPRNs)

This paper proposes “efficient approximate inference methods for GPRNs”

(1) First method : simple MF approach of GPRN

• show that…
  • 1) can obtain analytical expression of ELBO & closed-form update of variational params
  • 2) parameterize the corresponding covariances with only $O(N)$ params

(2) Second method : exploits VI

• non-parametric VI to approximate posterior of GPRN’s params
• approximate complex distn, which are not well approximated by single Gaussian
• needs $O(N)$ variational params

2. GPRNPermalink

Input : $\mathbf{x} \in \mathbb{R}^{D}$.

Output : $\mathbf{y}(\mathbf{x}) \in \mathbb{R}^{P}$.

• assumed to be linear combination of $Q$ noisy latent functions $\mathrm{f}(\mathrm{x}) \in \mathbb{R}^{Q}$
• corrupted by Gaussian noise

Mixing Coefficients : $\mathbf{W}(\mathbf{x}) \in \mathbb{R}^{P} \times \mathbb{R}^{Q}$

[ GPRN model ]

$\begin{aligned} \mathrm{y}(\mathrm{x}) &=\mathrm{W}(\mathrm{x})\left[\mathrm{f}(\mathrm{x})+\sigma_{f} \epsilon\right]+\sigma_{y} \mathrm{z} \\ f_{j}(\mathrm{x}) & \sim \mathcal{G} \mathcal{P}\left(0, \kappa_{f}\right), \quad j=1 \ldots Q \\ W_{i j}(\mathrm{x}) & \sim \mathcal{G} \mathcal{P}\left(0, \kappa_{w}\right), \quad i=1, \ldots, P ; j=1, \ldots Q \\ \epsilon & \sim \mathcal{N}\left(\epsilon ; 0, \mathrm{I}_{Q}\right) \\ \mathrm{z} & \sim \mathcal{N}\left(\mathrm{z} ; 0, \mathrm{I}_{P}\right) \end{aligned}$.

Advantage of GPRN model :

• 1) dependencies of outputs $y$ are induced via latent functions $\mathbf{f}$

• 2) mixing coefficients $\mathbf{W}(\mathbf{x})$ explicitly depends on $\mathbf{x}$

  ( = correlations are spatially adaptive )

Notation

• Observed Inputs : $\mathcal{X}=\left\{\left(\mathbf{x}_{i}\right)\right\}_{i=1}^{N}$

• Observed Outputs : $\mathcal{D}=\left\{\left(\mathbf{y}_{i}\right)\right\}_{i=1}^{N}$

• concatenation of latent function params & weights : $\mathbf{u}=(\hat{\mathbf{f}}, \mathbf{w}),$

• noisy version of latent function values : $\hat{\mathrm{f}}=\mathrm{f}+\sigma_{f} \epsilon,$

• hyperparameters of GPRN : $\theta=\left\{\boldsymbol{\theta}_{f}, \boldsymbol{\theta}_{w}, \sigma_{f}, \sigma_{y}\right\}$

Prior :

$\mathbf{u}$ : $p\left(\mathbf{u} \mid \boldsymbol{\theta}_{f}, \boldsymbol{\theta}_{w}, \sigma_{f}\right)=\mathcal{N}\left(\mathbf{u} ; \mathbf{0}, \mathbf{C}_{u}\right)$.

Conditional Likelihood :

$p(\mathcal{D} \mid \mathbf{u}, \boldsymbol{\theta})=\prod_{n=1}^{N} \mathcal{N}\left(\mathbf{y}\left(\mathbf{x}_{n}\right) ; \mathbf{W}\left(\mathbf{x}_{n}\right) \hat{\mathbf{f}}\left(\mathbf{x}_{n}\right), \sigma_{y}^{2} \mathbf{I}_{P}\right)$.

• $\mathrm{y}(\mathrm{x}) =\mathrm{W}(\mathrm{x})\left[\mathrm{f}(\mathrm{x})+\sigma_{f} \epsilon\right]+\sigma_{y} \mathrm{z}$.

• $\mathrm{z} \sim \mathcal{N}\left(\mathrm{z} ; 0, \mathrm{I}_{P}\right)$.

Posterior :

$p(\mathbf{u} \mid \mathcal{D}, \boldsymbol{\theta}) \propto p\left(\mathbf{u} \mid \boldsymbol{\theta}_{f}, \boldsymbol{\theta}_{w}, \sigma_{f}\right) p\left(\mathcal{D} \mid \mathbf{u}, \sigma_{y}\right)$.

3. VI for GPRNsPermalink

minimize KL-divergence :

• $\mathrm{KL}(q(\mathbf{u}) \| p(\mathbf{u} \mid \mathcal{D}))=\mathbb{E}_{q}\left[\log \frac{q(\mathbf{u})}{p(\mathbf{u} \mid \mathcal{D})}\right]$.

maximize ELBO :

• $\mathcal{L}(q)=\mathbb{E}_{q}[\log p(\mathcal{D} \mid \mathbf{f}, \mathbf{w})]+\mathbb{E}_{q}[\log p(\mathbf{f}, \mathbf{w})]+\mathcal{H}_{q}[q(\mathbf{f}, \mathbf{w})]$.

for mean-field method, we can obtain…

• 1) analytical expression for ELBO
• 2) need only $O(N)$ params for covariances

3-1. MFVI for GPRNPermalink

factorized distributions :

$q(\mathbf{f}, \mathbf{w}) =\prod_{j=1}^{Q} q\left(\mathbf{f}_{j}\right) \prod_{i=1}^{P} q\left(\mathbf{w}_{i j}\right)$.

• $q\left(\mathbf{f}_{j}\right) =\mathcal{N}\left(\mathbf{f}_{j} ; \boldsymbol{\mu}_{\mathrm{f}_{j}}, \Sigma_{\mathrm{f}_{j}}\right)$.
• $q\left(\mathbf{w}_{i j}\right) =\mathcal{N}\left(\mathbf{w}_{i j} ; \mu_{\mathrm{w}_{\mathrm{ij}}}, \Sigma_{\mathrm{w}_{\mathrm{ij}}}\right)$.

3-1-1. Closed-form ELBOPermalink

( full Gaussian mean-field approximation ) ELBO

(1) First term :

$\begin{array}{l} \mathbb{E}_{q}[\log p(\mathcal{D} \mid \mathbf{f}, \mathbf{w})]&=-\frac{N P}{2} \log \left(2 \pi \sigma_{y}^{2}\right) \\ &-\frac{1}{2 \sigma_{y}^{2}} \sum_{n=1}^{N}\left(\mathbf{Y}_{\cdot n}^{T}-\Omega_{\mathrm{w}_{\mathrm{n}}} \nu_{\mathrm{f}_{\mathrm{n}}}\right)^{T}\left(\mathbf{Y}_{\cdot n}^{T}-\Omega_{\mathrm{w}_{\mathrm{n}}} \nu_{\mathrm{f}_{\mathrm{n}}}\right) \\ &-\frac{1}{2 \sigma_{y}^{2}} \sum_{i=1}^{P} \sum_{j=1}^{Q}\left[\operatorname{diag}\left(\Sigma_{\mathrm{f}_{\mathrm{j}}}\right)^{T}\left(\mu_{\mathrm{w}_{\mathrm{ij}}} \bullet \mu_{\mathrm{w}_{\mathrm{ij}}}\right)\right. \left.+\operatorname{diag}\left(\Sigma_{\mathrm{w}_{\mathrm{ij}}}\right)^{T}\left(\mu_{\mathrm{f}_{\mathrm{j}}} \bullet \mu_{\mathrm{f}_{\mathrm{j}}}\right)\right] \end{array}$.

(2) Second term :

$\begin{array}{l} \mathbb{E}_{q}[\log p(\mathbf{f}, \mathbf{w})]&= -\frac{1}{2} \sum_{j=1}^{Q}\left(\log \left|\mathbf{K}_{f}\right|+\boldsymbol{\mu}_{\mathrm{f}_{j}}^{T} \mathbf{K}_{f}^{-1} \mu_{\mathrm{f}_{j}}+\operatorname{tr}\left(\mathbf{K}_{f}^{-1} \boldsymbol{\Sigma}_{\mathrm{f}_{\mathrm{j}}}\right)\right) \\ &-\frac{1}{2} \sum_{i, j}\left(\log \left|\mathbf{K}_{w}\right|+\mu_{\mathrm{w}_{\mathrm{i} j}} \mathbf{K}_{w}^{-1} \mu_{\mathrm{w}_{\mathrm{ij}}}+\operatorname{tr}\left(\mathbf{K}_{w}^{-1} \boldsymbol{\Sigma}_{\mathrm{w}_{\mathrm{ij}}}\right)\right) \end{array}$.

(3) Third term :

3-1-2. Efficient Closed-form Updates for Variational ParametersPermalink

Parameters for $q(\mathbf{f}_j)$

• $\mu_{\mathrm{f}_{\mathrm{j}}}=\frac{1}{\sigma_{y}^{2}} \Sigma_{\mathrm{f}_{j}} \sum_{i=1}^{P}\left(\mathrm{Y}_{\cdot i}-\sum_{k \neq j} \mu_{\mathrm{w}_{\mathrm{ik}}} \bullet \mu_{\mathrm{f}_{\mathrm{k}}}\right) \bullet \mu_{\mathrm{w}_{\mathrm{i} j}}$.
• $\boldsymbol{\Sigma}_{\mathrm{f}_{\mathrm{j}}}=\left(\mathbf{K}_{f}^{-1}+\frac{1}{\sigma_{y}^{2}} \sum_{i=1}^{P} \operatorname{diag}\left(\boldsymbol{\mu}_{\mathrm{w}_{\mathrm{ij}}} \bullet \boldsymbol{\mu}_{\mathrm{w}_{\mathrm{ij}}}+\operatorname{Var}\left(\mathbf{w}_{i j}\right)\right)\right)^{-1}$.

Parameters for $q(\mathbf{w}_{ij})$

• $\mu_{\mathrm{w}_{\mathrm{ij}}}=\frac{1}{\sigma_{y}^{2}} \Sigma_{\mathrm{w}_{\mathrm{ij}}}\left(\mathrm{Y}_{\cdot i}-\sum_{k \neq j} \mu_{\mathrm{f}_{\mathrm{k}}} \bullet \mu_{\mathrm{w}_{\mathrm{ik}}}\right) \bullet \mu_{\mathrm{f}_{\mathrm{j}}}$>
• $\Sigma_{\mathrm{w}_{\mathrm{ij}}}=\left(\mathrm{K}_{w}^{-1}+\frac{1}{\sigma_{y}^{2}} \operatorname{diag}\left(\mu_{\mathrm{f}_{j}} \bullet \mu_{\mathrm{f}_{j}}+\operatorname{Var}\left(\mathrm{f}_{j}\right)\right)\right)^{-1}$.

3-1-3. Hyper-parameters LearningPermalink

hyperparameters : $\boldsymbol{\theta}=\left\{\boldsymbol{\theta}_{f}, \boldsymbol{\theta}_{w}, \sigma_{f}, \sigma_{y}\right\}$.

learn by gradient-based optimization of ELBO

3-2. Non-parametric VI for GPRNPermalink

approximate posterior of GPRN, using mixture of $K$ isotropic Gaussian

$q(\mathbf{u})=\frac{1}{K} \sum_{k=1}^{K} q^{(k)}(\mathbf{u})=\frac{1}{K} \sum_{k=1}^{K} \mathcal{N}\left(\mathbf{u} ; \boldsymbol{\mu}^{(k)}, \sigma_{k}^{2} \mathbf{I}\right)$.

• in practice, $K$ is very small, so complexity is $O(N)$

3-2-1. Closed-form ELBOPermalink

$q(\mathbf{u})=\frac{1}{K} \sum_{k=1}^{K} q^{(k)}(\mathbf{u})=\frac{1}{K} \sum_{k=1}^{K} \mathcal{N}\left(\mathbf{u} ; \boldsymbol{\mu}^{(k)}, \sigma_{k}^{2} \mathbf{I}\right)$. cannot be computed analytically

$\rightarrow$ need approximation

Expectations decompose as … (using mean-field) :

(1) First term

$\begin{array}{l} \mathbb{E}_{q}[\log p(\mathcal{D} \mid \mathbf{f}, \mathbf{w})]= \\ -\frac{1}{2 K \sigma_{y}^{2}} \sum_{k} \sum_{n}\left(\mathbf{Y}_{\cdot n}^{T}-\Omega_{\mathrm{wn}}^{(k)} \nu_{\mathrm{f}_{\mathrm{n}}}^{(k)}\right)^{T}\left(\mathbf{Y}_{\cdot n}^{T}-\Omega_{\mathrm{W}_{\mathrm{n}}}^{(k)} \nu_{\mathrm{f}_{\mathrm{n}}}^{(k)}\right) \\ -\frac{1}{2 K}\left(\sum_{k, j} \frac{P \sigma_{k}^{2}}{\sigma_{y}^{2}} \mu_{\mathrm{f}_{j}}^{(k)^{T}} \mu_{\mathrm{f}_{j}}^{(k)}+\sum_{k, i, j} \frac{P \sigma_{k}^{2}}{\sigma_{y}^{2}} \mu_{\mathrm{w}_{\mathrm{ij}}}^{(k)^{T}} \mu_{\mathrm{w}_{\mathrm{ij}}}^{(k)}\right) \\ -\frac{1}{2 K}\left(\sum_{k} \frac{\sigma_{k}^{4}}{\sigma_{y}^{2}} N P Q+N P \log \left(2 \pi \sigma_{y}^{2}\right)\right) \end{array}$.

(2) Second term

$\begin{array}{l} \mathbb{E}_{q}[\log p(\mathbf{f}, \mathbf{w})]=-\frac{1}{2}\left(Q \log \left|\mathbf{K}_{f}\right|+P Q \log \left|\mathbf{K}_{w}\right|\right) \\ -\frac{1}{2 K}\left[\sum_{k, j} \boldsymbol{\mu}_{\mathfrak{f}_{j}}^{(k)^{T}} \mathbf{K}_{f}^{-1} \boldsymbol{\mu}_{\mathfrak{f}_{j}}^{(k)}+\sigma_{k}^{2} \operatorname{tr}\left(\mathbf{K}_{f}^{-1}\right)\right. \\ \left.+\sum_{k, i, j} \boldsymbol{\mu}_{\mathbf{w}_{i j}}^{(k)^{T}} \mathbf{K}_{w}^{-1} \boldsymbol{\mu}_{\mathrm{w}_{i j}}^{(k)}+\sigma_{k}^{2} \operatorname{tr}\left(\mathbf{K}_{w}^{-1}\right)\right] \end{array}$.

(3) Third term

$\mathcal{H}_{q}[q(\mathbf{u})] \geq -\frac{1}{K} \sum_{k=1}^{K} \log \frac{1}{K} \sum_{j=1}^{K} \mathcal{N}\left(\bold{\mu}^{(k)} ; \bold{ \mu}^{(j)},\left(\sigma_{k}^{2}+\sigma_{j}^{2}\right) \mathbf{I}\right)$.

(1)~(3) define tight ELBO

3-2-2. Optimization of Variational Parameters and Hyper-parametersPermalink

Optimization of variational params $\left\{\mu_{\mathrm{f}_{j}}^{(k)}, \mu_{\mathrm{w}_{\mathrm{ij}}}^{(k)}\right\}$ & hyperparameters $\theta$

3-3. Predictive DistributionPermalink

for non-parametric VI, predictive mean turns out to be…

$\mathbb{E}\left[\mathbf{y}^{*} \mid \mathbf{x}^{*}, \mathcal{D}\right]=\frac{1}{K} \sum_{k=1}^{K} \mathbf{K}_{w}^{*} \mathbf{K}_{w}^{-1} \boldsymbol{\mu}_{\mathbf{w}}^{(k)} \mathbf{K}_{f}^{*} \mathbf{K}_{f}^{-1} \boldsymbol{\mu}_{\mathbf{f}}^{(k)}$.