12.(EM algorithm) General EM for GMM

E-step & M-step for GMM

less than 1 minute read

Seunghan Lee

Seunghan Lee

Deep Learning, Data Science, Statistics

4. General EM for GMMPermalink

In this part, we’ll see how EM algorithm can be applied in GMM.

It will be as the below.

E stepPermalink

EM : For each point, compute
$q(t_i) = p(t_i|x_i,\theta)$
GMM : For each point, compute
$p(t_i|x_i,\theta)$

M stepPermalink

EM : Update parameters to maximize
$\underset{\theta}{max}E_qlogp(X,T|\theta)$
GMM : Update Gaussian parameters to fit points assigned to them
$\mu_1 = \frac{\sum_{i}p(t_i=1|x_i,\theta)x_i}{\sum_{i}p(t_i=1|x_i,\theta)}$

We’ll see how the equation became like the above.

M-step for GMMPermalink

Let’s take an example when t_i = 1,2,3 ( a total of three clusters(Gaussians) )

$\begin{align*} &\underset{\theta}{max}\sum_{i=1}^{N}E_{q(t_i)}logp(x_i,t_i|\theta) \\ &=\sum_{i=1}^{N}\sum_{c=1}^{3}q(t_i=c)\cdot log(\frac{1}{z}\cdot exp(-\frac{(x_i-\mu_c)^2}{2\sigma_c^2})\pi_c)\\ &=\sum_{i=1}^{N}\sum_{c=1}^{3}q(t_i=c)\cdot (log\frac{\pi_c}{z}-\frac{(x_i-\mu_c)^2}{2\sigma_c^2}) \end{align*}$

[ optimize for ‘mu’ ]Permalink

Let’s optimize the expression above with respect to mu_1.

$\begin{align*} &\frac{\partial }{\partial \mu_1} \sum_{i=1}^{N}\sum_{c=1}^{3}q(t_i=c)\cdot (log\frac{\pi_c}{z}-\frac{(x_i-\mu_c)^2}{2\sigma_c^2}) \\ &= \sum_{i=1}^{N}q(t_i=1)\cdot (0-\frac{2(x_i-\mu_1)(-1)}{2\sigma_1^2}) = 0 \end{align*}$

If we multiply it by sigma_1 ( = variance of the first gaussian ) and the solve the equation, the result will be like this.

$\mu_1 = \frac{\sum_{i}q(t_i=1)x_i}{\sum_{i}q(t_i=1)}$

[ optimize for ‘sigma’ ]Permalink

This time, let’s optimize the expression above with respect to sigma_c. Like the same way as the above, the result will be like this.

$\sigma_c^2 = \frac{\sum_{i}(x_i-\mu_c)^2\cdot q(t_i=c)}{\sum_{i}q(t_i=c)}$

[ optimize for prior weights ‘pi’ ]Permalink

( same as the above )

$\pi_c = \frac{\sum_{i}q(t_i=c)}{N}$

Twitter Facebook LinkedIn

You May Also Enjoy

8 minute read

2 minute read

5 minute read

14 minute read