12.(EM algorithm) General EM for GMM

E-step & M-step for GMM

less than 1 minute read

Seunghan Lee

Seunghan Lee

Deep Learning, Data Science, Statistics

4. General EM for GMM

In this part, we’ll see how EM algorithm can be applied in GMM.

It will be as the below.

E step

EM : For each point, compute
$q(t_i) = p(t_i|x_i,\theta)$
GMM : For each point, compute
$p(t_i|x_i,\theta)$

M step

EM : Update parameters to maximize
$\underset{\theta}{max}E_qlogp(X,T|\theta)$
GMM : Update Gaussian parameters to fit points assigned to them
$\mu_1 = \frac{\sum_{i}p(t_i=1|x_i,\theta)x_i}{\sum_{i}p(t_i=1|x_i,\theta)}$

We’ll see how the equation became like the above.

M-step for GMM

Let’s take an example when t_i = 1,2,3 ( a total of three clusters(Gaussians) )

$\begin{align*} &\underset{\theta}{max}\sum_{i=1}^{N}E_{q(t_i)}logp(x_i,t_i|\theta) \\ &=\sum_{i=1}^{N}\sum_{c=1}^{3}q(t_i=c)\cdot log(\frac{1}{z}\cdot exp(-\frac{(x_i-\mu_c)^2}{2\sigma_c^2})\pi_c)\\ &=\sum_{i=1}^{N}\sum_{c=1}^{3}q(t_i=c)\cdot (log\frac{\pi_c}{z}-\frac{(x_i-\mu_c)^2}{2\sigma_c^2}) \end{align*}$

[ optimize for ‘mu’ ]

Let’s optimize the expression above with respect to mu_1.

$\begin{align*} &\frac{\partial }{\partial \mu_1} \sum_{i=1}^{N}\sum_{c=1}^{3}q(t_i=c)\cdot (log\frac{\pi_c}{z}-\frac{(x_i-\mu_c)^2}{2\sigma_c^2}) \\ &= \sum_{i=1}^{N}q(t_i=1)\cdot (0-\frac{2(x_i-\mu_1)(-1)}{2\sigma_1^2}) = 0 \end{align*}$

If we multiply it by sigma_1 ( = variance of the first gaussian ) and the solve the equation, the result will be like this.

$\mu_1 = \frac{\sum_{i}q(t_i=1)x_i}{\sum_{i}q(t_i=1)}$

[ optimize for ‘sigma’ ]

This time, let’s optimize the expression above with respect to sigma_c. Like the same way as the above, the result will be like this.

$\sigma_c^2 = \frac{\sum_{i}(x_i-\mu_c)^2\cdot q(t_i=c)}{\sum_{i}q(t_i=c)}$

[ optimize for prior weights ‘pi’ ]

( same as the above )

$\pi_c = \frac{\sum_{i}q(t_i=c)}{N}$

Twitter Facebook LinkedIn

You May Also Enjoy

3 minute read

6 minute read

4 minute read

less than 1 minute read