3. Variational Transform
(1) Variational Transform
Non-linear functions are not easy to handle. It would be great if we would make it as a linear function. So, how could we transform it?
</br>
https://encrypted-tbn0.gstatic.com/images
Like the picture above, we can draw a tangent line at each x’s, and this would be the best linear function representing that non-linear function. But the problem is that as the x is getting bigger/smaller from the original x, the gap between two functions will be larger (becoming inaccurate).
But if we change that linear function every time the x changes, (which means that finding all the linear function for every x ), it would be a quite approximation.
Mathematical Expression
Let’s set
- the non-linear function : \(g(x) = lnx\)
- the linear function : \(f(x) = \lambda x + b(\lambda)\)
What we have to do is to minimze the gap between the two functions, \(g(x)\) and \(f(x)\)
\(\underset{x}{min} (f(x)-g(x))\)
If we take the derivative…
\(\frac{d}{dx} (f(x)-g(x)) = \frac{d}{dx} (\lambda x +b(\lambda) - lnx) = 0\)
so the lambda should be like the below
\(\lambda = \frac{1}{x}\)
Then we could express the equation like..
\(\lambda x + b(\lambda) = ln(x)\)
\(\lambda (1/\lambda) + b(\lambda) = -ln(\lambda)\)
\(b(\lambda) = -ln(\lambda) -1\)
What we can learn from this is that “(even though function is still non-linear with \(\lambda\)) We can treat the non-linear more easily with linear model (in the case of \(x\))”
(2) Variational Transform on Logistic Function
Logistic Function, which is also called sigmoid function, looks like below.
\(f(x) = \frac{1}{1+e^{-x}}\)
It is neither concave nor concave. But we can make this as a concave function by taking logarithm.
Let the function..
\(g(x) = lnx\)
\(f(x) = \frac{1}{1+e^{-x}}\)
Then \(g(f(x))\) would look like
\(g(f(x)) =ln(\frac{1}{1+e^{-x}}) = -ln(1+e^{-x})\)
( as you can see, the x range of the function is 0~1, and the y range is -inf~0 )
Then by taking exponential function, we could express like below!
\(f(x) = \underset{\lambda}{min} {e^{\lambda x –H(\lambda)}}\)
(3) Convex Duality
You have seen that we can approximate non-linear function into linear function. So how can we explain this in more general way?
If \(f(x)\) is a concave function, we can solve it by the way explained in (1) (above). But if \(f(x)\) is not a concave function, we can still solve it by ..
- 1) transform using the “log function”
- 2) re-transform using the “exp function”
Mathematical Expression
\(f(x) = \underset{\lambda}{min}{\lambda^T x-f^{*}(\lambda)}\)
\(f^{*}(x) = \underset{x}{min}{\lambda^T x-f^{*}(x)}\)