AAA (All About AI)

Time2Vec : Learning a Vector Representation of Time (2019,53)

Contents

1. Abstract
2. Related Works
3. Time2Vec
4. Experiments

0. Abstract

Time2Vec

• learn a model-agnostic vector representation for time

1. Related Work

Previous works

• assume that inputs are synchronous
  • arrive at regular time intervals
  • underlying process is stationary w.r.t time

propose a general representation of time

• related to time decomposition tehcniques,

  that encode a temporal signal into set of frequencies

• instead of using a fixed set of frequencies,

  allow the frequencies to be learned

  ( inspired by ND (Neural Decomposition) )

ND (Neural Decomposition)

• decompose a 1D signal of time into “several sine functions & linear function”,

  to extrapolate (+interpolate) the given signal

• [ difference ]

  • step 1) transform the time itself
  • step 2) feed its transformation into the model

2. Time2Vec

3 important properties

• 1) capture both “periodic & non-periodic” patterns
• 2) invariant to time re-scaling
• 3) simple

Time2vec

has the 3 identified properties above!

\(\mathbf{t 2 v}(\tau)[i]= \begin{cases}\omega_{i} \tau+\varphi_{i}, & \text { if } i=0 \\ \mathcal{F}\left(\omega_{i} \tau+\varphi_{i}\right), & \text { if } 1 \leq i \leq k\end{cases}\).

• \(\mathbf{t} 2 \mathbf{v}(\tau)[i]\) : the \(i^{t h}\) element of \(\mathbf{t} 2 \mathbf{v}(\tau)\)
• \(\mathcal{F}\) : periodic activation function
• \(\omega_{i}\) : learnable parameters
• \(\varphi_{i}\) : learnable parameters

Setting

• choose \(\mathcal{F}\) to be sine function
• then, for \(1 \neq i \neq k\) :
  • \(\omega_{i}\) : frequency of sine function
  • \(\varphi_{i}\) : phase shift of ~
• (note)
  • period of \(\sin \left(\omega_{i} \tau+\varphi_{i}\right)\) is \(\frac{2 \pi}{\omega_{i}}\) ( = has the same value for \(\tau\) & \(\tau + \frac{2\pi}{w_i}\) )

3. Experiments

• Time2Vec can be effective for long sequences

• using activation functions other than sine…..

  • not good….

• LAYER ( T2V embedding ) : \(\boldsymbol{a}=\boldsymbol{\Theta} \mathbf{t} \mathbf{2} \mathbf{v}(\tau)\)

  • \(\boldsymbol{\Theta}\) : first layer weights matrix, where \([\boldsymbol{\Theta}]_{i, j}=\theta_{i, j}\)

  • \(\boldsymbol{a}(\tau, k)[i]=\theta_{i, 0}\left(\omega_{0} \tau+\varphi_{0}\right)+\sum_{j=1}^{k} \theta_{i, j} \sin \left(\omega_{i} \tau+\varphi_{i}\right)\),

    • \(k-1\) : number of sine functions
      • [ term 1 ] linear term : \(\theta_{i, 0}\left(\omega_{0} \tau+\varphi_{0}\right)\)
        • can model “non-periodic components” & helps extrapolation
      • [ term 2 ] sum of the weighted sine functions : \(\sum_{j=1}^{k} \theta_{i, j} \sin \left(\omega_{i} \tau+\varphi_{i}\right)\)
        • used to model the periodic behavior of the \(\mathbf{a}[i]\) features

learning the frequencies & phase shifts ( rather than fixing ) improves performance!