How Can Time Series Analysis Benefit From Multiple Modalities? A Survey and Outlook - Part 2

How Can Time Series Analysis Benefit From Multiple Modalities? A Survey and Outlook - Part 2

Contents

1. Abstract
2. Introduction
3. Background and Taxonomy
   1. Taxonomy
   2. Background
4. TimeAsX: Resuing Foundation Models of Other Modalities for Efficient TSA
   1. Time As Text
   2. Time As Image
   3. Time As Other Modalities
   4. Domain-Specific TS Works

3. TimeAsX: Resuing Foundation Models of Other Modalities for Efficient TSA

(2) Time As Image

Quite natural! Similar to how humans perceive patterns!

a) Line-graphs

• Most popular way to convert TS2image
• To use vision foundational models (e.g., ViT)
  • E.g., VLMs for anomaly detection [200, 237] and classification [37].
• Examples) [137, 218, 200, 237, 37]
  • [137] ViTime: A Visual Intelligence-Based Foundation Model for Time Series Forecasting https://www.arxiv.org/pdf/2407.07311v3
  • [218] Time Series as Images: Vision TransformerforIrregularly Sampled Time Series https://www.arxiv.org/pdf/2303.12799
  • [200] Can Multimodal LLMs Perform Time Series Anomaly Detection? https://www.arxiv.org/pdf/2502.17812
  • [237] See it, Think it, Sorted: Large Multimodal Models are Fewshot Time Series Anomaly Analyzers. https://www.arxiv.org/pdf/2411.02465
  • [37] Plots Unlock Time-Series Understanding in Multimodal Models. https://www.arxiv.org/pdf/2410.02637

ViTime

ViTST

VisualTimeAnomaly

TAMA (Time Series Anomaly Multimodal Analyzer)

b) Heatmaps

• Visualize TS in a 2D space
  • Colors = Represent magnitudes
• Specifically useful for modeling LONG TS
• Examples) [143,219]
  • [143] VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time Series Forecasters https://www.arxiv.org/pdf/2408.17253
  • [219] Deep video prediction for timeseries forecasting

VisionTS

c) Spectrogram

Time series can be decomposed into the spectrum of frequencies and represented as a spectrogram. Wavelet transforms are a popular choice of representation for both univariate [220] and multivariate [144] tasks.

d) Other methods

Zhiguang and Tim [231] use Gramian Angular Fields (GAF) [20] to represent time-series. which visualize long and short termdependenciesbetter.Recurrenceplots(RP)Eckmannetal. [47] areanotherwaytocaptureperiodicpatternsintime-seriesused by [89] for classification and [110] forecasting. Time-VLM [161] combines information from Fourier coefficients, cosine and sine periodicity into a heatmap which is fed into a VLM encoder.