DALL-E; Zero-Shot Text-to-Image Generation

DALL-E; Zero-Shot Text-to-Image Generation

Ramesh, Aditya, et al. "Zero-shot text-to-image generation." International conference on machine learning. Pmlr, 2021.

참고: https://www.youtube.com/watch?v=-e-vW1j132A&t=2257s

Contents

1. Introduction
2. DALL-E
3. Visualization

1. Introduction

• GPT-3의 확장형태 (Auto-regressive), 12B params.
• CV + NLP \(\rightarrow\) Text-to-Image task
• Zero-shot performance

2. DALL-E

Text & Image token을 “single stream”으로 모델링

Issues

• (1) Memory issue (pixel-level)
• (2) Short-range dependence

(1) Stage 1

Image token 생성

• (1) Discrete VAE 사용

• (2) (256x256) \(\rightarrow\) (32,32) image token으로 압축

• (3) (codebook 내의) # Token = 8192
• (4) context size 192배 줄임
  • (256x256x3) \(\rightarrow\) (32x32)

픽셀을 순차적으로 생성하는 것보다 효율적!

• 나름, (디테일은 사라질지라도) 핵심 요소들은 잘 유지가 됨!

(2) Stage 2

Text token & Image token 합치기

Concatenate (a) & (b)

• (a) (최대) 256 BPE-encoded text token
• (b) (32x32) image token

\(\rightarrow\) Text & Image token의 joint distribution을 학습한다

아래의 과정을 1024개의 image token이 다 채워질때까지 autoregressive하게 반복한다!

(3) Interpretation

\(p_{\theta, \psi}(x, y, z)=p_\theta(x \mid y, z) p_{\varphi}(y, z)\).

• (1) \(p_{\varphi}(y, z)\): Transformer
• (2) \(p_\theta(x \mid y, z)\): Discrete VAE decoder
• Notation
  • \(x\): image
  • \(y\): text ( = caption )
  • \(z\): image의 token

Maximizing ELBO

\(\ln p_{\theta, v}(x, y) \underset{z \sim q_\phi(z \mid x)}{\mathbb{E}}\left(\ln p_\theta(x \mid y, z)-\beta D_{\mathrm{KL}}\left(q_\phi(y, z \mid x), p_\psi(y, z)\right)\right)\).

• \(p_{\theta}\): dVAE decoder ( image token으로 image 예측)
• \(q_{\phi}\): dVAE encoder ( image로 image token 예측)
• \(p_{\varphi}\): Transformer ( image token & text token의 joint distn 모델링 )

(4) Details

Stage 1. Learning the visual code book

• dVAE encoder & decoder 학습
  • 즉, \(\theta, \phi\)에 대해 ELBO를 maximize
• 오직 image와 관련된 부분 (text (X))

Stage 2. Learning prior

• Transformer 학습
  • 즉, (\(\theta, \phi\)는 고정해둔 채) \(\varphi\)에 대해 ELBO를 maximize
• Text token & Image token에 대한 prior를 학습함

Transformer: decoder-only model

• (1) Text-to-text ATT: standard causal mask
• (2) Image-to-Image ATT: row/col/convolutional attention mask

3. Visualization