LCM-LoRA; A Universal Stable-Diffusion Acceleration Module

LCM-LoRA: A Universal Stable-Diffusion Acceleration Module

Luo, Simian, et al. "Lcm-lora: A universal stable-diffusion acceleration module." arXiv preprint arXiv:2311.05556 (2023).

참고:

• https://aipapersacademy.com/from-diffusion-models-to-lcm-lora/
• https://arxiv.org/pdf/2311.05556

Contents

1. Introduction
2. Recap
   1. Diffusion models
   2. Consistency models
   3. Latent diffusion models
3. Latent Consistency Models
   1. LCM
   2. LCM-LoRA

1. Introduction

“LCM-LoRA: A Universal Stable-Diffusion Acceleration Module”

• A method to generate high quality images with large text-to-image generation models
  • e.g., specifically SDXL (Stable Diffusion XL)
• Make it dramatically faster.
• Works for both..
  • (1) Not only for SDXL
  • (2) But also for fine-tuned SDXL without going through another training process

2. Recap

(1) Diffusion models

(2) Consistency models

(3) Latent Diffusion models

Summary = Doing most of the work in the latent space makes the process…

• (1) more efficient
• (2) allows generation of high-quality images.

(4) Latent Consistency models

\(\rightarrow\) The proposed method!

3. Latent Consistency Models (LCM)

(1) LCM

LDMs = Still process with many iterations

\(\rightarrow\) Why not combine with consistency models?

\(\rightarrow\) Latent consistency models

For faster inference

\(\rightarrow\) Directly remove all of the noise in order to skip steps in the denoising process.

(2) LCM-LoRA

• Step 1) Use a pre-trained LDM weights
• Step 2) fine-tuning the LDM weights
  • Too costly (\(\because\) SDXL)
  • Solution = LoRA