LatentSync

LatentSync

🔥 Updates

- 2025/06/11: We released LatentSync 1.6, which is trained on 512 $\times$ 512 resolution videos to mitigate the blurriness problem. Watch the demo here.

- 2025/03/14: We released LatentSync 1.5, which (1) improves temporal consistency via adding temporal layer, (2) improves performance on Chinese videos and (3) reduces the VRAM requirement of the stage2 training to 20 GB through a series of optimizations. Learn more details here.

📖 Introduction

We present *LatentSync*, an end-to-end lip-sync method based on audio-conditioned latent diffusion models without any intermediate motion representation, diverging from previous diffusion-based lip-sync methods based on pixel-space diffusion or two-stage generation. Our framework can leverage the powerful capabilities of Stable Diffusion to directly model complex audio-visual correlations.

🏗️ Framework

LatentSync uses the Whisper to convert melspectrogram into audio embeddings, which are then integrated into the U-Net via cross-attention layers. The reference and masked frames are channel-wise concatenated with noised latents as the input of U-Net. In the training process, we use a one-step method to get estimated clean latents from predicted noises, which are then decoded to obtain the estimated clean frames. The TREPA, LPIPS and SyncNet losses are added in the pixel space.

🎬 Demo

        Original video
        Lip-synced video

(Photorealistic videos are filmed by contracted models, and anime videos are from VASA-1)

📑 Open-source Plan

- [x] Inference code and checkpoints
- [x] Data processing pipeline
- [x] Training code

🔧 Setting up the Environment

Install the required packages and download the checkpoints via:

bash
source setup_env.sh

If the download is successful, the checkpoints should appear as follows:

./checkpoints/
|-- latentsync_unet.pt
|-- whisper
|   -- tiny.pt

Or you can download latentsync_unet.pt and tiny.pt manually from our HuggingFace repo

🚀 Inference

Minimum VRAM for inference:

- 8 GB with LatentSync 1.5
- 18 GB with LatentSync 1.6

There are two ways to perform inference:

1. Gradio App

Run the Gradio app for inference:

bash
python gradio_app.py

2. Command Line Interface

Run the script for inference:

bash
./inference.sh

You can try adjusting the following inference parameters to achieve better results:

- inference_steps [20-50]: A higher value improves visual quality but slows down the generation speed.
- guidance_scale [1.0-3.0]: A higher value improves lip-sync accuracy but may cause the video distortion or jitter.

🔄 Data Processing Pipeline

The complete data processing pipeline includes the following steps:

1. Remove the broken video files.
2. Resample the video FPS to 25, and resample the audio to 16000 Hz.
3. Scene detect via PySceneDetect.
4. Split each video into 5-10 second segments.
5. Affine transform the faces according to the landmarks detected by InsightFace, then resize to 256 $\times$ 256.
6. Remove videos with sync confidence score lower than 3, and adjust the audio-visual offset to 0.
7. Calculate hyperIQA score, and remove videos with scores lower than 40.

Run the script to execute the data processing pi

……（后略，见原文）